Evidence Report/Technology Assessment: Number 59

Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems

Summary


Under its Evidence-based Practice Program, the Agency for Healthcare Research and Quality (AHRQ) is developing scientific information for other agencies and organizations on which to base clinical guidelines, performance measures, and other quality improvement tools. Contractor institutions review all relevant scientific literature on assigned clinical care topics and produce evidence reports and technology assessments, conduct research on methodologies and the effectiveness of their implementation, and participate in technical assistance activities.

Select for PDF File (84 KB). PDF Help.

Overview / Reporting the Evidence / Methodology / Findings / Conclusions / Future Research / Availability of the Full Report


Overview

The Nation's capacity to respond to bioterrorism depends in part on the ability of clinicians and public health officials to detect, manage, and communicate during a bioterrorism event. Information technologies and decision support systems (IT/DSSs) have the potential to aid clinicians (e.g., physicians, nurses, nurse practitioners, and respiratory therapists) and public health officials to respond effectively to a bioterrorist attack.

The Evidence Report from which this summary was taken details the methodology, results, and conclusions of a systematic and extensive search for published materials on the use of IT/DSSs to serve the information needs of clinicians and public health officials in the event of a bioterrorist attack. The information is intended to assist clinicians, public health officials, and policymakers to improve preparedness for a bioterrorism event.

Return to Contents

Reporting the Evidence

The University of California at San Francisco (UCSF)—Stanford Evidence-based Practice Center staff, in conjunction with a panel of expert advisors and the Agency for Healthcare Research and Quality (AHRQ), developed the following four Key Questions to be addressed in this report:

  1. What are the information needs of clinicians and public health officials in the event of a bioterrorist attack?
  2. Based on the information needs identified for these decisionmakers, what are the criteria by which IT/DSSs should be evaluated with respect to usefulness during a bioterrorism event?
  3. When assessed by these criteria, in what ways could existing IT/DSSs be useful during a bioterrorism event? In what ways are they limited?
  4. In areas where existing IT/DSSs do not meet the information needs of clinicians or public health officials, what functional and technical considerations are important in the design of future IT/DSSs to support response to bioterrorism events?

Return to Contents

Methodology

Conceptual Model

A conceptual model was developed to specify the decisions and tasks involved in diagnosis, management, prevention, surveillance, and communication by clinicians and public health officials in the event of a bioterrorist attack. The investigators used a process called task decomposition to specify the data requirements that need to be incorporated into an IT/DSS for it to assist clinicians and public health officials in making these decisions. This list of tasks and data requirements served as the foundation of the evaluation system of the currently available IT/DSSs.

Inclusion and Exclusion Criteria

Based on input from the expert advisory panel, the conceptual model, the task decomposition, and practical considerations, an inclusion-exclusion strategy was developed to identify articles that described or evaluated IT/DSSs.

Selection of Quality Scales

A scale developed at McMaster University was used to rate the quality of evidence from peer-reviewed evaluations of IT/DSSs for diagnosis, management, and communication. For reports of surveillance systems, an evaluation scale published by the Centers for Disease Control and Prevention (CDC) was used.

Literature Sources

In consultation with professional research librarians, a search strategy for references from three sources was developed: peer-reviewed articles, government reports, and Web-based information. For the peer-reviewed articles, five databases of medical, scientific, and government references likely to contain reports of relevant IT/DSSs were identified: MEDLINE® (January 1985 to April 2001), the Catalog of U.S. Government Publications, GrayLIT, the Library of Congress, and the National Technical Information Service. The investigators identified the 16 government agencies most likely to fund, develop, or use IT/DSSs that could also be used by clinicians or public health officials. Internet searches to retrieve reports of potentially relevant IT/DSSs from sites other than those operated by government agencies (e.g., academic and commercial sites) were also planned.

Search Strategies

Separate search strategies were developed, one for MEDLINE® and one for the government documents and Web-based information. Each included terms such as bioterrorism, biological warfare, information technology, decision support system, diagnosis, management, therapeutics, communication, surveillance, public health, and epidemiology. Additional articles were identified by members of the expert advisory panel, from conference proceedings, and by review of reference lists.

Data Collection and Analysis

Titles, abstracts, and full-length articles were reviewed as necessary to identify potentially relevant articles. All peer-reviewed articles that met the inclusion criteria were blinded to the investigators, two of whom independently abstracted study information on to a data-abstraction and quality-assessment form.

The following data were abstracted from all included articles: the purpose and description of the system (e.g., detection, diagnosis, management, surveillance, or communication), whether the system had been clinically evaluated and the results of these evaluations, what security measures the system uses, what kind of reasoning the system uses, and information about the quality of the report. A draft Evidence Report was critiqued by 16 expert advisors and 17 peer-reviewers who had expertise in nursing, clinical medicine, public health, hospital management, informatics, diagnostics, emergency management, epidemiology, national security, toxicology, and food safety.

Return to Contents

Findings

The investigators reviewed a total of 16,888 citations of peer-reviewed articles, 7,685 Web sites of government agencies, and 1,107 non-government Web sites. Of these, 251 articles, 36 government Web sites, and 54 non-government Web sites met the inclusion criteria. From these, descriptions were abstracted of 217 IT/DSSs of potential use by clinicians and public health officials in the event of a bioterrorist attack. They are comprised of 55 detection systems, 23 diagnostic systems, 18 management systems, 90 surveillance systems, 26 communication systems, and 7 systems that integrate surveillance, communication, and command and control functions (some systems have more than 1 function and are described in more than 1 section). Most reports only described IT/DSSs; however, 79 studies evaluated 58 systems for at least 1 performance metric. Some types of systems have been evaluated more than others. For example, 10 of the 18 management systems have been evaluated in at least 1 study; but none of the 7 integrated surveillance, communication, and command and control systems has been evaluated. Most of the 217 included systems were not designed specifically for bioterrorism; instead, they were created for detecting and managing naturally occurring illness. The few systems that were designed for bioterrorism are principally for detection and integrated command and control purposes, and most were designed by the military and are being converted for civilian use. There are almost no publicly available evaluative data on these systems, although the military developers may have performed comprehensive evaluations.

Key Question 1

What are the information needs of clinicians and public health officials in the event of a bioterrorist attack?

Based on the conceptual model and task decomposition, the information required by clinicians and public health officials while preparing for and responding to bioterrorist events relates to the decisions they have to make and the tasks they have to perform.

Clinicians require the necessary information to make diagnostic, management, prevention, and reporting decisions. Diagnostic decisions require information to accurately estimate the pre-test probability of disease for a given patient. Clinicians' interpretations of test results require information about the sensitivity and specificity of the test. Management decisions require information about how to appropriately distinguish between those patients who need treatment and those who do not, how best to treat the acutely ill, whom to isolate and how, how to manage scarce resources, and how to maintain personal safety. Prevention decisions require information about prophylaxis and vaccination protocols. Reporting decisions rely on information about what constitutes a reportable case or cluster of cases and about the kinds of data that public health officials seek.

The information that public health officials require to prepare for and respond to a bioterrorism event can be considered in relation to the decisions they must make: the interpretation of surveillance data; the investigation of outbreaks; the institution of epidemiologic control measures; and the issuance of surveillance alerts. The decision to perform outbreak investigation requires information about the baseline characteristics of the surveillance data and threshold levels that suggest that an outbreak resulting from naturally occurring or bioterrorism-related illness may have occurred. Once a bioterrorism event has been identified, public health officials require information that will enable them to perform ongoing surveillance in the midst of the crisis to track the extent and spread of the epidemic. The decisions regarding the institution of epidemiologic control measures that prevent the spread of disease require information about the transmissibility of the suspected biothreat agent(s) and about the criteria for and effectiveness of prophylaxis and quarantine strategies. Decisions to issue a surveillance alert require information about the nature of the suspected bioterrorist attack and the characteristics and expected natural history of the suspected biothreat agent(s). Other communication decisions relate to the specific information that needs to be conveyed to other public health officials, clinicians, the media, and other decisionmakers.

Key Question 2

Based on the information needs identified for these decisionmakers, what are the criteria by which IT/DSSs should be evaluated with respect to usefulness during a bioterrorism event?

The evaluation criteria vary depending on the purpose of the IT/DSS and the information needs of the users of the system as determined by task decomposition methodology.

Key Question 3

When assessed by these criteria, in what ways could existing IT/DSSs be useful during a bioterrorism event? In what ways are they limited?

The review identified 217 IT/DSSs, few of which were designed specifically for response to bioterrorism events. Most included systems had other intended purposes but could potentially be useful to clinicians or public health officials in response to a bioterrorism event. The evidence by which to judge the usefulness of these systems is limited. Many of the systems were not evaluated even for their intended purpose. Of the studies that did evaluate systems for their intended purpose, few adhered to published criteria for high-quality evaluations. In addition, even if a system received a favorable evaluation for its intended purpose, it may not necessarily be feasible to evaluate its usefulness for response to bioterrorism.

Detection systems. Fifty-five detection systems that collect and identify potential biothreat agents within environmental and clinical samples were identified. Many of these systems were developed for use by the military and some were adapted for civilian purposes. Few reports compare detection systems to a gold standard, and their sensitivity (i.e., the likelihood that the detection system will give a positive result when testing a sample containing a biothreat agent) and specificity (i.e., the likelihood that the detection system will give a negative result when testing a sample that does not contain a biothreat agent) remain poorly characterized in the publicly available literature. Most identification systems are limited in that each test cycle can evaluate a sample for only a single biothreat agent, often run only a limited number of samples at a time, and cannot test for many of the most worrisome agents (e.g., smallpox). No reports were found that directly compared two or more of the commercially available systems in any given category. The paucity of comprehensive evaluative information about these systems prevents conclusions about whether or not these systems are likely to serve the detection needs of first-responders, clinicians, and public health officials during a bioterrorist event.

Diagnostic systems. Twenty-three diagnostic systems with potential utility for enhancing the likelihood that clinicians consider the possibility of bioterrorism-related illness were identified. These systems are generally designed to assist clinicians in developing a differential diagnosis for a patient who has an unusual clinical presentation. The investigators found six general diagnostic systems, four systems designed to improve radiologic diagnoses, four telemedicine systems, four systems for the diagnosis of infectious diseases, one system for the diagnosis of dermatologic lesions, one system for the diagnosis of community-acquired pneumonia, and three systems for other purposes. None of these DSSs has been evaluated formally with respect to bioterrorism response. In an evaluation of a DSS for infectious diseases that has more than 20 biothreat agents in its knowledge base, the system was able to list the actual diagnosis in an output of possible diagnoses for nearly 95 percent of 495 actual and hypothetical cases. However, this system is limited in that it is specific for infectious diseases; consequently, even those clinicians with access to this technology may not use it if the patient does not present with a fever or other signs or symptoms of infectious disease.

Three of the general diagnostic DSSs have been evaluated for their intended (non-bioterrorism related) purposes. In these evaluations, the general diagnostic DSSs typically performed better than physicians-in-training but not as well as experienced clinicians. However, the accuracy of the DSSs decreased for difficult cases. The need for clinicians to manually enter patients' signs and symptoms into diagnostic DSSs—a laborious step that may be a barrier to the use of these systems and has been demonstrated to increase inter-user variability—is eliminated by the few systems that automatically collect patient data from an electronic medical record. For example, there are diagnostic DSSs currently available in hospitals with electronic medical records that provide clinicians with an estimate of the likelihood of community-acquired pneumonia or active pulmonary tuberculosis based exclusively on data collected from the medical record. Many diagnostic DSSs use probabilistic information about the likelihood of disease. Because bioterrorism-related illness is relatively rare, in the event of bioterrorism these systems will have inappropriately low pretest probabilities for biothreat agents. None of the reports of diagnostic DSSs described the ability to change the probability of disease based on information about suspected bioterrorism events.

Management and prevention systems. Management and prevention systems are designed to make recommendations to clinicians by abstracting clinical information from electronic medical records to make patient-specific recommendations. None of the 18 systems identified in this review has been specifically designed or evaluated for utility in providing management or prevention recommendations during a bioterrorism event; however, 10 of them have been evaluated for their intended purpose. These evaluations demonstrate that the expert systems that continuously search electronic medical records (including data from the laboratory, radiology reports, and clinician notes) for new evidence of infection and apply clinical practice guidelines to those data are able to affect clinicians' antibiotic selection decisions and increase compliance with clinical practice guidelines. No information was found as to whether the knowledge bases of these systems include comprehensive information about bioterrorism-related illnesses. The systems that are not linked to electronic medical records share many of the limitations of the general diagnostic systems—including that clinicians may not use the system to seek advice for patients presenting with common viral syndromes (i.e., the bioterrorism-related syndromes). Antibiotic recommendation programs are typically designed to provide recommendations for antibiotics with the narrowest possible spectra, thereby reducing the risk of developing resistant organisms. If clinicians make antibiotic selection decisions while unaware of the true bioterrorism-related diagnosis and select narrow-spectrum antibiotics, they may not be effective against biothreat agents. Therefore, whether the use of these systems would be helpful or detrimental is not known.

Surveillance systems. Ninety surveillance systems that collect a variety of surveillance reports were identified: 7 for syndromal surveillance, 6 for reports from clinicians, 11 for influenza-related data, 23 for laboratory and antimicrobial resistance data, 16 for hospital-based infections data, 10 for food-borne illness data, 6 for zoonotic illness data, and 11 for other types of surveillance data. For a surveillance system to detect a covert bioterrorist event, it must collect data that are sensitive and specific for biothreat agents, analyze the data, and report results to public health decisionmakers in a timely manner. None of 90 included surveillance systems has been evaluated for its utility in detecting a bioterrorism event. Forty of 61 reports of evaluations or descriptions of surveillance systems described the timeliness, importance of the health event under surveillance, and usefulness of the system. However, less than one-third of the reports of evaluations of surveillance systems described the representativeness, simplicity, sensitivity, specificity, acceptability, or flexibility of the system. The quality of the evidence regarding the effectiveness of the systems reported by these articles is therefore limited. Most of the evaluations of surveillance systems demonstrated that the electronic collection and reporting of surveillance data improved detection over older, manual methods. When the 90 surveillance systems described in this report are considered, there are relatively few systems collecting the earliest surveillance data—such as school and work absenteeism, calls to telephone care nurses, over-the-counter pharmacy sales, or veterinary or zoonotic illness—a potentially significant gap in available surveillance systems.

Communication systems. Eight of the 26 communication systems were designed for communication among public health officials at local, State, and Federal levels (e.g., Web-based discussion and reporting of surveillance data). In pilot evaluations directed by individual State health departments, these systems securely managed the disease reporting needs of local and State public health officials. However, these systems were limited to communication within a State. No single system was found that effectively links members of the public health community at national, State, and local levels. However, there are ongoing efforts (such as the Urban Security Initiative project of Los Alamos National Laboratory, EpiX, Health Alert Network and RHEACT) designed to integrate communication of public health information vertically and horizontally within the U.S. public health system. Five systems were designed for the automated communication of information from hospital-based electronic medical records to clinicians (e.g., alerting systems to notify clinicians of abnormal laboratory tests). These systems have been subjected to the greatest evaluation of all the communication systems. Despite being limited to institutions with electronic medical records, they could potentially play an important role in improving the timely recognition of bioterrorism-related illness. Three systems facilitated communication between emergency departments and first-line emergency response personnel. ProMED© has demonstrated the capacity for rapid reporting and dissemination of information on a wide range of infectious diseases resulting from both naturally occurring and bioterrorism-related events. During a bioterrorism event, clinicians must be able to rapidly communicate with their patients. Systems exist that enable Web-based communications between these parties in a manner compliant with the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Robust security measures that ensure patient confidentiality and resist cyberattack will be a necessary component of any bioterrorism-related communication system.

Key Question 4

In areas where existing IT/DSSs do not meet the information needs of clinicians or public health officials, what functional and technical considerations are important in the design of future IT/DSSs to support response to bioterrorism events?

No evaluations or studies that directly assess the functional and technical requirements that are important for future IT/DSSs were identified. This section provides the investigators' interpretation of factors that could be considered for the design of future IT/DSSs.

Return to Contents

Conclusions

IT/DSSs have the potential to help clinicians and public health officials make better decisions when responding to a bioterrorism event. IT/DSSs were identified that could potentially aid with detection, diagnosis, management, prevention, surveillance, and communication. However, most of these systems were not designed specifically for bioterrorism. Many of these systems have not been described in peer-reviewed literature, and fewer still have been evaluated rigorously. The existing evaluations primarily assess the usefulness of systems for their intended purpose, and often do not provide direct evidence about the usefulness of the IT/DSSs for bioterrorism.

The lack of evaluative studies creates difficulties in assessing the usefulness of IT/DSSs. For detection systems, almost no information is available on sensitivity and specificity. Without this information, interpretation of test results is highly problematic. Diagnostic DSSs have not been used widely, and several of the available systems require time-consuming manual input of patient data, which is impractical in many clinical settings. Whether management DSSs could be useful for bioterrorism-related disease remains unanswered. Surveillance systems hold promise, and although many are undergoing evaluation, the systems designed for bioterrorism response have been fielded only recently. Web-based communication systems are increasingly available to link public health officials with clinicians and the public; however, their efficacy in crisis situations is untested.

This review suggests important gaps in the available literature. One should note, however, that lack of evidence about effectiveness is not evidence for lack of effectiveness. Many of the systems reviewed may indeed be useful for response to bioterrorism and are reasonable candidates for further evaluation. Such evaluations would clarify their value both for response to bioterrorism and for the other purposes for which they were designed.

Return to Contents

Future Research

In addition to the development of systems described in the answer to Key Question 4, the following future research could provide additional insights into the information needs of clinicians and public health officials and the types of IT/DSSs that may best serve those needs:

Return to Contents

Availability of the Full Report

The full Evidence Report from which this summary was taken was prepared for AHRQ by the UCSF-Stanford Evidence-based Practice Center under contract No. 290-97-0013. Print copies may be obtained free of charge from the AHRQ Publications Clearinghouse by calling 1-800-358-9295. Requesters should ask for Evidence Report/Technology Assessment No. 59, Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems.

The Evidence Report is also online on the National Library of Medicine Bookshelf, or can be downloaded as a set of PDF files or as a zipped file.

Return to Contents

AHRQ Publication Number 02-E061
Current as of July 2002


Internet Citation:

Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems. Summary, Evidence Report/Technology Assessment: Number 59, July 2002. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/clinic/epcsums/bioitsum.htm


Return EPC Evidence Reports
Clinical Information
AHRQ Home Page
Department of Health and Human Services