U. S. Food and Drug Administration
Center for Food Safety and Applied Nutrition
March 2002

Initiation and Conduct of All 'Major' Risk Assessments within a Risk Analysis Framework

Part III. Conducting Risk Assessment within the Risk Analysis Framework

Once a risk assessment is selected as described in Part II of this document, the risk assessment is commissioned. Conducting a risk assessment within CFSAN's risk analysis framework includes the following steps--planning, performing, reviewing, and issuing. These steps, including activities and responsible parties for each, are described in detail in this part of the document.

Steps: Conducting the Risk Assessment

Step 1: Planning the risk assessment
Step 2: Performing the risk assessment
Step 3: Reviewing the risk assessment document(s)
Step 4: Issuing the risk assessment

This part of the framework document focuses on the activities directly related to the conduct of the risk assessment. However, it is important to understand the sequence of these activities as they relate to the responsibilities of risk managers and risk communicators. Figure III-1 shows the interrelationships of these risk analysis activities.

During the planning step, the risk assessment, risk management and risk communication teams are formed. The risk management team develops a 'charge' for the assessment, including the key assumptions that define the scope of the project. Once the risk assessment team receives the 'charge' they begin to conduct the risk assessment. During the performance step, there is regular interaction between the risk management and risk assessment teams to further refine the scope of the assessment, inform the risk management team of progress in completing the assessment, and eventually providing the results and a draft report for review. Also, during the performance step, the risk communication team is conducting an ongoing interaction with stakeholders to determine their concerns, perceptions, and information needs. During the review of the draft document, the risk managers may begin to develop an action plan and the risk communicators to initiate development of public health messages. Following the issue or publishing of the assessment, the teams should debrief and evaluate the effectiveness of the process.

Risk Assessment (RA)	Risk Management (RM)		Risk Communication (RC)
Assemble RA team	Assemble RM team		Assemble RC team		Plan
	Develop RA and RC charge
	Develop key RA assumptions

Receive RA charge from RM			Recieve RC charge from RM		Perform
Conduct Risk Assessment	Oversight of RA		Ongoing interaction with stakeholder/public to determine concerns, perceptions, and information needs.
Results of modeling	Review RA results	Develop RM Action Plan	Review RA and RM documents	Develop RC messages	Review
Draft report	Review RA report
Issue risk assessment documents Team debrief	Issue action plan Team debrief		Issue and evaluate effectiveness of public health and risk communication messages.		Issue/ Publish
Figure III-1. Summary of Risk Analysis Activities and Responsible Parties

Step 1. Planning the Risk Assessment

After a specific risk assessment is selected, the next step is to adequately plan the project. The purpose of planning is to identify project goals, resources, and participants. Planning ensures a sound foundation for the successful conduct of a risk assessment.

Goal: Develop scope and timeframe, allocate resources, and identify participants

Product: Assemble teams, risk assessment 'charge,' statement of resources, and schedule

Activities: Planning the Risk Assessment

Identify participants and assemble the risk analysis teams
Agreement of the scope, complexity, focus of the risk assessment
Develop "charge" for the risk assessment, including key assumptions
Allocate resources (technical and financial support)
Develop a timeframe and timeline for initiating and completing the risk assessment
Determine communication needs

Decisions needed during the planning step include:

how the assessment will be coordinated within the risk assessment team, among the risk analysis teams, with other offices, agencies, and stakeholders
how the risk assessment and risk management teams should communicate
how and when to communicate with stakeholders
how transparency in the process will be maintained (i.e., documentation)

Identify Participants/Form Teams

Once a decision has been made to conduct a risk assessment, the participants should be identified; the risk management, assessment, and communication teams formed; and a lead assigned for each team. The team leads, risk assessment project manager, and risk analysis coordinator should meet and plan the risk assessment. Planning includes an agreement on the goals, scope, and timing of the risk assessment as well as resources needed to complete the assessment.

Table III-1 provides lists of the typical members of risk assessment, risk management, and risk communication teams. The risk assessment team should include subject matter leads on consumption, pathogen prevalence, food processing (if appropriate), dose-response, and model. Administrative support staff is also needed to provide administrative/clerical assistance such as filing and technical writing expertise. Pending the complexity of the assessment, a RM liaison might be beneficial. The subject matter leads will report to a technical team leader. The team leader must interact closely with the risk assessment project manager. The risk assessment project manager in turn implements the plans of the risk analysis coordinator.

**Table III-1. Members of the Risk Analysis Teams**
Risk Assessment Team	Risk Management Team	Risk Communication Team
Project manager Technical team leader Technical writer/editor Administrative assistant Modeler(s) Technical subject matter experts, including: pathogen prevalence, food consumption, dose-response, and other appropriate experts (i.e., processing)	Team leader CFSAN Program Office Directors and/or deputy directors ORA representatives Technical subject matter experts, including economists and physicians	Team leader Technical writer (from the risk assessment team) Representatives from: Consumer research Food safety education Public affairs/press office EOS Advisory Committee OCO
CFSAN = Center for Food Safety and Applied Nutrition ORA = Office of Regulatory Affairs EOS = Executive Operations Staff OCO = Office of Constituent Operations

The teams should be formed after the team leaders have been appointed, in consultation with the Risk Analysis Coordinator, the Risk Assessment Program Manager, and Program Office/Staff Directors.

Define Scope and Develop Charge

The Program Office or component of CFSAN that requests a risk assessment should meet with the risk assessment coordinating staff to develop a prospectus of what will be involved in the assessment. An estimate of personnel, time, resources, and external expertise is developed during these initial discussions.

The risk management team must develop a formal charge to the risk assessment team. This should include the management question to be addressed and specific questions that the risk assessment is to answer. This charge should be developed, circulated for comment, and published in the Federal Register with the intent and request for additional data to be submitted. The charge should include a statement of purpose, define the risk management problems, and the specific risk assessment questions to be answered. Key assumptions must also be identified in the charge.

Risk Management Problem versus Risk Assessment Question. Risk assessment is one tool that managers can use to solve or address a risk management problem or issue. The questions that risk managers must answer are different from those that risk assessors should be asked. An example risk management problem is:

"How should FDA manage the risk of contracting listeriosis from eating ready-to-eat (RTE) foods served in restaurants?"

The answer to the risk management problem is important for the risk assessor to know but not to answer. Based on the above example risk management problem, the risk management team might pose questions to the risk assessor, such as:

"What is the exposure to Listeria monocytogenes from consuming RTE foods in restaurants?"
"What is the likelihood of the general population (or to a sensitive subset of the population) contracting listeriosis from eating RTE foods in restaurants?"
"How much is the risk reduced if FDA required 'use by dates' on RTE foods?"

What Should be Included in the risk assessment charge? The 'charge' to the risk assessment team is a critical document. It defines the overall scope of the problem and the importance of the issue to CFSAN. It should be as detailed as possible to guide the risk assessment efforts. The charge should at least include the following information:

The specific risk management problem or question, including the scope (assumptions/objectives)
The risk assessment question(s)
The type of risk assessment to be conducted
CFSAN's goal and how the assessment information and conclusions will be used by risk managers
Background information on the hazard or human health impact relevant to assessment
Expertise needed to conduct the assessment
Resources allocated
Timeline
List of members of the risk assessment, management and communication teams with the leader (contact) individual clearly identified.

What are the Key Assumptions? Assumptions are needed when data are lacking. Key assumptions define the scope of the project and must be stated in the 'charge' to the risk assessment team. These assumptions must take into consideration the objectives and scope of the project. The risk assessment team is responsible for reviewing the key assumptions, explaining any limitations of them in the modeling effort and for developing the assumptions related to the model.

Key assumptions or the objectives that define the scope of the risk assessment include the following examples.

Populations at-risk
End point of concern (i.e., illness, death) or adverse effects
Route(s) of exposure

Examples of assumptions related to modeling that risk assessors develop in consideration of the available data include the following. These are refined assumptions and should be arrived at following discussions between the risk management and risk assessment teams.

Grouping of food into categories
Surrogate exposure data
Data limited or inclusive from national, international, or states
Consumer practices or handling

Allocate Resources

CFSAN Personnel. Once personnel needs are identified, the risk assessment project manager submits the needs in the form of a request to the Program Office(s) with that expertise.

The request must include the following:

Type of expertise needed

Estimated duration of the project

Time per week that the individual will be needed

Peak periods of activity

While the risk assessment project manager might "name request" a specific individual, it should be understood that this is only a suggestion. It is up to the Program Office to assign an appropriately qualified individual. This request to the Program Office should be formal, and involve sign off on the request. This action is to protect the assigned employee. Feedback should be provided to the assigned employee's supervisor regarding performance. Commitment of a person and the time they need to be available must be considered along with other duties, which may need to be modified appropriately. A determination must also be made of whether the Union needs to be involved.

External Expertise. Any required external expertise should be identified during the planning of the risk assessment. This might includes consultants, advisory committees, research personnel, the interagency Risk Assessment Consortium, and others. Additional experts may be identified during the conduct of the risk assessment as new needs arise.

Other Resources. Sufficient resources needed to conduct the risk assessment should be made available through either the requesting Program Office or the Office of Science. An administrative lead should be identified and work on the project to provide and administer resources needed by the risk assessment team. These resources might include file cabinets to store/sort references, work space (such as tables, desks in a dedicated room), computers, software, access to copy machines, and other needs as identified.

Research needed to complete the risk assessment should be identified and conducted, if feasible within the timeframe developed for the completion of the assessment.

Project Budget. A budget for the project must be provided. Specific items will depend on the scope of the project but might include the following:

Travel for CFSAN staff to collect data/information
Travel for CFSAN staff to attend meetings to discuss/present the assessment
Travel to bring experts to CFSAN to discuss data, model, etc.
Costs associated with holding public meetings
Printing costs for the draft and final documents
Costs to photocopy publications and purchase books or other reference materials needed by the team and also for display in the public dockets
Specialized software such as project management, reference organization, or modeling tools
Costs associated with peer review of the document
Specialized training for risk assessment participants

Develop Timeline and Milestones

The identification of project timeline and milestones that are definable and clear must be established. Milestones will be used to judge whether the project is on track, delayed, or ahead of schedule. A sufficient number of milestones should be identified to ensure continual progress of the risk assessment. The risk analysis coordinator (with input from the risk assessment project manager) is responsible for keeping the risk management aware of any problems in meeting milestones. Whenever possible, these should be identified early enough so that corrective actions can occur to either realign the risk assessment and put it back on track or alter the timeframe, as appropriate. The risk management leader and the risk analysis coordinator should discuss the problems and propose solutions. The risk analysis coordinator discusses options with the risk management leader and the risk assessment project manager is responsible for implementing the recommended solutions.

The need for and timing of publications in the Federal Register and public meetings should also be discussed during the planning phase.

The timing of and amount of time needed to review draft and final models and documents should be discussed and agreed to so that the commitment and pathway is clearly understood. The risk assessment project manager in consultation with the risk analysis coordinator should discuss the type of review(s) needed, when the review is needed, and assist with making arrangements for acquisition of experts in a timely manner. During different stages of the risk assessment, different information needs to be reviewed by various parties. Examples of the types of information to be reviewed include:

Scope of risk assessment
Data
Modeling assumptions
Model
Results (summary tables)
Draft technical and summary documents

Determine Communication Needs

Interaction between the risk communication and risk assessment and risk management teams is critical throughout the risk assessment process. The risk communication team will need to identify risk communication needs for each unique audience. An analysis of the awareness and knowledge of the issues for each audience as well as the best method for reaching them is critical in preparing risk communication messages, materials, and determine the appropriate channels of communications. Once audiences have been characterized, the next step is to determine communication strategies for communicating that include both one-way communications (messages, materials) and two-way communication (listening to audience needs, open exchange of information). It is important that the broader risk communication messages meet the needs of the variety of audiences. See Appendix C for additional information about risk communication activities.

Step 2. Perform the Risk Assessment

Following the planning step, the risk assessment is commissioned. When the risk management team provides the 'charge' to the risk assessment team, the performance phase of the project can be initiated.

Goal: Answer the risk management question(s)

Product: Draft report of the results and conclusions of the risk assessment

Activities: Performing the Risk Assessment

Review and refine (as needed) the scope of assessment
Collect data and information
Develop and validate model
Review results
Draft technical and interpretive summary documents

Risk assessment is a process used to evaluate the likelihood of adverse human health effects occurring after exposure to a pathogenic microorganism. Or more simply stated, it describes what we know and how certain we are of what we know. The risk assessment is also a product--a mathematical model, predications based on simulations, and a report that documents the process.

The generally accepted risk assessment paradigm for food contaminates includes separating the assessment activities into four components: hazard identification, exposure assessment, hazard characterization or dose-response, and risk characterization. This process allows organization, characterization, assessment of uncertainties, and identification of data gaps. See Appendix A for a detailed description of the components of risk assessment.

Figure III-2 provides the overall process of conducting a risk assessment and the various activities. The iterative nature of the risk assessment is illustrated by the feedback loops. The various activities needed to perform the risk assessment are described in greater detail below.



Figure III-2. Activities in Performing a Risk Assessment

Clarify scope of the risk assessment

During the planning step, approval for team members to participate on the risk assessment team would have been obtained from program offices. The conduct of the risk assessment begins when the risk management formally issues a charge (the questions for the assessment to answer and key assumptions to use).

The risk assessment project manager and risk analysis coordinator will hold a kick off meeting with the risk assessment team members to discuss administrative and technical issues concerning the project. At this meeting the risk assessment team members should discuss any concerns or questions about the scope of the project and conduct of the assessment. The risk assessment project manager and the risk analysis coordinator will discuss these concerns. Any potential roadblocks to completing the assessment will be forwarded by the risk analysis coordinator to the risk management team and any other appropriate parties such as the science advisor. Most importantly, each team member needs to understand his or her duties and responsibilities for the conduct of the assessment.

Example agenda items for the kick off meeting might include:

Comments/ concerns about the scope of the project and key assumptions as presented in the "charge"

Timeline/timeframe/milestones for the project

Resources allocated and any additional needs not previously identified

How and when to communicate with the risk management and communication teams

Identify tasks and assign responsible parties

Development of conceptual model, if not done during the data feasibility evaluation (see Part II of this document)

Procedure for including documents in the repository for the project, including all references cited in the report and decisions made during the conduct of the assessment

Consultation with Risk Managers and Risk Communicators During the discovery and summarizing of data and assembling the model, it is likely that issues will arise that are appropriate for risk managers to decide. These issues need to be separated from technical and scientific issues that are appropriate for the risk assessment team to decide.

The risk management and risk communication teams will be briefed on the assessment at regular intervals during the conduct of the project. Risk management team approval/concurrence will be obtained on the assumptions used when data are lacking, types of data used in the model, additional research required, model structure and logic, and other milestones as agreed to during the planning step.

It is the risk assessor's responsibility to explain the quantitative results of the risk assessment in a way understandable to the risk management and risk communication team member; they should not be expected to translate the results. In addition, it is the risk assessors who need to describe their confidence in their conclusions, and the uncertainties inherent in the risk assessment, and the impact of assumptions made by the RM on the results. There should be an iterative dialog between risk assessors and risk communicators to ensure that the risk assessment reports are sufficiently clear and will answer questions and concerns raised by stakeholders.

The risk assessment team is responsible for:

maintaining the scientific credibility of the assessment including transparency and quality, and consistency

informing the risk management team of the impact of adding, removing, or altering data and assumptions

information risk managers of any deficiencies in data needs or resources

describing the uncertainty in the results

refining the key assumptions provided by the risk management team for use in modeling

assembling the data collected and incorporating these data into appropriate models chosen by the risk assessment team

explaining the quantitative results of the risk assessment in a way understandable to the risk management team

describing their confidence in their conclusions, taking into consideration the uncertainties and key assumptions incorporated in the model

Milestones. While risk managers, risk communicators, and risk assessors must confer through the process of conducting the risk assessment, the achievement of specified milestone events should trigger an interaction point. The risk management team should "check off" on what the risk assessors have completed for each milestone, thereby avoiding time-consuming revisions later in the process. These milestones will also be used to track progress of the risk assessment and determine whether the project is on schedule. Specific milestones should be selected during the planning process. Examples of milestone events include:

Receiving 'charge' from risk managers
Assembly of the risk assessment team
Agreement of key objectives and assumptions
Publishing the FR notice announcing the risk assessment
Assembly of data and assumptions to be used in modeling
Development of model
Results from model simulations
Results of sensitivity analysis
Draft report
Federal Register Notice of assessment availability
Receipt and review of public comments

Documentation Procedures

The risk assessment project manager is responsible to setup and maintain (with assistance from a clerical assistant) a repository for all documents generated during the conduct of the risk assessment. Documents related to risk management and risk communication should be separately maintained by the respective team leads. The files from the risk management and risk communication teams may be transferred to the risk assessment project managers for archiving when the project is completed.

Examples of risk assessment files to be included in the archived records:

Record of correspondence

Record of decisions

Federal Register notices

All references cited in the risk assessment

Contents of the public docket for the assessment (display items and submissions)

Collect/Assemble Data and Information

Under the leadership of the technical team leader and with assistance from the project manager, the team will collect data and information needed for the assessment. This will include the information and data obtained during the data feasibility evaluation conducted as part of the Identification and Selection process (see Part II of this document). A conceptual model may have been developed during the data feasibility evaluation. If available, the conceptual model will be used to guide the data collection. If not available, the risk assessment team should develop the conceptual model before extensive data collection efforts are initiated. Key assumptions provided by the risk management team will be refined for use in modeling, data will be assembled and the model constructed.

As data are collected and assembled for use in the assessment and as inputs for the model, the team must clearly identify the method used to collect data and the criteria used to determine the suitability of the data for use in the model.

Complete reference citations as well as a complete, legible copy of each reference must be given to the risk assessment project manager. These references will be filed in the risk assessment repository and included in the public dockets with the risk assessment.

Some of the information and data might be made available from research carried out or commissioned after the start of the assessment. The risk assessment team should decide what additional research they would like performed and make recommendations to the risk managers.

Develop and Validate Model

The process of developing the risk assessment model should be initiated in stages. Immediately after compilation and collation of all available data it will not be immediately evident what are the most relevant or key parameters of the process or how the relationships between parameters should be modeled. Consequently, the process of modeling is best viewed as an iterative process itself and the elements of the models as dynamic.

Initially, it can be of some advantage to develop a prototype of the risk assessment as quickly as possible in order to consider and evaluate as quickly as possible what types of modeling assumptions are likely to be the most influential. For example, in a process pathway model the distribution of a pathogen prior to the process may be very well characterized but the effect of the process on the pathogen levels may be relatively unknown. In such an event, concentration of effort on delineating plausible model structures for the effect of the process is likely to be more productive and facilitate interaction with RM in regard to plausibility of assumptions. Although uncertainty analysis of components of the model better characterized by available data should be considered, this may not be of paramount concern during the initial stages of model construction. The modeling process must be transparent in order to identify information and data gaps. The structure and flow of the risk assessment must be determined.

This includes specifying:

the individual data

how the separate data sets relate to each other (i.e., growth rates are multiplied by storage times to give the amount of growth)

what variations and uncertainties are assigned to a data set

what branching may exist in the data flow

how much detail (number of components) should be in each section of the risk assessment

Decisions must also be made regarding which data to include and how to organize the data in a manner suitable to modeling.

Assumptions. Examples of modeling assumptions include decisions on the type of distribution to use for various aspects of the model. Other assumptions include use of surrogate data for data gaps. The risk management team should be briefed on the assumptions and their likely impact on the results.

Data and model verification/audit. Before running the model, the data inputs will be audited/verified and if possible, the model programming will be audited. The risk management team will have oversight of the assessment process and should review and evaluate the data and model at regular intervals during the conduct of the risk assessment.

CFSAN's Quality Assurance unit might be helpful in assisting in the verification/audit of the data used in the assessment and model. Experts in mathematics and modeling techniques should perform the audit/verification of the model. These experts needed to audit the model may be available from within CFSAN, within FDA, from other federal agencies, from academia, or through contracts with private consultants.

Data quality. The development of data quality criteria must be consistent with the Office of Management and Budget (OMB) guidelines for ensuring and maximizing the quality, including objectivity, utility, and integrity of information disseminated by Federal Agencies (see www.whitehouse.gov/omb/fedreg/reproducible.html).

The Joint Institute of Food Safety and Applied Nutrition (JIFSAN) has initiated a project to define data quality criteria for data used in risk assessments that are posted on the JIFSAN Food Risk Clearinghouse web site. The following are a few exerpts from a public meeting held in December 2000, on data quality that was co-sponsored by the interagency Risk Assessment Consortium and JIFSAN.

Observations in a database should be 'model-free' to allow for data to be used in ways that the original investigator may not have intended.
Data are rarely collected with risk analysis in mind, so many types of data must be considered for inclusion.
The purpose and stage of the risk assessment process determines the data that will be used.
"Good" data are complete, relevant and valid; complete data are objective, relevant data are case-specific, and validation is time-dependent.
- Complete data includes such things as the source of the data and the related study information (sample size, species studied, seasonality, sensitivity, specificity and precision, and data collection method).
- Relevant data include age of data, region or country of origin, purpose of study and species involved. Relevant data is difficult to judge and may be case-specific.
- Valid data is that which agrees with others in terms of comparison of methods and development of tests.
Data that can be eliminated from the risk assessment depends on the stage of the assessment and the purpose of the assessment. In the early stages of risk assessment, small data sets or those with qualitative values may be useful whereas the later stages of risk assessment it may be more reasonable to include only those data that have been determined to have high quality standards.
Non-validated data can be included in certain circumstances, such as unique studies that are being reported for the first time.

Additional information concerning on data quality and other related issues can be obtained from the JIFSAN Clearinghouse web site (www.foodriskclearinghouse.umd.edu/Dec_5_2000dataquality.html).

Model Validation. Validation is a process by which a simulation model is evaluated for its accuracy in representing a system. All models are, by their nature, incomplete representations of the system they are intended to model, but, in spite of this limitation, some models are useful. Model validation is the process by which the model is evaluated on its usefulness. One of the more direct ways microbiological risk assessments are validated is to compare model predictions of illness with epidemiological data. Another way of validating is to compare model predictions with survey data (or other data independent of the data used in the model construction) at intermediate steps to the final prediction.

Model validation is not something that is done at the end of the risk assessment; it involves multiple steps:

creation and validation of a conceptual model
conversion of the conceptual model into a simulation program
verification of proper translation of model into a computer program
running of the simulation program
validation of results of the simulation program
communication of the results to the risk managers

Sensitivity/Uncertainty Analysis. If a risk assessment is complex, it may be difficult to ascertain which model components (and supporting data) are the most influential in terms of likely impact on the predictions made. A technique known as sensitivity analysis can be used to identify a part(s) of the model that is the most influential. In the broad sense, sensitivity analysis should encompass a systematic evaluation of the model, underlying and alternative model assumptions, and influence of input model parameters on the output distribution(s). Available Monte Carlo simulation programs, such as @Risk, commonly provide the option to perform sensitivity analysis by producing graphs or rank correlation statistics between input parameters and output parameter(s). These correlation statistics are useful with respect to evaluating the relative impact of each input distribution on the output distribution. However, these statistics are generated after the specification of a preferred model and therefore do not necessarily reflect the potential effect of alternative assumptions. Nevertheless, provided underlying model assumptions are considered reasonable, correlation statistics produced by a sensitivity analysis allows those input distributions where uncertainty has the greatest impact on the outcome to be identified. This, in turn will point to those parts of the risk assessment where additional research may be expected to have the most impact.

Where the distribution in the data may be assigned to variation and uncertainty, a two dimensional sensitivity analysis may provide additional insight. The parameters that are the most important for the central values may not be the most important for the tails of a population distribution. This is especially likely to be true if probability trees are used to represent model uncertainty in frequency distributions -- all the models may give similar results for central values, but diverge widely at the tails.

Another approach to understanding the impact of assumptions and data on the results is to run "what if" scenarios. For very complex models, this is a simpler way of determining the uncertainty. For each scenario, specific assumptions or data are changed and the results are compared to the baseline to determine what if any impact the change had on the results.

Evaluating the Value of More Data. Research proposals are often justified on the basis of an expectation that they will reduce uncertainty, sample variability, or both. Until the research is complete, it is not known what results will be achieved, and although the expectation is for uncertainty to be reduced, this may be not realized. It is always possible that additional research will reduce or increase the uncertainty. Nonetheless, additional research may be useful, and we have identified three general purposes for additional research.

Measurement of values used directly in risk assessment (e.g. in empirical distribution functions). For instance, if a risk assessment makes use of a population distribution for body weight, the risk assessment may be improved by measuring the body weights of more individuals, thus defining variability in a population.
Collection of data that will allow more accurate estimates of model parameters. Parameters are not measured directly -- they gain their meaning through their place in a model or theory. Therefore, the influence of additional data will depend on the way in which the data are used to judge the theories (e.g. the goodness-of-fit and parsimony criteria). If the models are only supported by data directly (there are no coherence arguments in support of the theory), then the problem is collecting more observations. If enough observations become available, then the theory may be discarded altogether in favor of an empirical distribution. However, if coherence arguments are used in support of the theory, then strengthening the theories that support the parameter estimates might be more important.
Collection of data that will allow discrimination among models. The value of additional data in discriminating among models will also depend on the logic used to identify preferable models. The most useful data for this purpose may differ from that used for parameter estimation. Observations made under conditions where the candidate theories diverge in their predictions are likely to have the most impact. Of course, producing data that favors a particular model or hypothesis will only reduce the uncertainty of a particular risk assessment if model uncertainty is represented in that assessment.

Initiating research is in itself a decision. Whether or not a particular avenue is pursued will also depend on the results that might reasonably be expected and the time, effort, and expense required to obtain them.

Prepare the Draft Document(s)

Review Results. The results of the risk assessment modeling must be reviewed for:

Reasonableness
General trends
Whether the risk assessment question(s) posed by the risk management team was answered
Need for sensitivity/uncertainty analysis

Technical report vs. Interpretive Summary. Documents should be available in several mediums. Printed copies should be available though the CFSAN Outreach and Information Center, electronic copies should be available on the CFSAN web site, and/or CD-ROM.

The purpose of the technical document is to provide all information needed to reproduce the assessment. This document will be lengthy and technical but must be written in plain language. For the most part, the data used in the assessment should be placed in appendices.

The purpose of the interpretive summary is to explain how the assessment was conducted, the results and conclusions, and importance of the conclusions in a way that non-scientists can understand. The interpretive summary is much shorter in length than the technical document, only about 20 to 25 pages (maximum).

Transparency and Clarity. The risk assessment document(s) should be transparent and clear. Transparency includes describing the approach, assumptions and their impact on the conclusions, use of data/distributions/surrogate data, assumptions and impact on the results, data gaps and future research needs, the uncertainty in the results. Clarity can be achieved by avoiding jargon, using plain English, defining technical terms, and including well designed tables and graphs.

Style. While it is more important that the chosen style be consistently used within the document, some guidelines follow:

Use Microsoft Word
Use an appropriate and consistent font size such as Times New Roman, 12 point
Use 1 or 1 1/2 lines spacing.
Left margin justify, do not justify right margin.
Use the Harvard system for references. For example cite last name and date of publication, for example Levitt et al., 2002. Do not cite references from a numbered list.
As a general guideline, tables and figures should contain enough information to "stand alone"

Format (Technical report). The specific sections to be included in the technical risk assessment document will vary pending the specific needs of the document.

Below is a guideline to the overall organization of a technical document for a major risk assessment.

Title page
Preamble [for a draft document this might include a list of questions for which CFSAN is requesting input; for a final document this might include a summary of the public comments submitted]
Contributors
Acknowledgements
Executive Summary
Abbreviations/acronyms
Glossary
Introduction
Hazard Identification
Exposure Assessment
Hazard Characterization [Dose Response Assessment]
Risk Characterization
Research Needs
Interpretation and Conclusions
References
Appendices (include data, description of model, ancillary information)

Note: The overall structure of the interpretive summary document should generally follow that of the technical document.

Content (Technical Report). The technical document should include sufficient detail to allow the results of the assessment to be independently verified.

Examples of the types of information included:

Purpose of the assessment including the risk management problem and risk assessment question(s) to be answered
List of key assumptions and modeling assumptions and description of the impact of these assumptions on the results
Tabular or graphic depiction of distributions
Justification for selected distributions
Identification of data source and strengths and weaknesses of the data used
Description of the models
Sensitivity analysis

Step 3. Reviewing the Risk Assessment Documents

A thorough review process is needed to prepare document(s) for release to the public. The review step includes peer review and agency approval or clearance. Review may occur at various points during the conduct of the assessment and the results of a review could necessitate that the model be re-run and document revised.

Goal: Critical review, approval, and agency clearance of the assessment

Product: Risk assessment document, ready for release to the public

Activities: Reviewing the Risk Assessment Documents

Risk analysis teams review
Peer review
Stakeholder input
Clearance/approval

Risk analysis team review. The risk assessment team will actively seek comments and consult with the risk management and risk communication teams throughout the conduct of the risk assessment. Specific milestones that identify critical points when the risk assessment team should brief the risk management team will be selected during the planning step. By the time the risk management and risk communication teams review the draft document, it is expected that they are thoroughly knowledgeable about the methodology used and general conclusions.

Peer Review. Currently, CFSAN does not have a formal peer review process policy. Such a policy should be established. For example, the International Life Sciences Institute (ILSI) had developed an independent Model Peer Review Center of Excellence to evaluate toxicity assessments and associated proposed toxicity values for chemicals identified as contaminants of concern at Superfund hazardous waste sites. This process could be adapted to meet CFSAN needs.

In order to evaluate data quality, the criteria used for data selection must be specified in the risk assessment document. Additionally, the risk assessment teams should solicit critical reviews of the methods and models from subject matter experts within and outside of the Center. In order to ensure that we have used the best information available, CFSAN should also make a draft of the risk assessment available to the public for review.

Government and Non-government Experts. Whenever possible, subject matter experts in the government and special government employees (SGE) should be involved in the peer review. SGEs are held to confidentiality agreements that allow draft documents to be reviewed before they are released to the public. This is important because the iterative nature of risk assessments often means that subsequently requested changes to data or model might change the results and conclusions of the document. Circulating preliminary documents to the public before they have undergone extensive Center review and approval causes confusion and should be avoided.
RAC. The interagency Risk Assessment Consortium^{( 1 )} (RAC) is an important resource for technical advice on the use of data and modeling techniques. As appropriate, the RAC members should be involved in reviewing and providing critical feedback on the draft risk assessments.
Advisory Committee. The advice and opinions of advisory groups such as the National Advisory Committee on Microbiological Criteria for Foods^{( 2 )} (NACMCF) should be solicited. An initial meeting should occur early in the conduct of the risk assessment. Additional meetings to discuss scientific issues such as the approach, assumptions, and model structure should also be held, as needed. An additional resource is the National Academy of Sciences.

Stakeholder input. Stakeholder input is a critical component of the process and necessary to meet CFSAN's requirement for a transparent process. Workshops and/or public meetings may be held to provide clarity and obtain input from stakeholders regarding both the scope of the assessment, assumptions made, and results. Obtaining stakeholder input is essential to the risk communication process. CFSAN will also provide feedback both formally and informally as to how stakeholder concerns and comments have been addressed. Stakeholders should be encouraged to submit comments to the public docket established for each risk assessment project. Informal meetings and technical presentations by risk assessors may also be made at the request of specific groups, industries, or trade associations. CFSAN should attempt to honor such requests whenever possible. At times it may be more efficient to have an informal meeting at CFSAN for specific identified stakeholders.

Means of informing and listening to stakeholders include:

Workshops

Public meetings

Comments submitted to the public docket

Informal meetings

Technical presentations, as requested

Federal register notices

Documents on the CFSAN web site

Clearance/Approval

CFSAN's process for clearing and approving of documents must be followed. This clearance/approval process ensures that materials released by CFSAN are scientifically and technically accurate. Form FDA 3451 (1/00) should be used for the purpose of tracking and documenting actions and clearances of responsible offices. A determination must also be made on whether the risk assessment documents require departmental review and FDA's commissioner and/or the DHHS secretary's signature. Collaborative projects with other government agencies will also involve consideration of the clearance process of those agencies.

The risk analysis coordinator should consult with the Regulations Coordination Staff to ensure that the Federal Register Notices are developed and reviewed using appropriate, standard operating procedures.

Step 4. Issuing the Risk Assessment Documents

In this step of the risk analysis framework, the risk assessment document(s) are issued. Following peer review and agency clearance, CFSAN will seek public opinion of the documents. A final (or revised) assessment will include a discussion of the public comments received and how they were addressed.

Goal: Develop a roll-out plan for issuing document(s)

Product: Publish the risk assessment technical and interpretive summary document(s)

Note: Some of the activities implemented during this step may have been initiated and discussed during the planning step; however, the focus at this stage should be implementing specific activities related to releasing the assessment to the public.

Activities: Issuing the Risk Assessment Documents

Develop and implement roll-out plan
Address public comments submitted to docket
Finalize and issue Report
Follow up and debriefing

Develop and Implement Roll-out Plan

In developing a roll-out plan, a number of issues need to be discussed and decided to determine how to best implement a plan. A roll-out plan cannot be prescriptive, it must be tailored to the specifics of the individual project and as such will be determined on a case-by-case basis. It is recommended that a one to two page analysis of the issues be developed to assist in implementing the roll-out plan. The risk communication team is essential to developing and implementing an efficient roll-out plan.

Some of the issues to be discussed include:

Audience. Who will be impacted by the assessment? Who needs to be informed - within and outside of CFSAN? Who is the target audience for each document? What tone should be conveyed by the messages? What groups will be interested (press/consumers)?

Means. How should the audience be informed so there is no misinterpretation of the results? Who needs to be involved within and outside of CFSAN? What vehicles should be used?

When/where. Timing of issuing documents and holding meetings? What immediate actions (if any) are needed? By whom?

Whenever possible, the plan should contain elements of proactive communication and outreach instead of being reactive. However, we recognize that this may not be possible in all situations.

Who Develops and Implements the Roll-out Plan? The risk communication team will be a major contributor to the development and implementation of key elements of the roll-out plan. The risk assessment project manager and coordinator should assist as needed. However, within CFSAN there are many resources and staff experienced in both communication and outreach activities that should be included. For example Executive Operations Staff, FDA's Office of Public Affairs, Regulations and Policy Staff, Food Safety Staff, Management Systems, and also staff located in individual program offices.

Example elements of the Roll-Out Plan

A press contact list

Talk points/ press announcements/briefing materials/Q and A's

Schedule and hold public meeting(s)

Plan and hold stakeholder briefing(s)

Plan and hold other briefings or presentations, as needed

Develop and publish the Federal Register Notice; Availability of document(s)

Make and distribute printed copies of document(s) to stakeholders and public

Prepare and post risk assessment document(s) on CFSAN's web page

[Materials developed and distributed by the risk communication team are described in detail in Appendix C.]

Public Dockets

Stakeholders must be permitted time to submit comments on the draft document. FDA's documents provided in the Federal Register and comments submitted are on display and available for public review at FDA's public dockets (FDA Dockets Management Branch, 5630 Fishers Lane, Room 1060, Rockville, MD 20852). A discussion of the comments submitted and how CFSAN addressed those comments in the risk assessment are presented in the final or next version of the document.

Issue Final Report

Following the public comment period, the draft risk assessment is revised and issued as a final (or next version) document. A notice of availability of the risk assessment must be published in the Federal Register.

Where Should the Risk Assessment and related documents be made available?

CFSAN's Outreach and Education Center (provides printed copies)

CFSAN's web site (electronic format)

JIFSAN Food Safety Risk Analysis Clearinghouse (link to CFSAN web site)

Public meetings (presentation)

Constitute Updates (brief description, faxed to stakeholders)

Other appropriate means identified in the roll-out strategy

Follow up or Debriefing

Following the completion of the risk assessment, the risk analysis teams should meet to evaluate the overall risk assessment process. Such an evaluation should focus on determining whether or not the procedures set forth in the this document were followed and, whether following the procedures set forth in this document facilitated and improved the overall process and resulting risk assessment document. To the extent that the procedures were not followed, or modifications made, the group should try and identify the reasons and determine whether modifications to these procedures should be made. The success of risk communication throughout and at the end of the process should also be evaluated (as described in Appendix C).

In particular, the group should look at the following:

Was the risk assessment 'charge' from the risk managers/program office sufficiently clear? Did the team take appropriate steps to resolve any concerns over the 'charge' in a timely manner such that the assessment could be modified or a decision to proceed/not proceed be made?
Were sufficient resources identified and provided? Were there any gaps identified that suggest the need for further capacity building?
Were key assumptions from the risk managers clearly provided? Did risk assessors/managers and communicators sufficiently discuss and understand the significance of such assumptions in a timely manner?
Were data gaps or other risk assessment assumptions identified by risk assessors clearly communicated to the other team members?
Were uncertainties clearly identified along with their resultant impact on the results? Was a sensitivity analysis performed?
Do risk assessors believe the results are scientifically sound and that the assumptions used followed best risk assessment practice? Were any conflicts between risk assessors and managers identified and if so, how were they resolved?
Were sufficient resources devoted to the conduct of the assessment including the assignment of appropriate staff that could adhere to schedule requirements?
Do risk managers feel they got an assessment that answered their question(s)? If not, why not and what could have been done differently?
At what point was the risk communication team brought into the process? Were stakeholder concerns identified early on in the process? Did these concerns result in any changes to the charge of the assessment/assumptions made, etc? Did the team feel it had sufficient information to facilitate the communication of results?
Were appropriate and sufficient project milestones identified? Did they facilitate the identification of problems and corrective actions?
Were any problems identified in a timely manner? Was there a clear process for resolving problems? What other steps (if necessary) would be needed to better resolve issues raised or problems identified?
Did sufficient communication between all team members occur? Was communication ongoing? Did team leaders sufficiently communicate? Did the appropriate level of information filter down to all team members?
Did people respond to the identified deadlines for providing comments? Were comments and changes made in the latter stage that had been identified earlier but not addressed? Did the review and release process significantly alter the assessment or result in the need for further work? Why?

In addition, after the risk assessment is completed, risk managers may ask for several types of additional research. For example, they may want epidemiological and statistical research to see how well the predictions of the risk assessment are corroborated by experience. As new information or data become available, CFSAN must determine whether the risk assessment should be revised. As new science and technology related to the hazard come into use, risk managers may want to see how those developments affect the conclusions of the risk assessment. If the risk assessment leads to a proposed regulation, risk managers will want research on the cost and effectiveness of potential risk mitigations.

¹ The interagency risk assessment consortium (RAC) was established upon the recommendation of the President's May 1997 Food Safety Initiative report. Its mission is to enhance communication and coordination among federal agencies to promote the conduct of scientific research that will facilitate risk assessments.

² NACMCF was formed in 1988 upon the recommendation of the National Academies of Sciences and is composed of experts on microbiological safety of foods. It serves as a scientific resource for the Departments of Agriculture, Health and Human Services, Defense, and Commerce.

National Food Safety Programs | Initiation and Conduct of All 'Major' Risk Assessments: Table of Contents

Foods Home | FDA Home | Search/Subject Index | Disclaimers & Privacy Policy | Accessibility/Help

Hypertext updated by dav 2002-JUL-22

U. S. Food and Drug Administration Center for Food Safety and Applied Nutrition March 2002