Practical Evaluation
of
Public Health Programs

Workbook

Practical Evaluation

of

Public Health Programs

Prepared by

The University of Texas-Houston Health Science Center
School of Public Health
and
The Texas Department of Health

Supported by

The Centers for Disease Control and Prevention
and
The Association of Schools of Public Health
Cooperative Agreement 97-0531(P)-BHPR

Provided through

The Public Health Training Network

PRACTICAL EVALUATION OF
PUBLIC HEALTH PROGRAMS
DEVELOPMENT TEAM
|

Principal Investigator
Hardy Loe, Jr., MD, MPH
Associate Dean, UT-Houston School of Public Health

Instructional Design
Allan J. Abedor, Ph.D.
Professor Emeritus
UT-Houston Health Science Center

Writing Consultant
Susan Griffin, B.S., M. P. Aff.

Nursing Consultant
Patricia Drehoble, RN, MPH
Director, Continuing Nursing Education Provider Unit
Centers for Disease Control and Prevention

PHTN Learner Support
Brian Siegmund, MS Ed
Supervisor, Distance Learning Support
Centers for Disease Control and Prevention

LISTING OF SECTIONS

Part I

Building Commitment

for Evaluation

PART I   OBJECTIVES

By the end of this program, you will be able to:

• Define program evaluation in practical terms
• Explain the importance of evaluation in the current public health environment
• Name three reasons evaluation is important to your program
• Using your program as an example, identify three barriers to evaluation and determine how they can be overcome

BUILDING COMMITMENT FOR EVALUATION

Why Do Evaluation?
With increased emphasis on accountability and quality of services from elected officials, the public, and the media, it is essential that health care organizations evaluate their programs focusing on improved outcomes, quality of services, and cost-effectiveness. The word "evaluation" often suggests a lengthy and costly process that requires the services of expert researchers and/or consultants. The purpose of this course is to present an alternative to the research model of evaluation. This approach is known as "practical evaluation."

Questions Evaluation Answers
In summary, an evaluation will provide a public health program with the answers to the following questions:

• What have you done?
• How well you have done it?
• How much you have done?
• How effective you have been?

Definition of Practical Evaluation

There are innumerable definitions for evaluation. The definition for evaluation used for this course is one proposed by Michael Quinn Patton in Practical Evaluation (1982)

This approach to evaluation is practical and doable, recognizing real world constraints such as lack of time, resources, knowledge, and political boundaries.

There are a number of important components to this definition of practical evaluation. The first component is that data is collected in a systematic manner. This requires that the collection of data for the evaluation be planned from the beginning. It is too late when report time arrives to discover that essential data elements have not been collected.

Secondly, this definition specifies that evaluation focus on the activities, characteristics, and outcomes of the program being evaluated. More and more grants and contracts are calling for performance measures, which is consistent with this component of the definition.

Another component of this definition states that the purposes of evaluation are to:

• reduce uncertainties
• improve effectiveness
• make decisions about the program

Practical program evaluation can provide feedback on program operations, expenditures, and results. It can be useful in deciding among budget alternatives, and in managing and reporting on uses of public funds.

Two other components emphasized in this definition of evaluation relevant to public health, are context and collaboration.

A practical evaluation is one that can reasonably be implemented within the context and constraints of a particular situation. Such constraints include inevitable limitations of time, resources, and knowledge as well as specific political boundaries. One example of context lies in the method of data collection and analysis for the evaluation. Data may not be available for certain evaluation questions within the time frame studied; personnel and automation resources may be limited in the collection of the required data; budgets may not include extra dollars to "out-source" the analysis of the evaluation data; and some of the evaluation questions may be politically "taboo" in some areas.

Collaboration is another important concept in the definition of evaluation. The team or collaborative approach involves a group of people who share the decision making for the major aspects of the evaluation. This approach to evaluation has many benefits. Using a collaborative approach to evaluation is one way to expand or maximize available resources. Collaboration is discussed in more detail later in this section.

Useful
Is it useful to evaluate your program? Will the results be used to improve practice or allocate resources better? Will the evaluation answer stakeholders' questions?

Evaluation is useful in developing new program proposals, in reauthorizing existing programs, in justifying requests for additional funding, in accounting for use of public funds, and especially in improving program performance.

Feasible
Is it feasible to evaluate your program? Given your specific political environment and current resources, can you afford to do it? Do you have adequate support to conduct the evaluation? Does your budget allow for contracting with outside professionals to conduct the evaluation? If not, do you have the personnel, time, and monetary resources to do it in-house? If you can not evaluate all aspects of your program, what parts can you evaluate?

Proper
Are you able to conduct the evaluation properly? Is the approach fair and ethical? For example, in conducting a survey, the results are intended to be kept confidential. Are you truly able to maintain that confidentiality within the constraints of your organization and with the types of data processing and analysis required?

Accurate
Is the evaluation accurate? It is important that appropriate data collection methods have been used and that the data have been consistently collected. For example, if you have several people involved in conducting interviews, have you provided adequate training to ensure consistency and quality in gathering the information?

PANEL DISCUSSION

TOPIC: The Importance of Evaluation in the Current Public Health Environment

WHY EVALUATION IS IMPORTANT TO YOUR PROGRAM

Improve Health Status

The primary focus of evaluation is to maximize your program's ability to improve health status in your community. For example, the Michigan Diabetes Outreach Network was established to ensure comprehensive diabetes management for persons with diabetes. The purpose of the evaluation study was to collect data on services delivered and their impact on patient outcomes, and to provide feedback to the partnering agencies to improve their performance. The evaluation results revealed a major reduction in hospitalizations, amputations, and mortality served by the network.

Demonstrate Accountability

Evaluation can help you demonstrate accountability to your funding sources, stakeholders, and community. For example, with the Michigan Diabetes Outreach Network, not only did results demonstrate that patient outcomes were improved, these positive outcomes resulted in significant increases in state appropriations for comparable programs throughout the state.

Manage Resources

Evaluation can help you examine what resources you need and/or whether you are using the resources you have effectively. For example, inspectors with a food and drug safety program were struggling with how to get all their inspections done and still have time to do the reports from each inspection in a timely manner. After evaluating the problem and the potential solutions, along with the program's resources, the program decided to purchase a laptop computer for each inspector. Now the inspectors take their laptops with them on their inspections and complete most of their reports on-site. The computers turned out to be the needed resources to allow the inspectors to use their time efficiently to get their jobs done.

Improve Program Operations

Evaluation can help you improve your program operations. In the previous example, the purchase of laptop computers significantly improved the food and drug safety program's operations.

In another example of using evaluation for program improvement, a nurse in a tuberculosis prevention and control program started reporting the number of patients on directly-observed therapy (DOT) by region. Tuberculosis program personnel in each region did not want to be lowest on the chart. The number of patients receiving DOT increased in every region, showing that collecting and sharing the numbers resulted in a change in staff behavior and an overall improvement in program outcomes.

An important benefit of carrying out the evaluation process is the opportunity for program personnel to learn about their own programs. Persons who view evaluation as an important way to learn about the program and make improvements are better able to influence their program activities and outcomes than those who view evaluation as a time-consuming bureaucratic requirement. They are also more competent in representing both accomplishments and resource needs to policy-makers and the public at large. A graphic representation of how programs can "learn" from a practical evaluation is shown on the next page.

BARRIERS TO PROGRAM EVALUATION

PRACTICE EXERCISE 1

INSTRUCTIONS: List barriers that prevent you or your program from doing program evaluation.

BARRIERS TO EVALUATION

Lack of Management Support
Barriers to evaluation include lack of management support. If not initiated from the upper levels of management, independent program evaluations will often not receive the latitude and resources necessary to conduct a proper evaluation. Program personnel are often expected to collect additional data and perform the analyses in additional to their original duties. The result is often a poorly constructed and conducted evaluation with unreliable information on which to base decisions.

Lack of Resources
Lack of resources is probably one of the most common barriers to evaluation. Lack of time, adequate personnel resources, and necessary equipment can seriously hamper the evaluation process.

Lack of Skills
Lack of skills and resources in the collection, analysis, and interpretation of data fosters incomplete or inaccurate evaluation results. Many organizations simply do not have the necessary automation or personnel with the skills to construct interview or data collection instruments, or analyze and interpret the data. This could result in false assumptions and conclusions from inaccurate, missing, or irrelevant data.

Lack of Relevant Data
Lack of relevant data can render the evaluation useless. Unless data collection instruments and methods are carefully planned from the beginning of the evaluation period, missing, inconsistent, and untimely data will result in an incomplete evaluation. Certain questions will not be answered and the "holes" in the data can render the results meaningless.

Fear of Consequences
Fear of the consequences of the results will often preclude an evaluation. Evaluation is often threatening to the administrators or program managers whose programs are being evaluated. Evaluations may provide ammunition for those who want to reduce program expenditures or dramatically change the program's direction.

COMPARISON OF THE ACADEMIC RESEARCH MODEL AND PRACTICAL EVALUATION

Perhaps the biggest misunderstanding about program evaluation is that it must follow an academic research model. This is not necessary. The following is a comparison of the academic research and practical evaluation models.

In the academic research model, the focus is on testing hypotheses. The intent of practical evaluation is to improve practice. For example, in the academic research model, a hypothesis might be tested that smoking behavior has decreased due to a public health promotion wellness program. In the practical evaluation model, the emphases might be on which public health promotion methods are getting the best results.

Most people think of research as requiring a controlled environment or control groups. In public health practices, this is seldom realistic. So practical evaluation focuses on the context in which people work, making this a more realistic model for public health practitioners. For example, in the academic research model, there would have to be a stratified random sample of both wellness program participants and a control group. The practical model would examine the data over time from the wellness program participants.

The research model requires sophisticated data collection and analysis. In practical evaluation, data collection needs to be well thought out and organized. Sophisticated statistical analyses may or may not be used, depending on what is being evaluated, the type of data collected, and the availability of resources for the data collection and analysis.

OVERCOMING BARRIERS TO EVALUATION

Many of the barriers to evaluation can be addressed through a collaborative approach. One way to meet the useful/feasible/proper/accurate standards for practical evaluation is to involve people who will assist in the design and implementation of the evaluation.

The collaborative approach means forming a team of people who make shared decisions about the purpose of evaluation, how it is going to be conducted, and how the results will be interpreted and used. Team members should be chosen based on the skills and knowledge needed to conduct the evaluation. Often people outside the program can bring some important skills, objective viewpoints, and fresh insights to the process. A team approach can help overcome the barrier of lack of resources.

The Collaborative Approach
A collaborative approach can have the following benefits:

Reduces Suspicion and Fear
By involving stakeholders in the evaluation process, you reduce suspicions and fears. If people are involved in the decision-making process about what to evaluate and how the evaluation will be conducted, they are more likely to support the process. If they are not involved, they can be suspicious about the process and fearful of the outcome. They may not trust the results or the people involved in the evaluation. People often fear that evaluation will result in the termination of the program or of their job. Involving them will help them understand that the focus of the evaluation is to improve the program.

Increases Awareness and Commitment
Through a collaborative approach, participants become more aware of and committed to the evaluation process. If people are involved in the evaluation process, they have ownership in the results. An open presentation and discussion of the program's data and information can lead to the development of a consensus about interpretation and follow-up action.

Increases the Possibility of Achievement of Objectives
A collaborative approach increases the possibility of achieving objectives. If people understand what is being evaluated and why, they are more likely to work toward improving those elements of the program. This approach promotes ownership of the process and responsibility for the outcomes.

Broadens Knowledge Base
A collaborative approach draws on a broader base of knowledge, skills, and experience in evaluation. In choosing the people to be on the evaluation team, the role each person will play in the process and the knowledge, skills, and experiences they bring to the evaluation should be considered. This is a good opportunity to look both inside and outside the organization to tap the needed resources for the type of evaluation planned. Team composition should include people with a variety of backgrounds, including those with front line service experience, statistical and epidemiological expertise, management and policy perspectives, and planning skills. All of these skills may not be available within the program or even the organization. If this is the case, it is often helpful to call upon a community college or university in your area for assistance from either faculty or student interns.

Teaches Evaluation Skills
A collaborative approach can serve as a method for teaching evaluation skills to team members. When people work together, they share ideas, knowledge, skills, and abilities. The evaluation team members can learn about program objectives, data collection methods, making evaluation decisions, and even how to work on a team. Team members learn by doing and come away from the process with a greater set of skills.

Teaches Stakeholders
A collaborative approach can serve as a method for teaching community stakeholders about public health. Involving people outside the program in the evaluation process can broaden their knowledge about your program and its role in public health. This is an excellent way to involve the community in a discussion of health problems and possible interventions. For example, you might involve members of a local clinic, local elected officials, representatives from community-based organizations such as the American Heart Association. Any of these representatives would walk away from the evaluation experience with a better understanding of the strengths of your program, the constraints under which public health programs operate, what is needed for improvement, and how the program relates to their interests.

Increases Possibility Findings Will Be Used
By involving stakeholders, there is a greater possibility that the findings will be used and implemented. Many organizations are in the process of developing quality assurance programs. When a variety of staff are involved in conducting chart audits, identifying problems, and determining solutions to correct the identified problems, they are more likely to buy into the implementation of the solutions decided upon by the group.

Allows for Differing Perspectives
A team allows for differing perspectives. This goes back to the old adages: "Two heads are better than one" and "The whole is greater than the sum of its parts." Including people outside the program provides unique points-of-view.

PANEL DISCUSSION

TOPIC: The Importance of Evaluation to
Your Program

PART I: BUILDING COMMITMENT FOR EVALUATION

SUMMARY

Evaluation Tells You

1. What you have done
2. How well you have done it
3. How much you have done
4. How effective you have been

Definition of Practical Evaluation:

Program evaluation is the systematic collection of data related to a program's activities and outcomes so that decisions can be made to improve efficiency, effectiveness, or adequacy.

Components of Practical Evaluation

1. Data is collected in a systematic manner.
2. Evaluation focuses on the activities, characteristics, and outcomes of the program being evaluated.
3. The purpose of evaluation is to:

• Reduce uncertainties
• Improve effectiveness
• Make decisions

4. Practical evaluation is implemented within the context and constraints of your particular situation.
5. Practical evaluation involves collaboration.

Reasons to Do Evaluation

1.  To measure program achievement
2.  To demonstrate accountability
3.  To examine what resources are needed and how effectively they are being used
4.  To improve program operations

Barriers to Evaluation

1.  Lack of management support
2.  Lack of resources
3.  Lack of skills and resources in the collection, analysis, and interpretation of data
4.  Lack of relevant data
5.  Fear of the consequences of the results of the evaluation

Comparison of the Academic Research Model and Practical Evaluation

Overcoming Barriers Through a Collaborative Approach

1. Reduces suspicion and fear
2. Increases participant awareness and commitment
3. Increases the possibility of achieving objectives
4. Draws on a broader base of knowledge, skills, and experience
5. Teaches evaluation skills to team members
6. Teaches community stakeholders about public health
7. Increases the possibility that findings will be used
8. Incorporates differing perspectives
   
   

Part II

Framework for Planning

and

Implementing Program

Evaluation

PART II OBJECTIVES

By the end of this module, you should be able to:

• List the six steps of the evaluation framework
• Name the four standards for "good" evaluation
• Recognize the component parts of a well written program objective
• Apply the evaluation framework to a case study
  

OVERVIEW OF THE EVALUATION FRAMEWORK
Evaluation is the key to creating a very special kind of public health practice, a practice characterized by programs that actually learn and constantly -- systematically and intelligently -- improve their chances for success. All across public health, there are real-world examples of a shift toward learning through practical program evaluation. The changing circumstances of the work demand it.

The following framework shows how evaluation helps programs learn. This framework summarizes the steps in evaluation practice and the standards that define good evaluation.

EVALUATION FRAMEWORK

This framework has been developed by the Centers for Disease Control and Prevention Evaluation Working Group.

EVALUATION FRAMEWORK
NOTES

Steps in Evaluation:

Step 1: Engage Stakeholders

Step 2: Describe the Program

Step 3: Focus the Evaluation Design

Step 4: Gather Credible Evidence

Step 5: Justify Conclusions

Step 6: Ensure Use and Share Lessons Learned

Standards of "Good" Evaluation

1.  Utility

2. Feasibility

 

 

 

3.  Propriety

 

 

 

4.  Accuracy

Step 1. Engage Stakeholders
Public health work nearly always involves partnerships. Stakeholders are people who have a stake in what will be learned from an evaluation and what will be done with the knowledge. Stakeholders include:

• Insiders -- people who manage or work in the program or organization
• Outsiders -- those who are served or affected by the program, or who work in partnership with the program to achieve its goals
• Primary intended users of the evaluation -- people who are in a position to do or decide something about the program

The evaluation cycle begins by engaging stakeholders. Once involved, they help to carry out each of the other steps.

The evaluation team should include a variety of program staff and external stakeholders. Potential team members should represent:

• management
• program staff, including clerical or support positions
• people from the community, including contractors
• people receiving services of the program
• those with relevant technical expertise.

Some or all of these people should also be involved in the planning stage of the program. Each evaluation team member should have a clear understanding of their role and responsibilities in the evaluation process.

Before stakeholders can discuss the value of a program, they must agree on the description and scope of the program. Different stakeholders may have different ideas about what the program is supposed to achieve and why. Evaluations done without agreement on program description are likely to be of questionable use.

Often, the very act of describing the program in a way that can be evaluated produces benefits for the program long before results from the evaluation are available. With a solid understanding of the program, it is possible to summarize its key features and set up a frame of reference for all the evaluation decisions that follow.

A complete program description should include:

• Need -- What problem/opportunity does the program address (e.g., clean food and water, poor nutrition, dental health, diabetes)?
• Expectations -- What are the anticipated outcomes or objectives of the program (e.g., few reported food-borne and water-borne illnesses, improved pregnancy outcomes, reduction in the incidence of dental caries, prevention of complications of diabetes)?
• Activities -- What are the tasks involved in operating the program (e.g., regulatory actions, direct services, education)?
• Context -- What is the operating environment of the program (e.g., limitations of time, skills, resources, political climate)?

Types of Objectives
In terms of evaluation, the most practical way to describe your program is to have well-written, measurable objectives. Outcome and process objectives provide a quantitative measurement of change that the program/health department can and should accomplish by some future date.

Outcome Objectives
An outcome objective is:

a statement of the amount of change expected for a given health problem/condition for a specified population within a given time frame.

An example of an outcome objective:

By December 31, 2003, the incidence of measles among people in Marvelous County born after December 31, 1956 will be reduced from 7.9 cases per 100,000 in 1996 to no more than 2.5 cases per 100,000 population.

Outcome objectives are:

• long term (usually 3-5 years)
• realistic
• measurable

Outcome objectives are usually measured by:

• levels of mortality, morbidity, and/or disability, e.g., infant mortality rate, low birth weight rate, number of cases of measles
• levels of health conditions, e.g., hypertension, hypercholesterolemia
• behavioral measures, such as rates of smoking.
  
  

Process Objectives

A process objective is:

a statement that measures the amount of change expected in the performance and utilization of interventions that impact on the outcome.

Interventions can include:

• health services
• health education
• counseling
• regulatory actions
• legislative and policy changes
  

An example of a process objective is:

By December 31, 1999, 95% of the children six years of age in Marvelous County will have completed the primary immunization series.

Process objectives are:

• short-term (usually one year)
• realistic
• measurable

There must be a logical, practical relationship between the outcome and process objectives. This relationship is based on educated projections of how much of what types of interventions will result in the expected or desired change in the identified health problem. For any one outcome objective, there may be several process objectives. That is, there may be several different interventions that all lead to the same desired change in health status.

An example of how process objectives relate to an outcome objective is:

Outcome objective:

By December 31, 2003, reduce the percentage of eligible patients ages 3-6 with dental caries in Utopia County from 60% in 1996 to no more than 20%.

Process objectives:

By December 31, 1999, increase the percent of eligible patients less than 4 years of age in Utopia County who have received sealants from 85% in 1997 to at least 90%.

By 1999, 80% of eligible patients in Utopia County will receive oral hygiene instruction and preventive education twice a year.

Other interventions to address this outcome might include: dental hygiene and nutrition education at appropriate grade levels, water fluoridation, and parent contacts/education.

Well-written Outcome and Process Objectives

All outcome and process objectives must include certain components to make them useful tools for measuring changes in health problems and interventions.

* When is expressed as a fiscal year (local, state, or federal) or calendar year. Whether fiscal year or calendar is used often depends on how the data is collected, e.g., vital statistics are collected on a calendar year basis, whereas clinic visit data may be collected by calendar year or by fiscal year. Units of time smaller than one year are used only in activities.

** Who is not always spelled out in objectives, particularly in outcome objectives, if the responsible staff or agency is self-evident or is later defined in process objectives or activities. The other five elements are usually included.

***Preferably, how much is written as a target rate, a percentage or a specific number, with the current rate, percentage, or specific number included for reference purposes.

With the increasing emphasis on accountability and performance measurement, objectives must include a measurable component and the data to measure the objective must be available. In reviewing objectives you have written, you may find that you do not have the data sets or data collection systems to measure some of the objectives. You may have to rewrite some of the objectives to reflect what you can measure. Afterward, you can work on developing the data collection systems pertinent to what you need to be able to measure in the future.

PRACTICE EXERCISE 1
STEPS 1 AND 2

ENGAGING STAKEHOLDERS and
DESCRIBING THE PROGRAM

A. Identify specific people in your program who could be involved in the evaluation.







B. Identify specific people, groups of people, or organizations who are served or affected by your program.







C. Identify specific people or groups of people who will use the evaluation findings to make decisions about your program.

D. What problem/opportunity does your program address? What are the anticipated outcomes of your program?






E. What are some specific activities that you or your program do to achieve the program goal?






F. What are current issues or trends affecting your program?

It is not possible for an evaluation to address all questions for all stakeholders. There must be a focus. Is the purpose of the evaluation to improve your program, to improve health status, or to demonstrate accountability? The evaluation team must do advance planning and reach agreement regarding exactly:

• what questions the evaluation will attempt to answer (e.g., what outcomes will be addressed, what is the real intent or purpose of the evaluation)
• the process that will be followed (e.g., scheduling of meetings, deadlines)
• what methods will be used to collect, analyze, and interpret the data (e.g., available census data, client records, logs, interviews, surveys, expenditure reports)
• who will perform the different activities (e.g., collecting the data, analyzing and interpreting the data, writing the report)
• how the results will be disseminated (e.g., who is the intended audience, which specific people are in a position to actually use the findings)

By utilizing a team approach to the evaluation, a wide range of resources, skills, experiences, and viewpoints can be brought into the process. This ensures optimal input from both internal and external sources and increases the possibility that the evaluation will be useful, feasible, proper, and accurate.

Step 4: Gather Credible Evidence

Credible information is the raw material of a good evaluation. It must be perceived by all stakeholders as trustworthy and relevant to answer their questions. Evidence should be viewed in a broad sense. Evidence consists of information and data from a variety of sources.

Types of data include:

• Demographic descriptors
  
• Indicators of health status
  Morbidity
  Mortality
  Disability
  Health behavior related to illness, injury

• Qualitative indicators
  Community values
  Public and private policies

• Quality of life indicators
  
• Inventories of interventions
  Implemented by health agency
  Implemented by other agencies, practitioners

• Routine statistical reports
  Census data
  Demographic information about populations at risk
  Vital statistics
  Reported morbidity
  National Health and Nutrition Examination Survey (HANES) - National Center for Health Statistics
  National Hispanic Health and Nutrition Examination Survey (HHANES) -
     National Center for Health Statistics

  ---

  Behavioral risk factor survey
  Health Plan Employer Data and Information Set (HEDIS) -
  National Committee for Quality Assurance (HMO data)
  Hospital discharge diagnoses
  Financial reports of agencies, hospitals, etc.
  Economic analyses

• Published studies and surveys in local, state, and national literature
  
• Official reports of governmental legislative, executive, and judiciary branches
  
• Documents from task forces, working groups, advisory groups, and health
  and medical care organizations
  
• Media articles and reports
  
• Program reports
  Surveillance reports
  Log sheets of various program activities
  Service utilization
      Agency program
      Other community programs relevant to agency
  Program user satisfaction survey
  Special surveys
  Personnel time sheets
  Expenditure reports

It should be recognized that there will be data limitations and that it is acceptable to use imperfect data (e.g., state vs. local data; under-reporting of conditions) as long as these data are analyzed and interpreted appropriately. These limitations should be clearly stated in any report or presentation.

• Ensuring that everyone is using the same versions of forms
• Thorough training of interviewers
• Spot checking of consistency of coding
• Quality assurance activities (e.g., record reviews)

It all depends on what kind of information stakeholders will find credible.

Once the evaluation evidence is collected, care must be taken in reaching justified conclusions. The process of reaching justified conclusions involves several questions:

• What are the findings?
• What do these findings mean? That is, how significant are they (e.g., statistical significance, program/policy implications, size of the effect)?
• How do the findings compare to the objectives for the program? How have we decided to what degree the program is successful or not (e.g., are the findings good/bad, high/low, favorable/unfavorable)?
• What claims or recommendations are indicated for program improvement (e.g., what, if anything should be done)?

Step 6. Ensure Use and Share Lessons Learned

Finally, it is important to ensure that the evaluation findings are used and that the lessons learned are shared. One of the main purposes for doing an evaluation is to improve practice. One way to increase program success is to learn from evaluation feedback and then to use that knowledge for program improvement. Evaluation provides a way to discriminate between program strategies that are successful and those that are not. Evaluations that are not used or inadequately disseminated are simply not worth doing. The likelihood that the evaluation findings will be used increases through deliberate planning, preparation, and follow-up.

Some of the activities that promote use and dissemination include:

• Designing the evaluation from the start to achieve intended uses by intended users.
  
• Providing continuous feedback to stakeholders about interim findings, provisional interpretations, and decisions to be made that might affect likelihood of use.
  
• Scheduling follow-up meetings with primary intended users to facilitate the transfer of evaluation conclusions into appropriate actions or decisions.
  
• Disseminating both the procedures used and the lessons learned from the evaluation to stakeholders, using communications strategies tailored to their particular needs.
  

  ---

PRACTICE EXERCISE 3

STEPS 5 AND 6

JUSTIFY CONCLUSIONS and
ENSURE USE AND SHARE
LESSONS LEARNED

A.    If you were to evaluate your program today, to what degree has your program achieved its
        objectives? On   what factors and information did you base your decision? How would you justify
        your decision? What  information do you wish you had to help justify your conclusion?

B.    Based on your conclusion, what recommendations would you make to improve your program?

C.    How and to whom will you communicate the findings of your program evaluation?

D.    What recommendations would you make to improve the evaluation efforts of your program?

STANDARDS

Evaluation is closely tied to program practice. Many of these activities are part of everyone's daily job activities. In a sense, everyone conducts evaluations all day long, but these evaluations are very informal and generally the stakes involved are lower. As the stakes increase, for example when there are hard decisions to be made regarding the future of the program, then evaluations become formalized.

The center of the evaluation framework depicts standards for a "good" evaluation: utility, feasibility, propriety, and accuracy. These standards help make evaluation practical. They are well-supported decision principles to follow when faced with having to make real-world tradeoffs or compromises. For example, the standards can guard against creating an imbalanced evaluation -- such as one that is accurate and feasible, but useless; or one that would be genuinely useful, but is not feasible or ethical. An optimal (as opposed to ideal) evaluation strategy is one that accomplishes all steps in the evaluation framework in a way that accommodates the program context and meets or exceeds all relevant standards.

Evaluation is a way to improve and account for public health programs using methods that are useful, feasible, proper and accurate. In a sense, we evaluate all the time as a routine part of program planning and operation. But do we evaluate well? And do we agree on what evaluation is? The framework presented in this program and summarized below provides guidance for improving existing evaluation practice and a standard for further improvement.