Chapter 2: What is program evaluation?

Program managers and staff frequently informally assess their program's effectiveness: Are participants benefiting from the program? Are there sufficient numbers of participants? Are the strategies for recruiting participants working? Are participants satisfied with the services or training? Do staff have the necessary skills to provide the services or training? These are all questions that program managers and staff ask and answer on a routine basis.

Evaluation addresses these same questions, but uses a systematic method for collecting, analyzing, and using information to answer basic questions about a program — and to ensure that those answers are supported by evidence. This does not mean that conducting an evaluation requires no technical knowledge or experience — but it also does not mean that evaluation is beyond the understanding of program managers and staff.

What are the basic questions an evaluation can answer?

There are many different types of program evaluations, many different terms to describe them, and many questions that they can answer. You may have heard the terms formative evaluation, summative evaluation, process evaluation, outcome evaluation, cost-effectiveness evaluation, and cost-benefit evaluation. Definitions of these terms and others and selected resources for more information on various types of program evaluations are provided in the appendix.

You may have also heard the terms "qualitative" and "quantitative" used to describe an evaluation. However, these terms, which are defined in the glossary, refer to the types of information or data that are collected during the evaluation and not to the type of evaluation itself. For example, an outcome evaluation may involve collecting both quantitative and qualitative information about participant outcomes.

This manual is designed to avoid the confusion that often results from the use of so many terms to describe an evaluation. Instead, all of the terms used here are directly related to answering evaluation questions derived from a program's objectives.

There are two types of program objectives — program implementation objectives and participant outcome objectives. Program implementation objectives refer to what you plan to do in your program, how you plan to do it, and who you want to reach. They include the services or training you plan to implement, the characteristics of the participant population, the number of people you plan to reach, the staffing arrangements and staff training, and the strategies for recruiting participants. Evaluating program implementation objectives is often referred to as a process evaluation. However, because there are many types of process evaluations, this manual will use the term implementation evaluation.

Participant outcome objectives describe what you expect to happen to your participants as a result of your program, with the term "participants" referring to agencies, communities, and organizations as well as individuals. Your expectations about how your program will change participants' knowledge, attitudes, behaviors, or awareness are your participant outcome objectives. Evaluating a program's success in attaining its expectations for participants is often called an outcome evaluation.

An evaluation can be used to determine whether you have been successful in attaining both types of objectives, by answering the following questions:

ü Has the program been successful in attaining the anticipated implementation objectives? (Are you implementing the services or training that you initially planned to implement? Are you reaching the intended target population? Are you reaching the intended number of participants? Are you developing the planned collaborative relationships?)

ü Has the program been successful in attaining the anticipated participant outcome objectives? (Are participants exhibiting the expected changes in knowledge, attitudes, behaviors, or awareness?)

A comprehensive evaluation must answer both key questions. You may be successful in attaining your implementation objectives, but if you do not have information about participant outcomes, you will not know whether your program is worthwhile. Similarly, you may be successful in changing participants' knowledge, attitudes, or behaviors; but if you do not have information about your program's implementation, you will be unable to identify the parts of your program that contribute to these changes.

These evaluation questions should be answered while a program is in operation, not after the program is over. This approach will allow you and your staff to identify problems and make necessary changes while the program is still operational. It will also ensure that program participants are available to provide information for the evaluation.

What is involved in conducting an evaluation?

The term "systematic" in the definition of evaluation indicates that it requires a structured and consistent method of collecting and analyzing information about your program. You can ensure that your evaluation is conducted in a systematic manner by following a few basic steps.

Step 1: Assemble an evaluation team. Planning and executing an evaluation should be a team effort. Even if you hire an outside evaluator or consultant to help, you and members of your staff must be full partners in the evaluation effort. Chapter 3 discusses various evaluation team options. If you plan to hire an outside evaluator or an evaluation consultant, Chapter 4 provides information on hiring procedures and managing an evaluation that involves an outside professional.

Step 2: Prepare for the evaluation. Before you begin, you will need to build a strong foundation. This planning phase includes deciding what to evaluate, building a program model, stating your objectives in measurable terms, and identifying the context for the evaluation. The more attention you give to planning the evaluation, the more effective it will be. Chapter 5 will help you prepare for your evaluation.

Step 3: Develop an evaluation plan. An evaluation plan is a blueprint or a map for an evaluation. It details the design and the methods that will be used to conduct the evaluation and analyze the findings. You should not implement an evaluation until you have completed an evaluation plan. Information on what to include in a plan is provided in Chapter 6.

Step 4: Collect evaluation information. Once you complete an evaluation plan, you are ready to begin collecting information. This task will require selecting or developing information collection procedures and instruments. This process is discussed in Chapter 7.

Step 5: Analyze your evaluation information. After evaluation information is collected, it must be organized in a way that allows you to analyze it. Information analysis should be conducted at various times during the course of the evaluation to allow you and your staff to obtain ongoing feedback about the program. This feedback will either validate what you are doing or identify areas where changes may be needed. Chapter 8 discusses the analysis process.

Step 6: Prepare the evaluation report. The evaluation report should be a comprehensive document that describes the program and provides the results of the information analysis. The report should also include an interpretation of the results for understanding program effectiveness. Chapter 9 is designed to assist you in preparing an evaluation report.

What will an evaluation cost?

Program managers are often concerned about the cost of an evaluation. This is a valid concern. Evaluations do require money. Many program managers and staff believe that it is unethical to use program or agency financial resources for an evaluation, because available funds should be spent on serving participants. However, it is more accurate to view money spent on evaluation as an investment in your program and in your participants, rather than as a diversion of funds available for participants. Evaluation is essential if you want to know whether your program is benefiting participants.

Unfortunately, it is not possible to specify in this manual exactly how much money you will need to conduct your evaluation. The amount of money needed depends on a variety of factors, including what aspects of your program you decide to evaluate, the size of the program (that is, the number of staff members, participants, components, and services), the number of outcomes that you want to assess, who conducts the evaluation, and your agency's available evaluation-related resources. Costs also vary in accord with economic differences in communities and geographic locations.

Sometimes funders will establish a specific amount of grant money to be set aside for an evaluation. The amount usually ranges from 15 to 20 percent of the total funds allocated for the program. If the amount of money to be set aside for an evaluation is not specified by a funding agency, you may want to talk to other program managers in your community who have conducted evaluations. They may be able to tell you how much their evaluations cost and whether they were satisfied with what they got for their money.

Although a dollar amount cannot be specified, it is possible to describe the kinds of information you can obtain from evaluations at different cost levels. Think of the process of building a house. If you spend a small amount of money, you can build the foundation for the house. Additional money will be required to frame the house and still more money will be needed to put on the roof. To finish the inside of the house so that it is inhabitable will require even more money.

Evaluation is similar. Some general guidelines follow on what you may be able to get at different evaluation cost levels.

Lowest cost evaluations. If you spend only a minimal amount of money, you will be able to obtain numerical counts of participants, services, or products and information about the characteristics of participants. You also may be able to find out how satisfied participants were with the services or the training. But this is only the foundation for an evaluation. This information will not tell you whether you have been successful in attaining your participant outcome objectives. Also, at this cost level you will not have in-depth information about program implementation and operations to understand whether your program was implemented as intended and, if not, what changes were made and why they were made.

Low-moderate cost evaluations. If you increase your evaluation budget slightly, you will be able to assess whether there has been a change in your participants' knowledge, attitudes, or behaviors, and also collect in-depth information about your program's implementation. However, this is only the framework of an evaluation. At this cost level, you may not be able to attribute participant changes specifically to your program because you will not have similar information on a comparison or control group.

Moderate-high cost evaluations. Adding more money to your evaluation budget will allow you to use a comparison or control group, and therefore be able to attribute any changes in participants to the program itself. At this cost level, however, your information on participant outcomes may be limited to short-term changes—those that occurred during or immediately after participation in the program.

Highest cost evaluations. At the highest cost level, you will be able to obtain all of the information available from the other cost options as well as longer term outcome information on program participants. The high cost of this type of evaluation is due to the necessity of tracking or contacting program participants after they have left the program. Although followup activities often are expensive, longer term outcome information is important because it assesses whether the changes in knowledge, attitudes, or behaviors that your participants experienced initially are maintained over time.

Basically, as you increase your budget for an evaluation, you gain a corresponding increase in knowledge about your success in attaining your program objectives. In many situations, the lowest cost evaluations may not be worth the expense, and, to be realistic, the highest cost evaluations may be beyond the scope of most agencies' financial resources. As a general rule, the more money you are willing to invest in an evaluation, the more useful the information that you will obtain about your program's effectiveness will be, and the more useful these results will be in helping you advocate for your program.