Chapter 1: Why Evaluate Your Program?

You should evaluate your program because an evaluation helps you accomplish the following:

• Find out what is and is not working in your program
• Show your funders and the community what your program does and how it benefits your participants
• Raise additional money for your program by providing evidence of its effectiveness
• Improve your staff's work with participants by identifying weaknesses as well as strengths
• Add to the existing knowledge in the human services field about what does and does not work in your type of program with your kinds of participants

Despite these important benefits, program managers often are reluctant to evaluate their programs. Usually this reluctance is due to concerns stemming from a lack of understanding about the evaluation process.

Common concerns about evaluation

Concern #1: Evaluation diverts resources away from the program and therefore harms participants. This is a common concern in most programs. However, because evaluation helps to determine what does and does not work in a program, it is actually beneficial to program participants. Without an evaluation, you are providing services with little or no evidence that they actually work!

Concern #2: Evaluation increases the burden for program staff. Often program staff are responsible for collecting evaluation information because they are most familiar with, and have the most contact with program participants. Despite this potential for increased burden, staff can benefit greatly from evaluation because it provides information that can help them improve their work with participants, learn more about program and participant needs, and validate their successes. Also, the burden can be decreased somewhat by incorporating evaluation activities into ongoing program activities.

Concern #3: Evaluation is too complicated. Program managers often reject the idea of conducting an evaluation because they don't know how to do it or whom to ask for help. Although the technical aspects of evaluation can be complex, the evaluation process itself simply systematizes what most program managers already do on an informal basis — figure out whether the program's objectives are being met, which aspects of the program work, and which ones are not effective. Understanding this general process will help you to be a full partner in the evaluation, even if you seek outside help with the technical aspects. If you need outside help, Chapter 4 provides some ideas about how and where to get it.

Concern #4: Evaluation may produce negative results and lead to information that will make the program look bad. An evaluation may reveal problems in accomplishing the work of the program as well as successes. It is important to understand that both types of information are significant. The discovery of problems should not be viewed as evidence of program failure, but rather as an opportunity to learn and improve the program. Information about both problems and successes not only helps your program, but also helps other programs learn and improve.

Concern #5: Evaluation is just another form of monitoring. Program managers and staff often view program evaluation as a way for funders to monitor programs to find out whether staff are doing what they are supposed to be doing. Program evaluation, however, is not the same as monitoring. Sometimes the information collected to monitor a program overlaps with information needed for an evaluation, but the two processes ask very different questions.

Concern #6: Evaluation requires setting performance standards, and this is too difficult. Many program managers believe that an evaluation requires setting performance standards, such as specifying the percentage of participants who will demonstrate changes or exhibit particular behaviors. Program staff worry that if these performance standards are not met, their project will be judged a failure.

This concern is somewhat justified because often funders will require setting such standards. However, performance standards can only be set if there is extensive evaluation information on a particular program in a variety of settings. Without this information, performance standards are completely arbitrary and meaningless. The type of evaluation discussed in this manual is not designed to assess whether particular performance standards are attained because most programs do not have sufficient information to establish these standards in any meaningful way. Instead, it will assess whether there has been significant change in the knowledge, attitudes, and/or behaviors of a program's participant population in general and whether particular characteristics of the program or the participants are more or less likely to promote change.

Guidelines for conducting a successful evaluation

You can maximize the benefits that evaluation offers by following a few basic guidelines in preparing for and conducting your evaluation.

Invest heavily in planning. Invest both time and effort in deciding what you want to learn from your evaluation. This is the single most important step you will take in this process. Consider what you would like to discover about your program and its impact on participants, and use this information to guide your evaluation planning.

Integrate the evaluation into ongoing activities of the program. Program managers often view evaluation as something that an outsider "does to" a program after it is over, or as an activity "tacked on" merely to please funders. Unfortunately, many programs are evaluated in this way. This approach greatly limits the benefits that program managers and staff can gain from an evaluation. Planning the evaluation should begin at the same time as planning the program so that you can use evaluation feedback to inform program operations.

Participate in the evaluation and show program staff that you think it is important. An evaluation needs the participation of the program manager to succeed. Even if an outside evaluator is hired to conduct the evaluation, program managers must be full partners in the evaluation process. An outside evaluator cannot do it alone. You must teach the evaluator about your program, your participants, and your objectives. Also, staff will value the evaluation if you, the program manager, value it yourself. Talk about it with staff individually and in meetings. If you hire an outside evaluator to conduct the evaluation, be sure that this individual attends staff meetings and gives presentations on the status of the evaluation. Your involvement will encourage a sense of ownership and responsibility for the evaluation among all program staff.

Involve as many of the program staff as much as possible and as early as possible. Project staff have a considerable stake in the success of the evaluation, and involving them early on in the process will enhance the evaluation's effectiveness. Staff will have questions and issues that the evaluation can address, and are usually pleased when the evaluation validates their own hunches about what does and does not work in the program. Because of their experiences and expertise, program staff can ensure that the evaluation questions, design, and methodology are appropriate for the program's participants. Furthermore, early involvement of staff will promote their willingness to participate in data collection and other evaluation-related tasks.

Be realistic about the burden on you and your staff. Evaluations are work. Even if your evaluation calls for an outside evaluator to do most of the data collection, it still takes time to arrange for the evaluator to have access to records, administer questionnaires, or conduct interviews. It is common for both agencies and evaluators to underestimate how much additional effort this involves. When program managers and staff brainstorm about all of the questions they want answered, they often produce a very long list. This process can result in an evaluation that is too complicated. Focus on the key questions that assess your program's general effectiveness.

Be aware of the ethical and cultural issues in an evaluation. This guideline is very important. When you are evaluating a program that provides services or training, you must always consider your responsibilities to the participants and the community. You must ensure that the evaluation is relevant to and respectful of the cultural backgrounds and individuality of participants. Evaluation instruments and methods of data collection must be culturally sensitive and appropriate for your participants. Participants must be informed that they are taking part in an evaluation and that they have the right to refuse to participate in this activity without jeopardizing their participation in the program. Finally, you must ensure that confidentiality of participant information will be maintained.

About this manual

This manual is designed to help you follow these guidelines while planning and implementing a program evaluation. Each of the chapters addresses specific steps in the evaluation process and provides guidance on how to tailor an evaluation to your program's needs. (Reminder: The ACYF bureau companion handbooks provide a discussion of evaluation issues that are specific to the type of program you manage.)

The manual is not intended to turn you into a professional evaluator or to suggest that evaluation is a simple process that anyone can perform. Rather, it is meant to provide information to help you understand each step of the evaluation process so that you can participate fully in the evaluation— whether you hire an outside evaluator or decide to do one with assistance from in-house agency staff and resources.