Chapter 3: Who Should Conduct Your Evaluation?

One decision that must be made before you begin your evaluation is who will conduct it. Evaluation is best thought of as a team effort. Although one individual heads the team and has primary responsibility for the project, this person will need assistance and cooperation from others. Again, think of building a house. You may hire a contractor to build your house, but you would not expect this professional to do the job alone. You know that to build your house the contractor will need guidance from you and assistance from a variety of technical experts including an architect, electrician, plumber, carpenter, roofer, and mechanical engineer.

Similarly, in conducting an evaluation, the team leader will need assistance from a variety of individuals in determining the focus and design of the evaluation, developing the evaluation plan and sampling plan (if necessary), constructing data collection instruments, collecting the evaluation data, analyzing and interpreting the data, and preparing the final report.

What are some possible types of evaluation teams?

There are many types of evaluation teams that you could assemble. Three possible options for evaluation teams follow:

• An outside evaluator (which may be an individual, research institute, or consulting firm) who serves as the team leader and is supported by in-house staff (Team 1).
• An in-house evaluator who serves as the team leader and is supported by program staff and an outside consultant (Team 2).
• An in-house evaluator who serves as the team leader and is supported by program staff (Team 3).

Whatever team option you select, you must make sure that you, the program manager, are part of the team. Even if your role is limited to one of overall evaluation management, you must participate in all phases of the evaluation effort.

Team 1 option: An outside evaluator with support from program staff

Possible advantages:

• Because outside evaluators do not have a stake in the evaluation's findings, the results may be perceived by current or potential funders as more objective.
• Outside evaluators may have greater expertise and knowledge than agency staff about the technical aspects involved in conducting an evaluation.
• Outside evaluators may offer a new perspective to program operations
• The evaluation may be conducted more efficiently if the evaluator is experienced.

Possible disadvantages:

• Hiring an outside evaluator can be expensive.
• Outside evaluators may not have an adequate understanding of the issues relevant to your program or target population.

Selecting this team does not mean that you or your staff need not be involved in the evaluation. You and other staff members must educate the evaluator about the program, participants, and community. Other staff or advisory board members must also be involved in planning the evaluation to ensure that it addresses your program's objectives and is appropriate for your program's participants.

When deciding on your option, keep in mind that although hiring an outside evaluator to conduct an evaluation may appear to be expensive, ultimately it may be less expensive than channeling staff resources into an evaluation that is not correctly designed or implemented.

Team 2 option: In-house evaluation team leader with support from program staff and an outside consultant

Possible advantages:

• An evaluation team headed by an in-house staff member may be less expensive than hiring an outside evaluator (this is not always true).
• The use of an agency staff member as a team leader may increase the likelihood that the evaluation will be consistent with program objectives.

Possible disadvantages:

• The greater time commitment required of staff may outweigh the cost reduction of using the outside professional as a consultant instead of a team leader.
• A professional evaluator used only for consulting purposes may not give as much attention to the evaluation tasks as may be needed. Like the Team 3 option, Team 2 may be perceived as less objective than using an outside evaluator.

This second option is a good choice if you feel that you have sufficient staff resources to implement the evaluation, but need assistance with the technical aspects. An evaluation consultant, for example, may help with developing the evaluation design, conducting the data analyses, or selecting or constructing appropriate data collection tools. You will also want the consultant to help you develop the evaluation plan to ensure that it is technically correct and that what you plan to do in the evaluation will allow you to answer your evaluation questions.

Team 3 option: In-house evaluation team leader with support from program and other agency staff

Possible advantages:

An in-house evaluation team may be the least expensive option, but this is not always true. An in-house staff evaluation team promotes maximum involvement and participation of program staff and can contribute to building staff expertise for future evaluation efforts.

Possible disadvantages:

• An in-house team may not be sufficiently knowledgeable or experienced to design and implement the evaluation.
• Potential funders may not perceive evaluation results as objective.

This option presumably avoids the expense of hiring an outside professional, so it is generally thought to be less costly than other evaluation teams. However, because it requires a greater commitment of staff time, you may discover that it is just as costly as using an outside evaluator either as a team leader or as a consultant. You may want to conduct a careful analysis of staff time costs compared to outside consultant costs before you decide on this team option.

How can you decide what team is best for you?

Before you decide on the best team to assemble, you will need to consider two important issues.

Your program's funding requirements. Often a funding agency requires that you hire an outside evaluator to conduct your evaluation. This type of evaluator is often referred to as a third-party evaluator and is someone who is not affiliated with your agency in any way — someone with evaluation experience who will be objective when evaluating your program.

Your program's resources and capabilities. You can assemble different types of teams depending on your agency's resources and how you will use the findings. To determine what internal resources are available, examine your staff's skills and experience in planning an evaluation, designing data collection procedures and instruments, and collecting and analyzing data and information.

Also, examine the information you already have available through program activities. If, for example, you collect and review information from the Runaway and Homeless Youth Management Information System or the Head Start Program Information Report (or any other organized participant database or information system), you may be able to use this information as evaluation data.

If you conduct entrance and exit interviews of participants or complete paperwork or logs on participants' progress in the program, this information may also be used as part of an evaluation.

The checklist on the following page can help you decide what type of team you may need. Answer the questions based on what you know about your resources.

Whatever team you select, remember that you and your staff need to work with the evaluation team and be involved in all evaluation planning and activities. Your knowledge and experience working with program participants and the community are essential for an evaluation that will benefit the program, program participants, community, and funders.

The checklist above can help you select your evaluation team in the following ways:

ü If your answer to all the resource questions is "no," you may want to consider postponing your evaluation until you can obtain funds to hire an outside evaluator, at least on a consultancy basis. You may also want to consider budgeting funds for evaluation purposes in your future program planning efforts.

ü If your answer to question 1 is "yes," but you answer "no" to all other questions, you will need maximum assistance in conducting your evaluation and Team 1 (an outside evaluator with in-house support) is probably your best choice.

ü If you answer "no" to question 1, but "yes" to most of the other resource questions, then Team 3 (in-house staff only) may be an appropriate choice for you. Keep in mind, however, that if you plan to use evaluation findings to seek program funding, you may want to consider using the Team 2 option (in-house evaluation team with outside consultant) instead and trying to obtain evaluation funds from other areas of your agency's budget.

ü If your answer to question 1 is "yes" and the remainder of your answers are mixed (some "yes" and some "no") then either the Team 1 or Team 2 option should be effective.

The next chapter provides advice on how to locate, select, hire, and manage an outside evaluator or consultant. This information will be particularly helpful in assembling Teams 1 or 2. If you plan to conduct the evaluation using the Team 3 option, Chapter 4 may still be useful, because it provides suggestions on locating resources that may assist you in your evaluation efforts.