Chapter 4: How Do You Hire and Manage an Outside Evaluator?

Careful selection of an outside evaluator can mean the difference between a positive and a negative experience. You will experience the maximum benefits from an evaluation if you hire an evaluator who is willing to work with you and your staff to help you better understand your program, learn what works, and discover what program components may need refining. If you build a good relationship with your evaluator you can work together to ensure that the evaluation remains on track and provides the information you and your funding agency want.

Finding an outside evaluator

There are four basic steps for finding an evaluator. These steps are similar to any you would use to recruit and hire new program staff. Public agencies may need to use a somewhat different process and involve other divisions of the agency. If you are managing a program in a public agency, check with your procurement department for information on regulations for hiring outside evaluators or consultants.

Step 1: Develop a job description. The first step in the hiring process is to develop a job description that lists the materials, services, and products to be provided by the evaluator. In developing your job description, you will need to know the types of evaluation activities you want this person to perform and the time lines involved. Evaluator responsibilities can involve developing an evaluation plan, providing progress reports, developing data collection instruments and forms, collecting and analyzing data, and writing reports. If you think you need assistance in developing a job description, ask another agency that has experience in hiring outside evaluators for help. Advisory board members may also be able to assist with this task.

Step 2: Locate sources for evaluators. Potential sources useful for finding an evaluator include the following:

Other agencies that have used outside evaluators. Agencies in your community that are like yours are a good source of information about potential outside evaluators. These agencies may be able to recommend a good evaluator, suggest methods of advertising, and provide other useful information. This is one of the best ways to find an evaluator who understands your program and is sensitive to the community you serve.

Evaluation divisions of State or local agencies. Most State or local government agencies have planning and evaluation departments. You may be able to use individuals from these agencies to work with you on your evaluation. Some evaluation divisions are able to offer their services at no cost as an "in-kind" service. If they are unable to respond to a request for proposal or provide you with in-kind services, staff members from these divisions may be able to direct you toward other organizations that are interested in conducting outside evaluations.

Local colleges and universities. Departments of sociology, psychology, social work/social welfare, education, public health, and public administration, and university-based research centers are possible sources within colleges and universities. Well-known researchers affiliated with these institutions may be readily identifiable. If they cannot personally assist you, they may be able to refer you to other individuals interested in performing local program evaluations.

Technical assistance providers. Some Federal grant programs include a national or local technical assistance provider. If your agency is participating in this kind of grant program, assistance in identifying and selecting an evaluator is an appropriate technical assistance request.

The public library. Reference librarians may be able to direct you to new sources. They can help identify local research firms and may be able to provide you with conference proceedings that list program evaluators who were presenters.

Research institutes and consulting firms. Many experienced evaluators are part of research institutes and consulting firms. They are sometimes listed in the yellow pages under "Research" or "Marketing Research." They also can be located by contacting your State human services departments to get a listing of the firms that have bid on recent contracts for evaluations of State programs.

National advocacy groups and local foundations, such as The United Way, American Public Welfare Association, Child Welfare League of America, and the Urban League. The staff and board members of these organizations may be able to provide you with names of local evaluators. They may also be able to offer insight on evaluations that were done well or evaluators especially suited to your needs.

Professional associations, such as the American Evaluation Association, American Sociological Association, and the Society for Research on Child Development. Many evaluators belong to the American Evaluation Association. These organizations can provide you with a list of members in your area for a fee and may have tips on how you should advertise to attract an evaluator that best meets your needs. Additional information on these organizations is provided in the appendix.

Step 3: Advertise and solicit applications. After you have developed a job description, identified possible sources for evaluators, and found ways to advertise the position, you are ready to post an advertisement to get applications. Advertising in the local paper, posting the position at a local college or university, or working with your local government's human resource department (if you are a public agency) are possible ways of soliciting applications. Agency newsletters, local and national meetings, and professional journals are additional sources where you can post your advertisement.

It is wise to advertise as widely as possible, particularly if you are in a small community or are undertaking an evaluation for the first time. Several advertising sources will ensure that you receive multiple responses. You should build in as much time as possible between when you post the position and when you plan to review applications.

If you have sufficient time, you may want to consider a two-step process for applications. The position would still be advertised, but you would send evaluators who respond to your advertisement more detailed information about your evaluation requirements and request a description of their approach. For example, you could send potential evaluators a brief description of the program and the evaluation questions you want to answer, along with a description of the community you serve. This would give them an opportunity to propose a plan that more closely corresponds to your program needs.

Step 4: Review applications and interview potential candidates. In reviewing applications, consider the candidate's writing style, type of evaluation plan proposed, language (jargon free), experience working with your type of program and staff, familiarity with the subject area of your program, experience conducting similar evaluations, and proposed costs.

After you have narrowed your selection to two or three candidates, you are ready to schedule an in-person interview. This interview will give you the opportunity to determine whether you and the evaluator are compatible. As you do for other job applicants, you will need to check references from other programs that worked with your candidate.

What to do when you have trouble hiring an evaluator

Despite your best efforts, you may encounter difficulties in hiring an outside evaluator, including the following:

Few or no responses to your advertisement. Many programs, particularly ones in isolated areas, have struggled to obtain even a few responses to their advertisements. Check with your Federal Project Officer to find out whether he or she can offer you suggestions, consult with other programs in your community, and check with your local State or county social service agency to obtain advice. Your advisory board may also be useful in identifying potential evaluators. Another source may be an organization that offers technical assistance to programs similar to yours.

None of the applicants is compatible with program philosophy and staff. If applicants do not match program needs, you may find it helpful to network with other programs and agencies in your State to learn about evaluators that agencies like yours have used. A compatible philosophy and approach is most important — tradeoffs with proximity to the evaluator may need to be made to find the right evaluator.

The outside evaluator's proposed costs are higher than your budgeted amount. In this instance, you will need to generate additional funds for the evaluation or negotiate with your evaluator to donate some of their services (in-kind services).

Another option is to negotiate with a university professor to supervise advanced degree students to conduct some of the evaluation activities. Information about participants and programs is a valuable resource, providing confidentiality is respected. For example, you can allow a university professor to have access to program information and possibly to other evaluation records in exchange for evaluation services such as instrument development or data analysis.

Managing an evaluation headed by an outside evaluator

Often, when the decision is made to hire an outside evaluator, program managers and staff believe that the evaluation is "out of their hands." This is not true. An outside evaluator cannot do the job effectively without the cooperation and assistance of program managers and staff.

An evaluation is like any activity taking place within your agency — it needs to be managed. Program managers must manage the evaluation just as program operations are managed. What would happen if your staff stopped interviewing new participants? How long would it be before you knew this had happened? How long would it be before you took action? How involved would you be in finding a solution? An evaluation needs to be treated with the same level of priority.

Creating a contract

A major step in managing an evaluation is the development of a contract with your outside evaluator. It is important that your contract include the following:

Who "owns" the evaluation information. It is important to specify who has ownership and to whom the information can be given. Release of information to outside parties should always be cleared with appropriate agency staff.

Any plans for publishing the evaluation results should be discussed and cleared before articles are written and submitted for publication. It is important to review publication restrictions from the funding agency. In some instances, the funding agency may have requirements about the use of data and the release of reports.

Who will perform evaluation tasks. The contract should clarify who is to perform the evaluation tasks and the level of contact between the evaluator and the program. Some program managers have found that outside evaluators, after they are hired, delegate many of their responsibilities to less experienced staff and have little contact with the program managers or staff. To some extent, a contract can protect your program from this type of situation.

If this problem occurs even after specification of tasks, you may want to talk with the senior evaluator you originally hired to offer the option of renegotiating his or her role. The resolution should be mutually agreeable to program staff and the evaluator and not compromise the integrity of the evaluation or program. The contract should specify the responsibilities of program staff as well as the evaluator. These responsibilities may vary depending on the structure of your evaluation and the amount of money you have available. The exhibits at the end of this chapter provide some guidelines on roles and responsibilities.

Your expectations about the contact between the evaluator and program staff. It is very important for an outside evaluator to keep program staff informed about the status of the evaluation and to integrate the evaluation into ongoing program operations. Failure to do this shortchanges program staff and denies the program an opportunity to make important changes on an ongoing basis. The contract could specify attendance at staff meetings and ongoing reporting requirements. Setting up regular meetings, inviting evaluators to program events and staff meetings, and requiring periodic reports may help solidify the relationship between the program and the evaluation. Other approaches that may help include asking a more senior agency staff member to become involved with the evaluation process or withholding payment if the evaluator fails to perform assigned tasks.

What to do if problems arise

Even with the best contract, problems can arise during the course of the evaluation process. These problems include the following:

Evaluation approaches differ (the program and evaluator do not see eye-to-eye). Try to reach a common ground where both programmatic and evaluation constraints and needs are met. If many reasonable attempts to resolve differences have been tried and severe conflicts still remain that could jeopardize the program or the evaluation, program staff should consider terminating the evaluation contract. This decision should be weighed carefully and discussed with your funder, as a new evaluator will need to be recruited and brought up to speed midstream. In some situations, finding a new evaluator may be the best option. Before making this decision, however, you will need to discuss this with your program funders, particularly if they are providing financial support for the evaluation.

Evaluation of the program requires analysis skills outside your original plan. You may find that your evaluator is in agreement with your assessment and is willing to add another person to the evaluation team who has expertise and skills needed to undertake additional or different analyses. Many times additional expertise can be added to the evaluation team by using a few hours of a consultant's time. Programmers, statisticians, and the like can augment the evaluation team without fundamentally changing the evaluation team's structure.

The evaluator leaves, terminates the contract, or does not meet contractual requirements. If the evaluator leaves the area or terminates the contract, you will most likely be faced with recruiting a new one. In some instances, programs have successfully maintained their ties to evaluators who have left the area, but this is often difficult. When your evaluator does not meet contractual requirements and efforts to resolve the dispute have failed, public agencies should turn the case over to their procurement office and private agencies should seek legal counsel.

The evaluator is not culturally competent or does not have any experience working with your community and the participants. It is not always possible to locate an evaluator with both experience in the type of evaluation that you need and experience working with specific groups and subgroups in the community. If your evaluator does not have experience working with the particular group reached by the program, you must educate this person about the culture (or cultures) of the participants' community and how it might affect the evaluation design, instruments, and procedures. The evaluator may need to conduct focus groups or interviews with community members to make sure that evaluation questions and activities are both understood by and respectful of community members.

You are not happy with the evaluator's findings. Sometimes program managers and staff discover that the evaluator's findings are not consistent with their impressions of the program's effectiveness with participants. Program staff believes that participants are demonstrating the expected changes in behavior, knowledge, or attitudes, but the evaluation results do not indicate this. In this situation, you may want to work with your evaluator to make sure the instruments being used are measuring the changes you have been observing in the program participants. Also, remember that your evaluator will continue to need input from program staff in interpreting evaluation findings.

You may also want your evaluator to assess whether some of your participants are changing and whether there are any common characteristics shared by participants that are or are not demonstrating changes. However, be prepared to accept findings that may not support your perceptions. Not every program will work the way it was intended to, and you may need to make some program changes based on your findings.