Environment |
PUBLIC INVOLVEMENT EVALUATION: Brevard Metropolitan
Planning Organization (Viera, Florida);
|
|
The need to conduct a second round of outreach provided the impetus for the agency to formalize its evaluation process and update its PIP. Before undertaking a second round of public involvement, however, the MPO conducted an evaluation of the Study outreach activities to gain insight into the agency's strengths and weaknesses in working with the public. |
To gauge the effectiveness of the second round of outreach, the agency conducted both an external and internal evaluation of the process. External evaluation entailed conducting a five-minute telephone survey with the approximately 1,500 participants from the first round of meetings. The survey was developed to solicit their opinions on the techniques employed, the materials presented and the inclusiveness of the process. The response rate was approximately 30 percent of the target audience (about 500 people) with the following results:
Equally interested in the perceptions of the internal audience (the MPO staff, local governments involved in the study, and MPO Board members), the MPO staff prepared a written survey to solicit their input on the public involvement process, the project information provided, and the activities of the study committee. Additionally, the survey asked participants to identify areas of improvement and lessons that could be applied to future studies. The survey was sent to the MPO board and 10 MPO and local agency staff members who were involved in the project. The response rate was approximately fifty percent of the target audience with the following results:
Internal Audience Written Survey
The input received from both the external and internal surveys, a review of the consultant study process and an informal assessment of the MPO's prior public involvement activities, informed the revision of the PIP and the development of the Evaluation Handbook.
LESSON 1: Understand The Purpose of Public Involvement Upfront
Faced with the challenge of delivering projects on time and on budget while addressing the needs and concerns of the public is no small task. Agencies are often accused of not adequately soliciting the advice of the citizenry in the planning process. Too often, agencies have relied on a set menu of activities to interact with citizens with little emphasis on the effectiveness of these techniques in a given circumstance. As such, additional resources may be deployed to expand the agency's ability to effectively engage the public in the decision-making process. Without policy and guidelines to direct these efforts, however, there is little information to substantiate an increase in efforts and guide their potential for success. This can be costly, both in terms of budget and goodwill from the public.
The Brevard MPO increasingly found itself asking consultants to provide a lot of public involvement "extras" (e.g., expand mailing lists, create newsletters, provide project web sites) without having either the formal policy or evaluation criteria to justify these requests and expenditures. The current PIP and Evaluation Handbook provides this guidance and positions the agency to be able to continually assess and refine its approach to better meet the needs of the public. This in turn will enable the agency to more effectively target its resources. The alternative can be costly and result in funding activities that do not contribute to engaging the public effectively.
LESSON 2: FIND OUT WHAT WORKS AND WHAT DOESN'T: REVIEW OF A PROCESS
When faced with the task of evaluating the public involvement efforts during the Southwest Brevard Transportation Study, we were quite apprehensive about calling citizens at home. We were fearful that those citizens that were not pleased with the outcome of the study would not be objective when answering our questions. However, we were very pleasantly surprised when an overwhelming majority of the people that we contacted were simply thrilled that we were interested in using their input to help us to refine our process to better meet their needs.
--Kama Dobbs, transportation planner, Brevard MPO
By listening carefully to the feedback provided on the first round of outreach and conducting the telephone survey with citizens that had participated in the second round, the agency obtained valuable information about its outreach process.
In the first round, the name of the study had caused confusion about who would and would not be impacted and limited participation. Additionally, the consultant's mailing list for the project newsletter did not accurately reflect the entire study area. By listening to the public's concerns, the MPO was able to modify the study name, mailing list, and means of meeting notification before launching the second round of outreach.
At the completion of the second round of outreach, the MPO once again turned their ear to the public by conducting a telephone survey of those who had participated in the process. Based on the level of workshop participation, the comprehension level of the attendees, and the success of the direct mailings to notify stakeholders about the meetings, it became evident that the MPO had been successful in notifying stakeholders and conveying project information.
By carefully reviewing what had been done in the first round, and making modifications in the second round, the agency was successful in identifying public concerns and responding to them in the recommendations.
It should be noted that the phone survey was subjective in that it included only those citizens that had participated in the workshops. Also, certain groups will not be represented in any evaluation and, therefore, there exists a need for agencies to continually vary their sample when assessing their outreach activities. This might be done by conducting a random survey or including comment cards in all agency mailings and publications to solicit feedback from those who have not yet had the opportunity to participate in the decision making or who have not had the time to attend public meetings.
LESSON 3: DESIGN A FRAMEWORK TO SUPPORT EFFECTIVE EVALUATION
The MPO shall strive to continuously improve public involvement. The MPO shall continuously evaluate public involvement techniques, according to the procedures contained in the Brevard MPO Public Involvement Evaluation Handbook.
-- Brevard MPO Public Involvement Plan (2000)
As the staff of the MPO began working on the Evaluation Handbook, it became apparent that the 1995 PIP did not provide the appropriate policy language to support evaluation and needed to be revised.
The recently completed PIP includes language specific to the evaluation of public involvement techniques based on the procedures outlined in the Evaluation Handbook and calls for a review of both the PIP and the Handbook at least every three years. By developing the PIP and Evaluation Handbook as companion pieces, the MPO has established an effective framework to conduct, evaluate and refine its public involvement policy and techniques on a regular basis. While the PIP provides the goals, policies and objectives of the MPO, as well as a list of public involvement techniques and forums, the Handbook details the actions required to evaluate the effectiveness of the techniques identified in the PIP.
The Handbook was developed over a period of a year and a half during which it was continually refined to reflect the activities of the MPO. Included in the Handbook is a Public Involvement Tools Evaluation Matrix, which lists specific techniques/tools, evaluation criteria, performance goals and methods to meet those goals. Developed by the MPO staff, the evaluation criteria include indicators such as the number of calls made and letters sent to the public for hearings, open houses and workshops and the number of people who respond to direct mailings and newsletters. (Similar criteria could be applied to response from press releases, newspaper articles and editorials.) The performance goals detail target levels of performance such as response by three to five percent of the affected population from a study area.
Assuring that the full range of issues raised by the public are identified and that they are reflected in decisions should be the ultimate goal. It is important that measures are developed to support Agency goals for effective public involvement and not be based solely on what is easy to measure.
LESSON 5: DEFINE METHODS OF EVALUATION
Evaluating public involvement means assessing how well we communicate with our customers. We have three basic customer groups - elected officials, other agency staff and the public at large. Each has different expectations, information needs, roles and levels of understanding. Different evaluation tools are needed for each group.
--Bob Kamm, staff director, Brevard MPO
Once tools/techniques, evaluation criteria and performance goals have been identified, the instruments for both internal and external evaluation must be developed.
INTERNAL
The Brevard MPO has developed a general evaluation form to be completed by agency staff each time an evaluation is conducted that specifies the type of study, the point at which the evaluation was conducted, the public involvement tools employed, the target audience and the type of evaluation conducted. A similar form is also used for project specific evaluations. Complementing these forms is an Improvement Strategies Form that has been developed to provide practitioners with the opportunity to provide input on potential improvements. Upon completion, these forms will be reviewed by the MPO staff and, where applicable, forwarded to the Florida Department of Transportation (FDOT). Copies of these evaluation forms will also be provided to the Technical Committee and the Citizen's Advisory Committee (advisory committees to the MPO Board) for review and comment with the aim of keeping staff and key stakeholders apprised of evaluation efforts and areas for improvement.
Recognizing the role of consultants in a number of its studies, the MPO has taken internal review one step farther by requiring consultants to provide an assessment of the public involvement process (what worked, what didn't, what they would do differently) as part of their final project write-up.
General Evaluation Form and Improvement Strategies
EXTERNAL
Once we began thinking about qualitative measures, rather than quantitative, our evaluations became much easier to develop. Answers to "How can we improve the workshop?" give us better guidance than will simply counting the number of people that show up.
--Bob Kamm, staff director, Brevard MPO
Externally, the agency has developed a workshop evaluation form that asks participants how they found out about the meeting, whether they felt the information provided was clear and informative, what their best source of information has been and how they would rate the public involvement process.
It also is important to find out the concerns of people who have not participated and mechanisms should be developed to determine who these people are and the reasons they are not involved.
LESSON 6: USE THE EVALUATION INFORMATION TO MAKE REFINEMENTS
What happens once all of this information has been collected? First and foremost a central point must be established to house the information collected. At this time, the Brevard MPO maintains binders of all evaluations conducted to date. It is anticipated, however, that this information will eventually be entered into a database. This would enable the agency to "group" findings in a manageable fashion where data could then be printed out, reviewed, debated and refined.
LESSON 7: TRACK COSTS CAREFULLY
The Brevard MPO estimates that fifteen percent (15%) of its annual budget is expended for public involvement activities (staff time and direct costs). Averaged out, twenty to twenty-five percent (20-25%) of the agency's staff time is dedicated to outreach activities. Consultant costs for public involvement are estimated at ten percent (10%)
of the study contract. As the need for public involvement in transportation planning increases, so too will these numbers. Therefore it is more important than ever to carefully track staff time, direct costs (e.g., newsletter production and mailing, web site development and maintenance) and consultant costs associated with these efforts both in terms of measuring success/failure against expenditure as well as to quantify the "savings" of eliminating a particular approach so that these funds can be applied to other activities.
LESSON 8: TAILOR PUBLIC INVOLVEMENT TO LOCAL NEEDS
The work done by the Brevard MPO provides a starting point for any agency intent on evaluating its public involvement process and outcomes. Recognizing that each agency needs to tailor its individual outreach activities, it is often helpful to review what other agencies have done as a means of comparison.
In the fall of 1999, members of the MPO staff attended a Public Involvement Training Course sponsored by the Florida Department of Transportation where they shared public involvement techniques with other MPO's in Florida. After completing the training, the staff decided to document some of the "best practices" identified so that they would have a list of public involvement "tools" to choose from for individual projects as well as everyday MPO activities. These tools are included in the PIP.
Brevard's Public Involvement Plan provides a comprehensive list of public involvement techniques and indicates when these techniques should be employed. Appendices to the Plan provide detailed examples of each of these tools, ranging from surveys, comment forms, and direct mail pieces to web site and database visuals.
The Evaluation Handbook is useful in the provision of both internal and external evaluation forms as well as providing detailed matrices against which these activities can be measured. Both of these documents provide an excellent starting point for practitioners intent on evaluating their public involvement activities.
CHALLENGES AHEAD
Public participation by definition is an open, ongoing, two-way communication, reflecting the local community of people and issues. It cannot be structured to any single right model or measured against any one size fits all criteria.
--Public Involvement Desk Reference, U.S. Department of Energy
Brevard is about to conduct its first evaluation since the adoption of the PIP and the Evaluation Handbook on the public involvement activities associated with the agency's Greenways and Trails Master Plan.
While Brevard has made great strides in the compilation of the Evaluation Handbook, the agency recognizes that it will need to be vigilant in continuing to assess what works and what doesn't. More importantly, to be successful in the eyes of the public, the MPO will need to be able to translate information into action.
The MPO identified three areas that might impede the agency's ability to effectively respond to the needs of the public in the future:
Another challenge is identifying people who have not been involved and understanding why they have not. Are there specific obstacles that limit their interaction with the MPO? Do they have concerns that are not being articulated? Public agencies have the responsibility to serve all people so it is important that some people are not left out of the process unintentionally.
By revising the PIP to support evaluation and developing an Evaluation Handbook to provide guidance on the actions required to evaluate the effectiveness of the public involvement techniques identified in the PIP, the MPO has made a strong start to enhance its relationship with the public. The challenge will be in continuing to find and refine cost effective means to assure that the issues and concerns from all members of the public are identified and reflected in the transportation decision-making process.
Contact:
Kama Dobbs
Transportation Planner
Brevard Metropolitan Planning Organization
2725 Judge Fran Jamieson Way, Building A
Viera, FL 32940
Phone: (321)690-6890
E-mail: kamad@brevardmpo.com
Web site: http://www.brevardmpo.com
The Brevard Metropolitan Planning Organization (MPO) is the agency designated by state and federal law responsible for the urban transportation planning and programming process for the Palm Bay-Melbourne-Titusville Urbanized Area. The jurisdiction of the MPO extends to the boundaries of Brevard County Florida, which is located along the east coast of the Central Florida region. Brevard County is approximately 20 miles wide, east to west and 72 miles long, north to south and is bordered by the Atlantic Ocean to the east and the St. Johns River to the west. Brevard County is adjacent to Indian River, Orange, Osceola, Seminole and Volusia Counties and located approximately halfway between Jacksonville and Miami and about 40 miles east of the greater Orlando area. NASA's Kennedy Space Center, Cape Canaveral Air Force Station, Patrick Air Force Base and the Port of Canaveral are all located along the east coast of Brevard County.
The estimated 2000 population of Brevard County is 483,000, with just over 27,000 residing in Melbourne and Palm Bay. Close to 87,000 (18% ) of Brevard County residents are over the age of 65. In 1990, approximately 90% of Brevard residents were white, 8% were black, with the other 2% being of asian, hispanic or other descent. The 1999 unemployment rate in Brevard was 4%. Approximately 55% of all land in Brevard County is undeveloped. |