 |
United States Department of
Agriculture Food Safety and Inspection Service Washington, DC 20250-3700 | |||||
| ||||||
| |||||||||||||
Evaluation Report
|
 |
E-mail surveys to a sample of intact work groups participating in the program. |
|
In-person and telephone interviews with a sample of program participants. |
|
Review of multi-source, 360-degree feedback instruments and their uses. |
Nearly all respondents agreed that a program for assessing training needs is useful. The MLDP is a useful program that should be continued with the condition that certain changes occur.
The use of 360-degree feedback was rated highly by respondents as an effective way to improve both individual and group performance, and focus training needs and resources. However, the strength of a program built on receiving 360-degree feedback to plan training is only as good as the instrument used to collect information. Respondents found the LEI instrument used for this pilot to be inadequate.
The most critical finding is to discontinue the use of the LEI instrument and find an assessment tool that helps identify more specifically the skills of FSIS employees that need improvement. Many questions in the LEI were not specific enough to result in an accurate or useful assessment of training needs, especially for field employees.
Another important finding is that intact work groups should only include those employees that work together regularly or perform similar duties along with their supervisors so accurate ratings can be done. This is especially relevant in the field. Participants were often asked to rate others whom they did not feel confident to rate. Suggestions for field intact work groups include: 1) District Manager, Deputy District Manager, Assistant Deputy Manager for Enforcement, and all Circuit Supervisors in the district; 2) Circuit Supervisor and all Inspectors-In-Charge reporting to him/her; and 3) Inspector-In-Charge and all in-plant inspectors reporting to him/her.
Finally, if the MLDP is to become a permanent program, participants must be provided with full information about all aspects of the program and how it relates to other Agency training initiatives and other assessment methods such as annual performance appraisals. Complete information will provide essential momentum for the program to flourish.
The 360-degree or multi-source assessment tool used in the MLDP pilot to identify training needs was the Leadership Effectiveness Inventory (LEI)1. Respondents cited a number of problems related to competencies being measured and the LEI instrument itself as described below2.
Some competencies being evaluated were not relevant and did not reflect the skills needed for certain positions. For example, some intact work groups were being rated on a "developing a new product for a customer" competency which many respondents felt was not relevant to their position.
Because of the irrelevancy of some competencies included in the assessment, respondents reported often choosing "not applicable" when answering questions.
|
Instructions about how to complete the LEI presented at the beginning of each diskette used to rate an individual were not clear enough. Raters were confused at viewing two very similar questions in a row about the same skill (1st to rate an individual’s required proficiency, 2nd to rate an individual’s current proficiency). |
|
Only one question per skill is needed. An individual’s supervisor should determine the required proficiency level for each skill. Raters could then rate their opinion of an individual’s current proficiency for that skill based on a pre-determined standard. |
|
Raters did not have an opportunity to give relative ratings (i.e., you excel in interpersonal skills but need work in leadership). |
|
Close-ended questions did not focus on technical or job-specific skills. Many close-ended questions were too general to provide useful feedback. |
|
Open-ended questions were redundant and were not worded to elicit information about specific topics/areas or should be reduced in number. |
Overall, narrative provided by raters in response to open-ended questions was considered most useful and insightful by participants. However, respondents cautioned that raters must be well-informed before-hand that their open-ended responses will be reported verbatim and may compromise their anonymity. Some respondents did not receive that important information before completing the instrument. Those individuals were quite surprised to read narrative feedback reported word for word by raters when they received their feedback reports.
Because of the shortcomings with the inclusion of certain competencies and the LEI instrument described above, some respondents reported that although they found the results interesting, they would not plan their professional development around those results. This fact coupled with the problems listed above and problems discussed later in the report related to poor LEI feedback report format cite serious flaws in the LEI instrument. These findings support the notion of discontinuing the use of the LEI and exploring other multi-rater instruments for collecting feedback.
Two multi-rater instruments reviewed during this evaluation as comparisons to the LEI were Benchmarks Developmental Reference Points survey and CCI Assessment Group instruments. These instruments include standard or generic questions that are task specific as well as an easy format to follow when rating a person. The Benchmarks Developmental Reference Points survey is geared toward assessing and developing managerial skills, which is the focus of most multi-rater instruments. CCI Assessment Group offers different instruments for managers, non-managerial personnel, and intact teams. These instruments should be considered for use, along with others, and further researched to determine if their feedback reports are user-friendly.
1
360-degree feedback involves one person
receiving performance ratings in anonymous form from their peers,
superiors, and subordinates (if applicable). This person also conducts a
self-rating using the same criteria and answering the same questions
posed to the others. These results are then compared, analyzed generally
with the help of a human resource specialist, and used to improve
performance by providing a better awareness of strengths, weaknesses,
and areas needing further development.
2 Respondent comments and
suggestions throughout the report include results received from both
e-mail surveys and personal and telephone interviews.
Nearly all respondents in this evaluation were very supportive of the value in receiving multi-rater feedback from peers, supervisors, and subordinates (if any). Although some participants initially were apprehensive about being rated, they felt that receiving objective feedback about their job performance overrode their anxiety. Participants felt that receiving both individual and group feedback results were useful in determining their training needs, respectively.
Supervisors who don’t often receive formal feedback from subordinates were supportive of receiving objective feedback from their staff and felt that subordinate ratings could be quite useful in providing them a unique perspective of and insight about their managerial skills and areas for improvement. A comment was made to provide separate instruments or sets of questions to supervisors who could be rated on their managerial skills by their subordinates and on their technical and communication skills by their peers and supervisor. Also expressed was the fact that supervisors’ ratings and training needs, that may likely be different from training needs of their staff, should be addressed and followed up with in addition to the entire intact work group’s training needs.
Many respondents felt that using multi-source assessment to improve group performance was an innovative and useful way to use the method. An effort should be made to replace the LEI with a more effective instrument with the following attributes (1) better identifies current skill levels of both supervisory and non-supervisory FSIS personnel through the use of customized competencies and questions; (2) allows supervisors to set a standard skill level expected of employees (if required by the instrument); (3) includes a user-friendly feedback report that can stand alone without extensive additional explanation; and (4) perhaps includes separate assessments based on position and who is rating (i.e., supervisors being rated by subordinates on their managerial skills, but being rated by their peers on their communication skills). With the use of a quality assessment tool and careful consideration of the composition of intact work groups discussed in more detail later in this report, 360-degree feedback was rated as an effective way to assess group performance and plan group training.
Related to the use of 360-degree assessment to determine group training needs and just as critical to its successful implementation as a good instrument, is the composition of intact work groups. Although the issue surfaced during interviews with field participants, it is equally important to headquarters participants. To informatively rate peers’ knowledge and skills, intact work groups should be composed of employees with similar job descriptions with the ability to rate each other’s job performance. A number of respondents felt they were not qualified to rate other’s performance because they did not have enough knowledge of the position or enough contact with the person filling a position.
Because the field work force is widely dispersed, many employees working in the same district and performing the same job do not have much contact with each other. This fact makes rating peers a difficult task because, more often than not, peers do not have sufficient information to rate each other on their performance. Therefore, the composition of field intact work groups must be carefully determined. Field respondents recommended that MLDP field intact work groups include:
|
District Manager, Deputy District Manager, Assistant Deputy Manager for Enforcement, and all Circuit Supervisors in the district; |
|
Circuit Supervisor and all Inspectors-In-Charge reporting to him/her; and |
|
Inspector-In-Charge and all in-plant inspectors reporting to him/her. |
These three groups individually encompass employees who should have the ability to rate each other’s job performance and can provide accurate feedback about each other’s knowledge and skills. Structuring field intact work groups this way will improve the accuracy of 360-degree feedback report results for field employees.
Similarly, carefully considering the composition of headquarters groups will improve the accuracy of their 360-degree feedback. Again, a headquarters intact work group should consist of peers who have similar job descriptions and perform similar duties on a regular basis and their supervisor. The training courses then planned as a result of their feedback will be relevant to the particular training needs of that peer group. As a result of including only peers who can accurately rate each other’s performance in an intact work group, all participants should be encouraged to involve their intact work group peers as raters.
Respondents found the LEI feedback report format very difficult to follow and understand without extensive assistance from an HRD facilitator. This fact made the feedback session an essential element in understanding report results. In fact, during the first few hours of the feedback session participants receive an example of a feedback report and spend time deciphering sample results before even receiving their own results. The lengthy amount of time needed to comprehend the report format and content illustrates how difficult the report is to follow.
The report content was considered useful by some, however nearly all respondents agreed that results would have been more useful if questions were more specifically adapted to reflect technical skills used on the job by each intact work group. Since the work of field inspectors and supervisors requires a strong technical knowledge base, it was suggested that a number of those individuals be included in customizing competencies or questions for field intact work groups. Below is a summary of comments received regarding the feedback report format and content.
|
|
Current format is very confusing. Participants needed much explanation to understand results. |
|
|
Without the inclusion of labels and color keys, and without averages being defined, the interpretation of results when the feedback session was over is difficult. |
|
|
Reports should be stand-alone documents and should not need a facilitator to explain results. Reports should be easy to read and easy to understand. |
|
|
Report would be more useful if results were based on questions related to more job-specific, technical skills. |
Partly because of the difficulty with interpreting LEI report results, the usefulness of the feedback session was rated highly by participants. Without the feedback session, participants would not have been equipped to understand their results. The length of the feedback session was adequate, however if in future another 360-degree instrument with an improved report format is used, the length of the feedback session could be considerably shortened and possibly combined with the planning session. Some participants suggested receiving their feedback reports shortly before the session begins to digest the information and read narrative comments before meeting as a group. This would alleviate the anxiety some feel about receiving potentially negative criticism during a group session. Again, this suggestion would only be relevant if an improved instrument and report is used so that participants could understand the data on their own before having the feedback session.
The usefulness of the planning session was mixed among participants. Some felt the planning session helped bridge their training needs with relevant training courses. These respondents felt their HRD facilitator helped guide them in choosing training courses that would be useful to the group based on LEI results and the group's knowledge of their training needs. Using the group's knowledge of their training needs in addition to 360-degree feedback results was considered useful. Many respondents stated that using HRD employees to develop and deliver in-house training is a good use of Agency resources.
Other respondents reported that not enough guidance was provided by the HRD facilitator and that they needed more direction to choose training courses. Suggestions were made that the HRD facilitator identify training courses that may meet the group's needs after studying the intact work group results before the session begins. The training recommendations by the HRD facilitator could then be discussed by the group, if more direct guidance is requested or needed, at the planning session for consensus.
Another suggestion was made to hold feedback and planning sessions concurrently in headquarters as well as the field, and hold them soon after completing the multi-source assessment instrument. Holding feedback and planning sessions back to back alleviates the need for participants to be refreshed and reintroduced to the data received at the feedback session. Holding them shortly after completing the 360-degree instrument helps to protect data integrity if a staff experiences a turnover of employees during the assessment process. For example, if after completing the multi-source assessment instrument, a staff experiences turnover, the data collected may not be accurate anymore because the staff member(s) who left may have had particular training needs that factored into the group results that the incumbents or new staff member(s) do not need. Also, the dates for feedback and planning sessions should be planned up-front at the start of the program to ensure that they can be scheduled consecutively and participants’ schedules can be planned accordingly.
Information about the MLDP was provided to participants in a few ways. After intact work groups agreed to participate in the program, a kick-off session was held for each group. During the kick-off session, a human resource employee discussed the different stages of the program such as completing the LEI and receiving multi-rater feedback in the form of a report, participating in the feedback and planning sessions, and receiving training from the in-house Human Resource Development staff or from an appropriate contractor. In addition to the kick-off session, each participant received a booklet titled MLDP Leadership Competencies. This booklet begins with a memo from the FSIS Administrator to senior managers announcing the MLDP pilot and urging senior management support for the program. The memo is followed by a discussion about and description of leadership competencies developed by the Office of Personnel Management (OPM) and examples of training to help employees develop those competencies. The booklet itself does not discuss the use of a multi-source instrument to collect feedback.
Although the information from these sources provided a general framework about the program, many pilot participants stated that they wished they had more information about the MLDP. Respondents explained that more written material about the program would be very useful. Many did not find the MLDP Leadership Competencies booklet helpful. Its content was confusing as it did not clearly explain how the use of competencies would be incorporated into the MLDP. The competencies were not explained in an MLDP context. In fact, aside from the booklet title and memo, the MLDP and 360-degree assessment was not mentioned at all.
Additionally, respondents reported that they were unclear about how each phase of participation led into the next stage of the program. They stated that the pieces of the program were not explained as building blocks necessary to reach the next stage, ultimately resulting in a cohesive, useful group training plan. Also, some specific information was not received by some participants including that raters' open-ended responses would be reported verbatim. These participants felt their anonymity (a critical element in multi-rater feedback) was compromised as a result of this lack of information.
Other respondents felt they lacked information about the big picture. Questions arose from participants such as what is the vision for the program, where is the program taking them in the long-term, is there support from senior management and the administrator, and how does MLDP fit with Tech 2000 and Workforce of the Future training initiatives. Another question and concern that was brought up by a number of respondents was whether there would be enough funding for them to complete the training they planned based on the group LEI results. Also raised was how the MLDP relates and fits into other forms of assessment such as the employee’s annual performance appraisal. Having answers to these fundamental questions would allow participants to stay energized about the program and not lose momentum.
Respondents requested that more comprehensive material encompassing the elements described above be provided about the MLDP. Any written material about the program should fully describe all aspects of the program and how they work together in reaching the goal of developing a useful training plan for each intact work group, and how they complement other assessments and development tools such as the annual performance appraisal and Individual Development Plan (IDP). Respondents also requested that information about the MLDP be included on the FSIS web site. These materials would allow participants to do their own research about the program and enhance the verbal descriptions they receive from the HRD staff.
The concept of the MLDP using 360-degree feedback to assess individual’s and intact work group’s training needs and plan group training around those needs is an effective way of enhancing group development and improving individual’s and group’s skills. As to be expected during a pilot, a number of areas for improving program implementation were discovered. Recommendations to improve the program appear below and are grouped by category.
Discontinue the use of the Leadership Effectiveness Inventory.
Explore other assessment tools with the following characteristics:
Simple, comprehensive instructions for raters
Use different instruments and/or groups of questions for supervisors. One instrument or group of questions should focus on managerial skills and be given to subordinates. A separate instrument or group of questions should focus on technical and communication skills and be given to a supervisor’s peers and supervisor. Follow-up with supervisor’s training needs in addition to intact work group’s training needs.
Encourage intact work group supervisor to review competencies for relevancy and to set expected performance standards (if required by the instrument) before implementing the instrument.
Assemble a small group of field employees to review competencies and questions for relevancy (in addition to intact work group supervisor) before implementing the instrument.
Intact work groups in the field should include:
Intact work groups at headquarters should include only members of staff that hold similar job positions and have the ability to rate each others performance as well as their supervisor.
Encourage participants to include their intact work group peers as raters.
|
Develop comprehensive written materials describing all elements of the program (i.e., 360-feedback, assessment tool, receiving results, developing group training plan, secured funding for implementation) and how each element leads to the next step and eventually results in a group training plan. |
|
|
Determine and report to participants how the MLDP complements other established assessment tools such as annual performance reviews and Individual Development Plans (IDP). |
|
|
Include written materials on the FSIS web site. |
|
|
Assure program participants that funding is available for completing training courses and seeing the program to completion. |
|
|
Continue to hold feedback session to discuss feedback report results. Shorten as needed after incorporating into the program an instrument with an improved report format. |
|
|
Hold feedback and planning sessions consecutively for headquarters intact work groups as is currently being done for field intact work groups. Schedule all sessions up-front at the initial participant meeting to ensure sessions can be scheduled close together and so participants can plan their schedules accordingly. |
|
|
Encourage HRD facilitator to study group results and identify recommendations for training courses based on group report results before the session begins. These recommendations should be used at the planning session if more direct guidance is requested or needed by participants. |
|
|
Encourage intact work groups to use their knowledge of the group’s needs in addition to feedback report results when planning training. |
|
|
Discuss either in group or individual planning session the supervisor’s report results and develop separate training plan for supervisor, if appropriate, in additional to group training plan. |
Continue to use HRD employees to provide training when possible.
|
|
Identify the Administrator’s and senior managers’ vision for merging and incorporating the MLDP’s purpose with those of other Agency training initiatives such as Workforce of the Future and Tech 2000. |
|
|
Secure funding for program implementation. |
|
|
Promote visibility of senior management support. |
The use of 360-degree feedback or multi-source feedback has been used in recent years in many public and private organizations as an effective development technique for a variety of positions, especially at managerial levels. The FSIS MLDP pilot enhanced the 360-degree feedback method by focusing on group development. The MLDP pilot uses a multi-source assessment tool to help evaluate individual and a staff’s strengths, weaknesses, and areas for improvement. Pilot groups include intact work groups, a supervisor and his/her subordinates. The entire staff completes the multi-source assessment instrument and receives individual and aggregate, or group, results. Group results are the focus of the program and training for the group at large is planned in response to those results.
To begin the assessment process, each intact work group leader, or supervisor, meets with a Human Resources Division (HRD) specialist and reviews 27 competencies. After reviewing the competencies and discussing with the HRD representative the general skills required by his/her particular staff, the supervisor determines which competencies s/he would like included in the assessment. The LEI includes questions designed to rate an individual’s required and current proficiency level for each competency. Once relevant competencies are chosen, the LEI is customized to include only questions that will rate the proficiency level of those competencies.
Next, each person receives seven diskettes and distributes them to individuals who can rate their performance level, such as co-workers or clients. Five diskettes are given to peers of each participant’s choosing, one is given to their supervisor, one is a self-assessment diskette. The supervisor must also give a diskette to all his/her subordinates. Each diskette begins with instructions for completing the LEI and follows with a series of closed-ended questions related to the competencies being rated. Each question has two parts. First, the rater is asked to rate the required proficiency of the person being rated for a particular skill. Second, the rater is asked to rate the current proficiency of the person being rated for the same skill. After all the closed-ended questions are answered, the rater is asked a few open-ended questions about areas of strength and areas for improvement of the person being rated. For these questions, the rater provides a narrative response. Once all questions are answered, the rater returns the diskette to a location where results of all diskettes for each individual and each intact work group, or staff, are compiled.
After all LEI results are compiled for each individual and intact work group, the results are included in separate reports prepared for each individual participating in the program. Each individual's report presents the average score for each competency as rated by themselves, their supervisor, their peers, and their subordinates (if any). The report also presents an average score for each competency of the intact work group as a whole. These results are displayed as four lines of a color-coded bar graph (supervisors have one extra bar graph line showing results from subordinates). Reports are ordered by competencies rated as needing greatest improvement for the intact work group to areas needing least improvement for the intact work group making group results the focus of the report.
LEI feedback reports are given out during a feedback session facilitated by a human resource employee. Due to the difficulty of gathering field employees together at one time, their feedback session is immediately followed the next day by a planning session. Headquarters employees who work in generally the same proximity, have their feedback and planning sessions staggered. During the planning session aggregate report results are reviewed and discussed, and two group training sessions are selected, possibly customized to meet the group’s particular needs, and scheduled. When at all possible, an HRD employee develops and delivers the training courses to the intact work groups. Only if special expertise is needed, training is contracted to an outside vendor.
[PEIS Home Page] [Evaluation Articles] [Final Reports]
Send mail to
webmaster with questions or comments about this web site. |
