EVALUATION OF COMPUTER-BASED TRAINING

In the Intervention Analysis Report for the Child Support Enforcement Model Office Project, Policy Studies, Inc., recommended:

...that the State, in conjunction with the Model Office counties, develop in-house training resources. Because location is a problem throughout the state, any training developed for the project could also be utilized elsewhere in the state.

In September 1996, the Steering Committee of the Model Office Project acted upon this recommendation by endorsing the development of a computer-based training module (CBT) dealing with the locate function in child support case processing. It appropriated funds to purchase software designed to permit the development of multi-media, interactive and self-paced training products. With project funds, three state-level child support workers received training conducted by PSI on Authorware software. During the ensuing 15 months, state staff developed a computerized training module that addresses all aspects of location: policy, procedure, system and recommended practices.

In November 1997, the completed training program was packaged and distributed to child support staff in Denver, Mesa and Archuleta counties for use and evaluation. Workers in the three pilot counties were randomly chosen to receive computer-based training or traditional, stand-up training on the locate function. Each worker completed a questionnaire designed to gauge the effectiveness of each training approach. The evaluation instrument included a set of knowledge questions on the location process that was administered to workers prior to and following their training experience. The attitudinal questions elicited user reactions to each training format.

This report presents the results of tests administered to child support workers prior to and following their exposure to computer-based or stand-up training. After comparing the relative gains made by those participating in each type of training, we consider the reactions of participants, trainers, workers and office administrators to each training format.

PROFILE OF TESTED WORKERS

Table 1 presents selected background characteristics of 77 workers in Denver, Mesa and Archuleta counties who participated in either computer-based or in-person training on child support locate work in November 1997. The vast majority of those participating in the training were legal technicians. They ranged from extremely experienced to relatively new workers. Most had at least some exposure to college and, prior to the training, most said they knew quite a bit about locate work. Most workers also reported feeling at least fairly comfortable with computers and were receptive to the idea of computer-based training.

TEST SCORES BY TRAINING METHOD

The evaluation questionnaire was administered to 34 legal technicians, supervisors and paralegals in Denver, Mesa and Archuleta Counties who participated in a computer-based training program on child support locate work. Thirty-four similar workers who received the same training in a conventional, in-person, training format also completed an evaluation questionnaire. The knowledge component of the questionnaire consisted of 21 true/false and multiple choice items. All training participants completed these items both before and after their training experience.

At the pre-test technicians, supervisors and paralegals exposed to computer-based and in-person training formats demonstrated similar levels of familiarity with the location process in child support. The average number of correct responses recorded by workers was 14.2 in the computer group and 13.9 in the in-person group.

At the post-test, technicians, supervisors and paralegals in the computer group averaged 18.0 correct responses, to 17.0 in the in-person training group. The difference between the computer and in-person training groups at the post-test was not statistically significant, indicating that knowledge gain for the two groups was identical (see Table 2).

Technicians, supervisors and paralegals who reported completing the computer-based training in one sitting did slightly better than those who took longer. However, there were no differences between those who completed the training at their own office, at a work station or somewhere else. Nor were there differences based on how comfortable workers said they were in using computers at the pre-test survey.

Table 2 Test Scores by Training Method Legal Technicians, Supervisors, and Paralegals
	Computer Training	In-Person Training
Average score at pre-test	14.2	13.9
Average score at post-test	18	17
Average points improved	3.8	3.1
N=	34	34

Neither Denver nor Mesa County showed any differences in post-test results based on the number of years the worker had been engaged in child support work, the size of the agency, the percentage of time spent on establishment versus enforcement work, or prior training exposure.

Responses by all staff members who were tested are reported in detail in Appendix A.

IMMEDIATE WORKER REACTIONS

Part of the post-test was devoted to eliciting the reactions of participants to each training format. Workers were generally quite positive about the training in which they participated. This was true for both those in computer-based and in-person training sessions. Most workers felt the training was convenient, useful, interesting and responsive to their questions.

Compared to those in-person training, those exposed to computer training were slightly more likely to describe this as "difficult or hard to follow", and slightly less likely to say the training answered most of their questions. At the same time, workers exposed to computer-based training were more likely to characterize it as "convenient" and "interesting and entertaining." In on-line comments, several users commented favorably on the graphics components of CBT. They also liked the incorporation of actual screens from the Automated Child Support Enforcement System (ACSES) in the training. The following comments by users are illustrative:

I feel this is an extremely valuable tool for all of Colorado Child Support workers. It was very useful for me and I've been in the program 9 years! I think this should be a mandatory training process which would ensure consistency and continuity within the program and eliminate a tremendous amount of duplication within document generation from many individuals who do not really know now to use locate. Computer based training should be a must!!! CBT makes learning fun!!!

I did enjoy this process of learning, it did not take up a lot of time and the graphics were a lot of fun and made it easier to continue through. I like that it was not wordy, only the important information was given and you could go at your own pace.

When asked how much they felt they knew about the location process following the training, the two groups were quite similar and both showed substantial improvements from the pre-test (See Figure 1).

One of the reputed benefits of computer-based training is its flexibility. It can be completed in one or multiple sittings, at different work stations and with and without interruption. The evaluation shows that workers did exercise a good deal of flexibility in how they handled the computer-based training process. Whereas nearly all Denver workers completed the module in one sitting, most Mesa County workers did it in two or more sessions. A third to one-half of workers were interrupted during their training experience. Finally, while 57 percent of Mesa County workers spent 45 minutes or less on the training package, an identical proportion of Denver County workers spent 75 minutes or more. Table 4 summarizes the experiences of workers in Denver and Mesa County with the training process and describes variation in where the training took place, how long it took and whether it was completed in one or multiple sessions. One child support worker in Archuleta County completed the CBT module. Her evaluation questionnaire is included in the analysis for Mesa County.

The final tables present worker reactions, likes and dislikes, about computer-based training. Workers in both Mesa and Denver Counties who used computer-based training clearly liked the process. They found it easy to use, sufficiently detailed and interesting. Virtually all surveyed workers would like to use CBT in the future. Only about 10 percent strongly preferred in-class training formats.

REACTIONS OF THE CBT DEVELOPERS

Three child support staff members in the State Child Support Enforcement Unit (CSE) worked together for 15 months to design and develop the computer-based training module. In order to shed light on the level of effort that this type of development project entails and the feasibility of expanding computer-based training to other child support topics, we spoke to them about their experiences and any problems or concerns they encountered while completing the project. We also discussed the state's existing training program.

STAND-UP TRAINING FORMATS

Stand-up training is the primary method currently used by the CSE to train child support workers in Colorado. State staff conduct a week-long orientation for new workers on a quarterly basis (held in the Denver metro area with an average class size of 35). They also hold an ACSES on-line class. This is a five-day program offered 10 times per year at the CSE Division in Denver. On average, 10 people attend on-line training. A maximum of 12 can be accommodated. Ongoing training is also offered whenever extensive changes are made to child support enforcement policies and procedures that also impact ACSES. These "as needed" trainings last, on average, from one-half to two days. Examples of topics for ongoing training include case processing, welfare reform, the administrative process, and document generation. These ongoing trainings are held in eight different regions, and are attended by anywhere from 25-75 people.

Approximately five different state workers representing the ACSES, interstate and program sections of the CSE participate in training in addition to their other job duties. Those who conduct training report that they do most of their preparation during evenings or weekends. Other job duties take priority during the days.

Although the state offers many training opportunities for county and state child support personnel, participation is not mandatory. Some counties are always well represented while other counties rarely send workers to training. As one training developer commented:

State training is not mandatory, so there is no rule that a director send everyone or even the right people. At some trainings we may get very few people. We may get the wrong people or new people (not the 5 year plus people who need to come). Or they show up late or go to lunch and don't return or return late and ask you to repeat everything.

The training staff reports that the stand-up format generally works well because the trainers can answer questions, present the state perspective, and attempt to remedy improper procedures followed at the county level.

On new initiatives, you need stand-up, we can't anticipate all the questions people need to ask. It's a great way to get the state perspective out to people.

However, while responsive to the audience's specific needs, the stand-up format necessarily creates inconsistencies. The trainers modify their presentation based on the previous questions encountered. As a result, workers who attend the same program on different days may hear different material.

Another downfall of stand-up is that we go to sites around the state and that we change the presentation as we go in response to the previous session, so it isn't the same script, it isn't the same message.

COMPUTER-BASED TRAINING FORMATS

Computer-based training, by contrast, is more consistent and standardized. Each worker receives exactly the same training. Training staff also believe it would be easier to update a CBT module than to update the many training handouts and manuals they use in stand-up training sessions. One trainer observes, "We could change the CBT modules after each legislative session." Not surprisingly, computer-based training is also viewed as less stressful than stand-up formats for training staff. As one trainer put it, "With practice and expertise, it is much easier to develop a CBT module than to go on a road show for a month."

Computer-based training, however, poses considerable challenges to state-level trainers. One is the time required to develop a module. The development team worked on the location module for 15 months. This was considerably longer than originally planned, although most of the delay was due to competing work obligations. As one training developer noted, "The biggest problem was none of us had time to work on this." Lacking consistent time to work on the module, it was hard to get comfortable with the software and generate some programming momentum. Development occurred in spurts with months of inactivity. As a result, the trainers had to re-acquaint themselves with the software over and over again. All the developers feel that doing a new module would go much faster, particularly if they had a block of time to work on development. As one developer put it:

If the three of us could do nothing else for 30 days, we could have it (new module) done. There would be bits of different CBT modules that we could cut and paste (from one module to the next), so it would have a multiplying effect.

The task of creating the computer training package proved to be more difficult than anticipated. Although the software trainers and federal officials who encourage computer-based training suggest that novices can pick up what they need to know quite readily, the reality is more complicated. The Colorado team agree that Authorware software is complex to use.

This was programming, it was hard to use. There is a huge learning curve, you can't come back after 5 days of training and do it.

Despite its difficulties, the developers found the project very rewarding. They found it valuable to work together and believe that their team effort has resulted in a strong product that has generated excitement for more module development. Developers believe that many child support topics are amenable to computerized training formats. Indeed, in their view, the only topics that are not suitable to computerized training formats are new legislative initiatives, although with the passage of time and implementation experience, these topics could also be incorporated into a CBT module for reference

At the same time, none of the developers see computerized training as a complete replacement for conventional, stand-up training formats. Rather, they see CBT as a useful way to enhance stand-up training. They regard CBT as a valuable resource for workers who want to review child support procedures on their own schedule. It is also a way around the fixed schedule of stand-up training classes. For example, one trainer notes, "It is a way to accommodate people hired months before the next training classes." They see CBT as a way to service the many workers who are unable to travel to Denver and the regional training sites for in-person training programs. Finally, they see CBT as a more efficient method of imparting the child support facts, policies, and procedures that are currently buried in a variety of dated manuals.

Stand-up training, on the other hand, is viewed as a way to address the real world problems that workers encounter. It is regarded as a more effective way to present new initiatives. Finally, state trainers feel that in-person training formats allow them to keep in touch with county workers and foster positive relationships between the county and state child support agencies.

EVALUATION OF CBT BY PILOT COUNTIES

There are some issues where you need the stand-up training -- like on the legal issues so you can ask questions. On the computer you can't ask questions.

Personal interaction between county and state workers is also viewed as an important aspect of stand-up training. Administrators feel that it is valuable for their workers to have a relationship with the state workers and " be able to shake hands and laugh with the state trainers."

To the extent administrators in the pilot counties have different reactions to CBT, it tends to reflect their geographic location. Mesa and Archuleta counties on the Western Slope do not send their workers to the state training on a regular basis. As one western-slope administrator commented, "it's absurd to drive all day to a two hour training." Even the five-day trainings can be too time consuming and costly for the smaller counties. As a result, workers in outlying counties frequently do not participate in state training programs. Denver, on the other hand, typically uses state training resources and finds it a successful instrument for training workers. Perhaps for this reason, Denver administrators feel that the process of creating and releasing new CBT modules may be unacceptably slow, especially when compared with the schedule for stand-up training.

All three counties agree that CBT allows more flexibility in training. One supervisor noted that "it was nice to be able to stop, work, and go back to it." CBT allows new workers immediate training, rather than having to wait for the next stand-up training session. CBT is also valuable for veteran workers who could pursue the ongoing training at their own pace. Finally, CBT is viewed as a way for administrators to monitor that their workers have mastered training material. Once it is in the field, the location training module will have a built-in pre- and post-test. The Mesa administrator looks forward to reviewing worker test results in order to ensure successful completion of each training program.

All three administrators feel that computer-based training can be extended to other topics within the child support system, and are anxious for additional modules to be created. When asked what other topics they would like to see on a module, Denver's administrator suggested that the CSE "start with the basics of the discipline: paternity, intake modules, interviewing techniques, case maintenance." The Mesa county administrator wants to see almost every topic in a module. As she explained, "After seeing the locate module, we want to utilize it more. The state should put it on the Local Area Network or on the Internet."

RECEPTIVITY TO COMPUTER-BASED TRAINING

To determine how receptive other Colorado counties would be to computer-based training, we spoke with administrators in Fremont, Morgan, Philips/Sedgwick, Custer, and Prowers counties. These counties were not involved in the computer-based training pilot project and thus had little or no direct experience with CBT. All administrators were asked to discuss their feelings about computer based training and the role it should play in Colorado's training mix.

For a variety of reasons, most related to time and money, the smaller counties located at great distances from the Denver-metro area were the most excited about the CBT. These counties tend not to send their workers to state training and view CBT as the only affordable way to train child support workers. As the following two administrators noted:

CBT is a very cost-effective way to go especially for the medium and smaller counties who do not have the time or money to send their workers to the metro-area for training.

We are such a small county that we don't have time to go to the training. Right now we are unable to send anyone to the state or regional training because the workers have more than one job and cannot take time off. The trainings are usually at the beginning or end of the month, and that is when other jobs require the most attention so they are not able to leave.

Most county administrators we interviewed feel as though their workers are proficient enough on computers to handle CBT. Most counties also report that they have computer support resources available if a minor problem with a CBT module does arise. Thus, they see little need for support by CSE staff once the modules are developed and distributed. With respect to the distribution of CBT modules, administrators report they would need either Internet access or would need to be connected to the LAN. Each administrator feels that a wide variety of subjects could be made into training modules on the computer, and if that occurred, they would definitely be used in their office.

CONCLUSIONS

With the support of the Model Office Project, Colorado purchased Authorware software and paid for three state child support employees to attend a five-day training program conducted by Policy Studies, Inc. The three employees, whose job descriptions include training along with many other duties, embarked on the process of developing a computer-based training program on the location process. Done in spurts, along with many other job responsibilities, development of the location module took 15 months. The result however, is a comprehensive, multi-media, interactive product that has been extremely well received by administrators and line staff in the counties participating in the pilot project.

Knowledge tests administered to workers exposed to CBT and to conventional, stand-up training on the location process reveal that the two formats are equally effective in imparting child support information. Attitudinal surveys reveal that both approaches meet with worker approval. To the extent that there are differences, workers tend to find CBT more convenient and entertaining. They are somewhat more apt to find stand-up training responsive to their questions and less apt to find it confusing and hard to follow .

Workers exposed to CBT, their administrators and administrators in other Colorado counties are eager for the state to develop new CBT modules dealing with other child support topics. Small counties and/or those that are distant from Denver report that they rarely send workers to training programs based in the Denver metro area. Many contend that they cannot even afford the time and money needed to participate in regional training programs. CBT promises to be a flexible and affordable training resource that can be used with new and veteran workers in a variety of office settings.

CBT developers are proud of their training product and excited about the possibility of developing new modules. They liked working together as a development team but believe that the process could be made much more efficient by having a block of time dedicated to development. Although the development process is definitely more complicated than the software architects and trainers contend, the developers believe that the first module was probably the most difficult and that future training modules will be much easier to create.

While there is little doubt that everyone would like to see CBT play a bigger role in the training menu offered by the State Child Support Enforcement Unit, there is also consensus that stand-up training should continue to be a part of the training mix available to counties. State and county personnel agree that stand-up training affords valuable opportunities for personal contact between state and local workers. There is also consensus that stand-up training is needed to explain new legislation like welfare reform. If new CBT modules are developed, it will be necessary to review the stand-up training schedules and determine which programs should be retained, altered, or discontinued in light of this new training resource.

Colorado appears to be the only state to have developed a state-specific, computer-based training module. Other states are either using more generic computerized training programs or have not ventured in this area at all. This pilot project suggests that child support workers and administrators in Colorado value what the CBT development team has created. Although future products will require additional development work, it appears that the biggest investment has already been made. Colorado should quickly purchase the software upgrades needed to ensure the distribution of the location module to all counties through the LAN or Internet. Based on this evaluation, there is little doubt that the decision to develop new CBT modules will meet the strong approval of county child support personnel.

by Training and Site