[ Main Page of Report | Contents of Report ]
This chapter examines the extent to which the mandatory NEWWS employment- and education-focused programs actually increased sample members' participation in employment-related activities. The findings presented below demonstrate that participation differences between program and control group members (in all seven sites with five-year survey data) and between LFA and HCD program group members (in the three LFA-HCD sites) are substantial. This chapter therefore confirms that the NEWWS Evaluation represents a legitimate test of the relative effectiveness of different welfare-to-work program approaches. In addition to participation outcomes, the chapter also discusses whether education-focused programs increased the percentage of sample members who attained GED certificates or other education credentials after random assignment, a key impact measure for these types of programs. Results are presented for the full sample and for subgroups of people who were high school graduates ("graduates") or high school nongraduates ("nongraduates") as of random assignment. Data on participation, from the Five-Year Client Survey, are available for seven of the NEWWS programs: the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside, and the Portland program.
[Go to Contents]
The analysis of sample members' levels of participation in potentially employment-promoting activities and degree attainment extends the discussion of program dimensions in Chapter 1. Participation levels for program group members and the types of activities in which people participate demonstrate how successfully employment- and education-focused programs implemented their strategies for self-sufficiency. For the seven programs for which Five-Year Client Survey data are available, this chapter will determine how consistently, over the full five-year follow-up period, programs increased participation levels or degree receipt beyond what welfare recipients would be expected to attain had they never enrolled in a mandatory welfare-to-work program. Results for control group members represent the alternative outcomes, and program-control group differences indicate the effect, or impact, of each program. It should be stressed that a program's effect on participation depends on the levels attained by members of both the program group and the control group. In previous evaluations of welfare-to-work programs, from 20 to 40 percent of control group members enrolled in education and training programs on their own over a two-year follow-up period.(1) As will be shown, rates of self-initiated participation for control group members in this study, where the follow-up is extended to five years, is significantly higher. This is important, as programs with similar rates of participation for program group members may have very different impacts, depending on how frequently their respective control group members engaged in employment-related activities on their own.
During the first two years of follow-up, all 11 NEWWS programs generated moderate to large impacts on participation,(2) and it was expected that these impacts would remain in later years as programs continued to enroll in activities individuals who finished their initial assignment without gaining employment and, in some cases, individuals who left welfare and then returned.(3) For participants in LFA programs, the intended program path started with job search activities such as job club, intended to last approximately five weeks, followed by short-term education and training only for those unable to find employment. The intended HCD sequence was longer-term education and training, generally lasting up to two years, after which clients were expected to test their gained skills in the labor market through some type of job search activity.(4) The Portland program employed a unique, mixed strategy: Program staff assigned most individuals to job search first, while the more disadvantaged members of the caseload were often first referred to education and training activities. (See Chapter 1 for a more in-depth discussion of the intended assignment patterns in — and the specific activities offered by — each NEWWS program.) As noted in Chapter 1, in all sites the program focus varied somewhat over the five-year follow-up period. Most notably among the sites included in the five-year participation analysis, the three HCD programs became more employment-focused over time, assigning more individuals to job search and work experience activities. To the extent that this occurred, over time impacts on job search participation should increase in these programs. In the LFA programs, program group members who did not find employment through job search and work experience could have been assigned to education and training, and thus over time education and training impacts can be expected to increase in these programs. Furthermore, as the more job-ready left welfare for work, programs were left with a more disadvantaged caseload at the end of the follow-up period. Thus, while impacts were expected to remain over the entire follow-up period, it was also expected that the patterns of impacts might change as the types of activities in which program group members participated changed over time.
The end of the embargoes on control group members receiving mandatory welfare-to-work program services in Atlanta and Grand Rapids two of the seven sites examined in this chapter is another factor that may have influenced long-term program impacts on participation. An analysis of welfare receipt and participation among controls who became eligible for welfare-to-work program services shows that most control group members in these four programs never received such services during the five-year follow-up period.
Participation levels presented in the chapter are estimated from survey responses. The analysis includes all instances of participation after random assignment, including activities that occurred outside the welfare-to-work programs. Most commonly, self-initiated or nonprogram participation among program group members occurred after sample members left the welfare or program-mandatory rolls; less commonly, they might have participated in a self-initiated activity while still enrolled in a welfare-to-work program that their case manager could not approve as a program activity because the type or intensity of the activity did not meet the program's standards.(5) Sample members are considered to have participated in an employment-related activity if they attended for at least one day. Most participants attended for a much longer period.(6) The Five-Year Client Survey asked about participation since random assignment and in follow-up year 5. Thus, cumulative five-year and year 5 participation data are available.
[Go to Contents]
Two-year participation findings, available for all 11 NEWWS programs, provide the basis for the longer-term rates, during a time period in which program participation was at its most intense. Across all 11 programs, program group members' two-year participation rates (in any employment-related activity) ranged from 19 to 29 percent in Atlanta, Columbus, and Riverside to approximately 40 percent in Grand Rapids, Detroit, Oklahoma City, and Portland.(7) Two-year participation rates were comparable to participation rates attained in most welfare-to-work programs studied in previous evaluations. Program group members most often participated in job search and basic education, including Adult Basic Education (ABE), high school completion, and General Educational Development (GED) certificate preparation classes, with levels varying by program approach and educational attainment subgroup. On their own initiative, without a mandate to participate in a welfare-to-work program, control group members participated most often in education and training activities (including post-secondary education and vocational training), somewhat less frequently in basic education, and least often in job search, work experience, and on-the-job training. Control group participation levels were notably high in Detroit, Oklahoma City, and Grand Rapids.
The four employment-focused programs produced large gains in job search participation — between 27 percentage points in Grand Rapids LFA and 32 percentage points in Portland — for both graduates and nongraduates. All education-focused programs except for Detroit and Oklahoma City achieved large increases in basic education participation for nongraduate sample members; however, education-focused programs had little effect on increasing participation in employment-related training for graduates. The three HCD programs and Portland's employment-focused, varied first activity program produced moderate to large increases in the attainment of a GED certificate among welfare recipients who lacked this credential at random assignment.
In sum, the two-year participation levels indicate that all sites except Oklahoma City and Detroit, both of which were low enforcement, successfully implemented their self-sufficiency approach. All LFA programs generated large increases in job search, and all HCD programs generated large increases in education and training activities. Education-focused program impacts were concentrated among nongraduates, who were most often assigned to basic education. Portland's mixed approach achieved large increases in job search and education and training activities.
Sanction rates were also examined at the two-year follow-up. In general, there was no clear association between a program's level of sanctioning and participation rates or impacts. Among the three programs that had the smallest increases in participation in any activity, two were low enforcement, while the third — Grand Rapids LFA — sanctioned a larger percentage of sample members than any other program. The two programs with the highest participation rates, Riverside LFA and Portland, both had mid-level sanctioning rates.
For the remainder of the chapter, the focus will narrow to the seven sites with five-year survey data. When two- and five-year trends are discussed, the sample is further restricted to those sample members for whom both two- and five-year survey data are available.
[Go to Contents]
In a random assignment research design, control group participation levels reflect self-initiated patterns of activity in employment-promoting activities. Participation among controls does not represent a compromise to the experiment; rather it shows what program group members would have likely done in the absence of a mandatory welfare-to-work program. In Portland and Riverside, control group members were kept from receiving welfare-to-work program services for the entire five-year follow-up period. For these two sites the control group participation levels presented in Figure 3.1 (and in the tables throughout the chapter) represent entirely self-initiated participation in activities that control group members sought out and attended on their own in their communities. In Atlanta and Grand Rapids, some of the control group participation, specifically participation in job search and work experience activities toward the end of the five-year follow-up period among sample members who were still receiving welfare at that point, did in fact occur in the context of a mandatory welfare-to-work program, after embargoes on such services were lifted. In Atlanta, it is likely that as much as, but certainly no more than, 15 percent of all controls were "exposed" to a mandatory welfare-to-work program in year 4 or 5, while the corresponding estimate for Grand Rapids is 7 percent.(8) Thus, most controls were kept from re-ceiving program services. Control participation in mandatory welfare-to-work program activities was further limited in several respects: First, it was limited to the end of the follow-up period (as late as the last quarter of follow-up year 5 for some controls in Atlanta); second, it was limited to job search or work experience activities, since it is unlikely that controls would have been assigned to activities other than these once they became mandatory to participate.
Figure 3.1
Two-Year and Five-Year Participaiton Rate in Employment-Related Activities
for Control Group Members
SOURCE: MDRC calculatios from the Two-Year and Five-Year
Client Surveys.
NOTE: See Appendix
A.2
Primarily on their own initiative, most control group members in the seven NEWWS programs participated in an employment-promoting activity during the five-year follow-up period, as shown in Figure 3.1. In all sites, controls mainly participated in education and training activities; across all sites from 35 to 55 percent of controls participated in education or training. More specifically, controls primarily enrolled themselves in vocational training programs and post-secondary education.(9) (Participation rates for these activities are presented in Appendix Table B.1.) More than 25 percent of control group members in every site except Atlanta took a post-secondary education course during the follow-up period. Participation levels were equally high for vocational training programs, ranging from 23 percent in Atlanta to 29 percent in Portland. For the most part, control group members' participation in education and training was consistent throughout the follow-up period, although in Atlanta participation was more common in years 3 to 5 than in years 1 and 2.
While education and training were the most common activities that controls participated in throughout the follow-up period, the largest increases in control group participation in years 3 to 5 were in job search. Job search was an activity in which very few controls had participated during the first two years of follow-up (two-year participation rates for this activity ranged from 4 percent in Atlanta to 7 percent in Portland). By the end of year 5, depending on site, from one-fifth to one-third of control group members had participated in job search. In Riverside and Portland, the two sites where control group participation levels represent entirely self-initiated participation, there were striking increases in job search participation in the last follow-up years. Such services were most likely provided to control group members by community colleges and other community-based organizations in the context of other types of education and training activities in which controls were participating, such as vocational training programs and courses at community colleges, after they had left welfare. In Portland, for example, the local community college system offered much-marketed job search services to low-income individuals independent of the welfare department-associated welfare-to-work program. In Atlanta and Grand Rapids, as previously noted, there is some evidence that some of the increase in job search participation was due to controls participating in mandatory welfare-to-work programs; however, even upper-bound estimates of likely program-related participation do not account for the bulk of the increase.
[Go to Contents]
The previous section described control group members' patterns of participation in largely self-initiated employment-promoting activities over the five-year follow-up period. As noted, control participation levels represent the extent to which program group members would have likely participated in such activities in the absence of a mandatory welfare-to-work program. Participation impacts, or program-control differences, represent the degree to which program group members in the seven mandatory NEWWS programs examined in this chapter participated in employment-promoting activities above and beyond the rate at which they would have participated in the absence of a mandate to participate.
Impacts for the full sample on three different measures of participation — participation in any activity, participation in job search or job club, and participation in any education or training activity — are presented in Table 3.1. Appendix Table B.1 presents impacts separately on participation in basic education, post-secondary education, vocational training, and work experience.
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 1,071 | 74.9 | 54.8 | 20.1 *** | 36.7 |
| Atlanta Human Capital Development | 1,146 | 74.2 | 54.8 | 19.3 *** | 35.3 |
| Grand Rapids Labor Force Attachment | 1,097 | 77.7 | 63.9 | 13.7 *** | 21.5 |
| Grand Rapids Human Capital Development | 1,109 | 81.1 | 63.9 | 17.1 *** | 26.8 |
| Riverside Labor Force Attachment | 1,219 | 78.2 | 57.4 | 20.8 *** | 36.2 |
| Lacked high school diploma or basic skills | 657 | 77.0 | 55.1 | 21.9 *** | 39.7 |
| Riverside Human Capital Development | 778 | 82.0 | 55.1 | 26.8 *** | 48.6 |
| Portland | 504 | 83.8 | 75.0 | 8.8 ** | 11.8 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 1,071 | 59.2 | 30.1 | 29.2 *** | 97.0 |
| Atlanta Human Capital Development | 1,146 | 43.7 | 30.1 | 13.6 *** | 45.4 |
| Grand Rapids Labor Force Attachment | 1,097 | 51.6 | 21.3 | 30.3 *** | 142.5 |
| Grand Rapids Human Capital Development | 1,109 | 39.5 | 21.3 | 18.2 *** | 85.6 |
| Riverside Labor Force Attachment | 1,219 | 53.5 | 20.4 | 33.2 *** | 162.9 |
| Lacked high school diploma or basic skills | 657 | 58.2 | 19.6 | 38.5 *** | 196.5 |
| Riverside Human Capital Development | 778 | 49.6 | 19.6 | 30.0 *** | 152.9 |
| Portland | 504 | 65.2 | 35.4 | 29.8 *** | 84.1 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 1,071 | 42.2 | 35.3 | 6.9** | 19.4 |
| Atlanta Human Capital Development | 1,146 | 61.2 | 35.3 | 25.8*** | 73.1 |
| Grand Rapids Labor Force Attachment | 1,097 | 54.4 | 55.2 | -0.7 | -1.3 |
| Grand Rapids Human Capital Development | 1,109 | 69.3 | 55.2 | 14.1*** | 25.6 |
| Riverside Labor Force Attachment | 1,219 | 50.1 | 48.9 | 1.3 | 2.6 |
| Lacked high school diploma or basic skills | 657 | 46.0 | 47.1 | -1.0 | -2.2 |
| Riverside Human Capital Development | 778 | 68.9 | 47.1 | 21.9*** | 46.4 |
| Portland | 504 | 67.5 | 55.0 | 12.5*** | 22.8 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2. |
|||||
Most program group members (from 74 percent in Atlanta HCD to 84 percent in Portland) participated in some type of employment-promoting activity over the five-year follow-up period. Overall participation rates (participation in "any activity") did not vary by program focus; for example, while the Portland employment-focused program achieved the highest overall participation rate, the Riverside HCD program achieved the second highest rate. All seven programs increased overall participation relative to control group levels. The magnitude of impacts varied somewhat by program: Most programs increased participation by approximately 20 percentage points or more, while the Portland program had a relatively small impact of only 9 percentage points.(10) The fact that the Portland program did not generate a larger increase in participation is likely due to substantial control group participation in the later follow-up years. Control group members in this site more actively participated in self-initiated activities, particularly in years 3 to 5, than controls in other sites.
As expected, job search was the most common activity among program group members in employment-focused programs; well over half of program group members in all four programs participated in this activity over the five-year follow-up period. All four of these programs increased participation in job search by approximately 30 percentage points or more. Work experience (including on-the-job training and unpaid work experience programs) was also a somewhat common activity in employment-focused programs, particularly the Portland and Atlanta LFA programs. All three LFA programs had moderate impacts on participation in this activity, while the Portland program did not increase participation relative to the control group.
Notably, job search participation among HCDs increased markedly in the last follow-up years. Across the three programs, five-year job search participation rates ranged from 40 to 50 percent.(11) As previously noted, these increases were anticipated for several reasons. It is natural that with the passage of time in a longer follow-up period, participation in all activities would increase. More specifically, over time in these programs additional program group members completed their initial education-focused activity assignments, and those who did so were then expected to test their skills in the labor market through some type of job search program. Finally, as discussed in Chapter 1, these programs became somewhat more employment-focused over time. As a result of all these factors, five-year impacts on job search participation were larger than two-year impacts in all three HCD programs. Despite these increases, cumulative five-year impacts on job search in the Atlanta and Grand Rapids HCD programs were still smaller than those in their LFA programs — and in employment-focused programs in general, while the Riverside HCD program had a job search participation impact of 30 percentage points, roughly equal to the impacts found in the four employment-focused programs.
Education and training remained the most common type of activity for program group members in the three HCD programs throughout the five-year follow-up period. Participation levels in all three programs were in the 60 to 70 percent range on this aggregate measure. Two of the three HCD programs, Atlanta and Riverside, had impacts of greater than 20 percentage points on overall education and training, while the third, Grand Rapids, had a more modest impact of 14 percentage points.
The aggregate "education and training" measure includes participation in basic education, post-secondary education, and vocational training. Participation rates and impacts were also examined for each of these activities separately. (See Appendix Table B.1.) Notably, all three programs increased basic education participation for the full sample. The Riverside HCD program, which enrolled only nongraduates or those deemed by program staff to be in need of basic education at the time of study entry, increased basic education participation by 31 percentage points. The other two HCD programs, Atlanta and Grand Rapids, enrolled both graduates and nongraduates, and these programs had more modest gains when both subgroups are considered together. (Impacts are discussed separately for graduates and nongraduates, below.)
For the full sample, impacts on vocational training and post-secondary education were not as consistent across the HCD programs and, where they did occur, were not as large as impacts on participation in basic education. The HCD programs in Atlanta and Grand Rapids increased participation in vocational training for the full sample, although the latter impact was just below the threshold for statistical significance. No HCD program increased the percentage of sample members who took a course at a community, two-year, or four-year college.
While there was substantial education and training participation among LFAs — roughly half of all program group members in the LFA programs participated in an education or training activity over the follow-up period — impacts on education and training participation were much less common and, where they did occur, smaller in the LFA programs than in the HCD programs. Of the three LFA programs, only Atlanta increased overall participation in education and training activities. This impact was due to a 10 percentage point increase in basic education participation, as the program did not increase post-secondary education or vocational training participation.
The Portland program, while employment-focused, differed from the LFA programs in that case managers did not always assign individuals to job search as their first activity, and the program made heavy use of available short-term education and training programs. Portland staff, more so than case managers in the LFA programs, emphasized education and training as a means of building skills necessary to acquire higher-paying, stable jobs. This emphasis is reflected in program group members' participation: Over the five-year follow-up period, 42 percent of program group members took a course at a community, two-year, or four-year college, and 25 percent participated in a vocational training program. While the program did not increase vocational training participation relative to control group levels, the program did increase college participation by a notable 16 percentage points.
Consistent with the findings for the full sample, all programs had substantial impacts on participation in any activity for both educational attainment subgroups, of roughly equal magnitude to those found for the full sample. These impacts were statistically significant in all sites except for Portland, where the lack of statistical significance is likely a product of small sample size.(12) All seven programs also had impacts on job search participation, equal in size to those found for the full sample, for both subgroups. (See Tables 3.2 and 3.3 for impacts on participation in any activity, job search, and any education or training activity, by educational attainment subgroup. Impacts on participation in basic education, post-secondary education, vocational training, and work experience are presented in Appendix Tables B.2 and B.3.) In the paragraphs that follow, impacts on participation in specific activities are discussed for both high school graduates and nongraduates.
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 407 | 70.1 | 49.8 | 20.3 *** | 40.7 |
| Atlanta Human Capital Development | 437 | 74.3 | 49.8 | 24.5 *** | 49.2 |
| Grand Rapids Labor Force Attachment | 434 | 77.0 | 64.1 | 13.0 *** | 20.2 |
| Grand Rapids Human Capital Development | 468 | 85.0 | 64.1 | 20.9 *** | 32.6 |
| Riverside Labor Force Attachment | 657 | 77.0 | 55.1 | 21.9 *** | 39.7 |
| Riverside Human Capital Development | 778 | 82.0 | 55.1 | 26.8 *** | 48.6 |
| Portland | 163 | 82.0 | 71.4 | 10.7 | 14.9 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 407 | 56.6 | 27.7 | 28.9 *** | 104.4 |
| Atlanta Human Capital Development | 437 | 43.2 | 27.7 | 15.5 *** | 56.1 |
| Grand Rapids Labor Force Attachment | 434 | 51.1 | 22.6 | 28.5 *** | 126.4 |
| Grand Rapids Human Capital Development | 468 | 39.3 | 22.6 | 16.7 *** | 74.0 |
| Riverside Labor Force Attachment | 657 | 58.2 | 19.6 | 38.5 *** | 196.5 |
| Riverside Human Capital Development | 778 | 49.6 | 19.6 | 30.0 *** | 152.9 |
| Portland | 163 | 64.8 | 36.4 | 28.4 *** | 78.1 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 407 | 40.3 | 31.0 | 9.3 ** | 30.0 |
| Atlanta Human Capital Development | 437 | 65.4 | 31.0 | 34.4 *** | 111.3 |
| Grand Rapids Labor Force Attachment | 434 | 57.4 | 54.4 | 3.0 | 5.5 |
| Grand Rapids Human Capital Development | 468 | 75.9 | 54.4 | 21.5 *** | 39.5 |
| Riverside Labor Force Attachment | 657 | 46.0 | 47.1 | -1.0 | -2.2 |
| Riverside Human Capital Development | 778 | 68.9 | 47.1 | 21.9 *** | 46.4 |
| Portland | 163 | 68.3 | 58.0 | 10.3 | 17.8 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2. |
|||||
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 664 | 77.9 | 58.1 | 19.7 *** | 34.0 |
| Atlanta Human Capital Development | 709 | 73.8 | 58.1 | 15.7 *** | 27.0 |
| Grand Rapids Labor Force Attachment | 663 | 78.0 | 64.0 | 14.0 *** | 21.9 |
| Grand Rapids Human Capital Development | 641 | 79.0 | 64.0 | 15.0 *** | 23.5 |
| Riverside Labor Force Attachment | 562 | 79.8 | 60.4 | 19.3 *** | 32.0 |
| Portland | 334 | 84.2 | 77.1 | 7.1 | 9.2 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 664 | 60.6 | 31.6 | 29.0 *** | 91.6 |
| Atlanta Human Capital Development | 709 | 44.1 | 31.6 | 12.4 *** | 39.3 |
| Grand Rapids Labor Force Attachment | 663 | 51.9 | 20.5 | 31.4 *** | 153.3 |
| Grand Rapids Human Capital Development | 641 | 39.5 | 20.5 | 19.0 *** | 92.6 |
| Riverside Labor Force Attachment | 562 | 47.3 | 21.2 | 26.0 *** | 122.7 |
| Portland | 334 | 64.2 | 36.0 | 28.2 *** | 78.2 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 664 | 43.8 | 38.1 | 5.7 | 15.0 |
| Atlanta Human Capital Development | 709 | 58.0 | 38.1 | 19.9 *** | 52.2 |
| Grand Rapids Labor Force Attachment | 663 | 52.9 | 55.9 | -3.0 | -5.4 |
| Grand Rapids Human Capital Development | 641 | 65.7 | 55.9 | 9.8 *** | 17.6 |
| Riverside Labor Force Attachment | 562 | 55.6 | 51.4 | 4.2 | 8.3 |
| Portland | 334 | 67.5 | 53.2 | 14.3 ** | 26.9 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2 |
|||||
All three HCD programs assigned nongraduate sample members to basic education as their first activity, and the majority of nongraduate HCDs (approximately 60 to 70 percent) in all three programs participated in basic education over the five-year follow-up period. These participation levels resulted in impacts ranging from 27 percentage points in Grand Rapids to 42 percentage points in Atlanta. A small increase in post-secondary education participation in Grand Rapids was the only impact found on participation in any education or training activity beyond basic education for this subgroup. (See Appendix Table B.2.)
Atlanta LFA and Portland, the only two employment-focused programs to increase education and training participation for the full sample, also increased this type of participation for nongraduates. The Atlanta LFA program had a 16 percentage point increase in basic education participation and the Portland program had a more modest 10 percentage point gain. None of the employment-focused programs increased participation in vocational training or post-secondary education for this subgroup.
The HCD programs in Atlanta and Grand Rapids enrolled sample members with a high school diploma or GED at the time of study entry, and they often assigned sample members with these credentials to vocational training programs. Basic education was also an assigned activity for some sample members with a diploma or GED certificate if they had low math or reading skills. Although assignments to college were not made in HCD programs, in some instances case managers approved post-secondary education courses that program group members had already enrolled in on their own as fulfillment of the participation requirement.
In Atlanta, 44 percent of graduate HCDs participated in a vocational training program over the follow-up period, resulting in a large impact (17 percentage points) on the measure for this subgroup. Participation in post-secondary education was less common, and the program did not increase participation in this activity. In Grand Rapids, participation levels were high in both college and vocational training for graduates among both program and control group members, resulting in no impacts on either activity. Both HCD programs generated small increases in basic education participation for this subgroup. (See Appendix Table B.3.)
As was found for the full sample and for nongraduates, impacts on education and training activities for graduates in the three LFA programs were not widespread. None of the three LFA programs had a statistically significant impact on the aggregate education and training participation measure. The only participation impact found among these programs for this subgroup was a small increase in basic education participation in Atlanta.
The Portland program increased education and training participation by 14 percentage points for graduates, principally the result of a large 21 percentage point increase in college participation. More than half of the program group members in Portland took a course for credit at a two-year or four-year college, which was the only NEWWS program to increase college participation.
[Go to Contents]
As noted, programs were not expected to continue to enroll individuals in employment-promoting activities at the same rate during follow-up years 3 to 5 as they did during years 1 and 2. If the NEWWS programs were achieving their goal of increasing sample members' self-sufficiency, program group members should have been leaving welfare for employment. As this occurred, participation levels should have decreased over the follow-up period, particularly in job search and work experience activities. Furthermore, changes in the welfare reform environment discussed in Chapter 1 might have led to a simultaneous increase in participation among control group members who were still on welfare in later follow-up years. All these factors would lead to small or no program impacts on participation in year 5.
As shown in Table 3.4, year 5 participation rates in "any activity" were similar for program and control group members in all sites (ranging from 18 to 26 percentage points), resulting in no statistically significant participation impacts. In year 5 program and control members most often participated in job search, post-secondary education, and vocational training. Very few sample members participated in basic education. (Results not shown in tables.)
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Any activity | |||||
| Atlanta Labor Force Attachment | 1,018 | 24.6 | 22.5 | 2.1 | 9.2 |
| Atlanta Human Capital Development | 1,092 | 20.9 | 22.5 | -1.6 | -7.1 |
| Grand Rapids Labor Force Attachment | 1,043 | 19.0 | 18.7 | 0.4 | 2.0 |
| Grand Rapids Human Capital Development | 1,048 | 21.6 | 18.7 | 2.9 | 15.5 |
| Riverside Labor Force Attachment | 1,072 | 26.0 | 22.0 | 4.0 | 18.2 |
| Lacked high school diploma or basic skills | 582 | 24.4 | 20.5 | 3.8 | 18.6 |
| Riverside Human Capital Development | 685 | 19.6 | 20.5 | -0.9 | -4.4 |
| Portland | 442 | 20.3 | 21.9 | -1.6 | -7.2 |
| Job search or job club | |||||
| Atlanta Labor Force Attachment | 1,018 | 14.5 | 11.0 | 3.4 * | 31.1 |
| Atlanta Human Capital Development | 1,092 | 13.4 | 11.0 | 2.4 | 21.3 |
| Grand Rapids Labor Force Attachment | 1,043 | 8.9 | 5.6 | 3.3 ** | 59.2 |
| Grand Rapids Human Capital Development | 1,048 | 8.8 | 5.6 | 3.2 * | 58.2 |
| Riverside Labor Force Attachment | 1,072 | 10.0 | 3.9 | 6.1 *** | 157.7 |
| Lacked high school diploma or basic skills | 582 | 9.4 | 4.5 | 5.0 ** | 111.7 |
| Riverside Human Capital Development | 685 | 10.6 | 4.5 | 6.1 *** | 136.8 |
| Portland | 442 | 10.6 | 10.8 | -0.2 | -1.9 |
| Any education or training | |||||
| Atlanta Labor Force Attachment | 1,018 | 11.0 | 13.4 | -2.3 | -17.4 |
| Atlanta Human Capital Development | 1,092 | 10.8 | 13.4 | -2.6 | -19.5 |
| Grand Rapids Labor Force Attachment | 1,043 | 10.6 | 15.0 | -4.4 ** | -29.4 |
| Grand Rapids Human Capital Development | 1,048 | 13.7 | 15.0 | -1.3 | -8.8 |
| Riverside Labor Force Attachment | 1,072 | 16.0 | 18.8 | -2.7 | -14.5 |
| Lacked high school diploma or basic skills | 582 | 14.5 | 16.7 | -2.2 | -13.3 |
| Riverside Human Capital Development | 685 | 10.6 | 16.7 | -6.1 ** | -36.4 |
| Portland | 442 | 10.7 | 12.4 | -1.7 | -13.7 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2. |
|||||
While none of the seven programs increased overall participation in year 5, five of the programs (all except Atlanta HCD and Portland) continued to generate small increases in job search participation. None of the seven programs, however, increased participation in education or training activities in year 5, although there was a statistically significant decrease in the Riverside HCD program, probably because some Riverside HCD program group members who would have participated in this type of activity on their own were instead assigned to job search activities by their case managers.
Year 5 participation levels were substantially higher for sample members who received at least one welfare payment in year 5. In more than half of the programs year 5 participation rates (in "any activity") for year 5 welfare recipients exceeded 30 percent. Participation levels were particularly high in job search; in many programs year 5 job search participation rates for recipients were about twice as high as those found for the full sample. Year 5 participation levels for those still receiving welfare indicate that the programs were still actively working with the on-welfare caseload at the end of the five-year follow-up period.
In Atlanta and Portland, job search participation levels were high for control group members still receiving welfare in year 5 as well, with roughly one-fifth of control group welfare recipients in both sites participating in job search in year 5. As has been discussed, this level of control group participation in job search in Atlanta is likely at least partially the result of some control group members receiving welfare-to-work program services in year 5. In Portland, as noted, this level of control group job search participation is likely due to the marketing and reach of the local community college system, which offers job search services to low-income individuals independent of the welfare department-associated welfare-to-work program.
Year 5 participation rates did not vary by educational attainment subgroup. However, significant increases or impacts in job search participation were concentrated among nongraduates. (Results not shown.) This cluster of impacts is not surprising, given that nongraduates have longer stays on welfare and were thus more likely to still be on welfare and subject to a mandatory welfare-to-work program participation requirement at the end of the follow-up period.
[Go to Contents]
Figure 3.2 presents two-year and five-year participation rates for program and control group members. For each program, the difference between the side-by-side black and white bars represents the program impact for the respective cumulative period. In all seven programs, as indicated by the stars representing statistical significance, program group participation levels were significantly higher than control group levels for both years 1 and 2 and years 1 to 5. Notably, two- and five-year cumulative impacts were similar. For the most part, impacts that were large after two years remained so after five years. Thus, during years 3 to 5, differences between program and control group participation rates were maintained; although, for the most part, impacts did not grow.
Figure 3.2
Two-Year and Five-Year Participation in Emploment-Related Activities, by
Research Group
SOURCE: MDRC calculations from the Five-Year Client Survey.
NOTES: See Appendix A.2.
Five-year cumulative impacts were somewhat smaller than those found after two years in the Atlanta HCD and Portland programs. In all three HCD programs, five-year impacts on job search participation were larger than impacts found after two years. (Results not shown.)
[Go to Contents]
Degree attainment can also play an important role in sample members' long-term labor market and welfare behavior. Individuals who received a GED or trade certificate during follow-up may have delayed entry into the labor market while they were attending school. In later years, however, those who attain new education credentials may have a better chance of finding a job or advancing to higher-paying and more stable employment. It is important to note that while research on the economic gains to welfare recipients of skill building, including degree attainment, has been rather inconclusive, a substantial body of research exists that shows the general importance of education credentials in the labor market. The outcomes described in this section thus may play an important role in the economic outcomes discussed in later chapters and in the future labor market success of program group members beyond the follow-up period covered in this report.(13)
In this analysis degree attainment was analyzed separately for high school graduates and nongraduates. For nongraduates, three measures were examined: receipt of a high school diploma or GED certificate, receipt of a trade license or certificate, and receipt of a high school diploma or GED certificate and a second education or training credential. For sample members with a high school diploma or GED at the time of program entry, two measures were analyzed: receipt of any education or training credential (either a trade license or an associate's, bachelor's, or graduate degree) and receipt of a trade license or certificate.
In all three HCD programs a majority of high school nongraduate sample members participated in basic education, including GED preparation classes, and the degree to which these programs increased receipt of a high school diploma or GED certificate relative to control group levels is a key indicator of the relative success of the basic education component of these programs. As shown in Table 3.5, across all sites from 3 to 17 percent of all control group members in this subgroup received a high school diploma or GED during the follow-up period. All three HCD programs increased program group members' receipt of this credential. The largest impact — 11 percentage points — occurred in the Riverside HCD program, where nearly 20 percent of nongraduate program group members received this credential over the follow-up period. The Atlanta and Grand Rapids HCD programs had slightly smaller impacts, 7 and 9 percentage points, respectively. While the Atlanta HCD program had a moderate impact on this measure, absolute levels of degree receipt were quite low in this program for both program and control group members.
| Site and Program | Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Received a high school diploma or GED | |||||
| Atlanta Labor Force Attachment | 407 | 4.1 | 3.0 | 1.2 | 39.0 |
| Atlanta Human Capital Development | 437 | 10.0 | 3.0 | 7.1*** | 237.6 |
| Grand Rapids Labor Force Attachment | 434 | 15.4 | 13.1 | 2.3 | 17.5 |
| Grand Rapids Human Capital Development | 468 | 22.5 | 13.1 | 9.4*** | 71.4 |
| Riverside Labor Force Attachment | 657 | 5.9 | 5.9 | 0.1 | 0.9 |
| Riverside Human Capital Development | 778 | 17.0 | 5.9 | 11.1*** | 188.5 |
| Portland | 163 | 25.9 | 16.8 | 9.1 | 54.2 |
| Received a trade license or certificate | |||||
| Atlanta Labor Force Attachment | 407 | 5.3 | 5.1 | 0.2 | 4.0 |
| Atlanta Human Capital Development | 437 | 6.0 | 5.1 | 0.9 | 17.1 |
| Grand Rapids Labor Force Attachment | 434 | 7.3 | 9.7 | -2.4 | -24.6 |
| Grand Rapids Human Capital Development | 468 | 9.7 | 9.7 | 0.0 | 0.4 |
| Riverside Labor Force Attachment | 657 | 11.7 | 10.7 | 1.0 | 9.4 |
| Riverside Human Capital Development | 778 | 11.6 | 10.7 | 0.9 | 8.5 |
| Portland | 163 | 16.6 | 4.4 | 12.2** | 277.5 |
| Received a high school diploma or GED and a second education or training credential | |||||
| Atlanta Labor Force Attachment | 407 | 1.3 | 1.1 | 0.2 | 20.3 |
| Atlanta Human Capital Development | 437 | 3.1 | 1.1 | 2.0 | 184.9 |
| Grand Rapids Labor Force Attachment | 434 | 2.6 | 3.6 | -1.0 | -28.0 |
| Grand Rapids Human Capital Development | 468 | 4.2 | 3.6 | 0.6 | 16.5 |
| Riverside Labor Force Attachment | 657 | 1.8 | 2.5 | -0.6 | -26.2 |
| Riverside Human Capital Development | 778 | 3.4 | 2.5 | 1.0 | 38.5 |
| Portland | 163 | 8.4 | 0.3 | 8.1** | 3171.6 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2. |
|||||
Absolute levels of high school diploma and GED receipt were substantially lower for program group members in the three LFA programs than in the HCD programs, and none of the LFA programs significantly increased any measure of degree receipt for high school nongraduates.
The Portland program had the most noteworthy effects on degree attainment for nongraduates 25 percent of nongraduate program group members received a high school diploma or GED over the follow-up period, and the program produced an increase, though one above the standard cutoff used in this report for statistical significance, of 9 percentage points on the measure.(14) The program also had a large impact on the receipt of a trade license or certificate, and for this subgroup it was the only program to have such an effect. This finding is somewhat surprising given that the program did not have a statistically significant five-year impact on vocational training for nongraduates (see Section V). There are several possible explanations: The program may have increased the likelihood that program group members enrolled in training programs offering trade licenses or certificates rather than nondegree programs, while not necessarily increasing participation in training programs overall. The program also may have increased the likelihood that individuals completed credential-offering programs. Finally, it is possible that impacts on participation in training found at the two-year follow-up point drove this impact on trade license receipt.(15) The most notable finding on degree receipt in this program, however, is that it had a substantial impact on the percentage of high school nongraduate sample members who attained both a GED certificate or a high school diploma and a second education or training credential (such as a trade license or college degree) over the follow-up period. Nonexperimental research conducted as part of the NEWWS Evaluation has found that high school nongraduate participants in adult education have substantially stronger improvements in longer-term earnings and self-sufficiency if they also participate in skills training or college, making the Portland impact on this measure a particularly promising finding.(16)
As shown in Table 3.6, education and training credential receipt was prevalent among control group members who already had a high school diploma or GED certificate at study entry, although less so in both Atlanta programs. Trade licenses were by far the most common type of credential earned for this subgroup; very few individuals obtained college degrees. Atlanta HCD, the only program to increase graduate sample members' vocational training participation, also increased trade license receipt for this subgroup. The only other program to have this effect was the Atlanta LFA program. In Grand Rapids, where graduate control group members' self-initiated participation in post-secondary education and vocational training was common throughout the follow-up period, the LFA program actually decreased credential receipt for graduates, likely by diverting to job search sample members who would have participated in education or training activities on their own (and thus possibly have obtained an education credential). This pattern is also reflected in the participation impacts discussed in Section V.
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Received any education or training credential | |||||
| Atlanta Labor Force Attachment | 664 | 20.4 | 13.2 | 7.2 ** | 54.7 |
| Atlanta Human Capital Development | 709 | 26.9 | 13.2 | 13.7 *** | 104.2 |
| Grand Rapids Labor Force Attachment | 663 | 18.3 | 29.8 | -11.5 *** | -38.7 |
| Grand Rapids Human Capital Development | 641 | 28.7 | 29.8 | -1.0 | -3.4 |
| Riverside Labor Force Attachment | 562 | 20.5 | 25.4 | -4.9 | -19.2 |
| Portland | 334 | 28.9 | 22.4 | 6.5 | 28.9 |
| Received a trade license or certificate | |||||
| Atlanta Labor Force Attachment | 664 | 16.8 | 12.0 | 4.9 * | 40.8 |
| Atlanta Human Capital Development | 709 | 23.6 | 12.0 | 11.6 *** | 97.3 |
| Grand Rapids Labor Force Attachment | 663 | 15.5 | 22.5 | -7.0 ** | -31.2 |
| Grand Rapids Human Capital Development | 641 | 26.1 | 22.5 | 3.6 | 16.1 |
| Riverside Labor Force Attachment | 562 | 18.1 | 22.9 | -4.8 | -21.1 |
| Portland | 334 | 23.6 | 18.6 | 5.0 | 26.8 |
| SOURCE: MDRC calculations from the Five-Year
Client Survey. NOTES: See Appendix A.2. |
|||||
[Go to Contents]
All three LFA programs in NEWWS were job search first programs, for all individuals who were enrolled. Compared with what would have happened in the absence of these programs, the LFA programs increased participation in job search — to quite a large degree — for both graduates and nongraduates. For the most part, these programs did not increase education and training participation or degree receipt, although one program — Atlanta LFA — increased basic education participation for both subgroups and increased trade license or certificate receipt for graduates.
All HCD programs were education-focused, although activity assignments differed depending on the educational background and skills level of enrollees. Sample members without a high school diploma or GED — as well as some sample members who had this credential but had low scores on math or reading achievement tests — were most often assigned to basic education, and basic education participation rates for nongraduates were much higher than control group rates in all three programs. The two programs that enrolled graduates in HCD programs also increased their participation in this activity, although to a lesser degree. The HCD programs were more likely than the LFA programs to result in nongraduates obtaining a high school diploma or GED at some point during the follow-up period. For those with a high school diploma or GED, assignments to and participation in vocational training were most common, although only one of the two programs enrolling graduates increased participation in this activity relative to control group levels. The difference in assignment patterns for graduate and nongraduate sample members is likely explained by the fact that a high school diploma or GED, or in some cases a certain score on a reading or math achievement test, is required for entry into most vocational training programs. For the graduate subgroup, only the Atlanta HCD program resulted in more individuals obtaining an education or training credential (usually a trade license or certificate). Notably, the Atlanta LFA program had the same effect for this subgroup.
As is clear from the findings presented in this chapter, the Portland program was unique among these seven NEWWS programs in that program staff employed a mixed strategy, using both job search and short-term education and training programs as a first activity, to enhance individuals' self-sufficiency. Program staff encouraged short-term education and training as a means of enhancing employability — specifically, as a means of obtaining jobs with higher wages and benefits — for all sample members, both graduates and nongraduates. As a result, the program increased participation in job search for the full sample and for both subgroups, as well as increasing participation in education activities across both subgroups. For nongraduates, increases in education and training participation led to an increase, relative to control group levels, in the percentage of sample members who received a high school diploma or GED and a trade license or certificate during the follow-up period.
All seven programs generated impacts on participation of a sufficient magnitude to verify that the programs studied provided a good test of the relative effectiveness of different welfare-to-work program models, including the side-by-side test of LFA and HCD programs in the three sites that simultaneously operated both program types. The subsequent chapters in this report examine the economic effects of all 11 NEWWS programs on adults, as well as on households and children, that were produced by these changes in employment-promoting activities and in degree receipt.
[Go to Contents]
1. See, for example, Riccio, Friedlander, and Freedman, 1994, Table 2.4; Kemple, Friedlander, and Fellerath, 1995, Table 3.5; Hamilton and Friedlander, 1989, Table 3.1.
2. See Freedman et al., 2000a, for details.
3. An awareness of welfare caseload dynamics is essential in understanding and interpreting welfare-to-work program participation rates. A number of studies have shown that many welfare recipients cycle on and off the welfare rolls, often leaving without any special intervention. For example, some people get jobs on their own or get married. To the extent that this occurs among individuals mandated for a welfare-to-work program before they enter their first program activity, a site's overall participation rate will be lowered. This rate will be further lowered to the extent that individuals obtain part-time employment, which, if it involves a specified number of hours per week, excuses clients from a program participation requirement. At the same time, welfare-to-work programs may induce some of these behavioral changes. For example, a desire to avoid a program participation requirement may lead some individuals to find employment or leave welfare sooner than they otherwise would have done, again lowering a site's participation rate if these actions are taken prior to starting an activity. Alternatively, some individuals might feel encouraged to remain on welfare longer to take advantage of a program's opportunities for education and training. Thus, participation rates, whether high or low, are influenced by normal welfare caseload turnover as well as by a welfare-to-work program's intervention. In any case, given welfare dynamics, participation rates in these programs should never be expected to reach 100 percent.
4. See Hamilton et al., 1997, Figure 3.1, Figure 3.2, and pp. 39-44, for a discussion of the sequencing of activities in both LFA and HCD programs.
5. In the benefit-cost analysis in Chapter 13, activities outside the programs are called "out-of-program" activities.
6. For detailed descriptions of the length and intensity of participation, including total hours of participation, in all 11 NEWWS programs, see the following: Hamilton et al., 1997, Tables 5.5 and 6.4 (for the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside); Scrivener et al., 1998, Table 3.4 (for Portland); Storto et al., 2000, Table 2.2 (for Oklahoma City); Farrell, 2000, Table 3.5 (for Detroit); and Scrivener and Walter, 2001, Table 3.4 for Columbus.
7. See Freedman et al., 2000a, Appendix A.1, for complete two-year participation rates and impacts.
8. The estimates of the proportion of control group members in Atlanta and Grand Rapids who were "exposed" to welfare-to-work program services after embargoes on such services were lifted are considered upper-bound ones for a number of reasons, but primarily because of client survey data limitations regarding the timing of participation. The Five-Year Client Survey did not ask respondents to specify the exact month (as opposed to year) when they participated in various activities, so months of post-embargo welfare receipt cannot be lined up directly with participation spells. The method of calculating these estimates thus erred on the side of counting as "exposure" some participation that, in fact, may have occurred while an individual was no longer receiving welfare.
9. "Post-secondary education" includes courses taken at community colleges or two-year and four-year colleges, including — but not limited to — courses taken for credit toward an associate's or bachelor's degree. "Vocational training" includes classes taken for training in a specific job, trade, or occupation, and does not include courses taken at community colleges or two-year and four-year colleges.
10. Five-year participation rates for program and control group members subject to mandatory welfare-to-work programs are generally not available, and thus statistics comparable to those presented in this section largely do not exist. Five-year participation rates were calculated in an evaluation of San Diego's Saturated Work Initiative Model (SWIM), a program operated in the mid to late 1980s. Among single parents in SWIM (comparable to the NEWWS samples analyzed in this report), 66 percent of program group members participated in activities intended to increase their employment during a five-year follow-up compared with 42 percent of controls, resulting in an impact of 24 percentage points. This measure includes participation in welfare-to-work program activities as well as self-initiated participation. However, control group members became mandatory for California's GAIN welfare-to-work program after three years if they were still receiving welfare, so SWIM's estimates of control group participation levels and participation impacts are not directly comparable to those in most NEWWS sites.
11. Specifically, the Atlanta HCD program's five-year job search participation rate was 3.6 times greater than its two-year job search participation rate. In the Grand Rapids and Riverside HCD programs, five-year job search participation rates were 2.9 and 2.7 times greater, respectively, than two-year rates.
12. For both subgroups, the impacts on participation in any activity are just above the threshold for statistical significance. The sample sizes for these subgroups, including program and control group members, are small: The nongraduate subgroup consists of only 163 sample members, while the graduate subgroup consists of 334 sample members.
13. Chapter 4 in Bos et al., 2001, presents a synthesis of findings related to this topic and of research on welfare populations. In particular, see Mincer, 1974; Polachek and Siebert, 1993; and Sum, Taggart, and Fogg, 1995.
14. The 9.1 percentage point increase has a p-value of .22, and the lack of statistical significance may be a product of small sample size (see footnote 12).
15. For a more detailed discussion of the types of training programs offered by the Portland program, and two-year participation rates and impacts, see Scrivener et al., 1998, Chapter 3.
16. Among nongraduate sample members who participated in basic education during an initial two-year follow-up period, those who went on to participate in post-secondary education or training had an additional $1,542 in earnings in the third year of follow-up compared with those who participated only in basic education and did not go on to participate in post-secondary education or training (Bos et al., 2001, Table 6.4).
Home Pages:
National Evaluation of Welfare-to-Work Strategies
(NEWWS)
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)