[ Main Page of Report | Contents of Report ]
Welfare-to-work programs have two primary goals: reducing welfare use and increasing employment. This chapter explores the success of the programs studied in the NEWWS Evaluation in achieving the second of these goals by presenting their effects on employment and earnings over the five years after people were randomly assigned. The chapter also investigates whether employment-focused programs had larger initial effects than education-focused programs, whether the effects of education-focused programs had surpassed those of employment-focused programs by the five-year point, and which approach was more successful for high school graduates and nongraduates. Special attention is given to a direct comparison of results for the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside, because these findings provide the most reliable evidence of the relative benefits of each approach.
[Go to Contents]
The programs pursued three different strategies to attain the goals of increased employment and earnings: job search first, education and training first, and a mix of job search and education as initial activities.
Employment-focused, job-search-first programs (Atlanta, Grand Rapids, and Riverside LFA are examples of this approach) encourage rapid entry into the labor market and are therefore expected to boost employment and earnings quickly. Proponents of this approach expect people to work their way up to better jobs by gaining experience and skills on the job. If this occurs, then initial employment gains may persist or increase over time, and earnings increases may grow larger in later years. On the other hand, effects of job-search-first programs may grow smaller over time, as control group members begin finding work on their own or if program group members work at low-quality and unstable jobs that they quickly lose.
Education-focused programs (they include Atlanta, Grand Rapids, and Riverside HCD; Columbus Integrated and Traditional; Detroit; and Oklahoma City) aim to increase enrollees' skills and credentials before they seek employment. Employment and earnings gains may be delayed while recipients participate in education and training activities. Education-focused programs may even reduce employment and earnings initially if people would work in the absence of the programs, but enroll in education and training activities instead. For these reasons, effects are likely to be smaller for the education-focused programs than for job-search-first programs in the short term. Toward the end of follow-up, however, the effects of education-focused programs may catch up to or surpass the effects of job-search-first programs, if program group members make up for initial forgone earnings by obtaining higher-quality jobs than they would have under the job-search-first programs. Employment and earnings gains may never occur, however, if enrollees drop out of education and training activities or if area employers have little demand for the skills and credentials that enrollees obtain.
Employment-focused programs that use a mix of initial activities (Portland used this approach) try to combine the best features of the job-search-first and education-focused approaches. For more job-ready individuals, they use the same approach as other employment-focused programs by trying to move people into jobs relatively quickly. Case managers, however, have more discretion to assign some people to short-term skill-building activities first. If this strategy is successful, boosts in employment should occur early in the follow-up, as people required to look for work find jobs. Initial gains may be smaller than they would be for job-search-first programs, however, because some enrollees participate in education or training activities before looking for work. These programs could achieve especially large gains in employment and earnings later in the follow-up by moving a large portion of the caseload into higher-quality jobs. Specifically, more job-ready participants (who are asked to look for work) may be able to advance to better jobs, and less job-ready individuals (who are allowed to enroll in education and training activities) may be able to use their new skills to find better jobs than they would have found if they had looked for work initially. If neither element of the employment-focused, varied first activity approach is effective, however, or if activities are targeted at the wrong people, effects on employment and earnings may be small and should not increase.
This discussion assumes that welfare-to-work programs affect employment and earnings by providing helpful services. It is also possible, however, that sanctions used to enforce the mandates encourage people to go to work in order to avoid participating in program activities. If this is true, then highly mandatory programs may have similar effects initially regardless of their self-sufficiency approach. If education-focused programs have smaller effects than employment-focused programs, it may indicate that services are responsible for the program impacts rather than enforcement of the mandates.
[Go to Contents]
Table 4.1 provides the first indication of how much the different programs affected the number of people who worked, the number of quarters they worked, and their total earnings over the five-year follow-up period for each of the 11 programs in the NEWWS Evaluation. In addition, results are shown for sample members in need of basic education for the Riverside LFA program so they can be compared with results from the Riverside HCD program, which included only people in need of basic education.(1) (All tables and figures in this chapter show results based on quarterly reports by employers to state unemployment insurance systems.)
Table 4.1 shows that most control group members worked at some point in the follow-up period, even though most of them were never eligible for program services and none of them were eligible for services during the first three years after random assignment. For example, about 88 percent of control group members in Grand Rapids and 86 percent in Columbus worked at some point. Except for Riverside (66 percent), the Oklahoma City control group was the least likely to work: about 79 percent of control group members worked at some point. The fact that so many control group members worked left little room for the programs to increase the number of people who ever worked during the follow-up period. In fact, only three of the programs increased the number of people who worked by more than 3 percentage points. Moreover, the two programs with the largest effects were both in Riverside, where control group members were the least likely to work.
Site and Program |
SampleSize | ProgramGroup | Control Group | Difference(Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Ever employed (%) | |||||
| Atlanta Labor Force Attachment | 2,938 | 83.0 | 80.3 | 2.8 ** | 3.4 |
| Atlanta Human Capital Development | 2,992 | 81.2 | 80.3 | 1.0 | 1.2 |
| Grand Rapids Labor Force Attachment | 3,012 | 89.8 | 88.3 | 1.5 | 1.7 |
| Grand Rapids Human Capital Development | 2,997 | 89.5 | 88.3 | 1.2 | 1.4 |
| Riverside Labor Force Attachment | 6,726 | 74.5 | 66.1 | 8.4 *** | 12.7 |
| Lacked high school diploma or basic skills | 3,125 | 70.5 | 61.1 | 9.4 *** | 15.4 |
| Riverside Human Capital Development | 3,135 | 66.9 | 61.1 | 5.8 *** | 9.5 |
| Columbus Integrated | 4,672 | 87.9 | 86.0 | 1.9 ** | 2.2 |
| Columbus Traditional | 4,729 | 87.0 | 86.0 | 1.0 | 1.2 |
| Detroit | 4,459 | 85.5 | 83.0 | 2.5 ** | 3.0 |
| Oklahoma City | 8,677 | 78.8 | 79.2 | -0.5 | -0.6 |
| Portland | 4,028 | 85.8 | 81.7 | 4.1 ** | 5.0 |
| Average number of quarters employed | |||||
| Atlanta Labor Force Attachment | 2,938 | 8.5 | 7.8 | 0.8 *** | 9.7 |
| Atlanta Human Capital Development | 2,992 | 8.3 | 7.8 | 0.5 ** | 6.9 |
| Grand Rapids Labor Force Attachment | 3,012 | 9.8 | 9.1 | 0.7 *** | 8.0 |
| Grand Rapids Human Capital Development | 2,997 | 9.5 | 9.1 | 0.4 * | 3.9 |
| Riverside Labor Force Attachment | 6,726 | 6.8 | 5.6 | 1.1 *** | 20.3 |
| Lacked high school diploma or basic skills | 3,125 | 6.0 | 4.7 | 1.3 *** | 26.9 |
| Riverside Human Capital Development | 3,135 | 5.5 | 4.7 | 0.8 *** | 16.6 |
| Columbus Integrated | 4,672 | 10.2 | 9.8 | 0.4 ** | 4.4 |
| Columbus Traditional | 4,729 | 10.1 | 9.8 | 0.3 * | 3.1 |
| Detroit | 4,459 | 8.3 | 8.0 | 0.2 | 3.0 |
| Oklahoma City | 8,677 | 6.6 | 6.7 | -0.1 | -1.3 |
| Portland | 4,028 | 9.4 | 7.8 | 1.6 *** | 21.1 |
| Average total earnings ($) | |||||
| Atlanta Labor Force Attachment | 2,938 | 19,838 | 17,380 | 2,459 *** | 14.1 |
| Atlanta Human Capital Development | 2,992 | 19,397 | 17,380 | 2,017 ** | 11.6 |
| Grand Rapids Labor Force Attachment | 3,012 | 22,322 | 20,770 | 1,552 * | 7.5 |
| Grand Rapids Human Capital Development | 2,997 | 21,616 | 20,770 | 846 | 4.1 |
| Riverside Labor Force Attachment | 6,726 | 17,438 | 14,889 | 2,549 *** | 17.1 |
| Lacked high school diploma or basic skills | 3,125 | 13,193 | 10,912 | 2,281 *** | 20.9 |
| Riverside Human Capital Development | 3,135 | 12,273 | 10,912 | 1,361 * | 12.5 |
| Columbus Integrated | 4,672 | 27,621 | 25,566 | 2,055 *** | 8.0 |
| Columbus Traditional | 4,729 | 26,977 | 25,566 | 1,410 * | 5.5 |
| Detroit | 4,459 | 21,968 | 20,508 | 1,460 * | 7.1 |
| Oklahoma City | 8,677 | 12,868 | 12,752 | 115 | 0.9 |
| Portland | 4,028 | 26,041 | 20,891 | 5,150 *** | 24.7 |
| SOURCE: MDRC calculations from state and
county administrative records. NOTES: See Appendix A.1. |
|||||
In contrast to their generally small effects on whether people ever worked, the programs generally increased how long people worked and how much they earned (see the middle and lower panels of Table 4.1). In 9 of the 11 programs, the program group worked more quarters on average than the control group; moreover, a slightly different combination of nine programs increased average earnings above the control group level.
The Portland program had the largest effect by far on number of quarters employed and earnings. Whereas control group members worked 7.8 quarters on average during the 20-quarter follow-up period, program group members worked 9.4 quarters on average, for an impact of more than 1.6 quarters, or more than 20 percent.(2) The program also increased earnings by more than $5,000 over five years, from just under $21,000 on average for the control group to just over $26,000 on average for the program group. The magnitude of effects in Portland should not be understated: Among welfare-to-work programs studied using random assignment, only the Riverside GAIN effects have been as large as the effects of Portland.(3) Portland's success may be due to its strong focus on employment, its emphasis on finding good jobs, and its mixed use of services that assigned some people to job search and some to education and training. Since it is the only program studied in NEWWS that had an employment focus with a mix of activities, it is difficult to say how important a factor that was in its success. However, in previous evaluations the most effective welfare-to-work programs have generally used similar mixed approaches to determining people's initial activities.(4) Portland's unusual success may also have been bolstered by its strong economy, although other programs that were operated in areas of high employment growth and low unemployment led to smaller impacts on employment and earnings (see Table 1.2). It should also be noted that case managers in Portland were less likely than program staff in other sites to refer people with serious personal or family problems or extremely low motivation levels. These individuals did not attend a program orientation and were not included in the research sample.(5)
The LFA programs, with their requirement that most participants look for work initially, had the next largest impacts as a group. Increases in the average number of quarters of work ranged from 0.7 quarter (Grand Rapids) to 1.1 quarters (Riverside). In addition, increases in total earnings over five years above the control group average ranged from about $1,500 per LFA (Grand Rapids) to about $2,500 (Atlanta and Riverside).
The education-focused programs led to smaller impacts on cumulative measures of employment and earnings. Columbus Integrated and Atlanta HCD were the most effective at increasing total earnings above control group levels. These two programs led to five-year earnings impacts of about $2,000 per program group member, which exceeded the increase for only one of the four employment-focused programs: Grand Rapids LFA. Every other education-focused program led to earnings gains of less than $1,500 over five years (or $300 per year) — a historically small increase. Moreover, the program-control group difference in total earnings was not statistically significant in Grand Rapids and Oklahoma City.
As shown in Table 4.1, the Oklahoma City program was the least effective among the 11 programs in the NEWWS Evaluation, probably reflecting the de facto voluntary nature of the program. During much of the follow-up period, the Oklahoma City program was hampered by large caseloads and limited resources. The program placed a priority on working with enrollees who expressed the greatest interest in participating. In addition, its integrated case management program staff spent most of their time performing financial functions (such as helping enrollees to find child care) and devoted little time to monitoring participation in employment-related activities. Detroit, which faced similar problems, also ran a de facto voluntary program for much of the period. Detroit did not significantly raise employment, although it did increase earnings of program group members by more than $1,400 over five years relative to the control group.(6)
[Go to Contents]
As discussed earlier, employment-focused programs were expected to boost employment and earnings immediately, and effects of education-focused programs were expected to occur more slowly, after enrollees completed their education and training activities. Table 4.1 shows that the cumulative impacts of education-focused programs fell short of the impacts of most employment-focused programs. It is possible, however, that the skills obtained through education and training helped program members find such good jobs that the impacts of the education-focused programs were continuing to grow at the end of the five-year follow-up period relative to the impacts of the employment-focused programs. If this pattern did occur, education-focused programs might have larger impacts than employment-focused programs after year 5 and larger cumulative gains over a longer follow-up period. Figure 4.1 and Appendix Table C.1 show, for each of the 11 programs for the five years of follow-up, the proportion of program and control group members who worked for pay.(7) In addition, the figure and table can be used to examine separately the period when all control group members were embargoed from receiving control group services and the later period. As discussed in Chapter 1, control group members could not receive program services in any of the sites during the first three years after random assignment. In Riverside and Portland, control group members in the report sample were embargoed from receiving services for all five years. It is important to keep in mind, however, that most evidence indicates that ending the embargo is unlikely to have changed the effects of most programs on employment and earnings very much (as discussed at the end of this chapter).
Figure 4.1
Impacts on Emloyment in Year 1 to 5
SOURCE: MDRC calculations from state and county
administrative records.
NOTES: See Appendix A.1.
The two Atlanta programs exemplify the basic differences in the effects of employment-focused and education-focused programs.(8) In the LFA program, the attempt to move people into jobs quickly produced an immediate payoff. During the first year after random assignment, about 49 percent of the program group worked compared with 44 percent of the control group, a 5 percentage point difference that was statistically significant. In contrast, the HCD program did not significantly increase employment in the first year, as was expected from the fact that many program group members were enrolled in education and training. However, the effects of the HCD program increased after year 1, and both the LFA and the HCD programs had similar effects on employment in years 2 and 3.
If education-focused programs are to "make up" the ground lost to employment-focused programs in the first year, then they must eventually have larger effects than the employment-focused programs. Figure 4.1 shows that, in fact, the Atlanta LFA program continued to significantly increase employment in year 4, but the HCD program did not. In year 5, neither program significantly affected employment. This finding suggests that it is unlikely that the HCD program will eventually catch up to or surpass the LFA program with longer follow-up.
The other employment-focused programs led to even larger impacts on percentage employed in year 1. Riverside LFA, arguably the most strongly employment-focused program in the evaluation, produced the largest initial increase in employment. (See Appendix Table C.1.) In Riverside, about 51 percent of LFAs worked for pay some time during year 1 compared with 34 percent of the control group. Impacts of the Grand Rapids LFA and Portland programs averaged about 10 percentage points above the control group level in year 1.
As discussed above, the positive effects of employment-focused programs may persist or grow larger over time. This pattern did not occur for the LFA programs in Grand Rapids and Riverside, however. For both programs, impacts on employment declined after year 1, although for different reasons. In Grand Rapids, LFA employment levels increased between years 1 and 2 and again between years 2 and 3, but control group employment levels increased even faster. This result suggests that the Grand Rapids LFA program succeeded initially by encouraging people who would have worked eventually to take jobs faster. In contrast, Riverside LFA employment levels diminished sharply between years 1 and 2, suggesting that a number of LFAs were encouraged to find work by the program but left employment quickly and had trouble finding another job. These differences are also consistent with the fact that Riverside had a fairly weak economy during the early part of the follow-up period.
Most other education-focused programs had small effects initially. Only two of the seven education-focused programs — Grand Rapids HCD and Riverside HCD — led to significant impacts on employment in year 1. For all of the education-focused programs except Oklahoma City (where impacts were statistically insignificant in each year), impacts were somewhat larger in year 2 or 3 than in year 1. Disappointingly, the effects of most education-focused programs declined after year 3. For all education-focused programs except Riverside HCD, employment levels were not significantly higher for program group members than for control group members in year 5. (Section V of this chapter presents further evidence that the end of the control group embargo is not responsible for declining impacts in most programs in years 4 and 5.)
Results in Grand Rapids and Riverside confirm that, as in Atlanta, it is unlikely that education-focused programs would surpass employment-focused programs with longer follow-up. In Grand Rapids, neither the LFA or HCD program increased employment levels above the control group levels in years 4 and 5, while in Riverside, the LFAs in need of basic education had slightly larger employment impacts than the HCDs.
The Portland program led to the most successful year-by-year pattern of employment impacts: It immediately increased employment (like the LFA programs) but continued to increase employment after year 1 (like the HCD programs). In fact, the year 3 impact on percentage employed (of 12 percentage points) was more than twice as large in Portland as any other program. Like both types of programs, its effects declined after year 3; however, the Portland program again led to relatively large effects (of 8 percentage points) and was one of only three programs to still have a positive effect on employment in year 5. This pattern suggests that Portland combined the best features of employment-focused and education-focused programs, assigning job-ready individuals to job search but allowing others to get short-term education or training initially.
Nonetheless, the decline in employment impacts for all programs after year 3 underscores the limitations of both employment- and education-focused welfare-to-work strategies that rely primarily on pre-employment services and mandates. More recently, policymakers and program administrators have attempted to strengthen the effects of welfare-to-work programs by adding a variety of post-employment services and financial incentives aimed at helping people stay employed and advance to better jobs. It remains to be seen whether such strategies accomplish these goals.
[Go to Contents]
Figure 4.2 and Appendix Table C.2 show the average annual earnings of program and control group members during years 1 to 5. These earnings comparisons provide the most comprehensive measure of the effects of the programs because they reflect differences in the number of people who went to work, how long they worked, and their average wages.
Figure 4.2
Impacts on Employment in Years 1 to 5
SOURCE: MDRC calculations from state and county
administrative records.
NOTES: See Appendix A.1.
The trends in program effects on earnings are similar to the effects on employment. Employment-focused programs had larger initial effects than education-focused programs, but most programs of both types significantly raised earnings in years 2 and 3 (the exception was Oklahoma City). Earnings impacts for both types of programs declined after year 3 and, especially, after year 4. By the end of the follow-up period, only the Riverside LFA and Portland programs continued to significantly raise earnings above the control group average.
The pattern of earnings suggests once again that employment-focused programs will continue to produce larger cumulative impacts than education-focused programs. Figure 4.2 and Appendix Table C.2 indicate that the LFA and HCD programs had similar effects on earnings in year 5, and, if anything, earnings impacts were slightly higher for the LFAs.
Results for Atlanta LFA and Portland deserve special attention. Like the other employment-focused programs (Grand Rapids LFA and Riverside LFA), Atlanta LFA and Portland significantly raised earnings in year 1. Unlike the other two programs, however, effects on earnings were greater in years 2 and 3 than in year 1. In other words, the pattern of their effects on earnings over time looked more like the education-focused programs and probably reflects these programs' greater emphasis on short-term education and training than the employment-focused programs in Grand Rapids and Riverside.
The findings for Portland are especially noteworthy. Portland continued to produce unusually large earnings impacts (of $1,200 and $900) in years 4 and 5 — well above the effects of any other program. As discussed earlier, the Portland program allowed some people considered in need of education to enroll initially in short-term education and training and encouraged job search participants to seek out relatively well-paying jobs with an opportunity for advancement.
[Go to Contents]
As discussed above, in theory both employment- and education-focused programs can lead to impacts on measures of employment stability and earnings growth. Participants in employment-focused programs are expected to start working quickly and acquire valuable skills and experience on the job that lead to employment stability and advancement. In contrast, education-focused programs aim to increase job-related skills and education credentials before employment, thereby helping people to obtain stable and high-paying jobs. Table 4.2 shows the effects of the 11 programs on the proportion of sample members who worked in all four quarters of year 5, and the proportion who earned $10,000 or more in year 5. Appendix Table C.6 shows the effects of the programs on two related measures: whether sample members worked in 9 of the 12 quarters in years 3 to 5 and whether they experienced earnings growth during the follow-up period.(9)
Site and Program |
Sample Size | Program Group (%) | Control Group (%) | Difference (Impact) | Percentage Change (%) |
|---|---|---|---|---|---|
| Employed in all four quarters of year 5 | |||||
| Atlanta Labor Force Attachment | 2,938 | 36.9 | 36.6 | 0.3 | 0.8 |
| Atlanta Human Capital Development | 2,992 | 36.5 | 36.6 | -0.1 | -0.3 |
| Grand Rapids Labor Force Attachment | 3,012 | 39.4 | 38.7 | 0.7 | 1.7 |
| Grand Rapids Human Capital Development | 2,997 | 38.3 | 38.7 | -0.4 | -1.0 |
| Riverside Labor Force Attachment | 6,726 | 26.3 | 23.2 | 3.1 *** | 13.5 |
| Lacked high school diploma or basic skills | 3,125 | 23.1 | 18.8 | 4.3 *** | 22.9 |
| Riverside Human Capital Development | 3,135 | 21.6 | 18.8 | 2.7 * | 14.5 |
| Columbus Integrated | 4,672 | 40.7 | 39.4 | 1.3 | 3.3 |
| Columbus Traditional | 4,729 | 40.4 | 39.4 | 1.1 | 2.7 |
| Detroit | 4,459 | 37.4 | 37.1 | 0.3 | 0.7 |
| Oklahoma City | 8,677 | 19.4 | 18.7 | 0.7 | 3.9 |
| Portland | 4,028 | 38.2 | 33.6 | 4.6 ** | 13.6 |
| Earned $10,000 or more | |||||
| Atlanta Labor Force Attachment | 2,938 | 25.7 | 25.0 | 0.6 | 2.6 |
| Atlanta Human Capital Development | 2,992 | 24.8 | 25.0 | -0.2 | -0.8 |
| Grand Rapids Labor Force Attachment | 3,012 | 26.0 | 25.4 | 0.6 | 2.3 |
| Grand Rapids Human Capital Development | 2,997 | 25.3 | 25.4 | -0.0 | -0.1 |
| Riverside Labor Force Attachment | 6,726 | 18.6 | 16.6 | 2.0 ** | 12.2 |
| Lacked high school diploma or basic skills | 3,125 | 14.3 | 12.0 | 2.3 ** | 19.6 |
| Riverside Human Capital Development | 3,135 | 12.7 | 12.0 | 0.7 | 5.9 |
| Columbus Integrated | 4,672 | 34.1 | 31.6 | 2.5 * | 7.9 |
| Columbus Traditional | 4,729 | 32.2 | 31.6 | 0.6 | 2.1 |
| Detroit | 4,459 | 28.4 | 27.1 | 1.4 | 5.0 |
| Oklahoma City | 8,677 | 13.9 | 13.2 | 0.7 | 5.3 |
| Portland | 4,028 | 30.4 | 26.6 | 3.8 * | 14.4 |
| SOURCE: MDRC calculations from state and
county administrative records. NOTES: See Appendix A.1. |
|||||
In general, there is little indication that either the employment-focused or the education-focused programs improved stable employment or high earnings in year 5. Only three programs — Riverside LFA, Riverside HCD, and Portland — resulted in significantly more people working in all four quarters of year 5. Likewise, only three programs — Riverside LFA, Columbus Integrated, and Portland — resulted in significantly more people earning at least $10,000 in year 5. Moreover, effects for all programs were relatively small — that is, less than 5 percentage points — on these measures.
A slightly different pattern emerges when stable employment and earnings growth are measured over more years of follow-up. (See Appendix Table C.6.) All four employment-focused programs, but only two of seven education-focused programs led to an impact on the measure of stable employment during years 3 to 5. However, on the measure of earnings over time, only the Riverside LFA and Portland programs, along with the Riverside HCD program, led to increases over the control group. The Portland program led to the largest impacts on both outcomes, which helps explain why it continued to produce earnings impacts in the last years of follow-up.
The effects of employment- and education-focused programs may vary according to education credential. Many proponents of skill-building activities have argued that education and training are particularly important for people who have not graduated from high school because nongraduates typically earn much less than graduates. Proponents of education-focused programs have asserted that job search programs will likely be ineffective for people with low skills and without education credentials, because these problems prevent many people from finding a job, even when assisted by a welfare-to-work program. Moreover, many nongraduates who do find employment will likely work at unstable jobs with low salaries and few if any benefits. On the other hand, people who had not graduated from high school may not attend long enough to benefit from the additional education provided in the education-focused programs; or, alternatively, local employers may have little demand for the skills they acquired.
There is also debate over whether employment- or education-focused programs provide greater benefit to enrollees who had already attained a high school diploma or GED certificate before entering a welfare-to-work program. According to one argument, many high school graduates have previous work experience and only require assistance in finding a job, or perhaps a referral to a better job than they could have found on their own. In these instances, employment-focused programs (particularly the job-search-first programs) should lead to larger effects than education-focused programs, which require people to forgo work. However, high school graduates in education-focused programs often take college courses or enroll in vocational training. If these types of education and training provide valuable skills, then education-focused programs may lead to more substantial effects for high school graduates than employment-focused programs.
The relative success of Portland's employment-focused program that used a mix of job search and education depends largely on its ability to determine who would benefit from education and training and who would benefit from job search. If identification of the two groups was quite successful, then the program might have had the largest effects over a long period for both high school graduates and nongraduates. It would have been able to increase earnings quickly for those capable of finding good jobs quickly and increase earnings later for those who would not have found good jobs quickly but who could have done so after short-term skills training.
Figure 4.3 shows how much the programs benefited high school graduates and nongraduates by showing the effects of the 11 programs on earnings over five years for these subgroups. (The next section of this chapter provides a more detailed comparison of the LFA and HCD programs for graduates and nongraduates.)
Figure 4.3
Impacts on Total Earnings in Years 1 to 5 for Sample Members With and Without
a High School Diploma or GED at Random Assignment
SOURCE: MDRC calculations from state and county
administrative records.
NOTES: See Appendix A.1.
Only 4 of the 10 programs significantly increased earnings for high school graduates.(10) While 3 of these 4 programs were employment-focused — suggesting that job search is more effective for this group — there were no large differences in either Atlanta or Grand Rapids in the effect of the LFA and HCD programs for graduates. Moreover, the largest impact on total earnings for high school graduates was in Portland, which allowed some high school graduates to obtain education and training and which resulted in a substantial increase in the use of vocational training and post-secondary education by high school graduates.
Only 5 of the 11 programs significantly increased earnings for nongraduates. Again, results were more positive for employment-focused programs than for education-focused programs. Three of the 4 employment-focused programs significantly raised earnings for nongraduates compared with only 2 of the 7 education-focused programs. Moreover, in each site the LFA program had a larger effect on earnings than the HCD program. However, the Portland employment-focused program, which used a mix of initial activities, had the largest effect for nongraduates.
There is also no clear indication that the welfare-to-work programs as a whole were more effective for one group or the other.(11) Five of the programs had larger effects for high school graduates than for nongraduates, but five had larger effects for nongraduates. These differences appear to be related more to site than to program approach. For example, both Atlanta programs had larger effects for high school graduates than for nongraduates, but both Grand Rapids programs had larger effects for high school nongraduates than for graduates.
[Go to Contents]
In comparing effects of the 11 programs, the section above provided informal comparisons of job search and education as alternative approaches to encouraging employment and increasing earnings. Fortunately, NEWWS was set up to allow a more formal comparison of the two approaches. In Atlanta, Grand Rapids, and Riverside (for those in need of basic education), people were assigned at random to either the LFA program, which required most participants to look for work intially, or the HCD program, which required most to enroll in education or training initially. Because people were assigned at random to the two programs, the difference in outcomes between individuals in the two programs provides a reliable indication of which approach was more effective. Table 4.3 makes this comparison for number of quarters employed and earnings over the five-year follow-up period.
| Site and Program | Sample Size | LFA | HCD | Difference (Impact) | p-Value |
|---|---|---|---|---|---|
| Average number of quarters employed | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 8.5 | 8.3 | 0.2 | 0.34 |
| Grand Rapids | 3,099 | 9.8 | 9.5 | 0.4 | 0.07 |
| Riverside | 4,980 | 6.8 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 7.0 | 6.4 | 0.7** | 0.04 |
| Grand Rapids | 1,268 | 8.7 | 7.8 | 0.9*** | 0.00 |
| Riverside | 3,182 | 6.0 | 5.5 | 0.5** | 0.02 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 9.5 | 9.6 | -0.1 | 0.73 |
| Grand Rapids | 1,827 | 10.6 | 10.6 | 0.0 | 0.96 |
| Riverside | 1,798 | 7.8 | n/a | n/a | n/a |
| Total earnings in years 1 to 5 ($) | |||||
| Full impact sample | |||||
| Atlanta | 2,936 | 19,838 | 19,397 | 442 | 0.59 |
| Grand Rapids | 3,099 | 22,323 | 21,616 | 706 | 0.39 |
| Riverside | 4,980 | 17,438 | n/a | n/a | n/a |
| No high school diploma or GED | |||||
| Atlanta | 1,190 | 13,439 | 12,344 | 1,095 | 0.26 |
| Grand Rapids | 1,268 | 16,243 | 14,299 | 1,945** | 0.04 |
| Riverside | 3,182 | 13,193 | 12,273 | 920 | 0.18 |
| High school diploma or GED | |||||
| Atlanta | 1,742 | 24,163 | 24,111 | 52 | 0.97 |
| Grand Rapids | 1,827 | 26,496 | 26,729 | -233 | 0.85 |
| Riverside | 1,798 | 23,019 | n/a | n/a | n/a |
| SOURCE: MDRC calculations from state and
county administrative records. NOTES: See Appendix A.1. |
|||||
For the full sample, Atlanta and Grand Rapids LFAs worked and earned about as much as HCDs.(12) For example, in Atlanta LFAs worked 8.5 quarters on average during the 20-quarter follow-up period and HCDs worked 8.3 quarters. Likewise, people in both programs earned between $19,000 and $20,000 on average.
Any significant differences between the two programs were in favor of the LFA approach. In particular, in Grand Rapids the LFA group worked 9.8 quarters on average and the HCD group worked 9.5 quarters. Other differences between outcomes for the full samples in the two sites were not statistically significant; however, employment and earnings were always higher in the LFA programs.
For people who had not graduated from high school, the results are clearer: Nongraduates in the LFA programs were more likely to work than their counterparts in the HCD programs. In all three sites the LFA group also had higher average earnings than the HCD group. Although the difference in earnings between the LFA and HCD groups was statistically significant for high school nongraduates only in Grand Rapids, the average difference across the three sites was statistically significant at the 1 percent significance level.
For high school graduates, the two program approaches had virtually the same effect. For example, sample members worked about 9.5 quarters on average in both Atlanta programs and about 10.6 quarters on average in both Grand Rapids programs. There were similarly small differences in earnings in both sites.
Although the job-search-first approach was sometimes more effective than the education-first approach at encouraging work and increasing earnings, it is important to keep in mind that the most effective program — by far — among the NEWWS programs was Portland, a program that used both job search and education and training with a focus on employment. Portland's success implies that a "one size fits all" approach is probably not the best approach and that using both job search and education may be better if sites can find strategies that can determine which individuals within a group might benefit most from school.
[Go to Contents]
This section considers whether the NEWWS Evaluation underestimated the longer-term effects on employment and earnings in sites where the embargo on control group services ended after year 3. It concludes that the effect of removing the embargo was likely to be small except in Portland, but that most programs' effects on employment and earnings would likely have diminished in years 4 and 5 even if the control group embargo had remained in place. Because the embargo did appear to have an effect in Portland, results in this report were limited in Portland to 499 control group members who were prohibited from receiving program services for five years.
The likely effect of ending the control group embargo was examined in four sites where some control group members gained access to program services in year 4 and some did not gain access until later. In those four sites — Atlanta, Columbus, Grand Rapids, and Portland — control group members were divided into a "short-term embargo" group whose members became eligible for program services in year 4 and a "longer-term embargo" group, whose members remained ineligible for program services for at least four years.
These comparisons were not made in Riverside, Oklahoma City, and Detroit. In Riverside, the test was unnecessary because all control group members remained ineligible for services for five years after random assignment. In Oklahoma City, the fact that the program did not significantly affect employment or earnings during years 1 to 3 — when all control group members remained ineligible for services — suggests that ending the embargo did not alter the program's effects. In Detroit, the test could not be performed because all control group members became eligible for services after three years of follow-up. It is possible that the ending the embargo caused impacts in the program to decline after year 3, but there is no way to know whether that was true.
In comparing the short-term and longer-term embargo groups, the rate of decline of impacts for the two groups in year 4 and the size of impacts for the longer-term embargo group in year 4 were examined. If impacts declined faster for the short-term embargo group than for the longer-term embargo group in year 4, it is evidence that ending the embargo did reduce impacts. In contrast, if a program did not significantly affect outcomes for the longer-term embargo group in year 4 — even though the group was still embargoed from receiving services — it is evidence that ending the embargo did not have a large effect because the effectiveness of the program ended before the embargo.
Portland provided the most reliable comparison. Several years into the Portland evaluation, 499 control group members were chosen at random from the control group and remained ineligible for services for five years after random assignment. This group of 499 constituted the longer-term embargo group while the short-term embargo group consisted of all other control group members. Because the groups were chosen at random, they did not differ systematically from each other in background characteristics, and both groups resembled the program group.
A comparison of the short-term and longer-term embargo groups in Portland suggested that ending the control group embargo did contribute to a decline in impacts there. The effect of the program on employment was about the same in year 3 whether it was calculated with the short-term or longer-term embargo group (12 versus 11 percentage points). In year 4, however, the effect was only 2.6 percentage points when measured with the short-term embargo group but 8.5 percentage points when measured with the longer-term embargo group. In year 5, the effect had largely disappeared when measured with the short-term embargo group but was 3.8 percentage points when measured with the longer-term embargo group. Differences in the trends in earnings impacts were similar to differences in the trends in employment impacts during years 3 to 5. Because of these differences, MDRC decided to include only members of the longer-term embargo control group in estimating impacts of the Portland program.
Comparisons were conducted less rigorously in the other three sites. Control group members in the short-term and longer-term embargo groups were randomly assigned at different times and therefore might have differed systematically from each other at the time of random assignment. Consequently, one cannot rule out the possibility that variation in impacts between the two groups resulted from differences in background characteristics, labor markets, or other external factors that may have affected employment.
Despite these measurement problems, it appears that ending the control group embargo was not responsible for the small effect on employment and earnings in Grand Rapids at the end of the follow-up period. For both programs, the effect on employment and/or earnings declined during year 3, when all control group members were ineligible for services. This downward trend in impacts continued in years 4 and 5 and was seen in both the short-term and longer-term cohorts for both the LFA and the HCD programs.
In Atlanta, the trend in impacts after year 3 suggests that the end of the control group embargo had only a limited effect. For the longer-term embargo group, the effect of the LFA program on employment and earnings was no longer statistically significant by the second half of year 4 — even though control group members were not yet eligible for program services. In other words, it appears that the effectiveness of the program had largely disappeared before control group members were allowed to receive program services. A similar pattern was seen for the HCD program in Atlanta.(13)
The evidence is similar for the Columbus Integrated program. During year 3, the estimated effect of the program on employment was 4.3 percentage points and statistically significant for the longer-term embargo group. In year 4, however, the effect on employment had decreased to 2.4 percentage points and was not statistically significant. The program's effect on earnings also began to decline during year 3 and continued downward slightly during year 4.
Ending the embargo may have made more of a difference for the Columbus Traditional program. Impacts on earnings for the first three years after random assignment were similar whether calculated for the short-term or the longer-term embargo groups. In years 4 and 5, however, the estimated impact on earnings was about $950 higher when calculated with the longer-term embargo group. On the other hand, Columbus Traditional had only a negligible effect on employment for either sample, either before or after year 3.
Whether or not allowing the control group to receive program services diminished the effect of the programs at the end of the follow-up period, the essential story is clear from the first three years of follow-up: The Portland program had the largest and most consistent effects by far, employment-focused job-search-first programs had the next largest effects, and education-focused programs had the smallest effects. Moreover, issues surrounding the control group embargo do not affect the comparison of LFA and HCD programs in Atlanta, Grand Rapids, and Riverside. That comparison indicates that the two program approaches were equally effective for graduates, that the LFA programs were more effective for nongraduates, and that there was no evidence that the cumulative effect of the HCD programs would surpass the cumulative effect of the LFA programs with longer follow-up.
[Go to Contents]
1. As discussed elsewhere, results for Portland use only the 499 control group members who were prohibited from receiving program services for five years.
2. Program impacts on average number of quarters employed can be greatly affected by the total number of quarters of follow-up during which employment is measured. For this reason, some researchers prefer to convert this measure to a standard format by dividing average number of quarters employed by the total number of follow-up quarters. The result (usually expressed as a percentage) is called the "average quarterly employment rate." For example, in Portland, program group members worked for pay for an average of 9.4 of the 20 quarters of follow-up, whereas control group members were employed for 7.8 quarters on average. These averages are equivalent to quarterly employment rates of 47 percent (9.4 / 20 x 100) and 39 percent (7.8 / 20 x 100), respectively. Similarly, in Portland the impact on average number of quarters employed (of 1.6 quarters) is equivalent to an impact of 8 percentage points in the average quarterly employment rate (1.6 / 20 x 100; or 47 percent - 39 percent).
3. Riccio, Friedlander, and Freedman, 1994; Friedlander and Burtless, 1995; Bloom and Michalopoulos, 2001. The Riverside GAIN program increased employment by 9.9 percentage points and earnings by about $1,000 per year (Freedman et al., 1996; Hotz, Imbens, and Klerman, 2000). Two other programs came close to Portland over a three-year period: the Canadian Self-Sufficiency Project (Michalopoulos and Berlin, 2001), which increased employment by 7.1 percentage points and earnings by about $725 per year, and the Connecticut Jobs First program, which increased employment by 7.8 percentage points but increased earnings by only about $425 per year (Bloom et al., 2000b).
4. Bloom and Michalopoulos, 2001; Michalopoulos and Schwartz, 2001.
5. This finding is based on field research. On the other hand, results in Chapter 7 demonstrate that Portland's approach led to large earnings impacts for sample members who faced at least one serious barrier to employment.
6. Unlike Oklahoma City, Detroit operated a traditional case management system, in which separate Income Maintenance staff determined recipients' eligibility for public assistance and calculated their monthly welfare and Food Stamp grants. However, the Detroit welfare-to-work program was not fully staffed during the first years of the evaluation, and its case managers reported spending most of their limited time working with clients who were participating on their own initiative and who had requested child care and other supportive services.
7. Results over time are also shown in Appendix Tables C.2 (impacts on total earnings in each year of follow-up), C.3 (impacts on cumulative measures of employment and earnings in years 1 to 3), and C.4 (impacts on employment and earnings in year 5). Appendix Table C.5 shows employment and full-time employment at the end of year 5 according to responses to the Five-Year Client Survey. Results in those tables confirm that few programs continued to have significant effects in year 5.
8. The two Atlanta programs are discussed first because they appear side by side in Figure 4.1 and because they used different self-sufficiency approaches. Section IV of this chapter presents a statistical comparison of employment and earnings in the LFA and HCD programs in Atlanta, Grand Rapids, and Riverside.
9. In addition, Freedman, 2000, describes results on employment stability and earnings growth through four years in much greater detail, and Michalopoulos, 2001, compares the effects of the NEWWS programs with effects from programs in Minnesota and Canada that supplemented earnings to encourage work.
10. The few high school graduates who were assigned to the Riverside HCD program were considered in need of basic education and included in the group of nongraduates shown.
11. Michalopoulos and Schwartz, 2001, report that impacts over a three-year follow-up period were larger for high school graduates than for nongraduates. Among the programs they studied, three of the four with the largest differences in earnings impacts between high school graduates and nongraduates were in the Alameda, Riverside, and San Diego GAIN programs, all of which used a mix of initial activities and none of which were in the NEWWS Evaluation. Even among the 20 programs that they studied, however, impacts were larger for high school graduates in 11 programs but larger for nongraduates in 9 programs.
12. Recall that people not in need of basic education were not randomly assigned to the Riverside LFA program, so that a comparison of the two approaches could not be made for the full sample in Riverside.
13. For unknown reasons, the effects of the HCD program were larger in years 4 and 5 when calculated with the short-term embargo group than the longer-term embargo group.
Home Pages:
National Evaluation of Welfare-to-Work Strategies
(NEWWS)
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)