[ Main Page of Report | Contents of Report ]
For the past 30 years, federal and state policymakers have been legislating various types of programs to increase employment among welfare recipients. How people can best move from welfare to work, however, has been the subject of long-standing debate. This report, summarizing the long-term effects of 11 mandatory welfare-to-work programs on welfare recipients and their children, represents a major advance in resolving this debate. The findings are the final ones from the National Evaluation of Welfare-to-Work Strategies (NEWWS), a multi-year study of alternative approaches to helping welfare recipients find jobs, advance in employment, and leave public assistance.
“What works best, and for whom?” is the central question animating this report and the NEWWS Evaluation as a whole. In particular, the evaluation compares the effects of two alternative pre-employment strategies, for different groups of welfare recipients: programs that emphasize short-term job search assistance and encourage people to find employment quickly (referred to as “Labor Force Attachment” [LFA] or, more broadly, “employment-focused” programs); and programs that emphasize longer-term skill-building activities, primarily basic education (referred to as “Human Capital Development” [HCD] or, more broadly, “education-focused” programs). The effects of each approach are estimated from a wealth of data pertaining to over 40,000 single parents (mostly mothers) and their children, and a five-year follow-up period (falling somewhere between 1991 and 1999, depending on the site), using an innovative and rigorous research design based on the random assignment of individuals to one or more program groups (with services) or to a control group (without services).
[ Go to Contents ]
The research designs that were implemented in the NEWWS Evaluation permit many comparisons. The key ones examined the programs’ economic effects on adults and the “spillover” effects on noneconomic outcomes and child well-being, as summarized below.
These results should not be taken as an indictment of the benefits of education and training in general in welfare-to-work programs. Nonexperimental work done as part of the NEWWS Evaluation has suggested that obtaining a GED and, especially, obtaining a GED and then receiving some type of vocational training, can result in employment and earnings gains for those who achieve these milestones.(1) However, in the context of mandatory welfare-to-work programs, few people make it this far, for many reasons, including: people leave welfare and therefore do not stay in welfare-to-work programs, and thus education or training classes, for very long; adults supporting families cannot afford an up-front deferment of employment and earnings that may or may not have a longer-run payoff; and only a small minority of welfare recipients report that, if given a choice, they prefer to go to school to study basic reading and math over going to school to learn a job skill or going to a program to get help looking for a job.(2) It should be noted as well that none of these programs made assignments to or emphasized college.
Average income levels among control group members over the five-year follow-up period were low. Despite the successes of these programs, no program, not even Portland’s, met the long-range goal of making enrollees substantially better off financially. Most program group members continued to have low incomes from various combinations of earnings, the EITC, welfare, and Food Stamps. In fact, among individuals who lacked a high school diploma or GED as of study entry, some programs had the five-year result of making them financially worse off. These findings suggest that the challenge of the future is to identify other types of programs or initiatives that can provide welfare recipients with better and more stable jobs, increase their income, and improve the well-being of their children.
[ Go to Contents ]
In 1988, the federal Family Support Act (FSA) established a system of mutual obligation within the Aid to Families with Dependent Children (AFDC) benefit entitlement structure: Government was to provide education, employment, and support services to AFDC recipients, who in turn were required to participate in the Job Opportunities and Basic Skills Training (JOBS) program created under that act. Through its mandates and incentives, the FSA encouraged state and local program administrators to serve welfare populations with whom they had had little if any contact in the past and to experiment with new types of services, messages, and mandates — often in advance of solid evidence of how well these innovations would work. For the first time, most single parents with children ages 3 to 5 (or ages 1 to 5 at the state’s option) were required to enroll in welfare-to-work programs. In addition, the FSA mandated that programs reserve at least 55 percent of funds for services to welfare recipients who were deemed to be at greatest risk of long-term welfare dependency. Furthermore, the FSA required enrollees to participate in employment-preparation activities for as long as they remained on welfare and eligible for services. Case managers were expected to monitor participation and to use a variety of informal and formal responses (including reductions of welfare grants) when enrollees did not attend. In addition, people were supposed to be assigned to additional activities if they completed participation in employment-preparation activities without finding a job.
The expansion of welfare-to-work programs and the requirement to work with more disadvantaged populations intensified a long-standing debate among program administrators and policymakers concerning how best to help welfare recipients — especially those facing serious barriers to employment — move from welfare to work. Rigorous research in the 1980s demonstrated that job search programs sped up the entry of welfare recipients into the labor market. Often, however, the jobs were neither long-lasting nor high-paying, and they did not increase family income. Furthermore, the programs generally did not benefit the most disadvantaged welfare recipients. During the years before passage of the FSA, administrators of welfare-to-work programs in several states and localities (most notably, California) began implementing programs that emphasized up-front investments in basic education and skill development as an alternative to job search. FSA regulations accelerated this trend by requiring states and localities to offer a variety of employment-preparation activities, including job search, basic education (classes in adult basic education, GED preparation, regular high school, and English as a Second Language), and vocational training and post-secondary education.
Proponents of education-focused programs argued that this approach offered the best chance of helping people — especially those who lacked a high school diploma or faced other barriers to employment — to get better and more stable jobs, increase their family’s income, and reduce returns to the welfare rolls. Some further hypothesized that education-focused programs would benefit children more than job search programs, because parents who attended education or training classes would become more involved in their children’s schoolwork and would serve as role models for succeeding in school.
There was little evidence at the time, however, that large-scale mandatory education programs for welfare recipients would achieve these goals. It was expected that programs that emphasized education and training would engage participants for months and perhaps years longer than programs that emphasized short-term job search assistance. As a result, education and training programs would be more costly to run than job search programs. To be considered cost-effective from a budgetary standpoint, these programs would have to produce savings in welfare and other benefits well in excess of what the less expensive job search programs would attain. Moreover, participants in education and training programs were expected to experience an initial “opportunity” cost (compared with participants in employment-focused programs) in the form of forgone earnings during the period when they were in the classroom rather than the workplace. It was hoped that participants in education-focused programs would make up for such forgone earnings as well as their later start in accumulating work experience and on-the-job skills by attaining better initial jobs and by advancing more quickly once employed than would otherwise be the case. However, it was not clear whether the people who were expected to attend education and training would actually remain in school long enough to attain credentials or enhance their job skills. This issue was especially crucial for people entering welfare programs with low levels of educational attainment or without the minimum literacy and math skills needed for employment.
The FSA also afforded program administrators an opportunity to address the shortcomings of low-cost job search services. Many job search programs before the FSA required participants to look for work but provided little instruction on how to find employment. In contrast, from the late 1980s onward, programs increasingly assigned enrollees to organized group job clubs, whereby participants received instruction on finding job leads, filling out résumés, and conducting job interviews. Many programs followed classroom instruction with one or more weeks of supervised job search, during which they provided job leads and the use of phone rooms to contact employers. Over time, program operators added new features to job club curricula, including career exploration, life skills and time management instruction, and self-esteem-building exercises. Some programs actively marketed their job placement services to area employers and engaged in job development activities to increase the pool of available jobs. Administrators also invested in new ways to communicate a pro-work message, although (as will be discussed below) the types of messages differed. Finally, the FSA’s ongoing participation requirement encouraged administrators to design follow-up activities (often short-term education and training) for job search participants who did not find employment.
In general, implementing these enhancements made job search activities more costly to operate, but not as costly as education-focused programs. It was hoped that these changes would be cost-effective by helping more disadvantaged welfare recipients find work and by helping people find employment sooner than they would have otherwise.
The most recent federal welfare reform effort, the 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), replaced AFDC with a flexible, state-directed block grant program, Temporary Assistance for Needy Families (TANF); set lifetime limits on most families’ receipt of federally funded TANF assistance; and created financial incentives for states to run mandatory, work-focused, welfare-to-work programs. If anything, TANF’s time limit, its focus on work, and its requirement that agencies work with the entire welfare caseload brought a new urgency to the question of which welfare-to-work approach was most effective.
In addition, both the FSA and PRWORA strengthened the requirement that single parents prepare for employment in exchange for welfare benefits, and they extended this mandate to parents with young children. These developments increased the importance of learning how welfare-to-work programs affected families. To what extent would programs be able to involve mothers who had young children? Would parents with toddlers and preschool-age children be able to find stable, affordable, and high-quality child care while they participated in pre-employment activities and while they were working? Would child care prove a financial burden and a barrier to employment after parents left government assistance? More generally, how would children be affected if there were program-induced changes in their mothers’ educational attainment, hours of employment, or self-esteem; in their family income; or in the amount, type, or quality of child care they experienced?
It was within this context that the NEWWS Evaluation was conceived and funded, in 1989, by the U.S. Department of Health and Human Services (HHS), with support from the U.S. Department of Education. The Manpower Demonstration Research Corporation (MDRC) conducted the evaluation. Child Trends, as a subcontractor, conducted the Child Outcomes Study, the part of the evaluation that examined effects on young children.
[ Go to Contents ]
The programs in the NEWWS Evaluation implemented many of the features described above. As shown in Table 1, the 11 programs in the NEWWS Evaluation were operated in seven sites across the country. Employment-focused programs were operated in Atlanta, Georgia; Grand Rapids, Michigan; Riverside, California; and Portland, Oregon. Atlanta, Grand Rapids, and Riverside also operated education-focused programs, as did Columbus, Ohio (two programs); Detroit, Michigan; and Oklahoma City, Oklahoma. The studied programs were initially administered under the FSA, which created the national JOBS program for recipients of cash assistance under AFDC. The programs continued (with some modification) under the FSA’s successor, PRWORA, which replaced AFDC with the TANF block grant program. Under both welfare reform acts, the programs’ primary goal was to move welfare recipients off government assistance and into paid work.
| Employment-focused approach | Education-focused approach | ||
|---|---|---|---|
| Job search first | Varied first activity | Education or training first | |
| High enforcement | High enforcement | High enforcement | Low enforcement |
| Atlanta LFA Grand Rapids LFA Riverside LFA |
Portland | Atlanta HCD Grand Rapids HCD Riverside HCD Columbus Integrated Columbus Traditional |
Detroit Oklahoma City |
| NOTES: “LFA” denotes the site’s Labor Force Attachment program. “HCD” denotes the site’s Human Capital Development program. |
|||
The four employment-focused programs — the LFA programs in Atlanta, Grand Rapids, and Riverside as well as Portland’s program — assigned most enrollees to job club as their first activity, and they encouraged enrollees to find work as quickly as possible. Further, both Portland’s and Riverside’s program employed full-time job developers to help place program enrollees in unsubsidized jobs.
In contrast to the three LFA programs, however, Portland’s program offered GED preparation classes to people who case managers thought had a good chance of attaining a GED certificate relatively quickly. Furthermore, Portland case managers, more often than those in the LFA programs, encouraged enrollees to hold out for jobs that paid well above the minimum wage (about 25 percent higher) and that offered the best chance for long-lasting and stable employment. Case managers in the LFA programs, especially Riverside’s, stressed the value of starting off with any job, even a low-paying one, and then advancing toward more stable and better-paying jobs in the future.
The HCD programs in Atlanta, Grand Rapids, and Riverside; the Columbus Integrated and Traditional case management programs;(3) and the programs in Detroit and Oklahoma City
can each be characterized as “education-focused.” A large percentage of program enrollees in these programs were initially assigned to some type of skill-building activity. The types of activities to which enrollees were first assigned depended, in part, on the level of educational attainment that individuals had achieved prior to entering the program. Those who had not completed high school or received a GED certificate but who were assessed by case managers as having high-school-level skills were assigned to GED preparation classes. Those with lower reading or math levels were assigned to adult basic skills classes. In addition, non-English speakers could be assigned to English as a Second Language (ESL) programs. Finally, those who had completed high school or held a GED certificate could be assigned to vocational training or employment-oriented skills courses at local community colleges. All in all, however, assignments to GED preparation or basic education courses predominated in these education-focused programs, and assignments to vocational training programs were less common, primarily as a result of welfare recipients’ low levels of educational achievement; enrollment in college played an even smaller role.
Other key program features varied across the 11 studied programs as well. All four employment-focused programs and five of the seven education-focused programs can be considered “high enforcement” programs: They worked with a cross-section of the welfare applicants and recipients who were required to participate; monitored participation closely; and, especially in the two programs in Columbus and the two in Grand Rapids, frequently invoked sanctions (reductions in welfare grants) for nonparticipation. The remaining two education-focused programs, in Detroit and Oklahoma City, did not have these characteristics (because of either lack of funds or program philosophy) and can be considered “low enforcement” programs.
The programs also differed in their child care policies and practices (within each site, however, child care assistance policies were identical for program and control group members). During the early to mid 1990s, the Atlanta, Oklahoma City, Portland, and Detroit programs provided the strongest staff support for arranging for child care, and the programs in Atlanta and Oklahoma City emphasized the use of licensed care; in contrast, case managers for both Riverside programs encouraged enrollees to find low- or zero-cost, informal child care.
The programs also differed in their case management strategies. Three programs — Columbus Integrated, Portland, and Oklahoma City — implemented an integrated case management staffing arrangement. The other programs used a traditional case management structure.
[ Go to Contents ]
The NEWWS Evaluation used a rigorous design — called a social experiment — to estimate the effects of employment- and education-focused programs. Welfare recipients were randomly assigned to one of two or three research groups, depending on the site.
As part of a largely unprecedented effort to determine which welfare-to-work program approach works better, three sites — Atlanta, Grand Rapids, and Riverside — simultaneously operated, expressly for the evaluation, two different programs: an LFA program and an HCD program. These programs were multidimensional but varied in terms of the key features that program operators and researchers thought most clearly differentiated employment- and education-focused programs. Each type of program communicated a different message to welfare recipients about the best route to employment, and each type differed from the other in the way program services were sequenced and emphasized. The programs were, however, mandatory to the same degree: Nonparticipants risked a reduction in their monthly welfare grant. In these three sites, welfare recipients were randomly assigned to an LFA program group, an HCD program group, or a control group. (Control group members were not subject to the participation mandate and received no services through either type of program but, on their own, could seek out similar services within the community.) This random assignment research design produces the most reliable comparison between employment- and education-focused programs. It ensured that, within each site, there were no systematic differences between the background characteristics of people in the LFA, HCD, and control groups within each site when they entered the study. Thus, any subsequent differences in outcomes between groups — comparing either the LFA or the HCD group to the control group, or the LFA and the HCD groups to each other — can be attributed with confidence to the effects of a particular type of program. These differences, referred to as the programs’ impacts, are the primary focus of this report, and all differences reported are statistically significant unless otherwise noted.
In the Columbus site, a three-group random assignment design was used as well. Here, the two program groups represented two case management models: “integrated” and “traditional.” The remaining three sites in the evaluation — Detroit, Oklahoma City, and Portland — used random assignment to test the effectiveness of established programs, as opposed to programs designed to meet research protocols; individuals were randomly placed either in a group that entered the program or in a no-program control group. Note that control group members in all sites were eligible for child care assistance, similar to that offered to program group members, if they were participating in nonprogram activities in which they had enrolled on their own.
In each site, individuals were randomly assigned to research groups over approximately a two-year period. Random assignment for the evaluation began in June 1991, in Riverside, and ended in December 1994, in Portland. Thus, the five-year results presented in this report cover the calendar period of June 1991 (the first sample member’s entry into the study) through December 1999 (the last month of the five-year follow-up for the last sample member randomly assigned, in Portland).
The research designs that were set up in the seven NEWWS Evaluation sites permit many comparisons. The key ones examine the programs’ economic effects on adults and the spillover effects on noneconomic outcomes and children. The central comparisons may be expressed as follows:
It should be noted that while control group members were not exposed to the services and mandates of the sites’ programs for the first three years of follow-up, their status differed by site in the fourth and fifth years. For a few programs, net impacts for years 4 and 5 are understated somewhat because a small portion of the control group (a subset of those still on welfare) received program services toward the end of the five-year follow-up period. (In these sites, the “start of the clock” for welfare time limits necessitated allowing control group members access to welfare-to-work program services.) Importantly, however, this situation does not affect the results of direct comparisons of the LFA and HCD program approaches (that is, the differential impacts). The three-group random assignment designs in the three sites in which the LFA and HCD programs were operated side by side permit a direct comparison of these two approaches, that is, one that does not need to take into account the services received by, or the behavior of, control group members.
This report includes data from administrative records (unemployment insurance [UI], state and county welfare payments, and Food Stamp data) and from surveys administered to mothers and children over the five years after individuals entered the study.
[ Go to Contents ]
Over the five-year follow-up period, a majority of control group members in each site (up to 75 percent) participated in some type of employment-related activity: job search, basic education, vocational training, or post-secondary education. Almost all of this activity was the result of control group members’ own initiative; despite the potential in several sites for controls to be subject to mandatory welfare-to-work programs at the end of the follow-up period, there is little evidence that much control group participation in such programs did, in fact, occur. Some of this self-initiated activity took place while control group members were receiving welfare; much of it took place after they left the welfare rolls. In most sites, the most common activities in which control group members enrolled themselves over the five-year period were vocational training and post-secondary education programs.
All programs increased overall participation levels above those achieved by control group members. The employment-focused programs increased participation in job search by approximately 30 percentage points relative to control group members. The education-focused programs increased job search participation as well, but to a much lesser degree. Most education-focused programs produced large increases in education and training relative to control group members. Among those who entered the study without a high school diploma or GED (“nongraduates”), increases were particularly large in basic education. Among those who entered the study with these credentials (“graduates”), few programs increased participation in vocational training. (While education-focused programs most commonly assigned graduates to vocational training activities, rarely did program group levels of participation in such activities exceed those of the control groups.) Increases in education and training participation in the employment-focused programs were much less common and, where they did occur, were smaller than those in the education-focused programs.
The Portland program, with its employment focus but mix of initial program activity assignments, produced five-year increases in both job search and education participation. While the program produced large increases in job search participation for both nongraduates and graduates, it also resulted in a 10 percentage point increase (though not a statistically significant one) in basic education participation among nongraduates and a large, 21 percentage point increase in post-secondary education participation among graduates.
Overall, program group members’ length of stay was longer in education and training activities than in job search activities. For example, within the first two years of follow-up, the typical participant in an adult education program received the equivalent of about two-thirds of a year of instruction in a high school.(5) Length of stay in any type of program activity was often curtailed because program group members started working for pay and/or left welfare — the goals, after all, of welfare-to-work programs.
Education-focused programs generally increased the proportion of nongraduates who obtained a GED or high school diploma over the five-year follow-up period, whereas only the Portland program among the employment-focused programs had such an effect (though this increase is not statistically significant). In addition, Portland had a notable increase in the proportion of nongraduates who obtained a high school diploma or GED as well as a second education or training credential. Overall levels of high school diploma or GED receipt, however, were low: By the end of the five-year follow-up period, less than one-quarter of initial nongraduates in the education-focused program groups had obtained a high school diploma or GED. Among graduates, only the two programs in Atlanta produced five-year increases in the receipt of some type of education or training credential — generally, a trade license or certificate.
The cost analysis considered all costs associated with providing employment services and associated support services to sample members (including case management costs). Costs paid by welfare departments and non-welfare agencies, and in-program as well as post-program or post-welfare costs, were included.
Five-year net per-person costs (the gross cost per program group member minus the gross cost per control group member) averaged $3,037 for the employment-focused programs and $3,972 for the education-focused programs. (These are 1999 dollars.) The most reliable comparison of the costs of these two types of program approaches, however, is one comparing LFA and HCD net program costs within each site. In Atlanta, Grand Rapids, and Riverside, HCD programs were 40 percent to 90 percent more expensive than their counterpart LFA programs.
NEWWS program costs were high compared with other programs that have been studied by MDRC. This is largely because of the greater use in all the NEWWS programs of high-cost education activities, such as post-secondary education and vocational training, relative to past welfare-to-work programs; as well as the enhancements made to job search activities by most NEWWS programs, relative to the above-described simple job search activities implemented in the 1980s. The average cost of the NEWWS programs was comparable to that of the two highest-cost California Greater Avenues for Independence (GAIN) programs, which operated in Alameda and Los Angeles Counties in the late 1980s and early 1990s.
[ Go to Contents ]
At the high end, 88 percent of control group members (in Grand Rapids) were employed at some point during the five-year follow-up period; at the low end, 79 percent (in Oklahoma City) and 66 percent (in Riverside) worked during this same time frame. In addition to illustrating the strong interest that welfare recipients have in going to work — regardless of any welfare-to-work program intervention — these figures suggest that there was little room left for programs to increase the proportion of program group members who “ever” worked. Employment levels for control group members grew steadily over the five-year follow-up period, although many control group members worked for less than one year and then experienced a spell of joblessness.
Not surprisingly, given the high levels of employment for control group members, programs generally had little effect on the percentage of sample members who “ever” worked. However, in 9 of 11 programs, the program group worked during more calendar quarters on average than the control group; and in 9 of 11 programs, the program group averaged higher total earnings than their control group counterparts.
Portland produced the largest, most consistent employment and earnings effects by far: Over five years, program group members worked 1.6 quarters more than control group members — a 21 percent increase in employment duration — and their average five-year earnings were about $5,000 greater than those of control group members. (See Figure 1, which depicts the impacts, or program-control differences, on earnings for all programs.) Portland’s program also produced the largest impacts on measures of stable employment and earnings growth among the 11 programs. Portland’s success may have resulted from its unique combination of a focus on employment, the use of both job search and education, and an emphasis on finding good jobs. In addition, the program made extensive use of job development, and staff were experienced in operating welfare-to-work programs. Portland’s relatively strong economy also may have contributed to the success of the program; however, other programs in localities where the demand for labor was similarly high did not do as well.
Figure 1.
Program Impacts on Total Earnings in Years 1 to 5
The employment-focused LFA programs in Atlanta, Grand Rapids, and Riverside also affected employment and earnings, but less so than the Portland program. Five-year earnings gains ranged from about $1,500 in Grand Rapids to about $2,500 in Atlanta and Riverside. The programs also increased employment duration by an amount ranging from 0.7 quarter in Grand Rapids to 1.1 quarters in Riverside.
The effects of the seven education-focused programs, as a group, were smaller than the effects of the employment-focused programs. Neither of the two programs with low enforcement of the participation mandate (Detroit and Oklahoma City) significantly affected employment. Among the other five education-focused programs, employment duration gains over five years ranged from 0.3 to 0.8 quarter, and earnings gains ranged from about $800 to about $2,000.
Employment-focused programs produced effects almost immediately, whereas education-focused programs generally did not have effects until more than a year after random assignment. In the middle of the follow-up period, most of the programs increased employment and earnings, but effects diminished during the final two years and were statistically insignificant for most programs by the end of year 5. (An example of this trend for the Grand Rapids LFA program can be seen by comparing the black bars and the white bars in Figure 2, and for the Grand Rapids HCD program by comparing the shaded bars and the white bars.) These results were especially disappointing for education-focused programs. As discussed above, education and training services were intended to help program group members eventually move into more stable and higher-paying jobs (compared with control group members and compared with those subject to employment-focused programs) in order to make up for forgone earnings early in the follow-up period. However, most programs — education- or employment-focused — had little or no effect on measures of stable employment and earnings growth.
A comparison of earnings impacts for nongraduates demonstrates more clearly the disappointing results for education-focused programs. For this subgroup, only two of the seven education-focused programs significantly raised five-year average earnings above control group levels, whereas three of the four employment-focused programs did so. The employment-focused program with the largest earnings impacts for nongraduates was Portland, however, which used a mix of initial activities that resulted in substantial use of education by nongraduates. Furthermore, the education-focused Columbus Integrated program led to earnings impacts among nongraduates that, along with those of the Grand Rapids LFA program, were the next-largest among all programs. This suggests that education activities, in some instances, may contribute to earnings impacts.
Employment-focused programs, compared with education-focused ones, also more consistently increased the earnings of high school graduates above control group levels. Once again, however, the largest earnings impact for high school graduates was found for the Portland program, which substantially increased the use of post-secondary education among graduates.
Cumulatively, over the five-year follow-up period, few LFA-HCD differences in employment or earnings (that is, differential impacts) are found when both graduates and nongraduates are included in the calculations (see the top panel of Figure 1).(6) Year by year, however, there were some differences. The first set of bars in Figure 2, showing only the Grand Rapids site, illustrates these. LFA average per-person earnings were higher than those of the HCD sample members in at least year 1 in Grand Rapids as well as in Atlanta and Riverside. The gap between the two types of programs narrowed, however, in year 2 of follow-up in Grand Rapids and in Atlanta and in year 3 of follow-up in Riverside. In addition, there was one LFA-HCD difference for the full sample on the measure of average quarters employed: In Grand Rapids, the LFA group worked more quarters than did the HCD group. Notably, the HCD programs, relative to the LFA ones, did not produce more earnings growth over the follow-up period or increase the likelihood of employment in “good” jobs. Finally, the yearly trends suggest that the story would not change if longer follow-up were available.
As shown in the lower panel of Figure 1, two LFA programs and one HCD program produced five-year earnings increases, relative to control groups, for nongraduates; as shown in the middle panel of this same figure, a different set of two LFA programs and one HCD program produced five-year net earnings impacts for graduates.
Contrary to expectations, earnings impacts generally were larger for nongraduates in LFA programs than in HCD programs. (Proponents of the HCD approach anticipated that education-focused programs might be particularly effective for those without high school diplomas or GEDs, inasmuch as their lack of skills or credentials might inhibit employers from offering them jobs. But this did not turn out to be the case.) Among this subgroup, LFA-HCD differences in five-year earnings were $920 in Riverside, $1,095 in Atlanta, and $1,945 in Grand Rapids. While only the Grand Rapids difference is by itself statistically significant, the average difference across the three sites is statistically significant at the 5 percent significance level. Furthermore, in no year of the follow-up were nongraduates’ average earnings higher in HCD programs than LFA programs; rather, earnings were generally higher in the LFA programs, with statistically significant differences found in the early years of follow-up in every site. (See the last set of bars in Figure 2 for an illustration of this pattern in Grand Rapids.) Among graduates, earnings impacts were very similar in the two types of programs.
The average control group member remained on assistance for about two to three years during the five-year follow-up period. Levels of welfare receipt fell steadily over time for control group members, reflecting “normal” welfare exits. Welfare receipt reached particularly low levels in Columbus and Portland, where less than 20 percent of control group members were receiving a welfare payment at the end of year 5.
All programs had an effect on the number of months that people received welfare. On average, employment- and education-focused program group members received AFDC or TANF assistance for two to six fewer months than their control group counterparts.
All programs also reduced total welfare payments below control group levels, and most produced savings of 10 percent or more (a historically large effect). (See Figure 3.) For many programs, welfare savings were larger and more persistent than earnings gains: Few programs continued to affect employment and earnings in year 5, but most programs continued to generate welfare savings at the end of year 5. This finding implies that some program group members who exited welfare for employment early in the follow-up did not return to assistance after leaving employment, even though they may have been eligible to do so. It is possible that the national welfare climate in the aftermath of the federal welfare reform legislation of 1996 contributed to this pattern, and since more program than control group members left welfare in the early years of follow-up (before enactment of the legislation), the climate may have had more of an effect on program than control group members.
Figure 3.
Program Impacts on Total Welfare Payments in Years 1 to 5
Welfare savings were generally larger for programs that had greater effects on earnings, but they varied for other reasons as well. Total payments were reduced more in higher-grant sites such as Riverside, Portland, and Grand Rapids and were reduced less in lower-grant sites such as Atlanta. In addition, welfare benefits were reduced more in sites that strictly enforced program participation mandates, such as Columbus and Grand Rapids, but benefits were reduced relatively little in sites that did not enforce mandates, such as Detroit.
The programs had similar welfare impacts for high school graduates and nongraduates. Most programs produced welfare savings for both groups, and there is little evidence that the effects were larger for one group than the other: In five programs, welfare savings were larger for graduates, but in five other programs, welfare savings were larger for nongraduates.
Over five years, program group members in all programs spent less time on Food Stamps and on average received smaller Food Stamp payments than control group members. Food Stamp impacts were generally smaller than welfare payment impacts, however, because some program group members appropriately continued to receive Food Stamps after they left welfare.
Cumulatively, over the five-year follow-up period, a statistically significant LFA-HCD difference (differential impact) in welfare expenditures was found in only one site (Grand Rapids), where the LFA program produced savings of $785 more than the HCD program (see Figure 3) and where the average number of months in which welfare or Food Stamps were received also was lower in the LFA program. In Grand Rapids, this pattern held for graduates as well as nongraduates. In Atlanta, welfare expenditures and months on welfare were lower for the LFA programs than for the HCD programs, but these differences are statistically significant only among the Atlanta nongraduates. In Riverside, the LFA and HCD programs led to similar reductions in welfare receipt and expenditures for nongraduates. The fact that in all three sites LFA sample members left welfare more quickly than HCD sample members represents a clear advantage in an environment where federally funded welfare months are time-limited.
Both employment- and education-focused programs helped sample members become more self-sufficient relative to the control group by increasing employment and earnings and reducing public assistance. As a result of these changes, program group members received a higher percentage of their income from earnings, compared with the control group. This impact averaged about 4 percentage points across all programs but was somewhat larger for employment-focused programs, especially the Portland one, than for education-focused programs.
Program group members, however, received about the same amount of income as their counterparts in the control group. Two programs — one employment-focused and the other education-focused — led to especially large decreases in welfare and Food Stamp payments and to decreases in combined income. In general, program effects on combined income were less positive (larger decreases or smaller increases) for nongraduates than for graduates. In addition, among nongraduates, LFA programs did better than HCD programs. Among nongraduates in the LFA-HCD sites, a simple average of the impacts across the three sites indicates that the three LFA programs as a group resulted in almost $1,000 more in combined income over five years than the HCD programs, a statistically significant difference.
Including estimates of sample members’ Earned Income Tax Credits and payroll taxes did not change the above results. (These findings do not account for program effects on other possible sources of income, such as child support payments or unemployment insurance benefits or income from spouses, partners, or other household members. The available data, however, suggest that the inclusion of these other income sources would have changed the impact estimates very little, if at all.)
As discussed above, FSA programs were required to target welfare-to-work resources on individuals at greatest risk of long-term welfare dependency. The NEWWS Evaluation was designed to determine which types of employment-preparation services provide the greatest benefit to at-risk populations.
In several respects, both employment- and education-focused programs were successful. Most programs raised earnings above control group levels for sample members with serious barriers to employment, such as no recent work history and a lengthy history of prior welfare receipt. Similarly, in most programs, the most disadvantaged group members — welfare recipients who did not have a high school diploma or GED, who had a history of welfare receipt, and who had not worked recently — earned more on average than their counterparts in the control group. Differences for several of these programs are small, however, and are not statistically significant.
In Atlanta and Grand Rapids, the LFA programs, compared with the HCD ones, led to considerably higher earnings impacts for the most disadvantaged sample members — especially in Atlanta, where the HCD program had no effect on this subgroup. However, earnings increases for this subgroup (relative to the control group) were similar for the two programs in Riverside.
Although the more disadvantaged groups had higher earnings as a result of most NEWWS programs, they still earned very little. In addition, most programs reduced welfare and Food Stamps by a larger margin than they increased earnings. As a result, programs did not raise the combined income of the most disadvantaged recipients above control group levels.
[ Go to Contents ]
The benefit-cost analysis extends the findings on program impacts and costs presented above. It considers program effects on additional outcomes, such as fringe benefits from employment, income and sales taxes, Medicaid expenditures, and the administrative costs of transfer programs. These additional outcomes were estimated or imputed from administrative records and published data. The analysis also considers program effects from the standpoint of sample members (referred to as the welfare sample perspective) and of government (referred to as the government budget perspective).
The benefit-cost analysis from the welfare sample perspective considers (in a more comprehensive way than presented above) whether programs increased program group members’ income — from any source — relative to the control group. (While society generally favors earnings over welfare payments, the benefit-cost analysis from the welfare sample perspective does not favor either income source.) The welfare sample derives a net gain from a welfare-to-work program if the program increases earnings (plus fringe benefits) by an amount that exceeds the sum of the welfare, Food Stamp, and Medicaid benefits lost and the increase in taxes paid (net of Earned Income Tax Credits).
From the government budget perspective, in contrast, increases in tax revenues and reductions in benefits are a net gain. The government budget perspective also counts as benefits any savings in administrative costs from reductions in receipt of transfer payments. These gains are compared with the program-control group difference in cost, that is, with the net cost of providing employment-related services.
Programs may lead to net gains from both the welfare sample and the government budget perspectives; they may also lead to net losses from both perspectives. Other times, programs may benefit either the welfare sample or the government budget.
From the benefit-cost perspective of the welfare sample, most of the programs produced net financial losses. Moreover, gains were close to zero in the few programs that resulted in them. (See Table 2.) In contrast, from the benefit-cost perspective of government budgets, the majority of programs broke even (that is, saved the government only slightly more or slightly less than they cost), a few produced clear savings, and one produced a clear cost. Government budget savings in several programs were larger for nongraduates than for graduates.
From the perspective of welfare sample members, neither LFA nor HCD programs consistently yielded gains or losses. (See Table 2.) Rather, in each site, both programs produced either a net loss for all enrollees (in Grand Rapids and Riverside) or a gain close to zero (in Atlanta). Nongraduates, however, experienced losses over the five years in both types of programs, but the losses were consistently greater in the HCD than the LFA programs. For graduates, results were mixed.
From the standpoint of government budgets, neither LFA nor HCD programs consistently produced budget savings or losses. In every site, however, for all sample members as well as for nongraduates and graduates, five-year government budget savings were greater, or losses were smaller, for the LFA programs. (See Table 2.) The differences between the gains or losses for the LFA and HCD programs within each site were quite large, particularly in Atlanta and Grand Rapids.
| Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Portland | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sample and Perspective | LFA | HCD | LFA | HCD | LFA | HCD | Integrated | Traditional | |||
| From the Welfare Sample Perspective | |||||||||||
| Full sample | |||||||||||
| Net gain (net loss) | 162 | 471 | (2,254) | (1,957) | (1,145) | N/A | (1,491) | (1,076) | 262 | (615) | |
| People with high school diploma or GED | |||||||||||
| Net gain (net loss) | 358 | 1,112 | (2,859) | (1,359) | (646) | N/A | (1,090) | (693) | 303 | (1,751) | |
| People without high school diploma or GED | |||||||||||
| Net gain (net loss) | (215) | (432) | (1,479) | (2,812) | (1,465) | (2,951) | (2,286) | (1,876) | 207 | 1,084 | |
| From the Government Budget Perspective | |||||||||||
| Full sample | |||||||||||
| Net gain (net loss) | (770) | (3,259) | 2,908 | (308) | 1,545 | N/A | 244 | (646) | (329) | 5,235 | |
| People with high school diploma or GED | |||||||||||
| Net gain (net loss) | (72) | (1,869) | 1,839 | (1,373) | 2,057 | N/A | (1,412) | (1,375) | (1,036) | 6,305 | |
| People without high school diploma or GED | |||||||||||
| Net gain (net loss) | (1,999) | (5,700) | 5,564 | 1,096 | 1,278 | 606 | 2,200 | 811 | 437 | 2,787 | |
To facilitate comparisons with the benefit-cost results of previously studied welfare-to-work programs, an additional measure of the cost-effectiveness of the NEWWS programs from the government budget perspective was used. This measure, called the return to budget per net dollar invested, is calculated by dividing the gains (taxes and savings in transfer payments and associated administrative costs) by the total net costs of services. Using this metric, government budgets come out ahead if programs produce more than a dollar’s worth of additional revenues and savings for each dollar spent on employment-related services for program group members (compared with control group members).
Using this measure, the Portland program and the Grand Rapids LFA program both produced over $2.00 in increased revenue and savings for every additional dollar spent on program group members. The Riverside LFA program also produced a considerable return, $1.47 per dollar invested. The Grand Rapids HCD and the Columbus Integrated programs essentially caused the government to break even ($0.92 to $1.06). The Atlanta, Detroit, and Columbus Traditional programs were not as successful, returning considerably less than one dollar for each dollar invested, ranging from $0.41 in the Atlanta HCD program to just over $0.80 in the other programs. The average return across all the programs was $1.29.
On average, these results are more positive than those found in benefit-cost analyses of prior, recent programs. The California GAIN programs, for example, had returns to government budgets that ranged from a low of $0.17 per dollar invested (Tulare County) to a high of $2.84 (Riverside County). The average across all counties in the GAIN evaluation was $0.76. The return on investments in the two NEWWS programs that were most successful from a government budget perspective — Portland ($2.83) and Grand Rapids LFA ($2.46) — compare favorably with previously studied programs that had high government budget returns: the Riverside GAIN program and the mid-1980s San Diego Saturation Work Initiative Model (SWIM) program, which returned $2.84 and $2.34, respectively.
[ Go to Contents ]
No aspects of the welfare-to-work programs studied as part of the NEWWS Evaluation were designed to directly change family circumstances — for example, to specifically affect marriage or fertility rates or to improve child well-being in specific ways. Theoretically, however, the programs could indirectly affect family circumstances or children through their impacts on such adult outcomes as educational attainment, employment, earnings, welfare status, and income. Data on five-year family and child outcomes are available for seven programs in four sites (Atlanta, Grand Rapids, Riverside, and Portland). The evaluation examined the net impacts on family circumstances and child well-being, that is, increases or decreases relative to the situations among control group members.
From 70 percent to 80 percent of sample members had health care coverage at the end of follow-up year 5, and most of them had coverage through public sources, such as Medicaid or other public programs, rather than through private sources, such as employers’ health plans. All of those who did not have coverage had left welfare. (Many received Transitional Medicaid but were not able to find alternative coverage when their transitional benefits expired.) None of the programs had an impact on health care coverage for adults or children, although most programs, because they increased employment, led to a shift from public to private coverage. The two programs in Riverside increased the use of Transitional Medicaid during the follow-up period, but neither program led to a gain in coverage as of the end of five years.
The Grand Rapids LFA and HCD programs led to the largest effects on living arrangements. This site’s two programs increased the likelihood that people would move (and move more than once), primarily to attain better housing. In addition, the Grand Rapids HCD program increased home ownership.
Information about abuse by intimate partners or others and about barriers to work put up by intimate partners or others is available for a subset of sample members — those who entered the study with a preschool-age child — in three sites (Atlanta, Grand Rapids, and Riverside), covering six programs. Similar percentages of program and control group members in these sites reported experiencing harassment, abuse (physical or otherwise), or other types of deterrence from working at some time in their lives. More positive results, however, were found at the end of the follow-up period. During follow-up year 5, reported rates of any abuse by intimate partners among control group members ranged from 19 percent to 22 percent. All six programs decreased reports of physical abuse (such as hitting) by intimate partners, by 3 to 6 percentage points, although the decreases are statistically significant only in the Atlanta LFA, Grand Rapids HCD, and Riverside LFA programs. There is some evidence that these reductions in reports of abuse were fostered by increases in employment (which may have increased individuals’ self-esteem or self-efficacy, ameliorated family stress, or simply reduced the amount of time that individuals spent with partners) and by program caseworker attention to support services. Notably, this evaluation did not try to identify women who might be in imminent danger related to abuse. For some women, work may lead to greater safety. For others, however, especially those in imminent danger of abuse, employment at a time of risk may not have such positive results, and such a risk may make it difficult for them to work or comply with welfare-to-work program requirements.
As discussed above, both employment- and education-focused programs increased employment levels and earnings during the early years of follow-up, but impacts grew smaller thereafter. The longer-term effects on use of child care for employment reflect these trends. During the first two years after random assignment, all four employment-focused programs plus three education-focused programs produced moderate to large increases in child care use while employed. However, at the five-year mark, only Portland’s program increased (by 7 percentage points) child care use during sample members’ most recent job. (It should also be remembered that most children who were studied intensively early in the follow-up were attending school by the end of year 5.)
A higher percentage of program group members received transitional child care benefits after random assignment, although the increase is statistically significant only for the two programs in Atlanta and the Riverside LFA program. This increase occurred because a higher percentage of program group members left welfare for employment (and became eligible for transitional benefits) and because program group members who became eligible for benefits more often received them than their counterparts in the control group.
The analysis also considered whether programs led to greater child care use for any purpose at the end of follow-up or otherwise altered how children spent their time. This issue was examined for a subset of sample members — those who entered the study with a preschool-age child — in six programs. Of these six, only one (the Riverside LFA program) had an effect on recent child care use — a decrease of 5 percentage points in the use of formal care. In addition, both programs in Riverside reduced the time that the 8- to 10-year-old children spent with their mothers, and these programs increased the time that these children spent with another adult.
The three-site Child Outcomes Study that was nested within the larger NEWWS Evaluation included nearly 50 measures of children’s academic functioning, health and safety, and social skills and behavior for children who were preschool age at study entry in six programs, providing about 300 program-control group comparisons. About 15 percent of these tests yielded statistically significant differences — a relatively small percentage, but more than would be expected by chance.
Most commonly, each of the six programs affected young children’s social skills and behavior, but in different ways. The two Atlanta programs led to favorable impacts: both a higher percentage of positive behaviors, such as being sensitive to others or making friends, and a lower percentage of negative behaviors, such as fighting or arguing with others. In contrast, the two Grand Rapids programs and the Riverside LFA program led to unfavorable impacts: decreased positive behaviors and/or increased problem behaviors.
Impacts on young children’s academic achievement or academic performance were few. However, results for measures relating to behavioral adjustment to school (these include disciplinary problems and the degree of engagement in school) were consistent with the impacts on social skills and behavior. Both the Atlanta LFA and HCD programs led to favorable impacts (although none were statistically significant in the LFA program), whereas the Grand Rapids LFA and HCD programs led to unfavorable effects.
Some negative results were also found for young children in Atlanta. Children in Atlanta’s program groups either missed or were late for school more often than their counterparts in the control group. Children of Riverside HCD program group members also averaged more days absent from school than children of control group members. There were few impacts on measures of children’s health and safety; however, the effects were all unfavorable and occurred mostly in these same three programs.
The evaluation examined program effects on a limited number of measures of academic functioning and of health and safety for children in different age groups in four sites (encompassing seven programs). Few effects were found, and these did not vary consistently by program approach or site. Some programs (in Grand Rapids and Riverside) led to some unfavorable impacts for young adolescents, but other programs (in Atlanta and Portland) led to few favorable or unfavorable effects for this same age group. In Riverside, for example, about 4 percent of adolescent children of control group members had ever repeated a grade in school, and both programs in this site increased this rate by 3 to 4 percentage points. In Grand Rapids, about 8 percent of control group adolescents had ever repeated a grade, and this rate increased by 4 percentage points as a result of both programs in this site. In Riverside, there were unfavorable effects on a few other measures as well, particularly in the site’s HCD program. For example, the Riverside HCD program increased the likelihood that an adolescent would drop out of school, and it increased the percentage of adolescents who had a physical, emotional, or mental condition that impeded their mother’s ability to go to work or school. In addition, both Riverside LFA and HCD programs — among families in which the parents lacked a high school diploma or GED — produced increases in the proportion of adolescents who had a baby as a teen. On the positive side, however, the Atlanta LFA program decreased the proportion of adolescents who had ever been suspended or expelled from school; the Portland program had no effects on adolescents; and on many other measures, either none or only one of the programs had effects on adolescents. In general, it is possible that adolescents’ academic functioning may have been especially vulnerable to the increased employment, decreased income, and/or changes in household composition that occurred among their mothers in several of the programs.
[ Go to Contents ]
1. Johannes M. Bos, Susan Scrivener, Jason Snipes, and Gayle Hamilton, Improving Basic Skills: The Effects of Adult Education in Welfare-to-Work Programs (Washington, DC: U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education; and U.S. Department of Health and Human Services, Administration for Families and Office of the Assistant Secretary for Planning and Evaluation, 2001).
2. See Gayle Hamilton and Thomas Brock, The JOBS Evaluation: Early Lessons from Seven Sites (Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education, 1994).
3. For a full discussion of the results of the direct test of integrated versus traditional case management in the Columbus site, see Susan Scrivener and Johanna Walter, Evaluating Two Approaches to Case Management: Implementation, Participation Patterns, Costs, and Three-Year Impacts of the Columbus Welfare-to-Work Program (Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families and Office of the Assistant Secretary for Planning and Evaluation; and U.S. Department of Education, Office of the Under Secretary and Office of Vocational and Adult Education, 2001).
4. The Riverside HCD program enrolled only individuals who did not have a high school diploma or GED, had low scores on baseline reading and math tests, or did not speak English. Comparisons throughout this summary between the Riverside HCD and LFA programs or between the HCD and control group members include only such individuals, and they are referred to as “nongraduates.”
5. See Bos, Scrivener, Snipes, and Hamilton, 2001.
6. Note that, in all the figures, “daggers” indicate statistically significant differences between the LFA and HCD impacts.
Home Pages:
National Evaluation of Welfare-to-Work Strategies
(NEWWS)
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)
Updated: 02/03/04