National Evaluation of Welfare-to-Work Strategies

How Effective Are Different Welfare-to-Work Approaches?
Five-Year Adult and Child Impacts for Eleven Programs:

Appendix I:
Impacts on Employment, Welfare, Income, and Related Outcomes for the Client Survey, Child Outcomes Study Survey, and Teacher Survey Samples

[ Main Page of Report | Contents of Report ]

This appendix presents a brief summary of program impacts on employment, earnings, welfare payments, combined income, and other outcomes calculated with administrative records and survey responses for the three survey samples. It then compares program impacts for each sample with impacts calculated for the larger samples from which its members were selected, including (for administrative data) the full impact sample. These findings provide context for interpreting program impacts on other outcomes measured for each survey sample, including child care use and cost and child outcomes.

Table I.1 (for administrative records data) and Table I.2 (for survey data) show the control means and impacts for the three survey samples and (for Table I.1) for the full impact sample. There are several reasons why impacts for a particular outcome may vary by sample. Samples may be chosen from different months of random assignment or selected from different subgroups. In addition, response rates may vary for different samples (or may differ by research group within a sample), and the samples may differ in the degree to which certain subgroups may be over- or underrepresented among respondents. This appendix does not attempt to explain which source of variation accounts for differences in impacts across samples.(1)

Appendix Table I.1
Impacts on Selected Outcomes, Based on Administrative Records Data, by Sample

Site

 Full Sample Client Survey Sample COS Sample Teacher Survey Sample
Control Group Impact Control Group Impact Control Group Impact Control Group Impact

Employed in years 1 to 5 (%)
Atlanta Labor Force Attachment 80.3 2.8** 84.6 -0.3 88.6 -1.0 90.0 1.9
Atlanta Human Capital Development 80.3 1.0 84.6 -1.2 88.6 0.0 90.0 -1.1
Grand Rapids Labor Force Attachment 88.3 1.5 88.4 2.5 91.7 4.7** 91.0 6.2**
Grand Rapids Human Capital Development 88.3 1.2 88.4 1.9 91.7 2.6 91.0 2.0
Riverside Labor Force Attachment 66.1 8.4*** 65.0 9.3*** 63.5 14.8*** 68.5 10.9**
Lacked high school diploma or basic skills 61.1 9.4*** 60.0 11.5*** 59.1 17.4*** 63.6 11.4*
Riverside Human Capital Development 61.1 5.8*** 60.0 9.2*** 59.1 13.8*** 63.6 13.7**
Portland 81.7 4.1** 84.0 0.5

Employed in all four quarters of year 5 (%)
Atlanta Labor Force Attachment 36.6 0.3 39.5 2.2 45.1 1.0 43.6 9.4*
Atlanta Human Capital Development 36.6 -0.1 39.5 0.8 45.1 -0.5 43.6 1.7
Grand Rapids Labor Force Attachment 38.7 0.7 41.7 3.5 43.3 7.9* 43.5 15.7***
Grand Rapids Human Capital Development 38.7 -0.4 41.7 0.8 43.3 2.4 43.5 6.1
Riverside Labor Force Attachment 23.2 3.1*** 22.7 3.1 20.4 7.6* 25.1 7.3
Lacked high school diploma or basic skills 18.8 4.3*** 17.5 5.3 14.2 11.4** 17.2 11.4**
Riverside Human Capital Development 18.8 2.7* 17.5 7.1** 14.2 13.0*** 17.2 13.0**
Portland 33.6 4.6** 34.7 2.8

Earnings in years 1 to 5 ($)
Atlanta Labor Force Attachment 17,380 2,459*** 18,801 2,531* 20,516 2,547 20,709 4,443*
Atlanta Human Capital Development 17,380 2,017** 18,801 2,484* 20,516 2,444 20,709 2,769
Grand Rapids Labor Force Attachment 20,770 1,552* 23,532 1,446 23,340 3,285 22,450 4,898*
Grand Rapids Human Capital Development 20,770 846 23,532 505 23,340 635 22,450 1,466
Riverside Labor Force Attachment 14,889 2,549*** 13,652 2,505* 10,805 6,537*** 13,810 6,727***
Lacked high school diploma or basic skills 10,912 2,281*** 9,412 2,697* 7,279 6,932*** 8,268 7,080***
Riverside Human Capital Development 10,912 1,361* 9,412 3,503*** 7,279 6,265*** 8,268 6,343***
Portland 20,891 5,150*** 21,893 1,119

Total months on welfare in years 1 to 5 (%)
Atlanta Labor Force Attachment 37.2 -2.9*** 38.5 -1.8* 39.1 -1.6 39.3 -2.8
Atlanta Human Capital Development 37.2 -1.9*** 38.5 -1.0 39.1 -0.7 39.3 -0.7
Grand Rapids Labor Force Attachment 31.1 -4.2*** 30.0 -5.1*** 31.6 -4.5*** 31.0 -4.5**
Grand Rapids Human Capital Development 31.1 -2.9*** 30.0 -2.6*** 31.6 -1.5 31.0 1.1
Riverside Labor Force Attachment 31.0 -3.2*** 34.9 -6.7*** 41.7 -8.0*** 41.1 -7.4***
Lacked high school diploma or basic skills 33.3 -3.2*** 37.9 -7.5*** 44.0 -9.2*** 45.2 -9.4***
Riverside Human Capital Development 33.3 -3.3*** 37.9 -3.7*** 44.0 -3.1* 45.2 -3.5
Portland 25.3 -5.6*** 26.1 -5.9***

Total welfare payments in years 1 to 5 ($)
Atlanta Labor Force Attachment 9,946 -881*** 10,383 -595* 10,892 -663 10,916 -982*
Atlanta Human Capital Development 9,946 -710*** 10,383 -340 10,892 -218 10,916 -120
Grand Rapids Labor Force Attachment 12,966 -2,552*** 12,602 -2,892*** 14,027 -2,987*** 13,958 -3,282***
Grand Rapids Human Capital Development 12,966 -1,767*** 12,602 -1,623*** 14,027 -1,249 13,958 -471***
Riverside Labor Force Attachment 18,294 -2,710*** 20,315 -4,613*** 25,450 -5,707*** 25,060 -5,548
Lacked high school diploma or basic skills 20,126 -2,955*** 22,634 -5,513*** 27,355 -6,817*** 28,342 -7,521***
Riverside Human Capital Development 20,126 -2,949*** 22,634 -3,080*** 27,355 -3,134*** 28,342 -3,353**
Portland 11,686 -2,746*** 12,240 -3,225***

Total combined income in years 1 to 5 ($)
Atlanta Labor Force Attachment 39,987 1,152 42,756 1,753 46,069 1,909 46,397 3,414*
Atlanta Human Capital Development 39,987 1,133 42,756 1,854 46,069 2,019 46,397 2,307
Grand Rapids Labor Force Attachment 42,172 -1,433* 44,750 -1,911 46,927 52 46,167 1,074
Grand Rapids Human Capital Development 42,172 -1,247 44,750 -1,398 46,927 -302 46,167 1,556
Riverside Labor Force Attachment 39,804 -875 41,446 -3,334** 45,432 -646 48,211 -310
Lacked high school diploma or basic skills 38,311 -1,280 40,334 -4,212** 44,323 -1,442 47,093 -2,271
Riverside Human Capital Development 38,311 -2,387*** 40,334 -525 44,323 2,300 47,093 1,521
Portland 41,807 1,870 44,691 -3,875*
SOURCE:  MDRC calculations from state and county administrative records.
NOTES: See Appendix A.

Appendix Table I.2
Impacts on Selected Outcomes, Based on Survey Data, by Sample
Site Client Survey Sample COS Sample Teacher Survey Sample
Control Group (%) Impact Control Group (%) Impact Control Group (%) Impact
Received education credential since random assignment
Atlanta Labor Force Attachment 10.7 4.8** 12.8 0.8 10.9 5.4
Atlanta Human Capital Development 10.7 10.9*** 12.8 14.6*** 10.9 12.9***
Grand Rapids Labor Force Attachment 26.1 -7.4*** 29.1 -11.3*** 27.4 -11.1**
Grand Rapids Human Capital Development 26.1 2.2 29.1 2.3 27.4 4.2
Riverside Labor Force Attachment 19.1 -0.4 19.5 3.2 19.6 3.0
Lacked high school diploma or basic skills 14.5 3.0 14.8 5.5 15.9 7.5
Riverside Human Capital Development 14.5 10.6*** 14.8 14.7*** 15.9 11.9**
Employed at interview
Atlanta Labor Force Attachment 54.3 2.9 59.6 -0.8 63.7 -1.6
Atlanta Human Capital Development 54.3 -1.4 59.6 0.3 63.7 -3.0
Grand Rapids Labor Force Attachment 67.7 -2.6 67.2 3.9 64.9 8.8
Grand Rapids Human Capital Development 67.7 -1.5 67.2 -1.4 64.9 7.2
Riverside Labor Force Attachment 48.9 6.1** 44.7 6.2 50.0 6.8
Lacked high school diploma or basic skills 43.0 5.9 37.3 9.7* 44.9 9.2
Riverside Human Capital Development 43.0 8.0** 37.3 13.5** 44.9 8.5
Employed full time at interview
Atlanta Labor Force Attachment 45.5 1.7 51.6 0.3 55.0 -2.2
Atlanta Human Capital Development 45.5 -1.4 51.6 -0.3 55.0 -3.3
Grand Rapids Labor Force Attachment 54.3 -0.8 55.2 0.7 51.7 6.8
Grand Rapids Human Capital Development 54.3 -1.3 55.2 -3.1 51.7 7.9
Riverside Labor Force Attachment 35.9 8.2*** 29.6 11.0*** 32.3 13.8***
Lacked high school diploma or basic skills 31.8 8.4** 23.7 15.5*** 28.6 19.3***
Riverside Human Capital Development 31.8 4.9 23.7 9.0* 28.6 6.5
Employed and using child care at interview            
Atlanta Labor Force Attachment 21.5 0.8 38.9 1.9 42.4 0.5
Atlanta Human Capital Development 21.5 -1.1 38.9 -2.0 42.4 -3.7
Grand Rapids Labor Force Attachment 30.7 4.3* 49.5 2.4 47.4 6.6
Grand Rapids Human Capital Development 30.7 1.4 49.5 -0.7 47.4 6.7
Riverside Labor Force Attachment 26.6 0.8 35.1 3.3 41.1 2.5
Lacked high school diploma or basic skills 25.2 0.3 31.5 3.3 40.7 -1.3
Riverside Human Capital Development 25.2 -0.2 31.5 2.1 40.7 -8.5
Respondent and all children had medical coverage in month before interview
Atlanta Labor Force Attachment 71.1 -1.0 74.3 -1.5 77.8 -7.5*
Atlanta Human Capital Development 71.1 0.2 74.3 0.2 77.8 0.0
Grand Rapids Labor Force Attachment 70.9 -1.7 76.4 -1.7 74.7 -0.3
Grand Rapids Human Capital Development 70.9 -1.2 76.4 -4.3 74.7 0.9
Riverside Labor Force Attachment 77.6 -1.9 83.7 -3.1 85.5 -3.0
Lacked high school diploma or basic skills 77.6 0.2 84.6 0.4 85.7 3.1
Riverside Human Capital Development 77.6 0.8 84.6 -2.9 85.7 -3.8
Married and living with spouse
Atlanta Labor Force Attachment 8.4 1.3 7.7 2.3 7.3 2.7
Atlanta Human Capital Development 8.4 -1.5 7.7 0.9 7.3 2.0
Grand Rapids Labor Force Attachment 20.5 2.3 22.8 2.6 24.0 3.4
Grand Rapids Human Capital Development 20.5 -0.2 22.8 1.1 24.0 -5.0
Riverside Labor Force Attachment 22.0 -1.4 21.2 2.4 25.6 -0.8
Lacked high school diploma or basic skills 18.1 0.5 18.4 2.7 21.6 -1.5
Riverside Human Capital Development 18.1 3.7 18.4 0.8 21.6 0.4
Gave birth to another child since random assignment
Atlanta Labor Force Attachment 12.4 -0.8 20.5 -1.4 24.3 -3.7
Atlanta Human Capital Development 12.4 0.1 20.5 0.1 24.3 -0.5
Grand Rapids Labor Force Attachment 21.7 0.9 29.4 -2.2 32.6 -9.2*
Grand Rapids Human Capital Development 21.7 0.5 29.4 0.4 32.6 -2.8
Riverside Labor Force Attachment 22.1 3.4 31.0 10.4*** 32.4 6.6
Lacked high school diploma or basic skills 23.1 5.1* 34.0 9.6** 37.5 1.8
Riverside Human Capital Development 23.1 1.0 34.0 -0.7 37.5 -5.7
Living with another wage earner in month before interview
Atlanta Labor Force Attachment 27.6 1.5 24.2 3.6 29.3 -2.2
Atlanta Human Capital Development 27.6 2.6 24.2 4.2 29.3 1.9
Grand Rapids Labor Force Attachment 43.0 3.8 41.4 4.7 42.8 5.8
Grand Rapids Human Capital Development 43.0 -2.9 41.4 2.3 42.8 -1.3
Riverside Labor Force Attachment 42.2 -4.7* 38.2 -2.6 39.6 -4.7
Lacked high school diploma or basic skills 39.7 -7.1** 37.2 -6.0 37.1 -5.3
Riverside Human Capital Development 39.7 -0.1 37.2 -0.9 37.1 -2.6
At least one household member received $1,000 or more in earnings
Atlanta Labor Force Attachment 9.8 0.4 11.8 0.9 15.2 -3.0
Atlanta Human Capital Development 9.8 -0.8 11.8 3.5 15.2 2.7
Grand Rapids Labor Force Attachment 21.9 0.7 26.4 1.7 27.4 0.4
Grand Rapids Human Capital Development 21.9 -0.7 26.4 -0.8 27.4 -1.6
Riverside Labor Force Attachment 21.1 -0.7 20.5 1.4 22.6 -2.1
Lacked high school diploma or basic skills 17.5 -2.5 17.3 -2.8 17.2 -1.6
Riverside Human Capital Development 17.5 2.6 17.3 0.9 17.2 0.1
SOURCE:  MDRC calculations from the Five-Year Client Survey, Child Outcomes study, and Teacher Survey.
NOTES; See Appendix A.2

As discussed below, for Riverside HCDs impacts are larger and more positive in each survey sample than for the full impact sample. In addition, effects were generally more positive for the COS sample and, especially, the Teacher Survey sample than for the full impact sample. In contrast, impact results were less positive for Portland's Client Survey respondents than for the full impact sample.

I. Impacts for the Five-Year Client Survey Sample

A. Administrative Records Outcomes

For the Client Survey sample, most programs did not increase job finding (percentage ever employed) over five years, but both programs in Atlanta and Riverside led to impacts on earnings of between $2,500 and $3,500. The other programs led to positive differences in total earnings that were small and not statistically significant. All programs also reduced months of welfare receipt and total payments over five years, though the Atlanta HCD-control group differences on these measures were small and not statistically significant. Impacts on combined income over five years showed greater variation by program. Both Atlanta programs led to small increases in income, whereas both Grand Rapids programs reduced income by a small amount. (None of these differences were statistically significant.) Relatively large losses in income (between $3,000 and $4,000) over five years were found for Riverside LFA and Portland. Riverside HCD had almost no effect on income.

The pattern of impacts for the Client Survey sample resembled impacts for the full impact sample for both programs in Atlanta and Grand Rapids, but varied substantially for Riverside LFA and HCD and for Portland. For Riverside LFA, employment and earnings impacts were similar for the two samples, but welfare reductions were considerably larger for the Client Survey sample. As a result, Riverside LFAs in the survey sample show a much larger reduction in combined income over five years than those in the full impact sample. A large disparity in impacts was found for Riverside HCD as well, but in the opposite direction. For the Client Survey sample, the program led to substantial gains in total earnings over five years ($3,503) and in stable employment in year 5 (7.1 percentage points); these effects were not seen for the full impact sample. Reductions in welfare were similar for the two samples, however. Therefore, the effect on combined income is more positive for Riverside HCDs in the Client Survey sample. Finally, the Portland program had little effect on total earnings over five years for the Client Survey sample, but unusually large increases for the full impact sample. Moreover, the Portland program led to a relatively large decrease in combined income for the Client Survey sample, but a small (and not statistically significant) increase for the full impact sample. Differences for these three programs should be kept in mind when interpreting impacts on other outcomes.

B. Survey Outcomes

Table I.2 shows program impacts on a selection of outcomes for parents and families that may indirectly affect the well-being of children. As discussed in the text, programs led to relatively few impacts on these measures for the Client Survey sample. Of note, the Atlanta and Riverside HCD programs led to relatively large increases in attainment of an education credential after random assignment and both Riverside programs increased employment at the end of five years. Grand Rapids LFA did not increase employment at the end of year 5, but led to a small increase in the percentage of Client Survey respondents who used child care for employment.

II. Impacts for the Five-Year Child Outcomes Study sample

A. Administrative Records Outcomes

Most programs led to positive impacts on employment or on earnings for COS sample — although some differences were not statistically significant. Both Riverside programs raised average earnings by more than $6,000 per program group member, an unusually large amount. These two programs and the Grand Rapids LFA program also led to impacts on stable employment in year 5. All programs reduced months on welfare and total payments over five years, but effects were small and not statistically significant for both Atlanta programs and for Grand Rapids HCD. Impacts on combined income were mixed. The Riverside HCD program and both Atlanta programs led to small and not statistically significant increases in income, whereas both Grand Rapids programs and the Riverside LFA program had almost no effect.

To some extent, the pattern of impact is more positive for the COS sample than for the Client Survey and full impact samples, particularly for the measure of combined income. The biggest differences in impact estimates are for the two Riverside programs, especially Riverside HCD. That program's five-year earnings gains for the COS sample exceeded impacts for the full impact sample by nearly $5,000 per sample member. The difference in impacts on combined income was nearly as large. For Grand Rapids LFAs, results were also more positive for the COS sample than for the Client Survey or full impact sample.(2)

B. Survey Outcomes

For the COS sample, the pattern of impacts on survey outcomes looks similar to impacts for all respondents to the Client Survey. The two Riverside programs led to the most positive results. They increased employment and employment with full-time hours at the end of five years. Riverside HCD also increased degree attainment by nearly 15 percentage points above the control group level.

Less positively, Riverside LFA increased the proportion of COS sample members who gave birth to another child after random assignment by 10 percentage points above the control group average. Only a small difference was found when all Client Survey respondents in Riverside were included in the calculation.

III. Impacts for the Teacher Survey Sample

A. Administrative Records Outcomes

In general, COS families who participated in the Teacher Survey experienced the most positive impacts of the four research samples. All programs increased five-year earnings above control group levels, although some differences were small and not statistically significant. Moreover, impacts on earnings averaged more than $800 per year for the three LFA programs and for Riverside HCD. Most programs also increased stable employment in year 5. All programs reduced total welfare payments over five years, but effects were small and not statistically significant for Atlanta and Grand Rapids HCD. Over five years, all programs except Riverside LFA increased combined income, although effects were small and not statistically significant in most programs.

The disparity of impact results across samples was greatest for Atlanta and Grand Rapids LFA. These programs led to much more positive effects on earnings, on stable employment, and on combined income for the Teacher Survey sample, than for the other samples. Furthermore, as for the COS sample, Riverside HCD led to much larger impacts on stable employment and earnings and more positive effects on combined income for the Teacher Survey sample than for the full impact and Client Survey samples.

B. Survey Outcomes

For survey outcomes, more program-control group differences of at least moderate size (+/- 5 percentage points) were found for the Teacher Survey sample than for the other survey samples. However, most of these differences were not statistically significant. As for the other samples, both Riverside programs increased employment at the end of year 5 above control group levels. In addition, a higher percentage of LFAs gave birth to a child during the follow-up, an effect also seen for all LFAs in the Riverside COS sample. Unlike its effect on the COS and Client Survey samples, Riverside HCD decreased below control group levels the proportion of Teacher Survey sample members who were working and using child care at the end of year 5 and decreased the incidence of childbearing after random assignment.(3)

The Grand Rapids LFA and HCD programs also led to larger effects for the Teacher Survey sample than for the COS samples, although not all differences were statistically significant. Each program increased employment and employment with full-time hours at the end of year 5. Grand Rapids LFAs were also more likely than control group members to be living with another person who was working for pay, more likely to be using child care for employment at the end of year 5, and less likely to have given birth to a child after random assignment. The Grand Rapids HCD program also increased child care use for employment at the end of year 5 and decreased the incidence of being married and living with spouse.

Finally, programs had as little effect on medical coverage at the end of year 5 for the Teacher Survey sample as they did for the other survey samples. However, Atlanta LFA led to a statistically significant decrease in coverage, a result not seen for the other samples.

Endnotes

1. See Appendix G for a discussion of these issues.

2. It should be remembered that both the full impact and Client Survey samples include parents whose youngest child was aged 6 or over at random assignment, but the COS sample does not.

3. For Riverside LFA, only the impact on full-time employment was statistically significant.

Where to?

Top of Page

Home Pages:
National Evaluation of Welfare-to-Work Strategies (NEWWS)
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)