[ Main Page of Report | Contents of Report ]
The primary aim of the NEWWS Evaluation was to test and compare the effectiveness of a variety of welfare-to-work approaches in different locales. This chapter describes the research designs employed, the samples of people studied and their characteristics, and the types of data used in the report. It concludes with guidelines for interpreting the results presented in the chapters that follow.
To assess the effectiveness of different welfare-to-work strategies, the evaluation used a random assignment research design. In each of the seven sites in the evaluation, people who were required to participate in a welfare-to-work program were assigned, by chance, either to a program group, which had access to employment and training services and whose members were required to participate in the program, or to a control group, which received no program services and whose members were not subject to a participation requirement but could seek out similar services on their own in the community.(1) Program group members who did not comply with the participation mandate risked incurring a sanction, that is, having their welfare grant reduced. Control group members, in contrast, could not be sanctioned because of the control embargo that precluded them from participating in program activities. Throughout the report, the program and control groups are referred to as research groups and the people in them as sample members. The random assignment design ensured that there were no systematic differences between the background characteristics of program and control group members when they entered the study. Thus, any subsequent differences between the groups' outcomes (called impacts) can be attributed with confidence to the effects of the programs.
Sample members in each research group were tracked over a follow-up period of five years after their date of random assignment. Average outcomes for control group members (such as employment and welfare receipt) after random assignment represent what could be expected of welfare recipients had they never enrolled in a welfare-to-work program. Past studies have shown that many people to whom welfare-to-work programs are targeted will leave welfare and find work on their own, that is, without being assigned to a welfare-to-work program.
The differences between outcomes for the program and control groups represent the impacts or effects of each program. Unless otherwise noted, all "increases" and "decreases" reported in this document refer to such program-control differences.
Four of the sites implemented a three-way random assignment research design in order to test the effectiveness of two different program approaches. In the three-way design, each person was assigned, by chance, to one of two program groups or to a control group. This design is especially powerful because members of both research groups have the same background characteristics and face the same labor market conditions and other environmental factors that can affect a program's success in helping people find jobs and advance toward economic security. In the three-way sites, the relative effectiveness of the two programs can be assessed by comparing outcomes for the program groups with one another directly, that is, without taking the control group into account. To assess the programs' absolute effectiveness, however, it is necessary to compare outcomes for each program with the control group's outcomes.(2)
Three of these four sites (Atlanta, Grand Rapids, and Riverside) operated two programs that were designed — for purposes of the evaluation — to magnify the differences between the employment- and education-focused approaches described in Chapter 1: Labor Force Attachment (LFA) programs, which emphasize rapid job placement as the best way for welfare recipients to develop their work habits and skills, even if the job pays low wages; and Human Capital Development (HCD) programs, which emphasize that welfare recipients have to develop their "human capital," that is, their knowledge and basic skills, through education and training in order to have a better chance of finding and keeping jobs and advancing toward well-paid and secure employment. In each site, the two program models were implemented to maximize the contrast between them, thus making the differences between their effects easier to detect. Figure 2.1 illustrates the process by which welfare recipients and applicants in Atlanta and Grand Rapids were randomly assigned to the research groups.
Figure 2.1
Steps Leading to Random Assignment in Atlanta and Grand Rapids
The random assignment process differed in Riverside because California's welfare rules mandated that only people "in need of basic education" — that is, people who lacked a high school diploma or GED, scored low on a welfare department math or reading literacy test, or required instruction in English as a Second Language (ESL) — could be assigned to the HCD group. This constraint meant that whereas the HCD group included only people determined to need basic education, the LFA group included both such people and people determined not to need basic education. To facilitate direct comparisons between the Riverside LFA and HCD groups in this report, the results for the subgroup of LFA group members determined to need basic education are provided in addition to the results for the full LFA group.(3) A second conse-quence of this constraint in Riverside is that the only way to make direct comparisons between the results of the Riverside HCD program and those of the programs in other sites in the evaluation is to focus on the subgroups of people in those other programs who lacked a high school diploma or GED.(4) Figure 2.2 illustrates the process by which welfare recipients and applicants in Riverside were randomly assigned to the research groups.
Figure 2.2
Steps Leading to Random Assignment in Riverside
Unlike the goal in the other sites that used a three-way design, the goal in Columbus was to test and compare the effectiveness of two different case management models. In the Traditional model, one worker handled the welfare department's employment and training function and another worker handled welfare eligibility and payment issues — often called "income maintenance." Both workers maintained relatively large caseloads. In the Integrated model, a single worker handled both the employment and training and income maintenance functions. In the Integrated model, the worker maintained a smaller caseload than either of the workers in the Traditional model since, on a per client basis, the worker was handling jobs "traditionally" done by two workers.
The remaining three sites in the evaluation (Oklahoma City, Detroit, and Portland) used a two-way random assignment design to test the effectiveness of program models already established in those sites. In other words, instead of implementing a program designed expressly for research purposes, as in the three-way sites, program administrators in each of the two-way sites determined their welfare-to-work program goals and practices and randomly assigned people to a group that entered the program or to a control group.(5) A summary of the research designs in all seven sites is presented in Table 2.1.
| Characteristic | Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Oklahoma City | Portland |
|---|---|---|---|---|---|---|---|
| Type of random assignment | Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Three-way
(2 program groups, 1 control group) |
Two-way
(1 program group, 1 control group) |
Two-way
(1 program group, 1 control group) |
Two-way
(1 program group, 1 control group) |
| Type of study | Differential impacts of HCD and LFA approaches | Differential impacts of HCD and LFA approaches | Differential impacts of HCD and LFA approaches | Differential impacts of Integrated and Traditional case management strategies | Net impacts of established program | Net impacts of established program | Net impacts of established program |
| Sample composition | AFDC applicants and recipients | AFDC applicants and recipients; teen parents (ages 18 and 19) | AFDC applicants and recipients | AFDC applicants and recipients | AFDC applicants and recipients; teen parents (ages 18 and 19) | AFDC applicants; teen parents (ages 16 to 19) | AFDC applicants and recipients |
| Age of youngest child | 3 | 1 | 3 | 3 | 1 | 1 | 1 |
| Point of random assignment | Program orientation | Program orientation | Program orientation | Income maintenance office: application or redetermination | Program orientation | Income maintenance office: application only | Program orientation |
[Go to Contents]
In each site, sample members were randomly assigned over a period of approximately two years. Random assignment began in June 1991 in Riverside, California, and ended in December 1994 in Portland, Oregon (see Table 2.2). Thus, the results presented in this report cover the calendar period from June 1991 (the month of the first Riverside sample member's entry into the study) to December 1999 (the last month of the follow-up period for the last sample member in Portland to be randomly assigned). Throughout the report, the five years of the evaluation's follow-up period are labeled year 1, year 2, and so on. These labels refer not to calendar years but to years after random assignment; for example, year 1 refers to the first year after sample members were randomly assigned, regardless of whether year 1 began in 1991 or 1994 for individual sample members.
Site and Program |
Full Impact Sample | Five-Year Client Survey Sample | Two-Year and Five-Year Client Survey Respondents | Five-Year Child Outcomes Study Sample | Five-Year Teacher Survey Sample |
|---|---|---|---|---|---|
Atlanta |
|||||
| Random assignment period | 01/92-06/93 | 03/92-06/93 | 03/92-06/93 | 03/92-06/93 | 03/92-06/93 |
| Labor Force Attachment | 1,441 | 519 | 491 | 289 | 184 |
| Human Capital Development | 1,495 | 594 | 565 | 367 | 226 |
| Control | 1,497 | 552 | 527 | 311 | 193 |
| Full sample | 4,433 | 1,665 | 1,583 | 967 | 603 |
Grand Rapids |
|||||
| Random assignment period | 09/91-01/94 | 03/92-01/94 | 03/92-01/94 | 03/92-01/94 | 03/92-01/94 |
| Labor Force Attachment | 1,557 | 535 | 506 | 214 | 144 |
| Human Capital Development | 1,542 | 547 | 511 | 196 | 120 |
| Control | 1,455 | 562 | 537 | 214 | 144 |
| Full sample | 4,554 | 1,644 | 1,554 | 624 | 408 |
Riverside |
|||||
| Random assignment period | 06/91-06/93 | 09/91-05/93 | 09/91-05/93 | 09/91-05/93 | 09/91-05/93 |
| Labor Force Attachment | 3,384 | 499 | 424 | 185 | 108 |
| Human Capital Development | 1,596 | 376 | 323 | 208 | 131 |
| Control | 3,342 | 720 | 648 | 348 | 222 |
| Full sample | 8,322 | 1,595 | 1,395 | 741 | 461 |
Portland |
|||||
| Random assignment period | 02/93-12/94 | 03/93-02/94 | 03/93-02/94 | ||
| Program | 3,529 | 281 | 238 | ||
| Control | 499 | 223 | 204 | ||
| Full sample | 4,028 | 504 | 442 | ||
Columbus |
|||||
| Random assignment period | 09/92-07/94 | ||||
| Integrated | 2,513 | ||||
| Traditional | 2,570 | ||||
| Control | 2,159 | ||||
| Full sample | 7,242 | ||||
Detroit |
|||||
| Random assignment period | 05/92-06/94 | ||||
| Program | 2,226 | ||||
| Control | 2,233 | ||||
| Full sample | 4,459 | ||||
Oklahoma City |
|||||
| Random assignment period | 09/91-05/93 | ||||
| Program | 4,309 | ||||
| Control | 4,368 | ||||
| Full sampleV | 8,677 | ||||
| Full sample size | 41,715 | 5,408 | 4,974 | 2,332 | 1,472 |
| SOURCE: MDRC-created database. | |||||
The differences in the random assignment procedures used in different sites affected the composition of the site samples and, thus, the comparability of the results for different sites and programs.(6) In five of the seven sites, welfare applicants and recipients who were required to participate in a welfare-to-work program (because they met certain demographic criteria: for instance, had no children below the minimum age set by the program) were randomly assigned while attending a program orientation at their local employment and training office. In Columbus and Oklahoma City, in contrast, people were randomly assigned at their local income maintenance office before being assigned to an orientation.
Not everyone assigned to participate in a welfare-to-work program actually attends an orientation. Reasons for not attending include leaving welfare shortly after being referred to the program, having one's welfare application denied, or simply failing to show up.(7) As a result, the people who attend a program orientation may not be representative of everyone in their locale who is required to participate in a welfare-to-work program. For example, when the waiting list for orientation "slots" is long, the people who find jobs and exit welfare before being randomly assigned are likely to be more employable, on average, than people who do not. As a result, those who enroll in the program are disproportionately likely to be "disadvantaged." Data on how many people who were required to attend an orientation actually did so are available for three sites (Riverside, Grand Rapids, and Columbus). About 66 percent in the Riverside, Grand Rapids, and Columbus Traditional programs and about 83 percent in the Columbus Integrated program attended an orientation.(8) Because outcomes in this report are reported as averages for all sample members in a group — including, in Columbus, those who did not ultimately attend an orientation — the relative effects of the Columbus Integrated and Columbus Traditional programs on sample members' participation and subsequent employment, earnings, and welfare outcomes reflect not only differences in case management strategies but in the capacity to enroll people.
The Oklahoma City results should be interpreted with the following in mind: Oklahoma City, unlike the other sites, randomly assigned only welfare applicants (including those whose application for assistance was not yet approved) to the research groups.(9) Moreover, about 30 percent of sample members in Oklahoma City were denied cash welfare assistance shortly after being randomly assigned.(10) Therefore, the impacts for the Oklahoma City program are based on a sample that included a larger proportion than in the other sites of people who never received a welfare payment after random assignment for reasons unrelated to the program. In addition, past research has shown that welfare-to-work programs have different effects on welfare applicants than on recipients, most likely because recipients tend to be more disadvantaged than applicants.(11)
[Go to Contents]
Table 2.2 shows the dates of random assignment and sizes of the samples used in this report, by site and research group. All the analysis samples used in the report are listed and described below.
Full impact sample. The full impact sample includes 41,715 program and control group members from all seven sites, for whom five years of administrative records data were collected (Figure 2.3, box A).(12), (13)
Figure 2.3
Sample Size and Data Sources
Five-Year Client Survey sample. Additional data on outcomes for adults and children were collected by interviewing sample members around two years after their date of random assignment and, in four of the seven sites, around their five-year anniversary. This report focuses on outcomes from the Five-Year Client Survey (detailed findings from the Two-Year Client Survey can be found in the two-year report from the NEWWS Evaluation).(14) The Five-Year Client Survey sample (Figure 2.3, box B) includes 5,408 members of the full impact sample in Atlanta, Grand Rapids, Riverside, and Portland. In each site, survey selection took place during some, but not all, months of random assignment. The survey sample was drawn from members of the full impact sample who had earlier been selected to be interviewed at two years, whether or not they actually responded to the two-year survey (4,974 sample members answered both surveys, and 434 responded to only the five-year survey; see Table 2.2).(15) Those selected to be interviewed at two years were a stratified random sample of the full impact sample members who were randomly assigned during the months when the survey sample was selected.
Certain subgroups were intentionally oversampled to produce large enough samples for special analyses such as the Child Outcomes Study (COS) and an intensive examination of adult education.(16) Results from all programs in this report have been weighted to reflect the overall demographic characteristics of the larger sample. The survey response rates exceed 70 percent for all programs and research groups (and 80 percent in Atlanta and Grand Rapids).(17)
Child Outcomes Study sample. The COS, which is part of the NEWWS Evaluation, includes the families of 2,332 sample members who responded to the Five-Year Client Survey in Atlanta, Grand Rapids, and Riverside (Figure 2.3, box C).(18) Individuals in these three sites who were selected for the Two-Year Client Survey and who had at least one child aged 3 to 5 at random assignment were randomly selected to be part of the COS. Each family included a child aged 3 to 5 at random assignment who was randomly selected to serve as the focal child, that is, the child about whom the most extensive information was collected. Response rates to the COS survey in Atlanta and Grand Rapids exceeded 75 percent; response rates in Riverside were between 63 and 67 percent for different research groups.(19)
Teacher survey sample. In a study of children's school progress, which is also part of the evaluation, COS mothers who were interviewed at the five-year follow-up point were asked for permission to mail a survey to the focal child's current elementary school teacher that asked about outcomes such as academic performance. (For details, see Section III.) The teacher survey sample includes responses from 1,472 teachers of focal children (Figure 2.3, box D).(20) Response rates were lower for the teacher survey than for the other surveys, ranging from 37 to 57 percent.(21)
Ethnicity. The racial and ethnic makeup of the full impact sample varied from site to site, reflecting general differences in the overall ethnic composition of the counties from which the samples were drawn. For example, whereas almost all sample members in Atlanta and Detroit were African-American, about one-half of sample members in Grand Rapids, Riverside, Columbus, and Oklahoma City and two-thirds of those in Portland were white. Only Riverside had a substantial proportion (one-third) of Hispanic sample members (see Table 2.3).
Characteristic |
Atlanta | Grand Rapids | Riverside | Columbus | Detroit | Oklahoma City | Portland |
|---|---|---|---|---|---|---|---|
| Demographic characteristics | |||||||
Gender (%) |
|||||||
| Male | 3.2 | 4.2 | 10.6 | 6.5 | 3.3 | 6.9 | 6.7 |
| Female | 96.8 | 95.8 | 89.4 | 93.5 | 96.7 | 93.1 | 93.3 |
| Age (%) | |||||||
| Under 19 | 0.0 | 5.4 | 0.9 | 0.2 | 2.9 | 9.8 | 0.0 |
| 19-24 | 9.5 | 34.2 | 15.5 | 12.9 | 26.2 | 27.0 | 23.1 |
| 25-34 | 54.9 | 40.6 | 49.7 | 55.7 | 43.3 | 42.9 | 50.8 |
| 35-44 | 29.6 | 16.5 | 27.8 | 26.8 | 22.7 | 17.3 | 22.2 |
| 45 or over | 6.0 | 3.3 | 6.1 | 4.4 | 5.0 | 2.9 | 3.9 |
| Average age (years) | 32.8 | 28.2 | 32.0 | 31.8 | 30.0 | 28.1 | 30.4 |
| Ethnicity (%) | |||||||
| White | 3.5 | 50.1 | 49.0 | 46.5 | 11.0 | 59.4 | 69.5 |
| Hispanic | 0.8 | 8.0 | 30.2 | 0.4 | 0.8 | 4.3 | 3.9 |
| Black | 95.2 | 39.3 | 16.7 | 52.0 | 87.3 | 28.9 | 20.1 |
| Black Hispanic | 0.1 | 0.3 | 0.0 | 0.0 | 0.0 | 0.3 | 0.3 |
| Native American/Alaskan Native | 0.1 | 1.5 | 1.4 | 0.1 | 0.2 | 6.4 | 3.0 |
| Asian/Pacific Islander | 0.2 | 0.4 | 2.7 | 0.8 | 0.2 | 0.6 | 2.2 |
| Other | 0.2 | 0.5 | 0.0 | 0.3 | 0.6 | 0.1 | 1.1 |
| Family status | |||||||
| Marital status (%) | |||||||
| Never married | 60.5 | 57.9 | 32.5 | 50.2 | 68.0 | 34.3 | 47.3 |
| Married, living with spouse | 1.4 | 3.3 | 8.1 | 8.2 | 2.7 | 3.8 | 1.7 |
| Separated | 20.7 | 18.4 | 31.4 | 22.3 | 15.8 | 35.7 | 21.8 |
| Divorced | 16.1 | 19.3 | 26.5 | 18.6 | 12.3 | 25.2 | 28.6 |
| Widowed | 1.4 | 1.0 | 1.5 | 0.8 | 1.2 | 1.0 | 0.6 |
| Number of children (%) | |||||||
| 1 | 36.7 | 46.9 | 38.6 | 40.0 | 43.7 | 50.6 | 40.6 |
| 2 | 33.2 | 35.3 | 32.0 | 33.4 | 29.6 | 30.5 | 33.6 |
| 3 or more | 30.1 | 17.9 | 29.4 | 26.6 | 26.7 | 18.9 | 25.8 |
| Average number of children | 2.1 | 1.8 | 2.0 | 2.0 | 2.0 | 1.7 | 2.0 |
| Age of children (%) | |||||||
| Any child aged 0-5 | 41.5 | 67.9 | 56.0 | 46.9 | 64.3 | 65.1 | 67.4 |
| Any child aged 6-11 | 63.0 | 38.3 | 56.2 | 57.3 | 44.3 | 40.5 | 47.6 |
| Any child aged 12-18 | 46.3 | 26.2 | 37.0 | 39.4 | 34.0 | 23.9 | 25.9 |
| Age of youngest child (%) | |||||||
| 2 or under | 0.3 | 46.3 | 6.2 | 1.8 | 39.3 | 41.4 | 40.2 |
| 3 to 5 | 41.2 | 21.6 | 49.8 | 45.1 | 25.0 | 23.8 | 27.3 |
| 6 or over | 58.5 | 32.1 | 44.0 | 53.1 | 35.7 | 34.9 | 32.6 |
| Had a child as a teenager (%) | 42.3 | 48.4 | 32.8 | 37.5 | 44.2 | 47.1 | 32.3 |
| Labor force status | |||||||
| Ever worked full time for 6 months or more | |||||||
| for one employer (%) | 71.4 | 63.8 | 71.0 | 42.5 | 48.1 | 68.8 | 76.9 |
| Any earnings in past 12 months (%) | 23.6 | 46.0 | 40.7 | 28.2 | 21.1 | 69.0 | 39.3 |
| Currently employed (%) | 6.9 | 11.4 | 11.2 | 4.0 | 6.8 | 8.6 | 9.6 |
| Education status | |||||||
| Received high school diploma or GED (%) | 59.7 | 59.0 | 56.2 | 57.4 | 56.5 | 55.1 | 67.3 |
Highest degree/diploma earned (%) |
|||||||
| GED a | 5.4 | 8.2 | 9.2 | 7.0 | 10.7 | 11.3 | 21.5 |
| High school diploma | 46.7 | 45.9 | 41.8 | 44.6 | 37.0 | 38.2 | 34.5 |
| Technical/AA/2-year college | 6.6 | 4.1 | 4.3 | 4.6 | 8.0 | 4.3 | 9.7 |
| 4-year (or more) college | 1.3 | 0.9 | 0.9 | 1.6 | 1.1 | 1.6 | 1.9 |
| None of the above | 40.0 | 40.9 | 43.8 | 42.3 | 43.2 | 44.6 | 32.3 |
| Enrolled in education or training in past | |||||||
| 12 months (%) | 13.4 | 39.2 | 19.6 | 9.5 | 20.0 | 23.7 | 21.1 |
| Currently enrolled in education or training (%) | 8.4 | 34.8 | 14.1 | 7.8 | 28.2 | 12.9 | 13.5 |
| Public assistance status | |||||||
| Total prior AFDC receipt (%) b | |||||||
| None | 0.3 | 0.1 | 1.0 | 10.0 | 2.8 | 44.4 | 1.2 |
| Less than 1 year | 18.9 | 22.1 | 33.8 | 8.3 | 13.7 | 18.8 | 20.9 |
| 1 year or more but less than 2 years | 10.1 | 18.6 | 11.3 | 9.0 | 9.1 | 12.5 | 16.6 |
| 2 years or more but less than 5 years | 24.6 | 30.0 | 26.4 | 27.9 | 24.0 | 15.3 | 32.1 |
| 5 years or more but less than 10 years | 22.4 | 16.4 | 15.6 | 22.7 | 22.5 | 6.5 | 21.1 |
| 10 years or more | 23.7 | 12.8 | 11.8 | 22.1 | 27.9 | 2.5 | 8.2 |
| Raised as a child in a household receiving AFDC (%) | 26.9 | 32.8 | 19.5 | 27.0 | 40.1 | 21.7 | 23.8 |
| First spell of AFDC receipt (%) c | 7.2 | 27.9 | 23.5 | 9.6 | 4.1 | 42.0 | 7.2 |
Level of disadvantage |
|||||||
| Most disadvantaged d | 24.2 | 15.1 | 24.7 | 19.0 | 25.1 | 4.9 | 15.3 |
| Housing status | |||||||
| Current housing status (%) | |||||||
| Public housing | 35.9 | 2.6 | 2.5 | 15.2 | 5.5 | 5.3 | 7.6 |
| Subsidized housing | 24.9 | 13.0 | 7.0 | 24.7 | 1.1 | 6.7 | 18.6 |
| Emergency or temporary housing | 0.7 | 2.4 | 1.4 | 1.4 | 0.8 | 14.4 | 3.7 |
| None of the above | 38.5 | 82.1 | 89.1 | 58.7 | 92.6 | 73.7 | 70.1 |
| Sample size | 4,433 | 4,554 | 8,322 | 7,242 | 4,459 | 8,677 | 4,028 |
| SOURCE: MDRC calculations from information
routinely collected by welfare staff. NOTES: Distributions may not add to 100 percent because of rounding. a The GED credential is given to those who pass the GED test and is intended to signify knowledge of high school subjects. b This refers to the total number of months accumulated from at least one spell on an individual's own or spouse's AFDC case. It does not include AFDC receipt under a parent's name. c This does not mean that such individuals are new to the AFDC rolls, only that this is their first spell on AFDC. This spell, however, may have lasted several years. d The "most disadvantaged" subgroup consists of sample members who did not have a high school diploma or GED at random assignment, did not work for pay in the year prior to random assignment, and had received AFDC for two years or more (cumulatively) prior to random assignment. |
|||||||
Family structure. Almost all the sample members in the evaluation were single parents. The "average" sample member was a 30-year-old single mother with two children.(22) She was likely to have had a preschool-age child at random assignment and to have had her first child as a teenager.
This characterization does not capture the diversity of the families who were subject to program participation mandates in these locales. In particular, it does not reflect important site differences in who was required to participate in a welfare-to-work program. Just under one-half of sample members in Grand Rapids, Detroit, Oklahoma City, and Portland — where mothers with a child as young as age 1 were required to participate — entered the program when their youngest child was under age 3. The remainder of the sample in these four sites and the full samples in the other three sites were evenly divided between mothers with a youngest child aged 3 to 5 and those with a youngest child aged 6 or over. In Grand Rapids, Detroit, and Oklahoma City, unlike in the other sites, teen parents are included in the full impact sample (see Table 2.1).
Educational attainment. Between 55 and 66 percent of sample members had a high school diploma or GED certificate when they entered the program, and some enrollees in all sites had some college or post-secondary schooling. On average, however, sample members had completed only 11 years of school before random assignment. Those sample members who had a high school diploma or GED certificate at random assignment are described in this report as graduates; those without a high school diploma or GED are described as nongraduates.
Employment history. Sample members' employment history varied by site. Less than one-half of sample members in all sites except Oklahoma City had worked at some point during the year before random assignment: from 21 percent (in Detroit) to 46 percent (in Grand Rapids). Not surprisingly, sample members in Oklahoma City, all of whom were welfare applicants, were far more likely to have worked in the year before entering the program (69 percent had done so).
In addition to having little recent work experience, less than one-half of the sample members in Columbus and Detroit had worked full time for six months or more for one employer at some point before random assignment; two-thirds to three-quarters of sample members in the other sites had done so.
Past welfare receipt. The majority of sample members in all sites except Oklahoma City had already received welfare for at least two years cumulatively before random assignment. Just 24 percent of those in Oklahoma City, compared with 54 to 74 percent in the other sites, had received cash assistance for two years or more. Excluding Oklahoma, between 28 and 50 percent of sample members had received welfare cumulatively for five years or more before random assignment.
"Most disadvantaged" status. The sample members considered to be the most disadvantaged were those who lacked a high school diploma or GED (or, in Riverside, who were determined to need basic education), lacked any work history in the year before random assignment, and had already received welfare for two years or more cumulatively before entering the program. The proportion of sample members in the most disadvantaged group ranged from 5 percent in Oklahoma City to 25 percent in Riverside and Detroit.
Housing status. The proportion of sample members who at random assignment were living in public housing developments or receiving housing subsidies through such programs as the Section 8 rental assistance program was highest in Atlanta (56 percent) and lowest in Detroit (7 percent). Some have argued that federal housing policies discourage people from working because — from the standpoint of residents of public and subsidized housing, who pay rent on a sliding scale — earnings increases mean rent increases. In addition, gross income limits on housing assistance eligibility could cause a newly employed person to lose her housing subsidy altogether.
Compared with people in the other sites, a fairly large proportion (14 percent) of people in the Oklahoma City sample lived in emergency or temporary housing — that is, lived in a shelter or were homeless — when they applied for welfare. Less than 3 percent of sample members in the other sites were experiencing this type of hardship at random assignment.
[Go to Contents]
The outcomes and impacts presented in this report are drawn from four primary data sources: unemployment insurance, welfare, and Food Stamp administrative records; surveys of sample members that were conducted at the two-year and five-year follow-up points (the Two-Year Client Survey and the Five-Year Client Survey); a survey of sample members focused on outcomes for children (the Child Outcomes Study survey); and a teacher survey.
Client characteristic data. Standard personal data, such as educational background and welfare history, were collected by welfare staff during routine interviews at the time of random assignment and are available for all 41,715 heads of the single-parent families in the full impact sample.
Private Opinion Survey. Data on attitudes and opinions about welfare-to-work programs and employment prospects were collected through the Private Opinion Survey (POS), a brief, self-administered survey that was completed at program orientation in four of the sites (Atlanta, Grand Rapids, Riverside, and Portland), and are available for 18,461 respondents in these sites. These sample members represent 93 percent of those randomly assigned in the four sites during the periods when the POS was being administered.
Reading and math tests. Reading and math achievement tests were administered in four sites (Atlanta, Grand Rapids, Riverside, and Portland) at random assignment. Test scores are available for 20,577 sample members. These sample members represent about 93 percent of those randomly assigned in the four sites during the period when the tests were administered.(23)
Field research. MDRC staff observed all 11 programs in operation and interviewed enrollees, case managers, service providers, and program administrators in each site. Information was collected about a range of issues, such as management philosophy and structure, the degree to which the participation mandate was enforced, the nature of interactions between caseworkers and program participants, the extent to which the program was able to work with all those mandated to participate in it, the availability of services, and the relationships that program staff had established with outside service providers and income maintenance staff in the sites.
Unemployment insurance, welfare, and Food Stamp administrative records data. Most employment, earnings, and public assistance impacts were computed using automated county and state unemployment insurance (UI), welfare, and Food Stamp administrative records data. Five years of follow-up data from the UI system are available for all members of the full impact sample; five years of follow-up data from welfare and Food Stamp administrative records are available for all sample members in all sites except Oklahoma.
UI earnings, which are recorded statewide, provide unbiased measures of program impacts on employment and earnings. These data, however, do not include earnings from out of state; from jobs not usually covered by the UI system, such as self-employment, federal employment, or informal child care (all types of work that may have been "off the books"); or from employers who do not report earnings. Some of the earnings missed by the UI system may be captured by earnings and employment data collected through the two-year and five-year surveys.
In all sites except Riverside, welfare and Food Stamp payments were also recorded statewide, and payments are captured for all sample members except those who moved out of state. In Riverside (as everywhere in California), welfare and Food Stamp payments were recorded only within each county, which means that payments received by sample members who moved outside the county were not included in the analysis. Although this could lead to an underestimate of the payments received in the Riverside sample, it should not bias the impact estimates because there is no reason to expect the program and control groups to show different patterns of moving between counties.
UI earnings data are collected by calendar quarter: January through March, April through June, and so forth. For purposes of the evaluation, these data were reorganized so that the quarter during which a sample member was randomly assigned is always designated quarter 1, followed by quarter 2, and so forth. These quarters are then grouped into "years." Quarter 1 is not included in year 1 because it includes some income earned before random assignment, especially for sample members randomly assigned near the end of a calendar quarter. Thus, year 1 covers quarters 2 through 5, year 2 covers quarters 6 through 9, and so forth. Welfare and Food Stamp payments were recorded monthly but were grouped into quarters and years to align with the earnings data.
Two-Year Client Survey and Five-Year Client Survey. As noted in a previous section, this report includes the results of a survey administered at the five-year follow-up point and some results of a survey administered at the two-year follow-up point. Both the two-year and five-year surveys provide information about sample members' participation in training and education activities, attainment of education credentials, views of work and welfare, employment history, income, receipt of noncash benefits such as health coverage, child care use, living situations, and children's well-being.
Survey responses are the only source of information about many key outcomes, such as participation patterns for control group members, work hours and wages, income from other people in the household, and outcomes for children. For some outcomes, such as employment, respondents provided information that was also recorded from administrative data. It is possible for data from these two sources to differ. Because the five-year survey respondents represent a subsample of the full impact sample that was selected during a shorter period of random assignment months, the impact and survey samples may differ with respect to observed characteristics (such as educational attainment or prior work history) or with respect to unmeasured characteristics (such as assertiveness or learning style) that might have affected their ability to find and retain employment. (For more information on survey response bias and the degree to which the survey sample and full impact samples differ, see Appendix G.)
In some cases, administrative records data may be more accurate than the survey data. The client survey depends on people's ability to recall information about events or jobs that they may have held up to five years prior to being interviewed, and failures of memory can give rise to discrepancies between the dates of employment or amounts of earnings reported in the survey and reflected in administrative records. In addition, some respondents may have been reluctant to provide information on employment and income that could be found in administrative records or, alternatively, may have exaggerated their earnings and income. In other cases, however, survey data may be more accurate, such as when respondents were working off the books or in short-term employment. The survey may also have captured earnings that employers failed to report or reported inaccurately to the UI system. (For more information on the differences between UI-reported and survey-based measures of earnings, see Appendix H.)
Additional COS survey data. COS respondents provided information on focal children's academic functioning, social skills, and health and safety. In addition, mothers and the focal children themselves completed a Self-Administered Questionnaire (SAQ). Mothers' SAQ included questions about domestic abuse; children's SAQ included questions about academic functioning and social skills.
Teacher survey. Current teachers of focal children in the COS were asked to assess them with respect to their academic standing, academic progress, school engagement, behaviors requiring disciplinary action, and social skills. The teacher survey complements the data collected from mothers and the children themselves. Reports from teachers and mothers sometimes differ. Possible explanations include the following: The children behaved differently in the presence of mothers and teachers, mothers and teachers perceived the children's behavior differently, or mothers and teachers based their reports on different criteria.
Cost data. The cost analysis used data drawn from state, county, and local fiscal records, supportive service payment records, administrative records, the Two-Year Client Survey, the Five-Year Client Survey, and case file participation records.
Benefit-cost data. The benefit-cost analysis is based on administrative records data (UI-reported earnings, welfare, and Food Stamp payments), Two-Year Client Survey data, Five-Year Client Survey data, and published data.
Published data and agency reports. Published data and reports from government agencies were used to gather additional information about the environments in each of the sites, including unemployment rates, welfare caseloads, and welfare grant levels.
[Go to Contents]
As discussed above, control group outcomes in this evaluation represent outcomes expected in the absence of a welfare-to-work program. Program-control differences show the effect, or impact, of each program. In the sites that conducted side-by-side evaluations of alternative program approaches, differences between the outcomes for each program group represent the relative effects of each program.
Although random assignment minimizes the likelihood of the research groups' differing systematically at the outset, there can be small differences in their average characteristics at random assignment. To control for these differences, the outcomes for each research group were regression-adjusted using ordinary least squares in all the analyses presented in the chapters that follow.
In this report, a difference between the program and control groups with respect to a particular outcome is considered statistically significant if the result of a statistical test indicates that there is less than a 10 percent probability that the difference occurred by chance (that is, when the p-value, or level of significance, of the difference is under .10). Impacts are generally reported only if they are statistically significant. This rule is intended to keep researchers from inferring an impact where none exists.(24)
Many analysts have noted that the greater the number of analyses conducted (regardless of the outcomes or domains studied), the greater the likelihood of chance findings and, thus, one needs to take the number of outcomes examined into account. However, some argue that this is relevant only when outcomes are not theoretically independent from each other. There are stringent statistical tests of multiple dependent variables that automatically adjust for the (limited) number of theoretically related outcomes (that is, multivariate analysis of variance, or MANOVA), as well as post-hoc corrections to p-values that can be applied to results from multiple individual analyses of "similar" outcomes (for example, the Bonferroni correction).
For this report, we have not attempted to adjust for the number outcomes such as employment and AFDC receipt that are examined because many of these outcomes are so highly statistically significant that they would pass the most stringent statistical correction for the fact that many outcomes are being measured. By contrast, because we are less certain about whether the nontargeted (child and family) outcomes examined in Chapters 9, 11, and 12 are theoretically independent from one another (and, thus, whether we may in part be capitalizing on chance by examining multiple measures of the same or similar underlying constructs), we calculate and report the number of findings we might expect by chance as if they were independent from one another. The proportion of statistically significant impacts across all family outcome measures (in Chapter 9), and by selected categories of these measures, or across all child outcome measures (in each of Chapters 11 and 12) and across all relevant programs was calculated. Specifically, given that the experiment-wise Type I error rate was set at .10, any one result will emerge as significant 10 percent of the time owing to chance alone. The number of chance significant outcomes were calculated and noted in drawing any conclusions about the effects of these welfare-to-work programs.
Some might argue that a more stringent standard is needed, requiring that the number of significant impacts within each program must exceed chance levels or that the number of significant impacts within each domain of child development must exceed chance levels. Because there is a lack of consensus on this issue among statisticians, and given that a goal of the analyses on family and child outcomes was to provide a thorough examination of program impacts, we did not adhere to a more stringent standard.
All impact estimates are based on the entire research sample, including program group members who did not participate in program activities (it is likely that nearly all "nonparticipants" in the program group encountered the program messages and participation mandates, which may have affected their decision to look for work or to leave welfare). Because all sample members are included in the analyses, the impacts must be interpreted as being the results of the welfare-to-work programs as a whole, not only of participation in specific program services. By the same principle, calculations of average earnings and welfare payments — which form the basis of many of the impact estimates — include sample members who were not employed (that is, earned $0) or did not receive welfare (that is, received $0 in welfare). To the extent that a program turns nonearners into earners or encourages welfare recipients to leave welfare, excluding these $0 values from the program and control group averages would lead to seriously biased underestimates of program impacts. For example, previous research has shown that some welfare-to-work programs dramatically increased the proportion of people who have earnings without affecting the average earnings of those who work. These programs led to a relatively large impact on earnings when all sample members were included in the calculation. However, omitting people with $0 earnings from the analysis would have suggested that these programs had no impact on earnings.(25)
Some analyses in this report focus on subgroups of the full impact sample. In one such set of analyses, presented primarily in Chapter 7, each site sample is broken down by various background characteristics (such as previous work history) measured at the time of random assignment. The impacts found for these subgroups can be confidently attributed to the programs under study because they are based on characteristics measured before anyone entered the program and because program and control group members are similar in other respects; that is, the only difference between the program and control subgroups with a particular background characteristic is exposure to the program. In the language of evaluations, such impacts are experimental. However, because they are based on smaller samples of people, the impact estimates for subgroups are less likely to be statistically significant than those for the full sample. Other analyses in the report compare outcomes such as average hourly wages for program and control group members who shared characteristics (such as being employed) acquired after random assignment. These nonexperimental comparisons should be interpreted with caution because the research groups may differ with respect to measured or unmeasured background characteristics that affect employment and, in turn, hourly wages. The report presents findings from such nonexperimental comparisons to explore underlying trends in the experimental impact estimates.
Because data were unavailable, the results for sample members in Atlanta who were randomly assigned during the last six months of the random assignment period are not presented in this report. Similarly, because welfare and Food Stamp data were not available for years 4 and 5 for sample members in Oklahoma City, the year 4, year 5, and cumulative impacts on welfare receipt, Food Stamp receipt, and combined income are not reported for Oklahoma City.
As discussed in Chapter 1, changes in the labor market and the environments in which the programs operated during the follow-up period could have affected program impacts. In particular, the economic expansion that began in the mid 1990s created a strong demand for entry-level jobs nationwide. However, because program and control group members in each site experienced these changes, it is difficult to know ahead of time which group was most affected by them. On the one hand, program impacts on employment and earnings may diminish as more control group members find employment. On the other hand, during an economic expansion welfare-to-work programs may help people advance more quickly to higher-paying or more stable employment, resulting in increasing impacts over time.
In addition, most of the programs in the evaluation became more employment-focused over time. As a result, in the last two years of the follow-up period, people assigned to education-focused programs who remained on welfare received services and messages similar to those that people in the employment-focused programs were exposed throughout the follow-up period. However, people in the education-focused programs were probably little affected by this evolution in program approach because — even under the original program model — many of them would have received job search assistance if they had not found employment after completing education and training activities.(26)
[Go to Contents]
Another development that has direct consequences for assessing the impacts of the NEWWS programs studied concerns changes in the treatment of control group members over time. As discussed in Chapter 1, in five sites some control group members who were still receiving welfare after year 3 became eligible for and were required to participate in welfare-to-work program services. In other words, in these sites the control embargo was no longer in effect in year 4 and/or year 5. (In Riverside and Portland, the control embargo was in force for the entire five-year follow-up period for the control group samples analyzed in this report.) This section discusses the treatment of controls in years 4 and 5 of the follow-up period in detail, to assess the extent to which the early lifting of the control embargo in five sites affected the impact estimates in those sites, if it affected them at all.
Figure 2.4 presents a time line of control group members' eligibility for welfare-to-work program services in each site, by quarter of random assignment. In Atlanta, Columbus, and Oklahoma City, there was a fixed date on which the control group embargo on welfare-to-work program services was lifted. In these three sites, then, controls randomly assigned early in the sample intake period would have had little opportunity for exposure to the programs under study while those randomly assigned later in this period would have had more opportunity. In Grand Rapids, as noted above, controls randomly assigned in 1993 retained their embargo on welfare-to-work program services for a full five years; those randomly assigned prior to 1993 would have had their embargo lifted when they reached the end of their particular three-year follow-up period. In this site, as in Atlanta, Columbus, and Oklahoma City, control group members thus would have had differing amounts of time during the five-year follow-up in which to possibly be exposed to program services. In Detroit, the control group embargo on welfare-to-work program services was lifted when controls reached the end of their three-year follow-up period. Consequently, in this site, all controls would have had an equal amount of time during the full follow-up period for possible exposure to the programs under study.(27) All control group members in Riverside and a randomly selected group of control group members in Portland had a five-year embargo on welfare-to-work program services. (For ease of presentation, the randomly selected group of Portland controls rather than all control group members in the site is shown in Figure 2.4.)
Firgure 2.4
Time Line of Changes in Control Group Members' Eligibility for Welfare-to-Work
Program Services over the Five-Year Follow-Up Period, by Quarter of Random
Assignment and Site
Key: Random assignment quarter Not eligible for welfare-to-work program services Eligible for welfare-to-work program services
NOTE: Control group members shown in Portland are the randomly selected group of 499 for whom a full five-year embargo on welfare-to-work program services was in effect.
The extent to which control group members actually received welfare-to-work program services once their embargo ended is not known. However, for several reasons it is likely that relatively few control group members did so. First, only control group members still receiving welfare when their embargo on welfare-to-work program services was lifted in follow-up year 4 or 5 would have been informed of and required to participate in such services. (Once the control embargo was lifted, controls would have been considered to be mandatory for participation in a welfare-to-work program, as long as they did not meet any of the exemption criteria established under the Family Support Act.) The following proportions of all control group members in each site had four years of "no treatment" follow-up, that is, years with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 86 percent in Atlanta, 67 percent in Grand Rapids, 91 percent in Columbus, and 37 percent in Detroit. Similarly, the following percentages of controls had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services: 54 percent in Atlanta, 66 percent in Grand Rapids, 76 percent in Columbus, and 36 percent in Detroit. Additionally, for the samples used in this report, 100 percent of controls in Riverside and Portland had five years of follow-up with no months in which they were receiving welfare and were eligible for welfare-to-work program services. (Some of the control group members receiving welfare when the embargo was lifted may no longer have been considered mandatory for welfare-to-work program participation, as all would have been as of random assignment, and probably would not have been told about the program or a requirement to participate in it because of changes since study entry in their personal situations, such as illness or disability, employment of more than 20 hours per week, or the birth of child.)(28)
Second, welfare-to-work programs in most sites had a phase-in schedule for assigning people to the programs if they were required to participate in them. It is likely that control group members who were required to participate in the programs would have waited up to six months before being assigned to a program.
Finally, once assigned to a program, control group members would have needed to attend a program orientation, be assigned to a specific program activity, and participate in that activity in order to actually receive welfare-to-work program services. Typically, less than one-half of those assigned to welfare-to-work programs as mandatory participants end up actually participating in welfare-to-work program services.(29)
It should be kept in mind, however, that any encounters with welfare-to-work program staff would have represented control group members' first exposure to a mandatory welfare-to-work program after random assignment. These encounters and possible participation in welfare-to-work program activities may have given control group members an unanticipated boost — to look for work, to pursue education or training, or to change other behaviors. While this boost would have occurred for program group members shortly after random assignment, and their exposure to program staff and participation in program activities would have continued for several years, the possible effects of this boost for control group members toward the end of the follow-up period cannot be ignored.
As discussed more extensively in Chapter 3, available data suggest that the proportion of control group members who actually received welfare-to-work program services in follow-up years 4 and 5 is likely to have been low. According to five-year survey data, which are available for four of the seven NEWWS sites, less than 6 percent of control group members subject to a five-year embargo received any welfare payments during the fifth year after random assignment and reported participating, at some point during the same year, in activities usually uniquely provided by welfare-to-work programs (job search workshops or work experience).(30) In comparison, participation rates measured in the same way were only slightly higher (between 7 and 12 percent, depending on the site) for control group members with a shorter embargo on program services.(31) Measured in a slightly different way, as the proportion of controls who ever received welfare in a "non-embargoed" month and participated in a job club or work experience activity in the same year, at most 15 percent of the Atlanta controls and 7 percent of Grand Rapids controls were likely to have received welfare-to-work program services during the five-year follow-up period.
While the level of participation of the control groups in welfare-to-work program services appears to be low, the behavior of control group members might have been affected by contact with program staff and by the messages they received about the advantages of working as opposed to receiving welfare.(32) In Portland, where a random sample of 499 control group members had an extension of the welfare-to-work program services embargo, it is possible to directly compare the employment and welfare behavior of control group members who had a full five-year embargo on welfare-to-work program services with those who had only a three-year embargo. As discussed more fully in Chapters 4 and 5, there were some differences in the behavior of these two groups. In follow-up year 3, employment levels were almost identical for the two Portland control groups. In years 4 and 5, employment increased much faster for Portland control group members whose embargo ended at the close of year 3 than for those whose embargo lasted the full five years. The employment rates of the former group were 6 percentage points higher in year 4 and 3 percentage points higher in year 5 than those of the latter group. Similarly, average earnings of the former group were $500 higher per year in these two follow-up years. Average amounts of welfare payments received by the two groups of controls in follow-up years 4 and 5 differed by year in direction and by a small amount ($57 less for the longer-term embargo group in year 4 and $186 more in year 5).
The Portland control group comparisons suggest that in this site the lifting of the embargo on welfare-to-work program services after year 3 did affect the behavior of some control group members. For this reason, the randomly selected 499 control group members, who had a five-year embargo on welfare-to-work program services, are used as the control group in all Portland analyses throughout the report.(33)
Unfortunately, a randomly selected alternative control group, similar to the one in the Portland site, does not exist in the other five sites where some control group members were eligible for welfare-to-work program services earlier than the end of the five-year follow-up period. In four of the sites (Atlanta, Grand Rapids, Columbus, and Oklahoma City) certain cohorts of control group members, that is, individuals who were randomly assigned in certain months, had much longer control service embargoes than did other cohorts. (See Figure 2.4.) In these four sites, there is no easy way to determine what effect the lifting of the embargo had on the behavior of controls toward the end of the five-year follow-up period, since the baseline demographics of sample members often differed between cohorts and the employment and welfare behavior observed for the various cohorts was often different even before the control embargo was lifted. In the fifth site, Detroit, all cohorts of controls had an equally long welfare-to-work program service embargo.
Program impacts on employment and earnings and other outcomes in the last two years of follow-up in a few of the five sites probably would have been somewhat larger had some control group members not been exposed to welfare-to-work programs. Impacts would likely have been affected more in follow-up year 5 than in year 4. However, as discussed in Chapter 4 of this report, many programs can continue to have effects long after control embargoes are lifted. Owing to the program group's early exposure to the programs, early gains in employment and earnings can continue in the later years of follow-up, reflecting a "head start" experienced by program group members.(34) This factor, combined with the findings of low year 5 control group welfare receipt and low year 5 control group use of program services, strongly suggests that ending the control group embargo earlier than the end of the five-year follow-up period did not change the impact findings very much. As a result, in this report all control group members in these five sites are included in the estimates of program impacts. Where appropriate, however, impact analyses for follow-up years 1 to 3 are separated from those for years 4 and 5.
Notably, the control group situations described above do not affect the assessments in this report of the relative merits of the Labor Force Attachment and Human Capital Development approaches in welfare-to-work programs. Owing to the research design in the three-way sites, the fact that the control group embargo ended after year 3 in several of these sites does not affect the estimates of the relative effectiveness of the LFA and HCD approaches over five years. Random assignment ensured that the background characteristics of LFA and HCD program group members did not differ systematically at the time of random assignment, which means that the outcomes for the two program groups can be compared directly with one another without taking the control group into account (Figures 2.1 and 2.2).(35)
[Go to Contents]
1. As will be discussed in more detail later in this chapter, in some sites control group members became eligible for program services before the end of the five-year follow-up period.
2. The following hypothetical example of a side-by-side evaluation of two program approaches illustrates these points. Control group members earned a total of $40,000 on average over five years, compared with $40,000 for program group 1 and $35, 000 for program group 2. Direct comparisons of earnings for the two program groups suggest that the first program was relatively more effective than the second, because its members earned $5,000 more on average over five years. However, comparisons with the control group show that neither program was effective because neither raised average earnings above the control group level.
3. The Riverside design has implications for calculating the LFA program impacts. Whereas the outcomes for sample members in the other six sites are unweighted, in Riverside the outcomes are weighted averages of the outcomes for LFA group members found to need or not to need basic education at random assignment. This weighting scheme compensates for the overrepresentation of those determined not to need basic education in the LFA and control groups.
Owing to the Riverside program design, impacts cannot be correctly calculated in an unweighted regression model (that is, one that includes all the sample members in Riverside and gives all observations equal weight). Instead, the LFA impact is calculated as (Wneed * BLFAneed) + (Wnot * BLFAnot). In this equation, BLFAneed represents the impact for the "in-need" LFA group members and BLFAnot the impact for "not-in-need" LFA group members. Wneed, the weight for the in-need sample, equals the fraction of LFA group members, HCD group members, and control group members who were classified by program staff to be in need of basic education at random assignment, and Wnot, the weight for the not-in-need sample, equals 1 - Wneed.
The Riverside LFA impacts were generated using a regression model that included all Riverside sample members, whereas the Riverside HCD impacts were estimated using a regression model that included only LFA, HCD, and control group members determined to need basic education.
For many outcome measures, the report presents the range of control group averages across the seven sites. For Riverside, the average for the entire control group will be included in the range, and not the separate average for control group members in need of basic education that is used to estimate the impacts of the HCD program.
4. Nearly one-quarter of the people in the Riverside in-need subgroup actually had a high school diploma or GED. These people were determined to be in need of basic education because they scored low on the math or reading portion of the appraisal test or were judged by program staff to need English remediation. See also Hamilton et al., 1997.
5. See Hamilton and Brock, 1994, for a more detailed description of the research designs in the seven sites.
6. For a discussion of enrollment practices in the sites, see Chapter 1. See also Hamilton and Brock, 1994, pp. 51-55.
7. See Hamilton and Brock, 1994, for a discussion of the implications of orientation attendance. A separate experimental analysis of the deterrence effects of a participation mandate and reasons for nonattendance was conducted in Riverside and Grand Rapids for the NEWWS Evaluation. For this study people who attended a meeting at income maintenance to determine their eligibility for welfare benefits were randomly assigned when income maintenance workers determined they were subject to the participation mandate. They entered either a "pre-orientation program group" and were assigned to attend a program orientation — or a "pre-orientation control group" and were not assigned. Members of the pre-orientation program group who showed up for their orientation during the sample intake period for this study were randomly assigned a second time — to either a program or control group. Only those who were randomly assigned to a program or control group at program orientation in Riverside and Grand Rapids are included in the analyses presented in this report. See Knab et al., 2001, for estimates of the deterrence effects of assignment to a mandatory welfare-to-work program.
8. Brock and Harknett, 1998; Scrivener and Walter, 2001; and Knab et al., 2001.
9. Although Oklahoma City included nonapplicants in its participation mandate, recipients were not included in the evaluation because including them would have required significant alterations to existing welfare department procedures.
10. Storto et al., 2000.
11. Friedlander, 1988.
12. The sample includes only the 499 control group members in Portland who had a full five-year embargo on the receipt of program services (more information on the control group embargo is included at the end of this chapter). Also, the sample includes only sample members in Atlanta who were randomly assigned between January 1992 and June 1993, excluding those randomly assigned after June 1993.
13. Approximately 15,000 more people were randomly assigned than are in the full impact sample. Excluded from this report's analysis are people randomly assigned before they attended a program orientation as part of the deterrence study, two-parent (AFDC-UP) families, and teen parents in Riverside (who faced different program requirements than older sample members).
14. See Freedman et al., 2000a.
15. The Two-Year Client Survey was conducted in all seven NEWWS Evaluation sites and included 9,675 respondents. For more information, see Freedman et al., 2000a.
16. For the two-year results of the former, see McGroder et al., 2000; for the results of the latter, see Bos et al., 2001.
17. For specific response rates by site and research group, see Appendix Table G.1.
18. Mothers and focal children in 2,594 families responded to the Five-Year Client Survey. A total of 262 of these families were later dropped from the analysis sample. Of these, 203 families had moved out of the survey area by the time of the five-year survey and therefore were not administered the special in-person COS survey sections (a phone interview was conducted to obtain information for the sections of the survey that were administered to all five-year survey sample members; these sample members remain in the five-year survey sample). Fifty-seven families were dropped because the focal child was not the mother's biological child; one duplicate case was dropped; and one family was dropped because the focal child was deceased at the five-year follow-up point.
19. For specific response rates by site and research group, see Appendix Table G.1.
20. A total of 1,489 teachers responded to the teacher survey. Seventeen teacher respondents were dropped from the final analysis sample because they taught focal children who were among the 262 respondents dropped from the COS sample.
21. For specific response rates by site and research group, see Appendix Table G.1.
22. As shown in Table 2.3, single fathers, or the husbands of disabled spouses, make up from 3 to 11 percent of the full impact sample, depending on site. Female pronouns will be used hereafter to describe sample members because most of them are women.
23. Of those who did not take the tests, about one-third did not speak English; the rest were unable to remain on site to be tested, spoke English but were unable to read or write it, or did not take the test for other reasons.
24. However, inferring that there is no impact when an impact really exists is another error of concern. In an effort to guard against this type of error, impacts with a probability between 10 and 20 percent of having arisen by chance are also occasionally discussed in the report, though these findings are referred to as program-control differences rather than impacts. These program-control differences are discussed if they are comparable in magnitude to a statistically significant impact of another program on the same outcome or if the impact appears to be part of a pattern of increases or decreases relative to the control group.
25. See, for example, the discussion of two-year earnings impacts for Riverside LFA in Freedman et al., 2000a, pp. 61-63.
26. See Chapter 3 for a discussion of education-focused program group members' participation in job search activities.
27. While the research design in Detroit specified a full three-year embargo on welfare-to-work program services, 8 percent of all Detroit controls ended up participating in the new Work First program in follow-up year 3. See Farrell, 2000, for details.
28. For example, as discussed in Chapter 9, between 12 and 23 percent of control group members, depending on the site, reported at the five-year follow-up point that a new baby had been added to their household since random assignment; the Family Support Act exempted women with children under age 3 from a mandatory welfare-to-work program participation requirement (or, at state option, women with children under age 1).
29. For a complete description of this process in welfare-to-work programs, see Hamilton, 1995.
30. Several situations could account for this 6 percent. While it is unusual for community college or other non-welfare programs to offer such activities, control group members may have found these programs on their own and voluntarily enrolled in these activities, a practice permitted under the NEWWS research design. NEWWS field research suggests that, over time, more non-welfare agencies in the evaluation sites began to offer job search assistance, particularly in Portland. In addition, while site welfare-to-work program staff were very diligent in screening for control group members at points of welfare application or program enrollment, some of these control group members could represent exceptions, or a few cases where controls "slipped through" the screening process. Field research, however, as well as periodic welfare case file reviews, indicated that the screening procedures in almost all sites were tight and that outside of Detroit very few control group members slipped through them.
31. Expressing these numbers as the proportion of control group members who received any welfare in follow-up year 5 rather than as a proportion of all control group members, among sites or groups of controls with a full five-year embargo on welfare-to-work program services, the proportion participating at some point during year 5 in activities usually uniquely provided by welfare-to-work programs was 10 percent in Grand Rapids, 4 percent in Riverside, and 24 percent in Portland (but the denominator, or the number of Portland controls receiving any welfare in year 5, was small). In contrast, among sites or groups of controls where the embargo on welfare-to-work program services was lifted at some point during the last two years of the five-year follow-up period, the proportion of those receiving any welfare in year 5 who reported similar participation was 18 percent in Grand Rapids and 22 percent in Atlanta. While these two sets of percentages are much higher than those mentioned in the text, the difference between them again suggests that the level of contamination with welfare-to-work program services was only somewhat higher in sites or among groups of controls where the control embargo was lifted during follow-up year 4 or 5 than where the embargo was in effect for the full five-year follow-up.
32. Both control and program group members probably would have been affected by publicity about the 1996 welfare law and, in the three sites where the count toward a welfare time limit began during the five-year follow-up, by the messages conveyed by welfare staff about the urgent need to find a job and leave welfare. In the three sites where the count toward a welfare time limit began during the five-year follow-up (Atlanta, Columbus, and Oklahoma City), the count began at the same time that the control embargo on welfare-to-work program services was lifted.
33. Data that would show direct evidence that the differences in employment behavior between the two control groups in Portland are due to exposure to welfare-to-work programs (evidence such as differences in measured program participation rates for the two groups) are not available. Using the smaller, five-year embargoed control group to calculate impacts, however, provides the "safest" estimates of the true effects of the Portland program.
34. Prior studies of the long-term effects of welfare-to-work programs have demonstrated that even after a control embargo on welfare-to-work program services is lifted programs can continue to have impacts, though perhaps diminishing ones, stemming from a labor market "boost" received by program group members early in the follow-up period. (See, for example, the five-year effects of the 1980s SWIM program in San Diego, presented in Friedlander and Hamilton, 1993.) Many of the NEWWS Evaluation programs examined here provide similar examples.
35. As noted above, direct comparisons between the LFA and HCD programs in Riverside can be made only by comparing the HCD group with those members of the LFA group who lacked a high school diploma or basic skills at random assignment.
Main Page of Report
Contents of Report
Home Pages:
National Evaluation of Welfare-to-Work Strategies
(NEWWS)
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)