SUMMARY: EVALUATION OF WISCONSIN'S TWO-TIER BENEFIT DEMONSTRATION
Background
In July 1992, Wisconsin was granted waivers to pilot test, in four counties, a two-tiered benefit policy under which individuals moving to Wisconsin and applying for AFDC cash assistance would have their eligibility and benefit payment calculated on the basis of the rules and payment levels of their prior state of residence.
Wisconsin sought approval to conduct this demonstration because there was concern that the state's more generous benefits for poor families, including significantly higher cash payments than their border state Illinois, significantly contributed to in-migration from other states.
Earlier research regarding the issue of "welfare magnet" states produced inconsistent conclusions about the importance of various factors (e.g., family members living in the new state, job prospects, affordable housing, safety, welfare benefits) contributing to interstate movement.
Key Features of the Demonstration
The demonstration was implemented in July 1994 in Milwaukee County and three other southeastern border counties: Racine, Kenosha, and Rock.
New AFDC applicants moving into these counties received AFDC based on the level of benefits in their previous state of residence, whether higher or lower than the Wisconsin amount, from the time they were found eligible until they had lived in Wisconsin for six months. The policy did not apply to families which had previously resided in Wisconsin.
The evaluation attempted to analyze the extent to which the level of welfare benefits, particularly the amount of cash assistance under AFDC, as opposed to other reasons, was influencing the decision of low-income families to move to Wisconsin.
The evaluation methodology included a non-experimental impact analysis based on a caseload trend analysis and telephone surveys of persons who had moved to Wisconsin and who applied for AFDC within six months of moving.
Findings
The trend analysis showed that in Milwaukee County (a demonstration site), there was a statistically significant decline after the demonstration was implemented in the number of in-migrant new AFDC approvals, compared to the levels that were expected from the baseline models.
Similarly, the number of in-migrant new AFDC approvals during the demonstration was also significantly below expected levels based on the predictive model in the three smaller demonstration counties (Racine, Kenosha, and Rock).
There was no difference between the predicted and actual caseload values in surrounding non-demonstration counties considered likely "spillover" sites where in-migrants might move to avoid the two-tier policy.
Among the overall sample responding to the telephone survey, there was a statistically significant decline after the demonstration was implemented in the percentage of respondents who cited welfare-related reasons for moving to Wisconsin. In the baseline period, 28.2 percent of respondents cited a welfare-related reason; this percentage decline to 21.2 percent during the demonstration period.
Among respondents who moved to a demonstration county, the percentage who cited a welfare-related reason for moving fell from 41.8 percent to 26.4 percent.
Among respondents who moved to a non-demonstration county, the percentage who cited welfare-related reasons for moving did not change significantly. The percentage was 18.9 percent at baseline and 17.7 percent during the demonstration.Interpretations
The evaluation findings are based on non-experimental impact analyses which utilized caseload trend modeling and telephone surveys. While the impact analyses suggest that the demonstration policy did have its intended effect, there are serious weaknesses in the evaluation methods which diminish the strength of the modeling and the findings based on the caseload trend predicted by the model. Further, lack of analysis of response bias lessens the credibility of the findings based on the survey.
The final report indicates that there were different models used for various counties or groups of counties, rather than a single model with dummy variables representing different program approaches. In addition, different variables were used to estimate what the caseload in different counties would have been in the absence of the demonstration and no out-of-sample testing was able to be conducted. As stated, this approach tends to diminish the strength of the modeling and the findings based on
the caseload trend predicted by the models. Moreover, the R-square values of the various models are low for forecasting models (i.e., below .8).
The weaknesses in the modeling place a greater burden on the findings from the telephone survey to answer the questions about the significance that welfare benefits play in decisions to move from one state to another. However, there was a very low, below 50 percent, response rate reported for the survey. Thus, the confidence which can be placed in the survey findings is greatly reduced. Such a low response rate introduces the possibility of bias in the reported findings. In the final report, the evaluators attempted to address this concern by conducting analyses based on client characteristics. However, they report that they were not able to conduct a very thorough analysis because the data file contained only limited characteristic information. The limited analyses, based on county to which the individual had moved, the state from which they had moved, and their age, did not reveal any non-response bias. While this additional analysis is helpful, limited confidence can be placed in findings based on responses from only about one-half of the sample.
The evaluation report places some appropriate caveats on the findings. However, in the draft report these caveats were fully discussed in Chapter IV of the report. Since it is likely that many readers will not read the full report to get to this discussion, we recommended that all caveats be included in the Executive Summary as well. The final report includes a discussion of some of the major limitations of the study methodology in the Executive Summary. However, it is likely that many non-technical readers may not fully appreciate the implications of such methodological weaknesses when interpreting the reported findings.