Core Performance Indicators for Homeless-Serving Programs Administrated by DHHS

Executive Summary

[ Main Page of Report | Contents of Report ]

Contents

Recent studies suggest that homelessness is a problem that afflicts many adults and children in the nation and can have a broad range of short- and long-term negative consequences. It is estimated that up to 600,000 people in the United States are homeless each night.(1) In developing programs to address the needs of the homeless, it is important to specify clearly the program goals and objectives to guide implementation of program activities, as well as a set of performance measures to facilitate documentation and analysis of the effectiveness of program interventions. This study explores the feasibility of developing a core set of performance measures for DHHS programs that focus on homelessness. It has two main objectives: (1) determine the feasibility of producing a core set of performance measures that describe accomplishments (as reflected in process and outcome measures) of the homeless-specific service programs of DHHS; and (2) determine if a core set of performance measures for homeless-specific programs in DHHS could be generated by other mainstream service programs supported by DHHS to assist low income or disabled persons.

A key focus of the study is on enhancing performance measurement across four homeless-serving programs administered by DHHS:

  1. Programs for Runaway and Homeless Youth (RHY),
  2. The Health Care for the Homeless (HCH) Program,
  3. Projects for Assistance in Transition from Homelessness (PATH), and
  4. The Treatment for Homeless Persons Program (formerly called the Addiction Treatment for Homeless Persons Program).

In addition, this project deals with an important government management requirement that has affected agencies and programs for the past several years: the Government Performance and Results Act of 1993 (GPRA), which requires government agencies to develop measures of performance, set standards for the measures, and track their accomplishments in meeting the standards.

This study mainly involved interviews with program officials knowledgeable about the four homeless-serving programs that were the main focus of this study, along with review of existing documentation. Interviews were conducted both by telephone and in-person. In addition, the research team conducted telephone interviews with program officials at four mainstream programs. Project staff also reviewed documents and interviewed program officials that operated homeless administrative data systems (HADS) or homeless registry systems in several localities across the country.

Characteristics of the Four Homeless-Serving Programs

Although the four homeless-serving programs shared the goal of providing services to the homeless, they also had significant differences. Some major findings from our interviews with program officials and review of documents are:

Differences among the four programs means that it will be a difficult and delicate task to develop a common set of performance measures for the four programs, and even more difficult for those measures to also be applicable to other DHHS programs serving homeless individuals. In addition, while federal agency officials are very willing to discuss their programs and share their knowledge of how they approach data collection and reporting, their willingness and ability to undertake change is uncertain. From our discussions, it appears that changes in how programs collect data and report on performance will require substantial efforts on the part of agency officials and programs. For example, with regard to RHY — which is currently involved in an effort to implement a streamlined data system — it would not only require change at the federal administering agency, but how over 600 grantee organizations collect and manage data.

Analysis of Measures Derived from Homeless Administrative Data Systems (HADS)

With input from DHHS, we selected five HADS (in New York City, Madison, Columbus, Kansas City, and Honolulu) for study. In the Summer 2002, we interviewed (by telephone) system administrators about the operations of each of the five HADS. We also conducted a site visit to New York City’s Department of Homeless Services to interview staff in greater depth and obtain additional background information on the operation of HADS. Major findings from the interviews are:

While the HADS reviewed for this report provide some useful measures of program inputs and process, they do not provide a set of measures of program outcomes or performance (with the possible exception of length of stay) that are readily adaptable to the DHHS homeless-serving programs that are the focus of our overall study. There are, however, some interesting implications that can be drawn from HADS for developing performance measures for DHHS homeless-serving programs and the systems capable of maintaining data that might be collected as part of such systems. Several of the systems we reviewed do collect data on duration of episodes of receipt of homeless services (i.e., length of stay in emergency shelters and transitional facilities). Such a measure is particularly helpful in understanding frequency and total duration of homeless individuals receipt of assistance (e.g., duration of each spell of use of emergency shelters). Such data would be particularly helpful in understanding the extent of chronic homelessness and types of individuals most likely to have frequent and lengthy stays in emergency or transitional facilities. This points to the need to collect client-level data on service utilization, which includes dates that services begin and end so that it is possible to examine duration and intensity of services received, as well as multiple patterns of service use (i.e., multiple episodes of shelter use). The HADS also show that it is possible to collect detailed background characteristics on homeless individuals served, and especially in the case of Hawaii’s HADS, to collect data at the time of entry and exit from homeless-serving programs to support pre/post analysis of participant outcomes.

Potential Core Performance Measures For Homeless-Specific Service Programs

In developing these measures, we took into consideration the following important factors:

Our earlier analysis of the four homeless-serving programs indicated that there are substantial cross-program differences that complicate efforts to develop similar performance measures and systems for collecting data. For example:

Our review of the performance measurement systems in existence across the four programs also indicates potential for both enhancement and movement toward more outcome-oriented measures. For example, the general approach to performance measurement used within the Treatment for Homeless Persons Program provides a potential approach that could be applicable to the other three programs. Of critical importance to our efforts to suggest core measures, all four of the programs are aimed at (1) improving prospects for long-term self-sufficiency, (2) promoting housing stability, and (3) reducing the chances that individuals will become chronically homeless. In addition, the four programs (some more than others) also stress addressing mental and physical health concerns, as well as potential substance abuse issues.

Based on the common objectives of these four programs, we suggest a core set of process and outcome measure that could potentially be adapted for use by the four homeless-service programs (see Exhibit ES-1). We suggest selection of the four process measures, which track numbers of homeless individuals (1) contacted/outreached, (2) enrolled, (3) comprehensively assessed, and (4) receiving one or more core services. We then suggest selection of several outcome measures from among those grouped into the following areas: (1) housing status, (2) employment and earnings status, and (3) health status. In addition, we have suggested a several additional outcome measures that could be applied to homeless youth.

Exhibit ES-1:
Potential Core Performance Measures for DHHS Homeless-Serving Programs
Type of Measure Core Performance Measure When Data Item Could Be Collected Comment
**PROCESS MEASURES**
Process # of Homeless Individuals Contacted/Outreached At first contact with target population  
Process # of Homeless Individuals Enrolled At time of intake/ enrollment or first receipt of program service  
Process Number/Percent of Homeless Individuals Enrolled That Receive Comprehensive Assessment At time of initial assessment May include assessments of life skills, self-sufficiency, education/training needs, substance abuse problems, mental health status, housing needs, and physical health
Process Number/Percent of Homeless Individuals Enrolled That Receive One or More Core Services At time of development of treatment plan, first receipt of program service(s), or referral to another service provider Core services include:
  • Housing Assistance
  • Behavioral Health Assistance (Substance Abuse/Mental Health Treatment)
  • Primary Health Assistance/Medical Treatment

**OUTCOME MEASURES — HOUSING STATUS**

Outcome — Housing Number/Percent of Homeless Individuals Enrolled Whose Housing Condition is Upgraded During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
Possible upgrade categories:
  • Street
  • Emergency Shelter
  • Transitional Housing
  • Permanent Housing
Outcome — Housing Number/Percent of Homeless Individuals Enrolled Who Are Permanently Housed During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 
Outcome — Housing Number/Percent of Homeless Individuals Enrolled Whose Days of Homelessness (on Street or in Emergency Shelter) During the Past Month [or Quarter] Are Reduced
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • HADS systems may provide useful data on shelter use (but not street homelessness)
**OUTCOME-MEASURES — EARNING/EMPLOYMENT STATUS**
Outcome — Earnings Number/Percent of Homeless Individuals Enrolled with Earnings During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly earnings data (matched using SSN) could be useful — though data lags, potential costs, and confidentiality issues
Outcome — Earnings Number/Percent of Homeless Individuals Enrolled with Improved Earnings During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly earnings data (matched using SSN) could be useful — though data lags, potential costs, and confidentiality issues
Outcome — Employment Number/Percent of Homeless Individuals Enrolled Employed 30 or More Hours per Week
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Hours threshold could be changed (20+ hours; 35+ hours); hours worked could be for week prior to survey or avg. for prior month or quarter
  • UI quarterly wage data not helpful (hours data not available); so follow-up survey probably needed
Outcome — Employment Number/Percent of Homeless Individuals Enrolled with Increased Hours Worked During the Past Month [Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly wage data not helpful (hours data not available); so follow-up survey probably needed
**OUTCOME MEASURES — HEALTH STATUS**
Outcome — Substance Abuse Number/Percent of Homeless Individuals Enrolled and Assessed with Substance Abuse Problem That Have No Drug Use the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Drug screening could be used
Outcome — Physical Health Status Number/Percent of Homeless Individuals Enrolled Assessed with Physical Health Problem That Have Good or Improved Physical Health Status During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • May be difficult to objectively measure “good or improved”
Outcome — Mental Health Status Number/Percent of Homeless Individuals Enrolled Assessed with Mental Health Problem That Have Good or Improved Mental Health Status During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • May be difficult to objectively measure “good or improved”
**OUTCOME MEASURE — YOUTH-ONLY**
Outcome — Family Reunification Number/Percent of Homeless & Runaway Youth Enrolled That Are Reunited with Family During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Reunification may not always be an appropriate outcome — and it is often hard to know when it is
Outcome — Attending School Number/Percent of Homeless Youth Enrolled That Attended School During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 
Outcome — Completing High School/GED Number/Percent of Homeless Youth Enrolled That Complete High School/GED During Past Quarter
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 

With regard to housing outcomes, we have identified three potential outcome measures intended to track (1) changes in an individual’s housing situation along a continuum (from living on the street and in emergency shelters to securing permanent housing), (2) whether the homeless individual secures permanent housing, and (3) days of homelessness during the preceding quarter (or month). Two earnings measures are identified — one that captures actual dollar amount of earnings during the past quarter (or month) and a second measure that captures whether an individual’s earnings have improved. Two employment measures are also identified — one relating to whether the individual is engaged in work 30 or more hours per week and another that measures whether hours of work have increased. Three health-related measures are offered, focusing on use of drugs, improvement in physical health status, and improvement in mental health status. Finally, three measures are offered that are targeted exclusively on youth (though the other outcome measures would for the most part also be applicable to youth): (1) whether the youth is reunited with his/her family, (2) whether the homeless youth is attending school, and (3) whether the homeless youth graduates from high school or completes a GED.

A pre/post data collection approach is suggested with respect to obtaining needed performance data — for example, collecting data on housing, health, and substance abuse status of program participants at the time of intake/enrollment into a program and then periodically tracking status at different points during and after program services are provided (i.e., at termination/exit from the program and/or at 3, 6, or 12 months after enrollment). Collection of data on homeless individuals at the point of termination can be problematic because homeless individuals may abruptly stop coming for services. The transient nature of the homeless population can also present significant challenges to collecting data through follow-up surveys/interviews after homeless individuals have stopped participating in program services..

Given difficulties of tracking homeless individuals over extended periods, the extent to which existing administrative data can be utilized could increase the proportion of individuals for which it is possible to gather outcome data (at a relatively low cost). Probably the most useful source in this regard is quarterly unemployment insurance (UI) wage record data, which can be matched by Social Security number (though releases are required and it may also be necessary to pay for the data). A second potential source of administrative data that may have some potential utility for tracking housing outcomes are HADS system maintained by many states and/or localities. HADS systems are not used principally for measuring program performance or outcomes, but they have the capability to provide analyses of length of stay.

Finally, in terms of tracking self-sufficiency outcomes, data sharing agreements with state and local welfare agencies may provide possibilities for tracking dependence on TANF, food stamps, general assistance, emergency assistance, and other human services programs.

Application of Suggested Core Performance Measures To DHHS Mainstream Programs Serving Homeless Individuals

With input from the DHHS Project Officer, we selected four DHHS mainstream programs for analysis: (1) the Health Centers Cluster (administered by Health Resources and Services Administration [HRSA]), (2) the Substance Abuse Prevention and Treatment (SAPT) Block Grant (administered by Substance Abuse and Mental Health Services Administration [SAMHSA]), (3) Head Start (administered by Administration for Children and Families [ACF]), and (4) Medicaid (administered by the Centers for Medicaid and Medicare Services [CMS]). While these programs are not targeted specifically on homeless individuals, some homeless individuals are eligible for services provided under each program by virtue of low income, a disability, or other characteristics. In comparison to the four homeless-serving programs, the mainstream programs:

Despite some differences, there are commonalities in terms of program goals and services offered by mainstream and homeless-serving programs. Three of the four mainstream programs (Medicaid, SAPT, the Health Centers Cluster) focus program services primarily on improving health care status of low-income individuals. Two of the programs — Medicaid and the Health Centers Cluster – are aimed directly at delivery of health care services to improve health care status of low-income and needy individuals. Though more narrowly targeted on homeless individuals, HCH and PATH are similarly aimed at improving health care status of the disadvantaged individuals. The third mainstream program — SAPT — aims at improving substance abuse treatment and prevention services. Under SAPT, block grants funds are distributed to states, territories, and tribes aimed at the development and implementation of prevention, treatment, and rehabilitation activities directed to diseases of alcohol and drug abuse. In terms of program goals and services, Head Start is quite different from the three other mainstream programs and the four homeless-serving programs. The Head Start program is aimed principally at increasing school readiness and social competence of young children in low-income families. Our main findings from the review of mainstream programs are:

Recognizing the difficulties faced by the mainstream programs in making changes to their well-established data sets, it would be very useful to work with mainstream DHHS programs to: (1) add a single data element to data systems that would capture living arrangement or homeless status at the time of program enrollment in a consistent manner across programs; (2) provide the mainstream programs with a common definition of what constitutes “homelessness” and, if possible, the specific question(s) and close-ended response categories that programs should use in tracking homelessness; and (3) if mainstream programs conduct a follow-up interview or survey with participants, request that they include a follow-up question relating to homelessness or living arrangement.

For all four of the mainstream programs and the four homeless-serving programs, a step beyond collecting homeless status or living arrangement at the time of enrollment would be to collect such data at the time of exit from the program or at some follow-up point following enrollment or termination from the program. However, determining a convenient follow-up point to interact with the participant may be difficult or impossible in these programs. With regard to collecting homeless or living arrangement status at a follow-up point, it may be best to focus (at least initially) on implementing such follow-up measures in the homeless-serving programs, where long-term housing stability is a critical program objective.

Finally, where collection of information about homeless status either at the time of enrollment or some follow-up point prove either impossible to obtain or too costly, DHHS should consider potential opportunities for collecting data on homelessness as part of special studies or surveys. Several of the mainstream programs (as well as the homeless-serving programs) are periodically the subject of either special studies or survey efforts. For example, the Head Start program has implemented the FACES survey, which is conducted in 3-year waves on a sample of over 3,000 children and families served by 40 Head Start centers. Working with a sample, rather than in the universe in large programs such as Head Start (nearly 1 million children) and Medicaid (about 40 million beneficiaries) has great appeal from the standpoint of reducing burden and data collection costs.

Endnotes

(1) An estimate provided by the U.S. Department of Health and Human Services website (see http://aspe.hhs.gov/homeless/).

(2) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(3) For example, New York City had over 800,000 records in its system, and Kansas City had 450,000 records.


Where to?

Top of Page | Contents

Main Page of Report | Contents of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation ASPE)
U.S. Department of Health and Human Services (HHS)