 |
Office of the Assistant Secretary for Planning and Evaluation
|
|
Policy Information Center
|
Performance Improvement 1995
Evaluation Activities of the Public Health Service
Contents
Making a Difference: Interim
Status Report of the McKinney Research Demonstration Program for Homeless
Mentally Ill Adults
Report to Congress on the Indian
Health Service With Regard to Health Status and Health Care Needs of American
Indians in California in Response to Public Law 100-713
Developing Effective Health
Communication Strategies for High-Risk Youth Outside of School
Evaluating Educational Outcomes
of School Health Programs
Evaluating the Effects of
School-Based Intervention Programs To Prevent Teenager Drug Use and Abuse
Indoor Allergens: Assessing and
Controlling Adverse Health Effects
HIV Service Networks in Four
Rural Areas
Outreach/Risk Reduction Strategies
for Changing HIV-Related Risk Behaviors Among Injection Drug Users
Adverse Events Associated With
Childhood Vaccines: Evidence Bearing on Causality
A Study of the Economic Underpinning
of Vaccine Supply
Advanced Practice Nursing Education:
Strategies for the Allocation of the Proposed Graduate Nursing Education
Account
Evaluation of Centers for Disease
Control and Prevention (CDC) and Agency for Toxic Substances and Disease
Registry (ATSDR) Training Activities
Evaluation of the Morbidity
and Mortality Weekly Report Series
Foreword
Continuous improvement in government
is a central theme of the Clinton Administration. As we move toward a balanced
Federal budget, funds available for all Departments will be increasingly
constrained. All programs must become more efficient and must demonstrate
their effectiveness in improving the lives of all Americans.
The Department of Health and Human Services (DHHS) is committed
to examining and redirecting its programs to ensure that its customers--the
American people--are being well served. In the spring of 1993, shortly after
Vice President Gore initiated the National Performance Review, the Continuous
Improvement Program (CIP) was established in DHHS. The CIP calls on all DHHS
employees to work together as agents of change to create a more
customer-oriented, cost-efficient, and effective government. It is important
to ensure that decisions for change are based on sound information about
programs and policies. A solid evaluation program can provide the information
needed to guide constructive change, including programmatic, budgetary, and
policy decisions.
The U.S. Public Health Service (PHS) recognizes the importance
of evaluation in improving the health and quality of life for all Americans.
PHS supports and encourages evaluation efforts that--
-
determine whether programs and services achieve their desired effect and
how to improve them;
-
measure the success of programs in reaching and helping target populations
or communities;
-
communicate knowledge about program successes and other lessons learned to
the public health community; and
-
develop or refine effective evaluation tools.
PHS Evaluation Program studies have had a major impact on
planning, budgeting, and legislative development. For example, a 1989 National
Academy of Sciences evaluation of the training and supply of biomedical and
behavioral scientists helped the National Institutes of Health (NIH) determine
the number of positions to be supported by National Research Service Awards
(NRSAs) and justify NIH training budget requests to meet future needs for
biomedical and behavioral scientists. In 1993, the Centers for Disease Control
and Prevention (CDC) evaluated the impact of an influenza vaccination program
for the elderly, documenting the cost-effectiveness of a vaccination promotion
campaign for reducing incidence of secondary acute respiratory bacterial
and viral infections. The results influenced the Department's decision to
make influenza vaccine reimbursable by Medicare as a cost-effective clinical
preventive service.
In addition, the PHS Evaluation Program has played a role
in supporting development of evaluation tools for use by the larger public
health community. In 1991, the National Research Council's Committee on AIDS
Research and the Behavioral, Social, and Statistical Sciences produced a
series of reports on strategies for evaluating CDC's AIDS prevention programs.
These evaluation methodology reports first helped provide the CDC with a
blueprint for designing more effective health communication strategies for
AIDS prevention. The reports also made these evaluation tools available to
public health practitioners, enabling them to assess the effectiveness of
AIDS prevention interventions.
In keeping with the PHS commitment to produce high-quality,
useful evaluations, we are pleased to present Performance Improvement
1995: Evaluation Activities of the Public Health Service. This report
is the first in a series of annual reports documenting evaluation efforts
across all PHS Agencies and Offices. The report has three audiences:
decisionmakers who need information on program results across the broad spectrum
of public health activities; program managers who want to know how other
public health programs are operating and how performance can be improved;
and the community of researchers, advocates, and practitioners who will use
the program information and the evaluation tools.
The report contains information on projects completed in fiscal
year (FY) 1994 and their applications. For example,
-
Research on adverse events associated with childhood vaccines is being used
to educate parents about vaccine-associated risks and to guide policies related
to childhood immunization and victim compensation.
-
An Institute of Medicine report on the impact of nonindustrial indoor allergens
spurred the creation of a new research program and numerous recommendations
for patient education, treatment, and methods to improve indoor environments.
-
A school health program evaluation produced designs that school systems can
use to evaluate school health interventions and their impact on student academic
performance and health.
This report samples the diverse array of evaluation studies
conducted by PHS in 1994 and provides information on evaluations in progress.
The report presents a comprehensive picture of the PHS Evaluation Program--its
past, present, and expected future activities.
Future studies will place even greater emphasis on outcome
or impact evaluations, increased methodological and scientific rigor, and
projects that are crosscutting and provide widely applicable results. Projects
also must meet information needs outside the PHS. They must contribute to
our national understanding of how public health and preventive approaches
improve the health of all Americans, reduce the risks to health, and enhance
the accessibility and availability of care.
Program Performance 1995 is organized into three chapters.
Chapter I describes the PHS Evaluation Program--its activities, funding,
planning and management, and future directions. Chapter II highlights the
results from 13 FY 1994 evaluations selected for their potential application
by the public health community. Chapter III presents the evaluation activities
of the seven PHS Agencies and the Office of the Assistant Secretary for Health
(OASH), including information on their evaluation programs, evaluations completed
in FY 1994, evaluations in progress, and future directions. A complete inventory
of the 71 PHS evaluation projects completed in FY 1994 is provided in appendix
A, and PHS Agency projects in progress are listed in appendix B. Review criteria
used by the special panel of senior editorial advisors for assessing program
evaluations can be found in appendix C.
We hope that you will find this report useful and
informative.
Donna E. Shalala
Secretary
U.S. Department of Health and Human Services
Philip R. Lee, M.D.
Assistant Secretary for Health
U.S. Public Health Service
Senior Editorial Advisors
The following individuals provided editorial guidance in the
development of Performance Improvement 1995:
David Blumenthal, M.D., M.P.P.
Chief, Health Policy Research and Development Unit
Massachusetts General Hospital
Medical Practices Evaluation Center
Boston, Massachusetts
Joel Cantor, Ph.D.
Director of Evaluation
Robert Wood Johnson Foundation
Princeton, New Jersey
Eleanor Chelimsky, Ph.D.
President
American Evaluation Association
Washington, D.C.
Gordon H. DeFriese, Ph.D.
Professor of Social Medicine, Epidemiology and Health Policy and
Administration
Cecil G. Sheps Center for Health Services Research
The University of North Carolina at Chapel Hill
Chapel Hill, North Carolina
Molly Engle, Ph.D.
School of Medicine
University of Alabama-Birmingham
Birmingham, Alabama
Karen Glanz, Ph.D., M.P.H.
Cancer Research Center of Hawaii
University of Hawaii
Honolulu, Hawaii
Lawrence W. Green, Dr.P.H.
Director, Institute of Health Promotion Research
Professor, Department of Health Care and Epidemiology
University of British Columbia
Vancouver, B.C., Canada
Margaret A. Hamburg, M.D.
Commissioner
New York City Health Department
New York, New York
Holly Korda, Ph.D.
Health Services Policy and Research
Cambridge, Massachusetts
John Kralewski, Ph.D.
Director, Institute for Health Services Research
School of Public Health
University of Minnesota
Minneapolis, Minnesota
Mark Krushat, Ph.D.
Office of the Inspector General
Department of Health and Human Services
Washington, D.C.
Kenneth McLeroy, Ph.D.
Health Promotion Sciences Department
University of Oklahoma
Oklahoma City, Oklahoma
Ricardo Millett, Ph.D.
Director of Evaluation
W.K. Kellogg Foundation
Battle Creek, Michigan
Joseph P. Newhouse, Ph.D.
John D. and Catherine T. MacArthur
Professor of Health Policy and Management
Director, Division of Health Policy Research and Education
Harvard University
Boston, Massachusetts
Cheryl Perry, Ph.D.
Division of Epidemiology
University of Minnesota
Minneapolis, Minnesota
Ira Raskin, Ph.D.
Health Policy Consultant
EEI
Alexandria, Virginia
Debra Rog, Ph.D.
Center for Mental Health Research
Vanderbilt University
Washington, D.C.
Mary Ann Scheirer, Ph.D.
Evaluation Consultant
Washington, D.C.
Lee Sechrest, Ph.S.
Department of Psychology
University of Arizona
Tucson, Arizona
In addition to reviewing the report, Drs. Chelimsky, Engle,
Glanz, Korda, Kralewski, Krushat, McLeroy, Raskin, Rog, Scheirer, and Sechrest
served on a panel charged with identifying the projects that would be highlighted
in this report.
Acknowledgments
Performance Improvement 1995: Evaluation Activities of
the Public Health Service describes the continuous efforts of the PHS
Agencies to examine public health programs for the efficiency of their operations
and their effectiveness in achieving their objectives. The planning, development,
and coordination of those evaluations is largely the responsibility of the
following PHS planning and evaluation offices:
Agency for Health Care Policy and Research
Office of Program Development
Phyllis Zucker, Director
Centers for Disease Control and Prevention
Office of Program Planning and Evaluation
Martha Katz, Director
Food and Drug Administration
Office of Planning and Evaluation
Paul L. Coppinger, Associate Commissioner
Health Resources and Services Administration
Office of Planning, Evaluation, and Legislation
Ronald H. Carlson, Associate Administrator
Indian Health Service
Office of Planning, Evaluation, and Legislation
Luana Reyes, Director
National Institutes of Health
Office of Science Policy and Technology Transfer
Lana R. Skirboll, Director
Office of the Assistant Secretary for Health
Office of Disease Prevention and Health
Promotion and Health Planning and Evaluation
Susanne A. Stoiber, Acting Deputy Assistant Secretary for Health
Substance Abuse and Mental Health Services
Administration
Office of Policy and Program Coordination
Frank Sullivan, Director
Office of Applied Studies
Daniel Melnick, Director
Preparation of this report and the appendixes was coordinated
by staff in the Office of Disease Prevention and Health Promotion and Health
Planning and Evaluation, under the supervision of J. Michael McGinnis, M.D.,
Assistant Surgeon General. Staff responsible include James Harrell; Paul
Johnson, Ph.D.; Janice Radak; Melanie Timberlake; and Valerie Welsh. Development
and production of the report was supported under contract by the services
of Muriel Levin, Ed.D.; the staff of EEI; and Miriam Davis, Ph.D., special
consultant.
The following persons from PHS Agencies contributed to the
writing, collection of information on evaluation activities, and reviews
of the report:
Agency for Health Care Policy and Research
Irma Arispe, Ph.D.
Centers for Disease Control and Prevention
Wilma Johnson
Nancy Cheal
Food and Drug Administration
Tim Hegarty
Mary Gamunev
Health Resources and Services Administration
Anabel Crane
Cassandra Lyles
Indian Health Service
Linda Arviso-Miller
Leo Nolan
National Institutes of Health
John Uzzell
David Chananie, Ph.D.
Anita Harris
Office of the Assistant Secretary for Health
Andy Lefton
Barbara Secka
Kirby Weldon
Substance Abuse and Mental Health Services
Administration
Anna Marsh, Ph.D.
Andrew Freeman
Executive Summary
Performance Improvement 1995:
Evaluation Activities of the Public Health Service is the first annual
report on evaluation activities of the U.S. Public Health Service (PHS).
As a report to Congress, it summarizes the findings of PHS evaluations completed
in fiscal year (FY) 1994. In general, the report is intended for three audiences:
decisionmakers who need information on program results across the broad spectrum
of public health activities; program managers who need information on how
they can make improvements in program operations and outcomes; and the public
health community, which can benefit by applying the knowledge and lessons
learned from PHS evaluations.
Evaluation plays an important role in the execution of the
Public Health Service mission. Every year PHS programs are engaged in efforts
to determine whether programs and services reach intended populations or
communities
and achieve desired results. The information obtained is applied to how those
programs and services can be improved. The theme of this report is
performance improvement, which is the focus of Secretary Donna Shalala's
Continuous Improvement Program (CIP), involving numerous changes and initiatives
throughout the Department of Health and Human Services to increase the
effectiveness and efficiency of public health programs. Of the approximately
$14 billion in the FY 1994 budget for program activities, PHS Agencies used
almost $27 million to conduct evaluations useful for understanding the outcomes
and improving the performance of PHS programs. In FY 1994, PHS Agencies produced
71 evaluation reports and supported more than 180 evaluation projects in
progress.
Chapter I
Chapter I provides a description of the PHS Evaluation Program.
The range of evaluation activities supported by PHS Agencies includes outcome,
impact, or process evaluations; policy assessments; cost-benefit or
cost-effectiveness analyses; survey data analyses; management studies; and
evaluation syntheses. Evaluation methodology projects, such as evaluation
feasibility and instrument development studies, are also performed. Other
project support activities include evaluation technical assistance, report
dissemination, training, and conferences. In this overview, the policies
and management of the PHS Evaluation Program are described, including annual
project planning procedures, funding levels from FY 1990 to FY 1994, quality
assurance systems, dissemination mechanisms, procedures for application of
evaluation results, and future directions for PHS evaluation activities.
Chapter II
Chapter II highlights 13 FY 1994 PHS evaluations selected
from recently completed studies covering six broad topic areas.
Access to Health Care for Special Populations
Making a Difference: Interim Status Report of the McKinney
Research Demonstration Program for Homeless Mentally Ill Adults. The
findings of this demonstration project indicate that provision of coordinated
multidisciplinary services can significantly reduce homelessness and improve
the mental health status of participants. Data were collected from participants
along five dimensions (psychiatric symptomatology, substance abuse, quality
of life, physical health, and residential stability) before and after the
various interventions were offered at the six study sites. The data indicate
that homeless people who are mentally ill will use services if they are
accessible and targeted to their needs, that these services will enable people
to find permanent housing, and that formerly homeless people with mental
illness can be valuable staff resources.
Report to Congress on the Indian Health Service With Regard
to Health Status and Health Care Needs of American Indians in California
in Response to Public Law 100-713 analyzes data from vital statistics,
databases, and patient records from tribal clinics to assess the health status
and access to health care services of American Indians from federally and
non--federally recognized tribes in California. Indians from non-federally
recognized tribes were found to have a poorer health status than Indians
from federally recognized tribes, and the health status of both groups was
substantially lower than that of other populations in the State.
Adolescent Health Problems
Developing Effective Health Communication Strategies for
High-Risk Youth Outside of School reports on a focus group study that
questioned 160 high-risk youths aged 10 to 18 years about their health practices
(tobacco use, substance use, unprotected sex, and violence). Although the
youths were fairly knowledgeable about health practices, they did not incorporate
this awareness into their behavior. They were willing, however, to listen
and follow the advice of trusted, credible adults in alternative settings.
Evaluating Educational Outcomes of School Health Programs
is a followup study to a large-scale, randomized, controlled field trial
that assessed the effectiveness of a Life Skills Training course. Fifteen
class periods were devoted to teaching social resistance and other personal
and coping skills to seventh-grade students in New York, with booster classes
offered in the eighth and ninth grades. Findings indicated that the probability
of students' smoking, drinking immoderately, or using marijuana was as much
as 40 percent lower for students who received the intervention and the
probability of using multiple drugs was as much as 60 percent lower.
Evaluating the Effects of School-Based Intervention Programs
To Prevent Teenager Drug Use and Abuse offers a blueprint for evaluations
of health programs for youths and for assessing the effect these programs
have on academic performance. Most studies reviewed concentrated on health
outcomes. A few studies, however, examined educational outcomes, demonstrating
improvements in basic academic skills, reductions in tardiness and absenteeism,
and reductions in drug use. Two designs for evaluating school health
interventions are proposed: analysis of existing data collected from previous
evaluations, and large-scale multischool demonstrations using experimental
and control group designs and random assignment.
Environmental Health
Indoor Allergens: Assessing and Controlling Adverse Health
Effects, performed by the National Academy of Sciences, is an extensive
review of the literature assessing the relationship between indoor allergens
and allergic diseases. Findings indicate that African Americans and children
are at greatest risk for developing complications associated with indoor
allergens; that children under 18 years old account for nearly half of all
emergency room visits for asthma; and that asthma mortality rates for African
Americans are two to three times greater than the rates for whites. Simple
steps such as regular washing of bedding, keeping carpeting to a minimum,
and use of special filters can significantly reduce the presence of indoor
allergens.
HIV/AIDS
HIV Service Networks in Four Rural Areas examines how
HIV services are organized and delivered in rural areas with low and high
AIDS prevalence and provides a portrait of HIV/AIDS epidemiology, HIV-related
service networks, barriers and gaps in service, and funding mechanisms. A
typology classifies areas based on degree of rurality, AIDS prevalence, and
the epidemiological and demographic characteristics of the infected populations.
Diagnosis and treatment of HIV infection are found to be significantly hampered
by stigma and lack of physician knowledge.
Outreach/Risk Reduction Strategies for Changing HIV-Related
Risk Behaviors Among Injection Drug Users looks at efforts to reduce risky
behaviors of injection drug users (IDUs). Interventions at 28 sites were
studied. Outcome measures looked at needle risk behavior, sexual risk behaviors,
and total frequency of drug injection. Data were collected on participant
demographics, and on behaviors and knowledge before and after the interventions.
Following interventions, IDUs reported significant increases in use of new
needles, bleach for cleaning injection equipment, and condoms. Changes in
needle-related behaviors were more dramatic than changes in sexual
behaviors.
Immunization
Adverse Events Associated With Childhood Vaccines: Evidence
Bearing on Causality reports the findings of an Institute of Medicine
expert panel that reviewed published books, articles, and abstracts and listened
to public testimony to determine whether a causal relationship exists between
specific vaccines and adverse effects in children. Of the 49 conditions in
which adverse effects from childhood vaccines were suspected, evidence in
12 cases substantiated a causal relationship; evidence in 4 cases rejected
a causal relationship; and evidence in the remaining 33 cases was equivocal.
The committee concluded that the available data often were insufficient to
establish or reject causality, and the risk of dying or developing serious
neurologic or immunologic complications appeared
to be very low.
A Study of the Economic Underpinning of Vaccine Supply
examined issues related to the economics of production and distribution of
vaccines. The study investigated (1) alternative models for purchase and
distribution; (2) the effect of scientific advances on research, development,
and purchase of vaccines; (3) the application of economic theory to the vaccine
market; (4) comparisons of distribution systems managed by manufacturers
and State agencies; and (5) the implications of purchasing vaccines from
foreign firms. Findings indicate that supplying vaccines at low prices to
physicians, encouraging parents to have their children vaccinated, and having
effective and convenient distribution networks help to increase vaccination
rates.
Public Health Infrastructure
Advanced Practice Nursing Education: Strategies for the
Allocation of the Proposed Graduate Nursing Education Account. The rapidly
changing health care delivery system is expected to cause a shortage of advanced
practice nurses (APNs) in the near future. The study analyzed policy options
on APN student characteristics, supply, overall education costs, and financing.
The advantages, disadvantages, and costs of each option for ensuring a stable
source of funds to support APN education were discussed.
Evaluation of Centers for Disease Control and Prevention
(CDC) and Agency for Toxic Substances and Disease Registry (ATSDR) Training
Activities. The study, using focus groups and site visits, examined the training
needs of local health departments. It was found that local health departments
were deficient in identifying their training needs and needed training in
advocacy, evaluation, and public information. Anticipating increasingly limited
resources at CDC and local agencies, recommendations emphasized the use of
new training technology such as interactive software and satellite
communications.
Evaluation of the Morbidity and Mortality Weekly Report
(MMWR) Series. A mail/telephone survey of MMWR readers
revealed that the CDC publication is read with great regularity, that each
issue is passed along to several readers, and that most people have been
reading the report for more than 5 years. Primary care physicians use the
information to update their knowledge about diagnosis, treatment, and disease
outbreaks. The MMWR was credited for its accuracy, relevance, and
concise reporting. Suggestions for change include improved electronic access
and larger page size.
Chapter III
Chapter III provides an overview of the evaluation activities
of the eight PHS Agencies--presenting information on the evaluation program,
a summary of evaluations completed in FY 1994 and evaluations in progress,
and a discussion of future directions for Agency evaluations.
Agency for Health Care Policy and Research
The goals of the Agency for Health Care Policy and Research
(AHCPR) evaluation program are to assess the Agency's effectiveness in meeting
major and long-term priorities and goals; obtain information quickly to respond
to critical Agency, PHS, and departmental concerns; and conduct internal
evaluations to improve the efficiency of key program areas. Several evaluation
mechanisms are used to achieve these goals: special evaluation studies; peer
review of grant applications; the User Liaison Program, which provides
information on the value of AHCPR-supported research to State policymakers;
and other efforts such as focus groups to provide feedback on AHCPR products.
During FY 1994, six evaluations were completed by AHCPR on two topics: the
process of developing clinical practice guidelines and the evaluation of
health care delivery. AHCPR currently supports 11 projects to facilitate
improvements in health care quality and continues its commitment to improving
the quality of AHCPR-supported clinical practice guidelines. For the future,
AHCPR's programs must generate the information and tools needed to improve
health care delivery and its outcomes and translate research findings into
forms of information that actively assist consumers, practitioners, payers,
and others in making effective health care decisions.
Centers for Disease Control and Prevention and Agency for Toxic Substances
and Disease Registry
The Centers for Disease Control (CDC) places a high priority
on evaluations seeking to answer policy, program, and strategic planning
questions related to the mission of the Agency. Evaluations typically focus
on programs to monitor the health of populations and communities, investigate
health or disease problems, develop public health policies, implement prevention
programs, promote healthy behaviors, foster safe environments, and provide
public health leadership and training. The Agency for Toxic Substances and
Disease Registry (ATSDR) evaluation program focuses largely on internal studies
of its mission of environmental protection established by the 1986 Comprehensive
Environmental Response, Compensation, and Liability Act (the Superfund).
CDC completed 12 evaluations in FY 1994 in the areas of training and information
dissemination, surveillance, program effectiveness, prevention, and costs
of disease. Two evaluations are highlighted in chapter II: an evaluation
of CDC and ATSDR training activities and a survey of readers of the
MMWR. CDC has a total of 32 evaluations in progress in four areas:
surveillance and data collection studies; program evaluations;
community/intervention effectiveness studies; and evaluation methodology
studies. In the future, CDC will be initiating projects designed to provide
data for performance indicators and to assess the effectiveness and efficacy
of such indicators.
Food and Drug Administration
Evaluation at the Food and Drug Administration (FDA) is largely
a line management responsibility rather than one for specialized evaluation
staffs. Projects are focused in the areas of performance management, customer
participation, and more rigorous rulemaking. In FY 1994, the FDA conducted
evaluations in three areas: implementation of the Prescription Drug User
Fee Act of 1992 (PDUFA); the negotiation process toward medical device user
fees; and the harmonization of international regulatory requirements--
"regulatory benchmarking"--in which specific components of the regulatory
process in different countries are compared and assessed by program managers.
Specific evaluations are currently in progress at the FDA on PDUFA management
and performance of the Mammography Quality Standards Act of 1992. Also FDA
has ongoing evaluation efforts to develop performance measures for additional
user fee programs and to develop a customer-sensitive dimension to the
commencement of negotiated or consensual rulemaking. Management changes in
government are creating systemic changes in the FDA's evaluation function:
FDA evaluation efforts are now largely driven by the mandates of the Government
Performance and Results Act (GPRA) of 1993 and its corollaries.
Health Resources and Services Administration
The objectives of the Health Resources and Services Administration
(HRSA) evaluation program are to improve program management and policy
development and to provide information that will enhance strategic planning,
budget decisions, and legislative planning. High priority is given to studies
that assess program effectiveness or outcomes, or improve capacity for measuring
performance. In FY 1994, HRSA completed evaluations in the following program
areas: effectiveness of community health centers; linkage of mental health
and primary care services; HIV service networks in rural areas; implementation
of the Ryan White Comprehensive AIDS Resources Emergency (CARE) Act of 1990;
rural health and telecommunications; populations served under the Hill-Burton
Uncompensated Services Program; beneficiaries of nursing education projects;
training for preventive medicine specialists; and strategic planning for
the Bureau of Health Professions. HRSA has more than 40 projects under way.
Subjects of the major program evaluations include the Healthy Start Initiative
(intended to reduce infant mortality), the community and migrant health centers,
the National Practitioner Data Bank, the Healthy Schools/Healthy Communities
Program, the Injury Prevention Implementation Incentive Grants Program, and
various health services funded under the Ryan White CARE Act. The top priority
in the future for HRSA evaluations will be development of performance measures
and data systems in conjunction with implementation of the GPRA. Future HRSA
evaluations will focus on the Agency's mission of expanding access to care
for the underserved, and such crosscutting priority areas as academic/community
partnerships in health professions education, community infrastructure building,
managed care, expanded roles for States, information technology, and HIV/AIDS.
Indian Health Service
The Indian Health Service (IHS) evaluation program serves
the Agency's program and policy objectives, developed in consultation with
tribal communities. It provides valid and reliable information to help the
Director promote the IHS vision and guide its implementation. In FY 1994,
the IHS completed nine evaluations on topics of service delivery, health
status, and management. Three evaluations are having a major impact on program
planning: an assessment of maternal and child health data in the Navajo area;
an assessment of health status and access to care for Native Americans in
California; and long-term health care projections for alcohol-related
hospitalizations. The IHS has 21 evaluations in progress. Two of these underscore
the IHS commitment to the prevention of child abuse and family violence.
One report examines the extent of child abuse and neglect among American
Indian tribes and the ways in which the IHS responds to these problems. The
other is a case study of family violence in four American Indian communities.
More than 100 unstructured interviews were conducted at the four sites to
probe the nature and extent of family violence--spousal abuse, child abuse
and neglect, child sexual abuse, and elder abuse. In the future, the IHS
will initiate evaluations in three program areas: mental health services
for urban Indians, regional treatment centers for substance abuse disorders,
and health services for elderly American Indians.
National Institutes of Health
Evaluation is an integral part of the role of the National
Institutes of Health (NIH) in supporting biomedical research, training, and
public education. The NIH peer review system is a major part of its overall
evaluation strategy: research proposals from scientists around the nation
are subjected to a rigorous assessment
by fellow scientists before they are funded. National advisory councils,
boards of scientific counselors, and consensus development conferences perform
regular research evaluation functions. Four of the eight major evaluations
completed by NIH in FY 1994 are highlighted in chapter II. The topics were
adverse reactions to vaccines, indoor allergens, drug abuse prevention
strategies, and interventions for high-risk behavior and HIV infection.
Currently, NIH has 24 evaluations in progress, ranging from small- to large-scale
assessments, from evaluation feasibility studies to full-blown evaluations.
Future plans for NIH evaluations include an examination of new ways to carry
out more efficient peer review of grant applications; a survey of public
understanding about biomedical science and how Americans get their information
about health; and development of new measures of NIH's internal performance,
required by the GPRA.
Substance Abuse and Mental Health Services Administration
The Substance Abuse and Mental Health Services Administration
(SAMHSA) is committed to evaluating its programs to assess the effectiveness
of treatment and prevention approaches and systems of care; accountability
for Federal funds; and the achievement of SAMHSA's program and policy objectives.
SAMHSA's evaluations of demonstration programs generate new knowledge to
lead the field in developing policies that improve services. During FY 1994,
SAMHSA completed six evaluations of programs targeted to four population
groups identified as being in greatest need of substance abuse and mental
health services: pregnant and postpartum women and their infants, children
with serious emotional disturbance, high-risk youth, and the homeless mentally
ill. Currently, SAMHSA has 10 major evaluations under way in the following
areas: program accountability, evaluation of demonstrations, reinforcing
behavioral health, and commitment to customer service. SAMHSA's future evaluation
activities will respond to emerging trends such as managed care and health
care reform. Evaluation activities will reinforce the critical role of behavioral
health in general health care.
Office of the Assistant Secretary for Health
The primary role of the Office of the Assistant Secretary
for Health (OASH) is coordination and development of evaluations across the
entire Public Health Service, often identifying potential program areas that
could benefit from a collaborative evaluation. In FY 1994, OASH completed
17 evaluations in the areas of health care reform, immunizations, adolescent
and school health, nutrition, primary care, emergency preparedness, international
health, and health services delivery. Currently, OASH has 14 program and
policy evaluations under way in two areas: health care reform and
population-based public health services. Two issues of health care reform
are being examined: (1) the role of Federal, State, and local health information
systems in achieving public health objectives, and (2) the impact of health
care reform on the health care workforce and the role of academic health
centers. Other major evaluations include an examination of the clinical
preventive services program, Put Prevention into Practice, and an assessment
of the cost-effectiveness of clinical preventive services. In FY 1995, OASH
will support evaluation activities in the PHS Agencies to examine the impact
of managed care and health information systems on public health objectives.
A complete inventory of the 71 PHS evaluation projects completed
in 1994 is provided in appendix A, and PHS Agency evaluation projects in
progress are listed in appendix B.
Review criteria used by the special panel of senior editorial
advisors for assessing evaluations can be found in appendix C.