INTRODUCTION

Overview of Evaluation Methods

Panel 1: Transtheoretical Model of Health Behavior Change

Panel 2: Educational Activities

Panel 3: Targeted Populations

Panel 4: OPO/Organizational Efforts

Panel 5: Expanded Donor Criteria

Conference Conclusions

APPENDIX A: CONFERENCE AGENDA

APPENDIX B: EXPERT PANEL

INTRODUCTION

The purpose of the National Organ and Tissue Initiative of the Department of Health and Human Services is to build on the experiences of public, private, and volunteer initiatives for increasing the number of donor organs available to the growing list of Americans awaiting transplants. In doing so, HHS seeks to learn what program interventions work in improving organ donation.

The main goal of this conference, "Increasing Donation and Transplantation: The Challenge of Evaluation," was to identify successful and promising approaches to increasing organ donation (and therefore, transplantation) that have been, or can be, evaluated. The conference presentations and discussions helped to identify measures and evaluation methods that can be used to determine the effectiveness of various approaches to increasing donation. Much of the content of the presentations of the conference dealt with descriptions of the various program interventions. Given the emphasis of this conference on the challenges of evaluation, this summary report draws on the evaluative aspects of the presentations (including hard copies of presentations and supporting material provided by the presenters), audience discussion, and expert panel comments. This report is to serve as a resource for future organ donation activities by highlighting the promising approaches for increasing donation and how they can be evaluated.

Margaret Hamburg, M.D., the HHS Assistant Secretary for Planning and Evaluation, opened the conference by discussing the initiative’s goal of increasing organ donation by 20% in two years, and regulatory changes underway to facilitate this goal. Kevin Thurm, J.D., HHS Deputy Secretary, opened the second day of the conference by reporting on the need for increased organ donations in the U.S. and outlining specific steps the Department is taking to ensure that organ donation goals are met. He described how proposed changes to the federal regulations have been developed to build a more systematic approach to national organ and tissue donation issues.

The abstracts presented at this conference were solicited by the HHS Office of the Assistant Secretary for Planning and Evaluation (OASPE) and describe evaluations of efforts to increase donation. The evaluations are in many stages, including those that are completed, ongoing, and planned for current activities. The presentations focused on a range of approaches, such as those to increase consent, improve the efficiency of the donation system, promote behavioral change, and reach specific target populations. The presentations were organized into five general program types according to the relative proximity of the specified activity and related evaluation to a measurable change in the organ donation rate.

Overview of Evaluation Methods

While the organ donation community has been very active in implementing diverse approaches for increasing organ donation, there has not been a concerted effort to determine the best methodologies for evaluating these activities. Although many activities have been evaluated at some level, e.g., counting donor card signatures, there is relatively little understanding of how these activities relate to the goal of increasing the number of organs available for transplantation. Well-planned and methodologically sound evaluations, coupled with program timeframes and goals, provide the cornerstone for understanding program impact. Rigorous evaluations of activities designed to increase organ donation will better inform resource allocation among alternative and complementary programs.

There are three essential components of a successful evaluation: 1) an activity to evaluate that is pursuant to the ultimate goal of the program, 2) tested performance indicators, and 3) rigorous evaluation methodology. Exhibit 1 provides an overview of these elements and how they inform and influence each other.

 

Despite their diversity in design, costs, performance indicators, and other factors, the aim of evaluation methods in use today is essentially the same, i.e., to assess the effect of an intervention on one group compared to the effect of a different intervention (or no intervention) on a similar group. All evaluations have a control or comparison group, whether explicit or implied. Exhibit 2 depicts a basic framework for considering the methodological rigor of evaluation types, their respective study elements, and examples of organ donation activity evaluations.

The evaluation types in Exhibit 2 are listed in rough order of most to least scientifically rigorous for internal validity, i.e., for accurately representing the causal relationship between an intervention and an outcome in the particular circumstances of a study. This ordering of methods assumes that each study is properly designed and conducted; a poorly conducted large randomized controlled trial (RCT) may yield weaker findings than a well-conducted study that is lower on the design hierarchy. This list is representative; there are other variations of these methodologic designs and some investigators use different terminology for certain methods. However, such a methods hierarchy can help programs to weigh different evaluation options, and implement an evaluation strategy that is both methodologically rigorous and feasible.

 

The selection of performance indicators affects the design of an evaluation, the resources required to conduct the study, and the utility of the results. For example, measuring changes in public awareness of organ donation activities may not ultimately provide insight on the impact the activity had on donation rates. Because of the inability to predict who might become a potential organ donator before the occurrence of a traumatic event, any activity targeted at the general public must cast a wide net in order to reach those few people who will become potential organ donors. It is difficult to attribute any measured change in organs retrieved to a given population-based program with acceptable levels of certainty. The evaluation of population-based programs requires careful selection of performance indicators based on the goals and resources of the organization conducting the evaluation.

To overcome limitations in measuring program effectiveness of actual organ retrieval, the organ donation community has used three related sets of performance indicators, each with varying degrees of separation from the actual intended outcome of organ donation. The three types of measures are shown in Exhibit 3 as concentric circles, illustrating their relative proximity to the donation event. Pre-event measures are used to gauge effectiveness in increasing organ donation, e.g., willingness to donate, before an actual donation opportunity arises. Post-event measures are used to measure five crucial steps in organ retrieval after a potential donor situation has occurred, including donor identification rate, referral rate, request rate, consent rate, and retrieval rate. Donation rates are the most direct measure of the success of programs to increase the number of organs available for transplantation. Exhibit 3 depicts the relative sizes of the populations captured in pre-event, post-event, and donation measures relative to the U.S. population.

The evaluation methods described above provided a framework for discussion among the invited presenters, expert panel members, and audience during the two-day conference. The descriptions included in this report of programs and findings shared by presenters at the conference are derived from their respective abstracts and presentations, and were not verified by HHS or The Lewin Group.

 
Panel 1: Transtheoretical Model of Health Behavior Change

This panel explored applications of the transtheoretical model (TTM) of health behavior change to evaluations of public awareness activities. Dr. Michael Rohr and Dr. Mark Robbins are collaborating on efforts to develop the TTM for evaluation activities in the organ donation community. Dr. Robbins is currently consulting on a project funded by the Health Resources and Services Administration (HRSA), Division of Transplantation, to develop tools to measure stages of behavior change in college-aged students to gauge the effectiveness of organ donation awareness programs. Together these presentations represented the first challenge of this conference, that is, how to strengthen evaluations of public awareness activities.

These presentations focused on applying the five stages of behavior change identified by Prochaska et al. to organ donation, i.e.:

• pre-contemplation: the person has not thought about donating organs
• contemplation: the person has thought about donating organs
• preparation: the person has made phone calls or requested information about organ donation
• action: the person has taken action to express his/her wishes about organ donation, e.g., signed a donor card, talked to a lawyer, or discussed wishes with family
• maintenance: the person may occasionally remind or reinforce statement of wishes to family members.

To develop this model for organ donation activities, Robbins et al. developed a structured family consent survey to interview next-of-kin who either did or did not consent to donate organs. The survey was designed to measure: 1) stage of change, 2) decisional balance, and 3) potential processes of change. The information from the survey can be used to target specific processes of change to promote advances towards the action phase and, ultimately, to increase the likelihood of consenting to donate. One limitation of this survey has been its small sample size of non-donor families. As indicated by the panel, this sample size needs to be strengthened to improve the validity of the survey findings.

The TTM could be used to guide the assessment of programs such as the Georgia Coalition’s "Life Takes Guts" college campus awareness initiative, presented by Von Roebuck, Chair of the Georgia Coalition. Based on the theory that values are chosen during the years 18-24 that guide adulthood decision making, Roebuck and his colleagues at the Coalition have implemented a college-level campaign for organ donation awareness and donor card signing. The campaign involves a variety of campus celebrities, food, local media, and other "attention grabbers" for students. The Coalition supported 5 campus programs in the first year, 10 campuses in the second year, and will support 15 campus programs in 1998. In some cases, the college campuses are taking over the program themselves. The Coalition has reached approximately fifty-percent of the students in Georgia, and is evaluating the awareness of students and their level of intent to donate. Over the past few years, the number of tissue and organ donors has increased in Georgia, perhaps due to this and other programs. The primary evaluative concern identified for this effort is the difficulty of measuring a causal link between the "Life Takes Guts" program and Georgia’s organ donation rates.

Conference participants expressed concern that the TTM needs to be further tested in the organ donation community, particularly given the contrast between the acute nature of organ donation and such chronic health behavior change models as smoking cessation. Suggestions included the following.

• A more effective survey instrument needs to be developed due to the logistical problems associated with administering detailed surveys to grieving families. To make the TTM more useful, concise surveys similar to the SF-36 quality of life questionnaires need to be developed. The survey results need to be reliable, reproducible, and able to be more readily administered in this difficult setting.
• To test the TTM more fully across its stages of behavior change, the survey should be given to a much bigger non-donor sample to improve the validity of results. With an oversampling of donor families, the results are likely to be skewed towards the action phase.

• The evaluation of the "Life Takes Guts" program could be improved by combining elements of the TTM. Measures could be taken of how many students were previously aware of organ donation activities, and the extent to which this intervention prompted progression through the stages of change. In addition, surveys could be taken and analyzed by class to see if the on-campus interventions moved students through stages of change as they advanced from being freshman to seniors. The Coalition could conduct pre- and post- surveys, tracking the classes by year and the number and types of impressions the classes have experienced. The college campus population is unique in that it is a relatively contained audience that allows for this type of tracking.
• The application of the TTM to this program should be expanded from the individual as the unit of analysis to the family. The family is most often the decision-making unit in these cases, and needs to reach the action stage in order to consent to donate. Parents’ weekends on college campuses may offer an appropriate opportunity to implement surveys of families.

• More efforts need to be made to analyze why the donation rate in Georgia has increased, to better identify potential causes of the increase. For example, if the increase in donations resulted primarily from a change in the consent rate, then it is more likely that the college and religious programs in Georgia may have affected the consent rate. On the other hand, if the increase in donations resulted primarily from an increased referral rate, then it is possible that the programs did not have as big an impact. To determine specifically what caused a family to consent to donate, surveys should be administered at the time of consent to determine what influenced that family’s decision.

To date, the organ donation community has focused much of its efforts on activities designed to educate and increase public awareness of organ donation. While many organ donation activities focus on educating the public-at-large through public awareness campaigns, a number of organizations have narrowed the focus of their activities to targeting school-aged children and teens. Evaluations of the impact of these activities tend to target pre-event measures rather than organ donation itself. The challenge in this instance is, first, to establish that changes in pre-event measures are due to the education activity and, second, to establish a link between that pre-event measure and the outcome goal of organ donation. The three education activities presented demonstrate the challenges of evaluating this type of activity.

Gloria Bohrer, Director of Public Education/Information of the Southern California Organ Procurement Center, presented DiscoveriesÒ , a school education program designed to teach students in grades 6-12 about organ and tissue donation and transplantation and the life-saving importance of both. The program consists of a 22-minute instructional video, an instructional guide, and a training component. The goal of the program is for students to gain knowledge and understanding about organ donation that will lead to the decision or the process of deciding to become an organ donor. Further, the program encourages students to talk with their families about their decision. The program is currently in place in a single pilot site, and plans are underway to expand it to three areas. The program, a prospective, self-controlled study, was evaluated using pre- and post-assessment surveys. The pre-assessment survey determined baseline understanding and attitudes towards organ donation, existence of "misinformation," and any issues and concerns about organ donation in general. The post-assessment survey determined the extent to which students had gained knowledge about organ donation and measured the change in their willingness to donate. The results of these surveys indicate that students (1) had gained knowledge and understanding of organ donation, (2) had engaged in discussions with their families, and (3) had either made a personal decision about donation or were thinking about the decision.

Warren Riley, Executive Director of The James Redford Institute, presented "Teen Talk on Transplantation," an informational program consisting of a short, high-impact film with companion materials designed to increase awareness of organ donation among teens. Before creating the film, a survey of 50 organ procurement organizations (OPOs) was conducted to gauge interest in the film as a medium for conveying this information. As a result of the OPO survey, the film was modified to emphasize the need to share the decision with family members and to stress that organ donation is the "gift of life." When completed, the film will be disseminated nationally to teen-targeted television such as MTV, drivers’ education classes, high school health classes, and the like. The evaluation of this program will consist of written pre- and post-tests to assess how the respondents’ attitudes on organ donation changed, whether respondents expressed a willingness to be an organ donor, and whether they shared the decision with family members.

•  Awareness is an intermediate outcome and does not necessarily correlate with eventual outcomes. Follow-up and correlation of increased awareness with actual donation is necessary to determine if the program has any direct impact on organ donation rates.

• Ruling out confounding factors in these activities is very difficult. Because individuals assimilate information from many sources, it is difficult to use a pre-test/post-test design to determine whether changes in attitudes towards organ donation are due to the education program or some other factor.

Audience members also showed support for these educational programs with comments that "it is good to instill information at an early age" and "the target audience of today are the students of 30 years ago." Concern was expressed about very young children being exposed to death-related issues. Audience members suggested that these programs could be improved by creating developmentally appropriate programs that build on each other as students progress through grades in school, and that the programs should be administered by experts from outside the school rather than by the teachers. One audience member stressed that reaching into the schools should begin with educating children about chronic disease prevention, not just focusing on the "rescue" of receiving a donated organ.
 

Panel 3: Targeted Populations

Another group of organ donation activities focuses on increasing donation rates or awareness in specific groups within the population as a whole. These "targeted populations" may comprise, for example, specific age groups, ethnic groups, or neighborhoods. For various reasons, organ donation behavior of these groups may differ from that of the general public, and therefore may benefit from organ donation programs tailored to meet their specific circumstances. Depending on the proximity of the evaluation measures to the goal of increasing organ donation, the evaluation of these programs can be challenging. The following three "targeted population" studies were presented at the conference.

Denise Kinder, Regional Manager of California Transplant Donor Network, presented a study on ways to increase consent rates among the Hispanic population. The study revealed a large difference in donation rates among Caucasians and other ethnic groups in Fresno, California. In 1996 the Caucasian consent rate was 70% compared to 24% among Hispanics. The initial goal of the study was to determine the factors leading to lower consent rates among Hispanics, and then to use that information to increase consent rates. A survey of 520 Hispanic households indicated that non-assimilated Hispanics had the lowest consent rate, and that factors such as utilizing the language spoken at home, using trusted messengers from the community, and honoring families that have donated in the past could play roles in increasing consent rates. The researchers used this information to disseminate organ donation literature and awareness campaigns. To evaluate the program, the study tracked every potential donor whether or not consent was given using a donor-tracking tool. This tool includes questions about the referral and request process, outcomes of the donation, and family disposition towards donation. At the end of the study, 10 out of 22, or 46% of potential Hispanic donors in the Fresno area gave consent, an increase of 92% over the previous year.

Jennie Perryman, Director of the Georgia Leadership Commission on Organ, Tissue, Blood and Marrow Donation, and Stephen Thomas, Assistant Professor and Director, Institute for Minority Health Research, Rollins School of Public Health of Emory University, presented a "conceptual project" intended to improve collaboration among blood, tissue, marrow, and organ donation organizations to create best practices guidelines for increasing donation, especially among the African-American population. The study calls for strategies to identify national scholars and hold hearings in which the public can voice concerns. The project aims to improve activities by starting with a "formative evaluation" to determine if the activity can be shown to influence a specific targeted audience prior to commencing the activity. Once an activity can be shown to influence a target population, a "summative evaluation" can be conducted to determine if the activity is effective in changing outcomes, e.g., organ donation rates.

• The success of the ICFA program in increasing organ donation will depend on the ages of their pre-need planning consumers, and whether or not the information will be passed to the consumers’ families, i.e., potentially younger relatives who may make more suitable organ donors in the event of a traumatic event. If the pre-need population is too old or otherwise not appropriate as organ donation candidates, e.g., because of disease, the success of the program in increasing organ donation rates may be limited.

• Although larger sample sizes are preferred, it is possible to derive meaningful findings from studies with smaller sample sizes using non-parametric statistics. Programs should collaborate with statisticians and other health services researchers to further refine the data analysis methods.
• For a program to be evaluated and demonstrated as effective, it must follow families through the entire utilization process to determine the relationship between those who signed a donor card or received organ donor literature and those who donated. Such longitudinal follow-up can be achieved in carefully defined populations.

 
Panel 4: OPO/Organizational Efforts

This panel of presenters discussed a variety of efforts from within OPOs and hospitals to improve the identification of potential donors and facilitate donation. In contrast to the programs presented in previous panels, these settings offer more immediate links between post-event measures, e.g., referral rate and consent rate, and measures of actual donation rates. However, evaluation of these programs must be designed to determine whether these proximal measures are causally linked at statistically significant levels.

Holly Franz, R.N., from the Partnership for Organ Donation, spoke on "Evaluating Hospital Donation Performance by Monitoring Family Satisfaction with the Donation Process." Arguing that the quality of care received by a potential donor’s family is a significant predictor of its willingness to donate, Ms. Franz presented results of a survey, conducted in cooperation with the Harvard School of Public Health, of families who had been through the request process. A comparison of responses of donor and non-donor families on scales measuring quality of care received, brain death knowledge, and quality of request revealed a significant difference in the experiences of these families on all three scales. She suggested that such instruments would be useful tools in continuous quality improvement of hospital procedures. Further, she described certain methodological weaknesses of the evaluation that the team could correct or adjust for in subsequent studies. These included a significantly lower response rate from non-donor families, partly due to less complete information retained by the OPO for such families and a resulting inability to track them down, and the time elapsed between the donation opportunity and the interview, which may hinder accurate recall.

Phyllis Weber, R.N., the Executive Director of the California Transplant Donor Network presented "Exceptional Requestors: Best Organ Procurement Organization (OPO) Consent Practices." This initiative, being conducted on behalf of the UNOS Council for Organ Availability, seeks to identify exceptional requestors, defined as those with consent rates of more than 75%, and to use surveys to identify their relevant personality traits and practices. The feedback from these surveys will be used to craft a personality profile that can be used to inform the hiring of new requestors, and instructional tools to be used for current requestors and new hires, including vignettes for role-playing. Plans for evaluation include a detailed tracking of individual and institutional consent rates and comparison of rates between those who have undergone best practices training and those who have not. Additional comparisons will be made between participants’ consent rates before and after training, and between new hires and trained "historically employed" staff. Consent rates of cohorts of trainees who participated in the same workshop or instructional module also will be compared. Finally, participants in any workshop or instructional module will be asked for qualitative evaluations of the program.

Aside from selection of appropriate performance measures, the panel raised multiple specific issues for consideration in design of program evaluations, as follows.

• Evaluators must strive for representative samples and otherwise account for selection bias. For example, when studying donor versus non-donor families, panelists suggested that families who chose to respond to the survey might be more satisfied with the quality of care received than families who chose not to respond. This sample bias may have skewed the study results and attempts should be made to reach a truly random sample of non-donor families.
• Evaluators must account for other factors that may confound results. Examples given were accounting for geographic regions and socioeconomic factors. For instance, it might not be appropriate to compare changes in consent rates of hospitals in different parts of the country if there are underlying differences in quality of care or compliance with required-request procedures.
• Another goal in good evaluation is prospective design, which helps to limit selection bias in study participants. For example, the Franz study encountered a low response rate from non-donor families. In addition, other researchers noted a difficulty in tracking down non-donor families for a retrospective study because OPO coordinators do not collect contact information unless a family consents to donate. Rather than requesting a family’s involvement months after the donation event, it might be advantageous to ask families if they are willing to participate in a study prior to consenting to donate. Follow-up information could be collected at the time of the event, and the response rate would likely be higher if families expected to be contacted.
• Careful consideration must be given to sufficient sample sizes in program evaluations. The panel stressed the need for more data in nearly all of the programs presented. For example, in the case of the donor family surveys, the presenters acknowledged the desirability of conducting a larger study that would allow multivariate analysis of the relative importance of quality of care, quality of request, and brain death knowledge on donation decisions.
• Finally, the importance of applying evaluation results to program redesign was noted. Once validated, the Franz survey of families and the results of Weber’s effective requestor study should both provide useful mechanisms for hospitals to improve and continually monitor their performance. Also, a hospital’s key person, as in the Katz study, is in a good position to facilitate a continuous quality improvement process, especially in regard to enforcing protocol compliance.

Panel 5: Expanded Donor Criteria

This panel included presentations on initiatives designed to improve the organ donation rate, either by expanding the criteria that have traditionally defined the donor pool or by novel measures to achieve greater conversion rates among those who are already recognized as potential donors. As in the programs detailed in the previous panel, these efforts are hospital-level initiatives and thus lend themselves to evaluations using donation rates and post-event intermediate measures that are proximal to donation rates. However, they also have faced unique challenges to implementation, whether from public or media responses to preservation techniques or from OPOs that consider some organs to be more costly or of marginal quality.

Jimmy A. Light, M.D., of the Washington Hospital Center spoke on a "Legislative Initiative to Preserve the Family Donation Option in the Potential Non-Heart Beating Donor." Dr. Light described the process by which the Washington, DC, city council approved the enactment of a law allowing in situ cannulation of potential donors to maintain the viability of kidneys for four hours to permit families to be contacted, preserving their right to opt for donation. The importance of positive media impressions and effective presentations to stakeholders, including the public, was stressed as vital for the eventual success of any similar legislation in other jurisdictions. The evaluation procedures used by Dr. Light’s team included recording the number of media exposures about the program and administering surveys to people attending informational presentations about their opinion on the procedure, yielding a 90% approval rate. Dr. Light’s team also analyzed donation rates both in cases of cannulation pending consent and in all cases, to determine whether the cannulation policy had an adverse effect on donation. Among the six families of patients undergoing cannulation since the policy was enacted in May 1996, none objected to the procedure, three gave permission to donate, one declined, one could not be reached during the four-hour window of time, and one was lost due to technical failure. Donation rates for the hospital as a whole remained unchanged while the policy was in effect.

Louise M. Jacobbi, Executive Director of the Louisiana Organ Procurement Agency (LOPA), spoke on "Increasing the Number of Transplantable Donor Organs by Expanding Donor Criteria." She described efforts by LOPA to increase the availability of organs by making use of more liberal donor criteria, including a broader age range and accepting as donors patients with hypertension or diabetes. Recognizing benefits and costs inherent in expanded donor protocols, LOPA tracked relative costs of "traditional" and "expanded" donors over the course of the year. The analysis included costs both to the OPO (including hospital costs, personnel time, donors per million population, discard rate, and increase in recovered organs) and to the patient (including patient and graft survival rates and length of stay). Over the study period, 642 organs were transplanted from expanded donors, increasing the number of organs available for the year by 40%. In terms of patient and graft survival, success varied by organ, with expanded hearts performing slightly better than traditional ones, expanded kidneys somewhat worse, and expanded livers significantly worse. The results of the latter two, however, might have been related to the placement of the organs, since some organizations were more inclined to take the risk associated with expanded donors for placement in sicker, more desperate patients. In terms of costs to the OPO, the procurement of the expanded organs incurred an average 17% more in direct hospital costs, 30% more personnel time, and 40% more indirect costs per organ than for traditional organs. These results suggested that expanded donor criteria can be a successful way to increase the number of organs donated, but that careful studies are necessary to understand whether and to what extent such expansion is cost-effective.

• Broadly construed to include both direct and indirect costs to the OPO, donor and recipient, the use of cost-benefit analysis and related economic analyses is prevalent in health services research but relatively unexplored in the field of organ donation. The LOPA example is instructive in its attempt to quantify the organizational costs of expanded donors while also evaluating clinical outcomes. Recognizing the importance of economic analyses, the panel recommended collaboration with economists in design and analysis of program evaluations.
• The panel noted the importance of stakeholder relations to successful organ donation programs. An example was Dr. Light’s mention of the considerable effect on public opinion of media presentations of organ donation procedures, and the need for securing public and policymaker buy-in before pursuing such potentially controversial practices. On an organizational level, the LOPA experience reinforces the need for good communication between OPOs and transplant centers. Clarity about what risk factors a given transplant center will accept in a donor can help to reduce some of the indirect costs incurred by staff trying to place organs unsuccessfully.
• The expert panel again stressed the need for more complete and extensive data collection and larger sample sizes. In the case of expanded donors in particular, there was a call for more information, including patient and graft survival rates over longer periods of time, data on immunologic compatibility, and the extent to which recipient health status contributes to diminished viability of organs from expanded donors. With larger sample sizes, these factors could be isolated more effectively in a cost model using multivariate analysis.

Conference Conclusions

The conference offered a diverse range of organ donation activities and evaluations with performance indicators of varying proximity to organ donation rates. Conference deliberations recognized the practical challenges of evaluating programs in a wide variety of health care and social environments, including the resources needed to implement rigorous evaluations. Organizers of the conference sought to aid program managers and researchers to understand what might be needed to improve evaluations of their programs and better demonstrate what does and does not yield greater organ donation.

Presentations on the first day focused primarily on pre-event activities, i.e., public awareness and educational activities, and the challenges of evaluating such programs relative to measures of organ donation. Presentations on the second day focused on post-event activities that tend to be more proximal to the "bull’s-eye" of performance indicators (Exhibit 3), but still face significant methodological challenges. Although the specific evaluation methodologies may be different for pre- and post-event activities, each type of activity shares similar challenges of evaluation. A distillation of the conclusions made by conference participants regarding program evaluation is as follows.

1. Evaluations of the impacts of program interventions need to be more rigorously designed in order to determine the causal links between the interventions and the appropriate performance indicators. More rigorous evaluation design will rule out factors that may confound causal effects, e.g., differences in geographical or socioeconomic attributes that may affect performance indicators rather than the interventions themselves. Among the design attributes that contribute to establishing causality are:

• prospective studies
• careful identification of populations or target groups (including stratification into subgroups as appropriate)
• contemporaneous and otherwise well-matched control groups
• random assignment of interventions
• sample sizes of adequate size to detect any true causal relationships between interventions and changes in performance indicators.

1. Researchers must demonstrate an impact on the intended target population and a change in the associated performance indicators. The performance indicator(s) chosen to measure the impact of a program intervention should be as proximal as possible to organ donation rate. Clearly, making the causal connection to organ donation rates is difficult or impractical for many programs, particularly those involving pre-event interventions. To the extent that causal links between less proximal pre-event measures such as measures of public awareness or number of people registered as organ donors, or post-event measures such as referral rates or request rates, and organ donation rate can be demonstrated, pre-event measures would be more useful.
2. The target groups of evaluation and the timeframes of evaluation need to be commensurate with the chain of events or other stages of progress from initial program intervention through organ donation and follow-up. For example, programs intended to change the behavior of younger people may have to be evaluated over longer periods of time and include their family members to determine if the intervention affected donation-related behaviors.
3. Potentially useful evaluative paradigms such as the transtheoretical model of behavior change, and tools such as survey instruments that have been developed or applied in other fields should be validated in organ donation settings, e.g., post-event in hospitals, and in representative populations, e.g., with donor and non-donor families.
4. To improve their "generalizability" or external validity, program evaluations should be conducted in multiple geographic regions and socioeconomic groups.
5. Organ donation researchers could benefit from increased collaboration with researchers in other fields including evaluation design experts, statisticians, health services researchers, health economists, and other academic researchers. For example, statisticians can provide assistance with regard to "power calculations" to determine adequate sample sizes and identify appropriate statistical tests, e.g., using non-parametric statistics and multivariate analysis. Economists can provide assistance with identifying and quantifying the direct and indirect costs of programs and conducting cost analyses. Engaging such experts in these efforts will strengthen the longer-term evaluation capacity in the organ donation community.
6. Given considerable tradeoffs in the costs and outcomes of programs for improving organ donation, cost-benefit analysis and related economic analyses should be used to compare programs to improve organ donation and to demonstrate their value relative to other types of health care programs.
7. Program evaluations should not overlook what one presenter called "formative evaluation." That is, programs should specify implementation milestones and provide the means to measure progress against these.
8. Organ donation programs should increase collaboration in evaluation. Researchers should become more familiar with other programs in similar areas and evaluations, including those reported in the literature and other sources. Organizations involved in efforts to improve organ donation should engage in larger-scale collaborative efforts to plan and implement programs, e.g., through multicenter evaluations, registries, and related data collection and sharing efforts. More efforts need to be undertaken to refine pre- and post-event measures, and to further establish causal links between these measures and organ donation rates. Some of these efforts can be coordinated at a national level, e.g., through organizations like UNOS, and other nationally active organizations.
9. Evaluation findings should be used to improve the programs that were evaluated and disseminated more widely for incorporation into other efforts to improve or ensure the quality of organ donation programs.

APPENDIX A: CONFERENCE AGENDA

 
APPENDIX B: EXPERT PANEL

Christopher Bladen, M.Sc., has served a variety of posts in the Office of the Assistant Secretary for Planning and Evaluation, including Acting Deputy Assistant Secretary for Health Policy, Deputy to the Deputy Assistant Secretary for Health Policy, and Director of the Division of Health Care Financing Policy. Mr. Bladen is now a private consultant.

Jacques Corman, M.D., FRCS (C), FACS, is a transplant surgeon at CHUM (Hospital Center Montreal University), and is a Full Professor of Surgery at the University of Montreal. A founder and former Chairman of Quebec-Transplant, Dr. Corman now serves as its Medical Director.

Ann Mongoven, Ph.D., is a Professor of Religious Studies at Indiana University with a research interest in ethics, particularly medical ethics. Dr. Mongoven was formerly on the staff of the Physician Payment Review Commission, and last year served on a DHHS panel on issues in liver allocation.

Robert J. Rubin, M.D., is President and Chief Operating Officer of The Lewin Group, a health care policy and management consulting firm based in Fairfax, VA. In the early 1980s, Dr. Rubin served as Assistant Secretary for Planning and Evaluation at DHHS. Dr. Rubin also holds an appointment as a Clinical Professor of Medicine at the Georgetown University School of Medicine.

Valerie Petit Setlow, Ph.D., is Deputy Director of the Tulane/Xavier Center for Bioenvironmental Research. She was formerly Director of the Division of Health Sciences Policy at the Institute of Medicine, National Academy of Sciences. Dr. Setlow also has a clinical appointment in Environmental Health Sciences at the Tulane School of Public Health.

Moderator

Clifford Goodman, Ph.D., is a Senior Manager at The Lewin Group, and an international expert in medical technology assessment. Among his posts, Dr. Goodman directed the Institute of Medicine’s Council on Health Care Technology and coordinated projects at the Swedish health care technology assessment agency in Stockholm.