Objective 11-3. Research and Evaluation of Health Communication Programs
Drafter
Christine Prue, Ph.D., National Center on Birth Defects and Developmental
Disabilities, Centers for Disease Control and Prevention, U.S. Department
of Health and Human Services
Small Group Participants
John Burklow, National Institutes of Health, U.S. Department of Health and Human Services
Robert Hornik, The Annenberg School for Communication, University of Pennsylvania
Craig Lefebvre, Prospect Center
Terry Long, National Heart, Lung, and Blood Institute, National Institutes of Health, U.S. Department of Health and Human Services
Susan Middlestadt, Academy for Educational Development
Note taker: Sara Alden, National Cancer Institute, National Institutes of Health, U.S. Department of Health and Human Services
Back to Top
Text of Objective
Increase the proportion of health communication activities that include research and evaluation.
Scope
The objective is intended to represent an important aspect of the assessment of health communication activities but cannot encompass all issues raised by and related to the subject. The objective means to serve the specific purpose of providing data on progress for research and evaluation of a subset of health communication activities and the general purpose of stimulating discussion about a wide range of health communication research and evaluation issues.
Measurement
The objective is developmental. Currently, no mechanisms are available to define or measure the total number of health communication activities conducted in the United States. Also, no broad-based consensus has been reached regarding which criteria should be used to determine if an activity includes research and evaluation components.
Any data source would have to specify the universe of health communication activities that should have research and evaluation components (the denominator) as well as the actual number of health communication activities that involve research and evaluation (the numerator). The members of the action plan group determined that the short-term goal should be the measurement of research and evaluation within federally funded communication activities, or even more narrowly, U.S. Department of Health and Human Services- (HHS-) funded communication activities. The long-term goal should be to identify and target for improvement those health communication programs that have not typically been expected to include research and evaluation, including programs run by non-HHS agencies. Research and evaluation should become a standard element not only for federally funded research grants to investigators but also for all funded programs, especially service programs at the community level.
Back to Top
Issues Pertaining to Measurement
Data collection and analysis efforts in support of measuring this objective are not an integral part of any stakeholder's mission or mandate. Although stakeholders are active in encouraging research and evaluation in health communication initiatives, they do not have dedicated resources for systematically or routinely collecting data about the number of health communication initiatives that do or do not include research and evaluation activities.
Back to Top
State of Knowledge About Issues Represented by the Objective
The objective is stated broadly enough to encompass any type of health communication activity. The unifying idea is that relevant activities would be strategic and be designed to "inform and influence individual and community decisions that improve health" (Freimuth et al., 2001). Examples of these activities include one-on-one physician counseling of an individual patient, print health information materials for specific diseases or conditions distributed by public health departments, public relations efforts to place health issues on the public agenda, mass media "edu-tainment
" with health themes, and national public health information campaigns. For the purposes of the objective, "activity" is synonymous with "intervention" and all that is implied in terms of purposefulness and articulated goals, audiences, methods, and outcomes. Change in health behaviors and social context may occur by other pathways. In a recent compilation of evaluations of public health communication, Hornik (2002) argues that
"intervention"
may be too narrow a concept to understand change and that public health communication would do equally well to begin with effects. He states,
The process might start with trying to understand the extraordinary secular trends in some health behaviors that are already in place. There have been some extraordinary changes in smoking and other cardiovascular risk behaviors, in sexual risk behaviors, and in others. Rather than starting with an intervention and its evaluation, health communication research might sometimes start with its effects, the substantial secular change in behavior, and try to explain it.
The scope of the objective does not preclude such a starting point. Rather, in recognition of the limitations imposed by the need to measure progress as well as provide data on strategic investments of publicly funded programs, the objective seeks to promote meaningful research and evaluation as a means to improve the quality of interventions. Research and evaluation of any type that will provide better data and richer explanations of the "messy treatment" of public communication are welcome. At the same time, when interventions are standard features of health communication, the objective focuses attention on research and evaluation as essential components of a sound intervention. Furthermore, research and evaluation should be used to influence all phases of an intervention, including design, implementation, adoption, and redesign.
There is increasing need to focus on the outcomes and effectiveness of any type of health intervention, whether it is communication, biomedical, or some other type. Scarce resources force the issue of "getting our money's worth." Policymakers want to know and program planners need to explain if an investment in a particular intervention was worth it, however the intervention goals are defined. At the same time, scarce resources indicate that program planners and implementers may be reluctant to take away from the actual activity to fund research and evaluation that may be perceived as not directly serving intended audiences.
Evaluation may be perceived as the domain of academic researchers and too expensive; program implementation may be seen as weakened because of the drain of resources for evaluation activities. Often different individuals are responsible for different parts of a program, and there may be a lack of understanding and operational integration of the overall project to assess accurately how resources should be assigned. Lack of integration among planning, implementation, and evaluation "teams" can result in evaluations receiving too little or too much funding to provide the data needed to understand why change did or did not occur.
Information on how to conduct research and evaluation of health communication interventions of various types is abundant. Research and evaluation findings, however, are not always easily available, and consensus is not necessarily reached regarding what needs to be evaluated and how extensive evaluation should be. One challenge in agreeing on what needs to be evaluated relates to the number of different disciplines involved in health communication research. Evaluations of health communication programs are published in many disciplines (e.g., communication, marketing, health education, psychology, journalism/mass media), so finding relevant articles can be difficult. Also, the published literature represents only a fraction of the research that is done: much of the qualitative research conducted for program development is never published. Some of this has to do with the findings not being generalizable, but some of this has to do with the voluminous data that come from qualitative research. Synthesizing data
and findings from multiple studies would be an important step in making research and evaluation findings more useful to various audiences and building consensus on the key evaluation factors. The Centers for Disease Control and Prevention (CDC) is currently updating its Internet-based health communication research database (www.cdc.gov/od/oc/hcomm) to help practitioners access the published literature.
Back to Top
Stakeholders
Organizations actively encouraging systematic research and evaluation
of health communication activities include the following: Federal
agencies (specifically, CDC, National Cancer Institute [NCI], Office
of Disease Prevention and Health Promotion [ODPHP], and National
Heart, Lung, and Blood Institute); public health professional organizations
(e.g., the American Public Health Association's Healthy People
2010 Committee of the Health Communication Working Group, the
National Public Health Information Coalition, the Society of Public
Health Educators); communication professional organizations (e.g.,
National Communication Association, International Communication
Association); social marketing professional organizations (e.g.,
Institute for Social Marketing); philanthropic foundations (specifically,
The Robert Wood Johnson Foundation and WK Kellogg Foundation); numerous
private, for-profit, and not-for-profit social marketing firms (e.g.,
Academy for Educational Development, Prospect Center); and health
information creators and providers (e.g., Healthwise, a creator
and provider of print and electronic health information).
Back to Top
Status of Selected Stakeholder Activities
ODPHP convened a meeting of communication research and evaluation experts on October 18, 2000, to discuss measurement of health communication research and evaluation.
The American Public Health Association's Healthy People 2010 Committee of the Health Communication Working Group has convened a group to discuss and offer recommendations to ODPHP regarding each of the Healthy People 2010 objectives. The group invited input from all key stakeholders. The Working Group ranked making progress on the research and evaluation objective as its top priority.
CDC's Office of Communication is encouraging systematic research and evaluation through four main strategies:
- Developing tools that facilitate research—(1) CDCynergy, a CD-ROM tool outlining the health communication planning, implementation, and evaluation process and (2) HealthComm KEY, an online database of health communication research
- Offering access to and expert consultation in marketing databases
- Offering access to and expert consultation in purchasing the services of research and evaluation contractors
- Developing a research agenda and supporting communication research that is consistent with the agenda.
Since 1994, the small business grant program for media technology health communication conducted by NCI has actively tracked grant research, evaluation, and outcomes. Grants and outcomes are synthesized; developed products are showcased to the National Institutes of Health (NIH) community; research and product information is made available to the public online; and NCI is currently having the Small Business Innovative Research (SBIR) products evaluated. This SBIR grant program is also currently building a repository of all evaluated media technology health communication grants and contracts within NIH.
The characteristics of all non-small business grants in NCI's Division of Cancer Control and Population Sciences are currently tracked through its Portfolio Management Analysis system. This means that all health communication research grants within the Division are monitored and evaluated. In addition, NCI has made a major commitment to research dissemination and diffusion through its Translating Research into Improved Outcomes program. In the Division of Cancer Control and Population Sciences, each program and branch has formed knowledge transfer teams that include scientists and health communication experts from NCI and, in some cases, from other parts of HHS. These teams are developing and acting upon strategic plans to close the gap between research discovery and program delivery.
Back to Top
Factors That Can Influence Change
Many challenges and issues arise in getting communication activities to include systematic research and evaluation.
- Securing approval and resources for research and evaluation
of communication programs and significant communications components
within larger interventions is a challenge. Many communication
professionals advise that communication be better integrated with
other program elements, yet the communication elements require
their own analysis. How can communication research outcomes be
"teased out" from endpoint or programmatic outcomes?
(Many programs do not have explicitly stated logic models that
guide evaluation activities of the overall program, let alone
the communication components.)
- Working with many public health and healthcare professionals trained in medical
model research in which the "gold standard" is the randomized
control trial (RCT) can present difficulties. Communication activities
have as their goal synergistic effects of messages from many messengers
through many types of media. Health research has as its goal the
controlling of as many confounders as possible to prove specific
cause-and-effect relationships. Applying the RCT model to prove
communication effects has been unproductive, especially when considering
relatively small differences in exposure of messages between treatment
and control groups (Hornik, 2002; 1996). In fact, most resource
gatekeepers expect robust data to support communication as an
effective intervention, but they do not acknowledge the attributes
of communication interventions that make them challenging to study.
Hornik argues that for communication interventions to be effective,
"the necessary level of exposure and extent of diffusion
through a social system is such that it is impractical to maintain
clear treatment and control conditions." Although the communication
literature offers good data to support what communication can
accomplish, it does not do so in the manner deemed robust enough
for key gatekeepers. Hornik recommends a number of approaches
(e.g., natural experiments, time-series designs, other quasi-experimental
approaches). These approaches are often used in other research
domains where it is unethical to have a true control group (e.g.,
welfare, social services, employment, criminal justice, education).
What are the best models for showing the impact of communication
interventions? How can these models be legitimized among resource
gatekeepers who see the randomized control trial as the gold standard?
- It can be difficult to acquire resources to conduct robust process and summative evaluation. Much progress has been made in encouraging programs to conduct audience research in the formative phases of their program development. However, less progress has been made in developing affordable, timely, and doable evaluation of communication exposure, delivery, and effects. More important, the assumption is that more evaluation is better. But when resources are limited (as they always are) and decisions are made between providing a communication service or product and evaluating that service or product, production wins out over evaluation. What is the optimal resource allocation mix for developing, delivering, and evaluating health communication products or services? What data are available to support the optimal resource allocation mix for research and evaluation of health communication products or services?
- Policies and related systemic barriers faced by Federal agencies
include difficulties in obtaining timely Office of Management
and Budget (OMB) approvals for conducting audience research and
surveys; lengthy clearance processes that effectively prevent
audience research; limited support for funding optimal research
and evaluation for communications programs; and the Government
Performance Reporting Act's emphasis on the quantity, not quality,
of Government activity.
- It is difficult to make generalizations about best practices for health communication that are not tied to specific disease topics or specific audiences. Research is typically topic or audience-specific and does not aim to make general conclusions about communication. Programs are built on logic models that try to get at contextual clues of the target behavior and the target audience, not generate general principles.
- There is an increasing demand for more accountability.
- There is growing sophistication in the field of health communication that is changing norms and expectations about what health communication can achieve.
- The demand for health information, especially online health information, is increasing (Boston Consulting Group, 2001; Fox et al., 2000). For example, a national survey of a representative sample of approximately 2,500 U.S. adults found that the percentage of respondents who said that they agreed or strongly agreed with the item, "I need to know about health issues so I can keep myself and my family healthy" increased from 66.9 percent in 1999 to 75.2 percent in 2000. In response to the statement, "It is important to me to be informed about health issues," affirmative responses increased from 66.8 to 69.2 percent. Both of these increases were statistically significant (Pollard, no date).
- The establishment of health communication objectives in the Healthy People 2010 health objectives for the Nation was a critical first step in getting health communication on the public health agenda. Although the process for developing the 2010 health objectives for the Nation was clear (that is, there were publicly known mechanisms for providing input through the Web, in writing, and in public hearings), a similar process does not exist for discussing implementation efforts on behalf of the objectives. Not having a known mechanism for interacting, discussing, or convening stakeholders beyond the development phase is a barrier to achieving the stated objectives. It is important for a mechanism to be identified and implemented so that health communication stays on the Nation's public health agenda.
Back to Top
Strategies and Solutions
- Develop and widely promote the use of recommendations on how to design appropriate communication-specific research and evaluation for interventions with significant communications components. (Develop industry minimum standards for communication program research and evaluation as well as criteria to assess when summative evaluation is absolutely needed.)
- Develop viable and credible evaluation models for communication and disseminate them through professional channels to communications and evaluation decisionmakers. Educate professionals about the value of health communication research and its contributions to understanding other phenomena. Explain why randomized control trials are not appropriate for communication research (e.g., no control of dose/exposure, confounders good).
- Develop evidence-based recommendations for allocating resources for use by communication professionals justifying program plans and resource requirements. Research the costs and return-on-investment for various communication approaches to provide guidance for decisionmakers.
- Explore the feasibility of an expedited review process (or exemption) from OMB clearance for Federal research and evaluation of health communication interventions.
- Develop criteria or a framework for culling out best practices in health communication that can be used across disease and audience contexts.
Back to Top
Action Steps
Obtain stakeholder support endorsing more and better research and evaluation in health communication interventions. This could be tied to the release of the Healthy People 2010 Health Communication Action Plans.
Renew the commitment of academic institutions to train health communication professionals in research and evaluation. Institutions should train "generalist" health communication professionals (those who understand the basics of research and evaluation) and "specialist" health communication research and evaluation professionals (those who can conduct rigorous communication research) (Riegelman and Persily, 2001).
Promote discussions of revisions to OMB rules to faciliate the conduct of audience research (expedited procedure or exemption).
- Explore other ways to support communication research and evaluation, such as innovative collaborations with non-Federal entities.
Develop consensus on alternative models for evaluating communication effects that have credibility among health communication and key resource gatekeepers.
- Summarize evidence for determining best research designs for conducting communication research and evaluation.
- Publish and promote the integration of findings into practice.
Develop "model" language—for inclusion in Requests for Proposals (RFPs), Requests for Applications (RFAs), and program announcements—that supports health communication research and evaluation.
- Encourage the inclusion of health communication research and evaluation in Federal RFAs, RFPs, and program announcements.
Encourage Federal stakeholders (HHS) to collaborate on conducting a benchmarking study.
- Determine the extent to which HHS agencies are currently conducting or funding health communication research and evaluation.
- Identify policies and procedures that either help or hinder health communication research and evaluation.
- Offer recommendations for changing policies and procedures that will support achievement of this objective.
Explore the potential for HHS support to establish a Federal (Office of Personnel Management) professional job series to recognize health communication research and evaluation personnel.
Delineate competencies of health communication professionals.
Promote additional resources for health communication and health communication research and evaluation. Few Government health agencies have ever been given the resources needed to ensure exposure of health messages; they rely on public service advertising. Purchasing media time is one way to ensure exposure, which is a critical first step in evaluating the effectiveness of health communication interventions (Hornik, 2002; 1996). However, purchasing media time is costly.
Develop better methods for measurement in health communication research and evaluation.
- Develop measurement guidelines.
- Develop models for measuring communication effects at the group or community level (not just the individual level).
Offer recommendations for how to deal with issues arising from differentiating or integrating communication evaluation in overall program evaluation efforts.
- Offer recommendations for allocating resources for different types of evaluation.
Increase the number of health communication articles published in public health and medical journals.
- Educate editors and reviewers regarding appropriate models for evaluating the merits of health communication research. Note: Most best-practices documents use medical-model strength of evidence criteria that often disqualify health communication interventions.
Back to Top
References
Boston Consulting Group. Vital signs update: the e-health patient paradox. May 2001.
Fox S, Rainie L, et al. The online health care revolution: how the Web helps Americans take better care of themselves. The Pew Internet & American Life Project: Online Life Report, November 2000.
Freimuth V, Cole G, Kirby S. Issues in evaluating mass-media health communication campaigns. In: Evaluation in health promotion: principles and perspectives, I. Rootman et al., eds. Copenhagen: World Health Organization, 2001.
Hornik R. Introduction. In: Public health communication, R. Hornik, ed. Mahwah, NJ: Lawrence Erlbaum Associates, 2002.
Hornik R. Public health communication: making sense of contradictory evidence. Paper presented on October 31, 1996, at the Annenberg School for Communication, Philadelphia, PA.
Pollard W. Data from Porter Novelli's Healthstyles. No date.
Riegelman R, Persily NA. Health information systems and health communications:
narrowband and broadband technologies as core public health competencies.
Am J Pub Health 2001;91(8):1179-83.
Back to Top
|
 |
|
