You are viewing a Web site, archived on 04:39:56 Oct 15, 2004. It is now a Federal record managed by the National Archives and Records Administration.
External links, forms, and search boxes may not function within this collection.
Click here to skip navigationDepartment of Health and Human Services logo

Communicating Health: Priorities
and Strategies for Progress

Office of Disease Prevention and Health Promotion logo

Objective 11-4. Disclosure of Information to Assess the Quality of Health Web Sites

Cynthia Baur, Ph.D., Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services

Small Group Participants
Chuck Bell, Consumers Union

Carol Cronin, Markle Foundation

Guy D'Andrea, URAC

Robert Denniston, Office of National Drug Control Policy

Joan Dzenowagis, Ph.D., World Health Organization

Note taker: Marianne Glass Duffy, National Institute of Nursing Research, National Institutes of Health, U.S. Department of Health and Human Services

Back to Top

Text of Objective
Increase the proportion of health-related World Wide Web sites that disclose information that can be used to assess the quality of the site.

The objective is intended to represent an important aspect of quality improvement for health Web sites but cannot encompass all issues raised by and related to the subject. The objective means to serve the specific purpose of providing data on progress for health Web site quality improvement and the general purpose of stimulating discussion about a wide range of health Web site quality issues.

The objective is developmental. Several elements are needed to make this a measurable objective. These elements include a reliable estimate of the number of health-related Web sites and the number of health-related Web sites that disclose information that can be used to assess quality; consensus about which information should be disclosed to assess quality; and an organization or governmental entity to undertake assessment and measurement. It may be possible to work with Internet search engines and directories to develop a methodology to create a proxy number for the total number of health-related Web sites.

The explicit purpose of the objective is to ensure that Web sites properly disclose certain information about their operations. The supporting text for the objective identifies six types of information that should be disclosed to users of health Web sites.

  • Identity of the developers and sponsors of the site (and how to contact them) and information about any potential conflicts of interest or biases (Note: "Bias" is not intended to mean simply a point of view or perspective but a systematic prejudice or predisposition that is intended to influence a user unfairly.)
  • Explicit purpose of the site, including any commercial purposes and advertising
  • Original sources of content on the site
  • How the privacy and confidentiality of any personal information collected from users is protected
  • How the site is evaluated (in terms of impact on users as a result of interacting with a Web site's functions)
  • How the content is updated

Although not a matter of information disclosure, Web sites should also be accessible to all users, including those with disabilities and low-end technology.

Back to Top

Issues Pertaining to Measurement
The inclusion of this objective in the Nation's public health agenda represents the belief that the implementation and measurement of the diffusion of disclosure and quality standards for health Web sites is vital, not only to improve individual Web sites, but also for the credibility of the e-health field as a whole and the overall improvement of the public's health. The challenges to achieve improvements should not be underestimated, although numerous groups have made substantial progress to identify ethical practices, disclosure mechanisms, and quality standards for health Web sites.

Defining the universe of health-related Web sites is difficult. An infinite and ever-expanding number of Web sites offers an ever-increasing variety of functions. Some Web sites are clearly providers of mainstream health information and services; others mix health with other subject areas, leaving the boundaries on "health" information unclear. For example, should all lifestyle information be considered "health-related"? How should the categories of "alternative and complementary medicine" be treated? Should personal Web sites or even pages posted by an individual to express a point of view or personal experience with a health issue be considered health-related Web sites? If the most expansive definition of "health-related" is used, even more Web sites may need to be considered, and it is not yet clear how this term should be applied in either the short term or the long term.

The objective is based on the principle that public disclosure of critical information about Web site operations will benefit users. Disclosure is considered an essential first step in the process of quality improvement. However, disclosure alone is unlikely to accomplish the larger goal of the objective, which is to improve the quality of health Web sites, not just increase the amount of information disclosure. In lieu of an oversight and enforcement mechanism for Web site quality standards, disclosure is used as a proxy for quality, not quality itself. Quality is constituted by the actual characteristics of a Web site, its practices and policies, and the value of the site and its content to users. Moreover, disclosure doesn't necessarily change consumer behavior to seek out the highest quality Web sites, although disclosure may make the behavior more likely. One of the limitations of a disclosure approach for Web sites is that, to date, it has been a purely voluntary effort, and it is not clear how broad and deep the support among Web site sponsors is for disclosure. Also, little effort has been made to make information about Web sites easily understandable for consumers.

An issue that underlies disclosure is the difficulty associated with developing and implementing Web site quality standards. Elaborating, operationalizing, and maintaining core elements and definitions of quality standards require considerable work. Once the standards are developed, they need to be implemented consistently, presented understandably, and disclosed publicly. Mechanisms to monitor their use and enforce their proper application do not yet exist and will require a substantial commitment of resources and effort. An additional issue is the applicability of standards, particularly voluntary ones, across the spectrum of Web sites. Government Web sites, for example, may have specific obligations or standards based on legal or regulatory requirements or the mandate of the agency or office that sponsors the Web site. Private-sector Web sites that engage in a specifically regulated activity, such as the sale of prescription medication, may be subject to their own sets of criteria. It is not clear how many different sets of standards can be meaningfully applied to a given Web site or a set of content, or if a single set of criteria could be meaningfully applied to all health Web sites.

The proliferation of Web site standards may present another problem for creating an environment for meaningful disclosure. Inundating users with information about every aspect of a Web site's operations will likely create a situation of information overload, resulting in user frustration and lack of interest or time to process all the different approaches and pieces of information. Standard criteria and templates, whether in human or machine-readable languages, have been proposed as a solution by some organizations. Web site sponsors, search engines, and directories may be reluctant to agree on standards, however, if they perceive that their own ability to respond to market conditions and their user base will be limited as a result.

Back to Top

State of Knowledge About Issues Represented by the Objective
The objective intends to promote greater disclosure of information about health Web site content and operations so that users can determine for themselves the degree of quality and appropriateness of the site and its content or features. A disclosure approach is not intended to assess whether Web site content is accurate. Rather, disclosure of key information allows potential users to compare Web sites and decide which they can trust and which are the most suitable for their needs. Disclosure of key information has been used in situations where users may have differing perceptions of what constitutes "appropriate" and "high quality." In the context of Web sites, "quality" is not limited to the caliber of information on the site; it extends to multiple elements that bear on reliability, value, and user protections.

The idea of disclosure of information about Web site operations as a means to assess appropriateness and quality borrows from other consumer sectors and health interventions. The assumption is that consumers—whether of manufactured products, medical procedures, or Web sites—need information to make the most informed decisions. Examples include the practice of giving patients information about the risks and benefits of medical treatments or experiments or giving potential purchasers information about consumer products or services. The nutritional food label is one of the most well-known implementations of the disclosure approach. The label is required to appear on most types of packaged and prepared foods and to provide specific information about the nutritional characteristics of food items so that consumers can decide for themselves if the food fits their own dietary needs. The label and its constituent elements have been subject to extensive public comment and are standardized. The Food and Drug Administration, the Federal agency with oversight responsibility of the food label, has put a great deal of effort into research and evaluation of consumers' understanding and use of the label to make the label "consumer-friendly." Research has shown that consumers do use information from food labels to make purchase choices and that those who do can positively influence their nutritional intake.

In the Internet context, much of the debate about disclosure of information has occurred in relation to Web sites' handling of users' personal information. Congressional committees have held numerous hearings to discuss the information practices of Web sites. The Federal Trade Commission (FTC) and numerous research organizations and advocacy groups have conducted studies and issued reports documenting that in general Web site users are ill-informed about Web sites' information practices. Even savvy Web users often have a difficult time understanding the privacy policies that Web sites post to disclose their information practices. The FTC has proposed that Web sites adopt fair information practices to inform consumers of the uses of their personal information. Notably, the Health Insurance Portability and Accountability Act of 1996 (HIPAA), which is the latest set of Federal regulations on health information privacy, will not affect most health Web site operations. Technology tools, such as the Platform for Privacy Preferences Project (P3P) standard for matching users' preferences with the elements of Web sites' privacy policies, are also emerging and contributing to the advancement of a disclosure approach. However, credible research is lacking on the impact of making users more aware of privacy and disclosing specific practices (e.g., disclosure, fair information practices, P3P).

Although not specific to Web-based information or the health sector, information quality standards will achieve a new level of rigor and prominence with the implementation of Federal "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies" (Section 515, Public Law 106-554; H.R. 5658). As of October 1, 2002, all Federal agencies are obligated to develop and abide by information quality standards that will apply to almost all information disseminated to the public, including Web-based health information. The guidelines are likely to have an impact on information practices beyond the Federal Government. Federal agencies will need to review not only the information they create but also third-party information that agencies disseminate as their own. Non-Federal and private-sector organizations that supply information to Federal agencies may need to change their own information practices to meet agency standards. In this way, the standards may shift expectations and practices across all sectors, including health information.

Back to Top

The ultimate goal of the objective is to increase the proportion of reliable health Web sites. Those who would benefit from this are numerous and varied. However, some Web site sponsors benefit from the continued dissemination of misleading or false information because it helps to sell their products, however ineffectual or dangerous. Although these types of Web sites can never be completely eliminated, their visibility may be diminished if the proportion of reliable Web sites increases.

The following list includes any group that has a reasonably substantial interest in the disclosure of information about health Web sites.

  • Web site sponsors
  • Health information seekers (consumers, patients, families, and caregivers)
  • Healthcare professionals and their professional societies
  • Healthcare provider organizations (private and public)
  • Health insurance providers
  • Managed care organizations
  • Public health organizations
  • Third-party reviewers and accrediting bodies
  • Government—regulators, information providers, protectors of the public interest, consumer protection groups, healthcare providers, and researchers
  • Search engines
  • Foundations
  • Academic and research institutions
  • Media watchdogs
  • Consumer advocates
  • Mass media
  • Libraries
  • Health info-mediaries—organizations that help their constituencies access and use health information
  • Health educators
  • Literacy and health literacy experts
  • Trade associations
  • Health industry groups—pharmaceuticals and medical products and devices
  • Technical community
  • Sellers of health information—online and offline bookstores (e.g., Amazon, Barnes & Noble)
  • Net advisory services

Some groups will be more interested and motivated than others. Web site sponsors want to be perceived as credible and reliable providers of information and services and need mechanisms to signal their credibility and reliability to consumers. Internet search engines enhance their own popularity, credibility, and economic viability when they direct searchers to the information they need in the most efficient, reliable manner, which may be easier to do with disclosure and quality criteria. Health information seekers need credible and reliable information to make informed decisions and high-quality services to realize the maximum improvements in their health, and disclosure and quality criteria may be important tools to help them. Healthcare professionals are also information seekers, as well as sources for referrals, and they need quality information so that they know they can rely on the Web sites they use and to which they refer their patients. Health services and insurance providers need their own Web sites to be perceived as credible, and they need to know that the patients they treat and insure are using reliable information to make decisions that affect their health. In addition to their functions as health insurers and service providers, governments are responsible for providing health information in a reliable manner, protecting consumers from harm, and promoting the public interest in having a safe Web environment and reliable health information and services.

Back to Top

Status of Selected Stakeholder Activities
Various mechanisms have been proposed to help users assess Web site quality, which include mechanisms to protect users' privacy. All mechanisms rely at least in part on Web sites' disclosure of information about their operations. Without disclosure, it would be virtually impossible to answer most of the questions on the quality checklists, seal assessment forms, and other tools, or to check for compliance with quality standards that have been developed by the initiatives described below. Numerous groups have created checklists that consumers can use to assess Web sites. Seal programs, such as the HON code and TRUSTe, provide minimum criteria that Web sites must meet to receive the right to display the seal.

Some e-health companies collaborated to draft a code of ethics for health Web sites (Hi-Ethics). The Internet Healthcare Coalition (IHC) convened a diverse group of interested parties, including some Hi-Ethics members, to draft a separate code of ethics. URAC, an accrediting body for healthcare organizations, has created a program to allow health Web sites to apply for accreditation. The URAC standards encompass elements of the Hi-Ethics and IHC codes. Professional societies have created their own referral programs and networks to direct consumers to reputable organizations and Web sites. The American Medical Association (AMA) has created quality guidelines that it uses to govern operations on AMA-affiliated Web sites.

Government agencies have created portals and Web sites to distribute reliable information and links to other organizations and sources of information. The World Health Organization proposes to create and manage a dot-health domain name that would function as a label for trustworthy health information. The European Union funded an initiative called MedCERTAIN, envisioned as a self- and third-party rating system with metadata tagging technology, standard quality vocabulary, and content filtering labels. MedCIRCLE, a new initiative, expands on MedCERTAIN and aims to realize the application of standard vocabularies and the interoperability of rating services and gateways.

The U.S. Department of Health and Human Services (HHS) is implementing several ongoing activities. In response to the Federal law on "Guidelines for Ensuring and Maximizing the Quality, Utility, and Integrity of Information Disseminated by Federal Agencies," HHS has drafted departmental and agency-specific guidelines on information quality. HHS also has draft guidance on developing agency policies about linking to Web sites external to the Department. The Multimedia Technology Health Communication small business grant program at the National Cancer Institute (NCI) has funded two projects to develop Web assessment tools. NCI staff has developed a tracking model for grantees and contractors that includes consistent procedures for usability testing and for assessing the quality of site content.

Back to Top

Factors That Can Influence Change
The Internet is a very fluid environment, and the factors that can influence the disclosure of information about Web sites and their content and the diffusion of quality standards can emerge and recede quickly. The following list identifies the most likely reasons that Web sites may begin to disclose information, the likely leverage points to encourage disclosure, or potential barriers that could impede disclosure or make disclosure less effective. Some of these factors may be specific to health Web sites, although many reflect the broader Internet environment and public perceptions of the Web and Web site practices.

Pending or actual legislation

  • Self-regulation. Web site sponsors and their business partners have been concerned about the potential for regulation of Web site operations, specifically regulation of the use of consumers' personal information. The industry has responded by launching numerous self-regulatory efforts and testifying frequently to Congress about the viability of self-regulatory mechanisms. More robust disclosure could be an important element in convincing legislators that Web sites are providing enough information to consumers to make self-regulation workable.
  • Legislation in other sectors. The notices mandated by the Gramm-Leach-Bliley Act (GLB) have brought attention to the process of disclosing information to consumers. Under GLB, financial institutions are required to provide notices to their customers about their information-collection and information-sharing practices. The financial services sector has been publicly criticized for the quality of these notices. The FTC held a workshop in December 2001 for companies to learn more about disclosure and providing notices about privacy for consumers. Privacy regulations under HIPAA will also be a high-profile test of how to give consumers notice about information-handling practices. In addition to increasing the amount of notice in the financial services and healthcare sectors, GLB and HIPAA may become important models of how to disclose information about many types of business practices, including Web site operations.

Marketplace considerations

  • Failure of dot-coms. As a result of difficult market conditions, Web site sponsors may be more interested in finding ways to retain customers and develop long-term relationships with them. Companies may perceive disclosing information about their Web operations as one way to enhance their reputations and attract and retain customers.
  • Image and public relations. The ability to enhance Web sites' reputations through positive publicity, or fear of potential damage to image from adverse publicity, may motivate information providers to adopt disclosure mechanisms and quality standards.
  • Transparency. Disclosing practices that Web site sponsors may be uncomfortable defending in public may motivate them to change and improve their practices.
  • Burden of disclosure. Web site sponsors may view the perceived financial and administrative burdens of reporting information and redesigning Web sites to make information easy to find and use as a barrier to disclosure of information.
  • Financial costs of disclosure. Disclosure may impose financial costs on Web sites for several reasons. Web sites may need to be redesigned to accommodate the disclosure of information about quality criteria. Web site sponsors that decide to seek accreditation through URAC or other means have to pay a fee to the accrediting body as well as cover the costs associated with staff time to prepare for the review and to redesign business functions to meet standards.

Government Web sites

  • Applicability of disclosure and standards to Government Web sites. Government Web sites may be reluctant to participate in disclosure and quality standard mechanisms that either are designed to fit the operations of non-Government Web site operations or require submitting the site to marketplace considerations, such as paying for accreditation or being listed in a directory or search engine. Several outcomes are possible if Government Web sites decide not to use the same quality standards and disclosure mechanisms as those used by non-Government Web sites. Non-Government Web sites may decide that disclosure and quality criteria cannot be that important if Government Web sites are not willing to use them. An artificial distinction could be made between Government Web sites and non-Government Web sites because Government information is available on many different types of Web sites, some of which are clearly dot-gov and some of which aren't, including those that are clearly commercial.

Media activities

  • Media activism. The media have reported numerous electronic mishaps related to the unauthorized release of identifiable personal information. Mistaken e-mails, files that should have been erased but were left on computers, and unauthorized sharing of personally identifiable health information are a few examples that have appeared in the headlines. In national surveys, consumers repeatedly report concern about how much information is collected about them while they are on the Internet and who has access to that information. Public comment about the recently enacted privacy regulations for healthcare transactions also indicates concern by consumers about the electronic information flow among organizations that provide health care. If media coverage feeds public fears about abuses, disclosure may be seen as insufficient to deal with the magnitude of the problem.
  • Media framing of quality issues. Consistent use of media for advocacy of quality standards could enhance the media's watchdog role and promote consumer awareness regarding Web content. Responsible (instead of inflammatory) stories on fraudulent and unethical practices could promote outrage and action on the part of consumers, rather than broadly dampen enthusiasm for using the Web as a health resource.

Attention on quality

  • Break-out of one (or more) of the quality initiatives. If some of the most popular health Web sites begin to use one of the more robust approaches described above and they receive large amounts of favorable publicity, public demand for disclosure of information and adoption of quality standards may increase.
  • Best practices. The development of best practices for disclosure and quality standards could reinforce the behavior of those Web sites that are implementing the practices and be a prod for those sites that aren't.
  • Healthcare providers' adoption of Web tools. Healthcare providers could take on more of an opinion leader role in recommending the best Web resources. This would help dilute the typically negative opinions that many healthcare providers express about resources for consumer health information and provide an important conduit for awareness and education activities with patients.
  • Emergence of a market leader for Web site seals and recommendations. Seals and recommendations are based on the ability to assess Web sites' activities. If a small number of seals and related identifying marks or logos emerge as credible and meaningful to consumers, disclosure would consequently be enhanced.
  • Visible support. Employers, other sponsors of health insurance, and information intermediaries could insist that their agents—such as health plans, employee assistance programs, and disease management programs—use only health information services that disclose information about their Web site operations and meet quality standards.

Technical factors

  • Emergence of technical solutions. Standards, such as P3P, and one-click access to an "About Us" section that relays core information to users could make it easier for Web site sponsors to engage in disclosure.
  • Filtering software. The proliferation of tools to recommend sites could direct users to sites that disclose meaningful information in an easy to understand way and direct users away from sites that don't.

User considerations

  • Information overload. In the interest of full disclosure or perceived liability considerations, Web sites may provide so much information about themselves and their operations that users become overwhelmed by the amount of information that they need to assess the quality of a Web site. Or, because of the lack of standardization in disclosure, users may find they need to learn multiple approaches to seek out information on Web pages as they move among sites, which could also create information overload. The likely consequences are frustration and an eventual lack of interest or at least attention to the information provided.
  • Disparities. The success of disclosure as a tool to help consumers will be influenced by the characteristics of Web site users. Users differ greatly in their health status, their fluency in the language used to present information, and their skills in information processing, general literacy, technology literacy, and health literacy. Also, the sophistication of the information presented on Web sites, in terms of both health content and information about Web site operations, tends to be very high, which would make the site difficult to use for a large number of health information seekers.

Back to Top

Strategies and Solutions

Premise: Promoters of disclosure and quality standards need to find solutions that put the least amount of burden on users. Three general strategies have been identified.

Create market advantage for quality (of interest to search engines, directories, other aggregators of Web content, and evaluators of Web sites and their content).

  • Encourage dissemination of information about trusted sites.
  • Promote, publicize, and reward good information practices; recognize the best and the worst.
  • Promote editorial "brands" for the most reputable sites.

Build demand for quality.

  • Encourage public education to alert Web site users to disclosure, quality criteria and standards, how to identify them, and how to interpret seal programs or other disclosure mechanisms.
  • Make communicating with Web site users about quality a priority.
  • Engage health info-mediaries: they are the conduits to particular user groups and audiences to discuss quality and how to use the Web.
  • Engage in counteradvertising: Web advertising could be used to direct users away from unreliable and low-quality sites and toward reliable and high-quality sites. One method might be ads that remind users of corporate and other influences on site content.

Build capacity to use quality criteria.

  • Rewrite Web site policies in a simple and focused manner to convey only the most relevant information to consumers.
  • Develop a template to allow users to identify the information being disclosed easily and quickly.
  • Develop users' critical information-seeking skills.
  • Collaborate with K-12 educators to integrate health content into school-based programs that teach technical skills.
  • Educate Web publishers about the importance of incorporating quality criteria.

Back to Top

Action Steps
This section identifies the necessary steps for measuring the objective or supporting the three strategies outlined above. They are not in sequential order. Many of these actions could occur simultaneously and in coordination with each other.

  • Define and characterize major categories of health-related sites for identification and quantification.
  • Identify core elements and definitions of quality standards.
  • Conduct consumer testing to find out what consumers understand about disclosure and quality standards on health Web sites.
  • Create a communication tool (e.g., label) to help Web site users understand the information disclosed about Web site operations.
  • Explore U.S. Department of Education programs for teaching "skills for the digital age" to see if health content can be included.
  • Mobilize health info-mediaries to help stimulate consumer demand for disclosure and quality.
  • Engage groups involved in health literacy in skill building around the use of Web resources.
  • Identify potential template approaches (e.g., P3P, the food label).
  • Identify industry leaders in quality improvement for Web sites.
  • Create a mechanism to monitor adherence to quality standards.
  • Develop self-applied tools for Web publishers to enhance their understanding of quality issues.
  • Research the operations and practices of Internet search engines and directories to understand their methodologies and market influences.
  • Open discussions with search engines about the possibility of incorporating quality criteria and disclosure templates in the search and directory processes.
  • Identify ways to publicize search engines and directories that steer users to more reputable sites.

Back to Top

For More Information
American Medical Association


Health on the Net

Internet Healthcare Coalition




Back to Top

Baur C, Deering MJ. Proposed frameworks to improve the quality of health Web sites: review. Med Gen Med. September 26, 2000:E35. Available online at http://www.medscape.com/viewpublication/122_toc?vol=2&iss=3&templateid=2.

Baur C, Deering MJ. Commentary on the "Review of Internet health information quality." J Med Internet Res 2001;3(4). Available online at www.jmir.org/2001/4/e29/index.htm.

Eysenbach G. An ontology of quality initiatives and a model for decentralized, collaborative quality management on the (semantic) World Wide Web. J Med Internet Res 2001;3(4). Available online at www.jmir.org/2001/4/e34/index.htm.

Mack J, ed. The new frontier: exploring ehealth ethics. Washington, DC: URAC/Internet Healthcare Coalition, 2001.

Risk A, Dzenowagis J. Review of Internet health information quality. J Med Internet Res 2001;3(4). Available online at www.jmir.org/2001/4/e28/index.htm.

Back to Top

Background on the Health Communication Focus Area in Healthy People 2010
Introduction to the Health Communication Action Plans
Objective 11-1. Internet Access in the Home
Objective 11-2. Improvement of Health Literacy
Objective 11-3. Research and Evaluation of Health Communication Programs
Objective 11-4. Disclosure of Information To Assess the Quality of Health Web Sites
Objective 11-5. Centers for Excellence in Health Communication
Objective 11-6. Healthcare Providers' Communication Skills
Appendix. Examples of Stakeholders Involved in Technology Diffusion and Internet Access Initiatives