On October 24, 2002, the Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare & Medicaid Services (CMS) hosted a live, interactive Web chat to answer questions about the Hospital-CAHPS project. Questions concerning this project were answered by a panel consisting of representatives from AHRQ, CMS, and the CAHPS® II grantees. Questions and answers from the Web chat follow.
Note: The "Hospital State Pilot Project" refers to the CMS pilot study to be conducted by the Quality Improvement Organizations (QIOs) to test the Hospital-CAHPS instrument.
*Asterisks preceeding questions indicate new material added to the transcript.
Intent
Development of Hospital-CAHPS Instrument
Domains of Hospital-CAHPS
Survey Submissions
Project Officers, Grantees, and Instrument Development
Hospital-CAHPS Survey Structure
Hospital-CAHPS Population
Alternative Languages and the Illiterate
Implementation of Hospital-CAHPS
Reporting and the National Database
Pilot Study and Hospital-CAHPS
Evaluation and Feedback
Miscellaneous
Conflict of Interest and HCAHPS
Question 1: Why is the Government developing a hospital survey? Why not just let the industry do this as it has to date?
Answer: CMS has requested a hospital patient survey as a way of providing comparison information for consumers who need to select a hospital and as a way of encouraging accountability of hospitals for the care they provide. There are many excellent patient surveys out there, but most of them are proprietary and are not constructed in a way that would allow patient assessment of hospital care across the country—that is, the instruments are not standardized. AHRQ plans to develop a standardized instrument and allow free and open use by any interested party. We hope that this will: allow consumer to make "apples to apples" comparisons among hospitals, allow hospitals and hospital chains to self compare, and provide State oversight officials with useful data.
Question 2: Is it true that the ultimate purpose of a Hospital-CAHPS (HCAHPS) instrument is for public accountability and not for quality improvement?
Answer: CMS has requested a hospital patient survey as a means of obtaining comparative information for consumers who need to select a hospital and as a way of encouraging accountability of hospitals for the care they provide. The ultimate goal of this effort, both for AHRQ and CMS, is high quality care for hospital patients. So discouraging quality improvement efforts based on HCAHPS data would be counterproductive.
Even though the main purposes of HCAHPS are consumer choice and hospital accountability, we hope that the collection and reporting of these data stimulate the desire for quality improvement. But since the HCAHPS instrument needs to be general enough to apply across many different types of hospitals, we doubt that the data resulting from this instrument will be specific enough to support development of targeted QI interventions. Fortunately, there are a number of excellent survey firms who specialize in collecting this type of detailed data and working with hospitals to bring about improvements.
Question 3: Is it true that AHRQ has already selected the instrument on which Hospital-CAHPS (HCAHPS) will be based?
Answer: No. Members of the CAHPS® team will work with AHRQ staff and CMS staff to review all submitted instruments and make a first-cut selection of items. It is likely that items from different instruments will be selected rather than an instrument in its entirety.
*Question 4: Is this CMS work the basis for the Hospital-CAHPS instrument?
Answer: CMS has requested a hospital patient survey to provide consumers with comparative information on patient hospital experiences; however, AHRQ, as a research partner to CMS, is charged with development of the instrument.
*Question 5: Is this an effort to combine "technical" performance and "non-technical" performance measures?
Answer: No. This is a survey that will report patients' experiences with hospital services, not the "technical" or clinical performance of the care provided.
*Question 6: Couldn't it be used to collect patient satisfaction data also?
Answer: The HCAHPS survey is intended to capture reports and ratings of patients' hospital experience. As indicated in the literature, patient satisfaction surveys continually yield high satisfaction rates that tend to provide little information in the way of comparisons between hospitals. Patient experiences tend to uncover patient concerns about their hospital stay, which can be of value to the hospitals (in quality improvement efforts) as well as consumers (for hospital selection).
Select for Revised Answer.
*Question 7: Would this survey replace current vendor surveys or would it regulate them?
Answer: No. The Hospital-CAHPS survey is intended to complement the current vendor surveys. The primary purpose of this survey is public reporting; however, it may also help to identify broad areas in need of improvement. Most current surveys are focused on internal quality improvement and more actionable steps for improvement. At this time CMS is considering different options for implementing the survey.
*Question 8: Who will be targeted for possible sponsorship of this initiative? Who will be funding, developing, and managing this initiative?
Answer: CMS is sponsoring and funding the development of a hospital patient experience instrument with AHRQ as its research partner. After the instrument is developed, CMS may decide to require its use. In this case, it is likely that CMS would solicit proposals for data collection, analysis and reporting.
*Question 9: Proprietary research shows that a high majority of patients have no input, or give no input to the choice of hospital. That is determined by their doctor or which ambulance picks them up. Doesn't this, at least currently, reduce the value to the ultimate consumer of what all this surveying is intended to measure?
Answer: Not necessarily. The HCAHPS survey can be used to encourage discussion between physician and consumer with regard to patient preferences prior to the need for admission. In addition, the literature indicates that public reporting of quality data has led to quality improvement.
*Question 10: This is somewhat "off the wall" but why a survey? AHRQ's research shows that nurse staffing levels impact quality (e.g., lower mortality). If the purpose is to give a tool to consumers to judge hospitals, then the concept "more nurses = better care" seems a simpler approach than trying to educate consumers with complex survey results.
Answer: Nurse staffing levels are one aspect of the multidimensional concept of quality of care. Through consumers may be interested in this measure, we know that they are also interested in assessments of the quality of health services from the point of view of other consumers. Hospital patient assessment of care is the dimension for which CMS has asked AHRQ to develop measurement and reporting tools.
Question 11: What is your timeline for development/release of Hospital-CAHPS?
Answer: The current schedule for development of the HCAHPS survey requires submission of the survey instrument, as well as the sampling and data collection procedures to CMS in summer 2003.
Question 12: What is the research plan to document the validity and reliability of the standard questions that are to be used?
Answer: CMS has funded a pilot test of the hospital patient experience instrument through Quality Improvement Organizations (QIOs) in three States (Arizona, Maryland, and New York). In addition, the CAHPS® team will perform cognitive testing of the draft instrument to the extent possible given CMS' time frame for this work. Also, hospitals in addition to those included in the pilot test have approached the CAHPS® team and offered to participate as possible test sites.
Question 13: By what criteria will the items be selected for inclusion in the Hospital-CAHPS project test questionnaire?
Answer: The criteria that will be used in the initial review of submitted measures are those included in the call for measures, e.g., documented validity and reliability of the instruments. The selected measures will then be subjected to additional field testing and cognitive testing.
Question 14: May we review the framework that will be established for evaluation of questions; the methodology which will be used to establish validity and reliability, etc.?
Answer: In the original CAHPS® RFA published in 1995, AHRQ listed a number of design principles that would guide the process of survey design and development. For example, we required that CAHPS® instruments go through iterative rounds of cognitive testing, rigorous field testing, and process and outcome evaluations in the settings where they would be used. We revised the instrument after each round of testing. As the CAHPS® work progressed, the team developed additional design principles.
To make the CAHPS® methodological process as transparent as possible, the team published these principles in a document called the "CAHPS® Survey Submitters' Kit." This kit is publicly available free of charge through our Web site www.CAHPS-sun.org and provides researchers who desire the CAHPS® trademark for instruments they have developed with a detailed map of the testing process. This same map will guide the development of the HCAHPS instrument. Anyone who wants to know how the HCAHPS instrument will move from the concept stage to a final product can consult the kit. CAHPS® researchers also published articles regarding the actual cognitive and field testing of CAHPS® 1.0 instruments in Medical Care Supplement, March 1999; Vol. 37, No. 3. Interested parties who want to see how the CAHPS® team puts the design principles into practice can consult that resource. Once the CAHPS® instrumentation team has met to review the submitted items and determined the need for further testing, they will develop a work plan based on these design principles that will guide the process of testing and revision.
*Question 15: What linkages have been established between the current survey and hospital outcomes either clinical or financial?
Answer: No survey or survey items have been selected at this point, furthermore, those kinds of linkages are not in the scope of work as specified by CMS.
*Question 16: What kind of information is planned on the survey about hospitals' adverse outcomes?
Answer: At present, the Hospital-CAHPS survey is under development, therefore we can not specifically comment on what the survey will contain. In our call for measures, we requested surveys and items within the following domains: respect for patients' values, preferences and expressed needs; coordination and integration of care; communication and education; physical comfort; emotional support—i.e., relieving fear and anxiety; involvement of family and friends; continuity and transition; and access to care. It is very likely that a draft of the survey will be available for public comment at a later date.
*Question 17: Will there be a survey of consumers/future patients to identify what is relevant to them about hospital quality or patient satisfaction measures?
Answer: The CAHPS® team is conducting a literature search to identify issues of importance to consumers in selecting a hospital and will also conduct some qualitative research in response to this question.
*Question 18: Given that there are multiple stakeholders (consumers, providers, purchasers, etc.), has the cognitive testing been conducted with other than the traditional definition of consumer?
Answer: The key feature of the CAHPS® approach to instrument development is collaborative, both within the CAHPS® team and with other organizations including consumers, providers, and others interested in the development process. CMS and AHRQ plan to provide multiple opportunities for stakeholder input into this process. AHRQ will also be doing qualitative research with consumers and hospital personnel to determine what will be included in the survey. The reports will also be tested with input from both consumers and appropriate hospital personnel.
*Question 19: Since many consumers are using the CAHPS® (health plan surveys) when picking a managed care organization (MCO) for their care has consideration been given to keeping the same look/feel of the health plan surveys for the hospital one?
Answer: The CAHPS® team has performed extensive cognitive testing with various audiences to illuminate many questions pertaining to both instrument and report development for health care quality information for consumers. Many lessons from this experience will likely apply to the development of the instrument and report formats for hospital patient experience, so it is likely that this instrument will have the same look and feel of existing CAHPS® instruments.
*Question 20: If the measures are altered, then essentially it becomes a different measurement tool, requiring its own validity and reliability testing. Will you be performing validity and reliability testing along with the "field testing" and "cognitive testing?"
Answer: Yes, the CAHPS® instrumentation team will be performing validity and reliability testing.
*Question 21: The proposed nine questions that will easily be added to a hospitals survey—are these the nine you mentioned earlier and these additions are paid for by CMS?
Answer: The number of questions for the survey and the methodology have not been determined yet.
*Question 22: Early on in this chat you said the purpose was to develop a standardized survey to use in all hospitals. What would/will be more difficult is standardizing the patients in all hospitals. There could be significant differences in survey results due to different patient characteristics hospital to hospital. Does CMS have any plans to address this?
Answer: Casemix adjustment of patient populations will be taken into account in the instrument development process. AHRQ is charged with development of the survey, and the instrumentation team will evaluate casemix adjustment.
Question 23: The Call for Measures that AHRQ published in July contained a number of categories within which items were requested. Where did these categories come from?
Answer: Most of the categories came from a report published by the Institute of Medicine called Crossing the Quality Chasm: A New Health System for the 21st Century. Other categories were suggested by AHRQ staff based on our experience developing consumer assessment instruments through the CAHPS® programs over the past 7 years.
Question 24: Will HCAHPS include measurement of the patient's holistic comfort (physical, psychospiritual, sociocultural, and environmental)?
Answer: As stated in our Call for Measures, we anticipate that we will be developing items within these dimensions of patient-centered care:
As we reviewed the submitted surveys, we may discover additional dimensions to include. These dimensions are consistent with some of the concepts you mention but until we begin actual item selection or construction, we will not know how close the fit is. If you join the HCAHPS LISTSERV® by going to http://www.ahrq.gov/qual/cahps/hcahplist.htm, you will receive updates about progress on this project. Also, available is the HCAHPS mailbox (Hospital-CAHPS@ahrq.gov), which can be used to ask questions or otherwise communicate with AHRQ staff about various issues that arise during the survey development process.
*Question 25: Are these domains identical to those in CAHPS®?
Answer: CAHPS® domains (communication, getting care quickly, being treated with dignity and respect, and health care customer service) relate to health plan selection while Hospital-CAHPS domains relate to acute care facilities.
*Question 26: What is the underlying model for these domains and how was it validated for value, importance, predictive ability?
Answer: The six domains from the IOM were developed through focus groups. The domains of transition of care and access are based on previous CAHPS® work founded on qualitative research, i.e., focus groups and psychometric correlates with overall rating.
*Question 27: Can you tell us if you received suggested questions that would enable you to test measures in each of these domains?
Answer: Yes, the submissions included questions specific to each of the domains cited in the Call for Measures. However, it should be noted that these domains are starting points in the development process and may change as the cognitive and field-testing continue.
*Question 28: It appears as though AHRQ included two dimensions of care that were not mentioned in the Institute of Medicine's Crossing the Quality Chasm:
If not from the IOM report, where do these two additional dimensions of care come from? Could you define, from a hospital perspective, "access to care" and provide some examples where hospitals have a significant impact upon "access to care"?
Answer: The domains of transition of care and access are based on previous CAHPS® work founded on qualitative research, i.e., focus groups and psychometric correlates with overall rating. Access to care means whether a consumer can get the needed care from the healthcare delivery system. Hospitals can have an impact on access with regard to patient/staffing ratios (inadequate staffing may force the hospital to close beds), or when admissions are suspended when all available hospital beds are full.
Question 29: How many responses were received to the request for proposals and who responded? Are the responses public, and if so how do we get copies?
Answer: We received eight responses. Since we offered anonymity to submitters, we cannot, at present, identify them or disseminate copies of their submissions.
*Question 30: The deadline has passed for submission of instruments for consideration. How can an organization submit another instrument(s) for consideration?
Answer: The deadline for instrument submission was September 23, 2002. Instruments submitted after this date will not be considered. However, this will not preclude any additional call for measures if the need arises during the development and testing phases of the process.
*Question 31: Will the eight organizations have an opportunity to defend their submissions?
Answer: Input from the organizations that presented instruments for consideration will be requested by the CAHPS® instrumentation team when required such as in the case of missing information or questions about the materials submitted.
Question 32: Will CAHPS® be developed by AHRQ staff or an external grantee or contractor? Who is the project director at AHRQ?
Answer: The CAHPS® survey will be developed by the CAHPS® Consortium, which includes the CAHPS® grantees from the American Institutes for Research, RAND, and Harvard Medical School and, the project officers from AHRQ (Chris Crofton and Chuck Darby), and from CMS (Liz Goldstein).
Question 33: How was the CAHPS® Team selected as the group that would develop HCAHPS? How were the three CAHPS® II grantees selected?
Answer: CMS asked AHRQ to serve as their research or science partner in development of a hospital inpatient experience questionnaire and asked that the work be performed through the CAHPS® II cooperative agreements. CMS made this request because:
Even if CMS had not specified the CAHPS® team, AHRQ would not have been able to quickly and simply choose a different or preferred organization—a "referee committee," as one enquirer put it—to perform this work. Federal legislation generally requires the award of grants and contracts through a process of full and open competition and AHRQ's authorizing statute and regulations require us to convene a panel of experts to review grant applications and contract proposals and assess each applicant's ability to respond to the issues described in the RFA or RFP. In the case of CAHPS® II, 12 reviewers met in March of this year to review applications and make recommendations for awards. (Of the 12 reviewers, 2 were government employees, 1 from the National Cancer Institute and 1 from CMS.) AHRQ accepted the reviewers' recommendations and selected the top three applicants to receive the awards. We are confident that the reviewers identified three superbly qualified organizations to perform this work, both in terms of their ability to produce high-quality, rigorously tested instruments and in terms of their ability to make objective, professional decisions throughout the instrument development process.
Furthermore, AHRQ decided to award CAHPS® grants to three individual organizations, rather than a single "mega-team," as extra insurance against a single point of view dominating any aspect of the instrument or report development process. Unrestricted by roles in a single hierarchy, the three CAHPS® grantees often disagree with each other, challenge each other to present evidence to support their assertions, and scrutinize all aspects of each others' draft products in the course of their work.
*Question 34: What health care disciplines are represented by the team answering questions?
Answer: The team is made up of expert health care researchers that developed previous CAHPS® surveys experienced in qualitative survey research, report development and document design, process and outcome evaluation and analysis. Agency staff from AHRQ and CMS in consultation with other healthcare professionals including physicians, nurses, statisticians and legal counsel rounds out the team.
*Question 35: Who will be judging and selecting the surveys submitted?
Answer: Members of the CAHPS® Consortium (which includes the CAHPS® grantees from the American Institutes for Research, RAND, and Harvard Medical School) will work with AHRQ and CMS staff to review the instruments submitted and select items to be tested for the HCAHPS instrument. It is likely that items from different instruments will be selected rather than an instrument in its entirety.
Question 36: As it was stated in AHRQ's Federal Register request for submission, you are anticipating creating a mandate that requires hospitals to use an all-inclusive survey instrument. What research have you found that favors this method instead of allowing hospitals to include a few mandated questions in their current survey?
Answer: AHRQ is not a regulatory agency and so cannot "create a mandate" to require hospitals or other organizations to use any survey instrument. CMS, however, does have this power. It is not clear what you mean by "all-inclusive survey instrument." We anticipate that the hospital industry will need questionnaires in addition to HCAHPS, because it is likely that data obtained through the HCAHPS instrument will not be detailed enough to allow hospitals to plan and implement quality improvement interventions. In terms of the choice between "a set of mandated items" vs. a "survey instrument," it seems that resolution of this question depends more on the purpose of the survey and an organization's use of the resulting information than it does on empirical research. CMS' intention in funding AHRQ to develop HCAHPS is to produce a reliable and valid instrument that any organization can use at no cost to obtain patient data about hospital experiences. This will allow consumers to make "apples to apples" comparisons in selecting a hospital and will provide standardized measurement of hospitals across the country.
*Question 37: Will the standardized instrument allow for various research vendors to ask and report the standardized questions rather than mandated participation in one single process?
Answer: CMS is considering different options for national implementation.
*Question 38: Will the survey incorporate measures currently used by vendors such as Press-Ganey to distinguish between satisfaction with physician-care and satisfaction with all other aspects of inpatient care?
Answer: Those instruments submitted either in full or in part in response to the Call for Measures will be considered for inclusion in the HCAHPS survey. Items are being considered that measure specific aspects of physician care in addition to other aspects of inpatient care.
*Question 39: Will there be one standardized survey tool and provider of the tool or do you anticipate a standardized listing of reportable questions?
Answer: We are trying to create one standardized survey tool with sampling specifications. At the same time, we are trying to build in some flexibility by allowing survey vendors/hospitals to add their own questions at the end of the survey and by providing some options for administering the survey.
*Question 40: Is it expected that vendors will incorporate the standard questions into their tools, and then have "proprietary" additional questions? Will having additional questions be acceptable at all in terms of standardization efforts for public reporting?
Answer: It is uncertain at this time, but for CAHPS® instruments as a whole, users have been able to add their own questions with guidance from the CAHPS® surveying and reporting kit.
Question 41: Is HCAHPS for general population, for all hospital consumers regardless of coverage?
Answer: The HCAHPS survey is being developed to be administered to a general population discharged from hospitals. Results of this Web chat and a more general description of HCAHPS will appear on the AHRQ and CMS Web sites.
Question 42: Will Medicare patients or all patients be surveyed?
Answer: In the pilot, all patients will be surveyed. The questions will be designed for all patients and CMS believes that the information will be useful and of interest to Medicare beneficiaries.
*Question 43: Do you intend for the instrument to be all-inclusive covering private, public, and non-profit hospitals?
Answer: The HCAHPS survey currently being developed is intended for use in measuring patient experiences in the hospital setting including private, public and non-profit institutions.
*Question 44: Will you look at specific departments (e.g., high volume departments) within hospitals?
Answer: The Hospital-CAHPS instrument will be designed to measure patient care experiences within the hospital setting and therefore must be general enough to apply across many different types of hospitals. It is a means of obtaining comparative information for consumers who need to select a hospital and also a way of encouraging accountability of hospitals for the care they provide.
*Question 45: Do you have plans to develop a parent/child survey, a maternity survey and a rehab survey?
Answer: Development of other patient care experience surveys is a possibility in the future. However, funding for this effort and anticipated use of the instrument are two factors that would need to be considered before the CAHPS® team proceeded in this direction.
*Question 46: The average person is only admitted to an inpatient setting once every 13 years or so, however, the use of hospital Emergency Departments (EDs) is more frequent. Are other hospital services going to get CAHPS® surveys?
Answer: While there is interest in the other aspects of hospital services, ED services will not be explicitly addressed at this point.
Question 47: Will the surveys be available in languages other than English?
Answer: CAHPS® Surveys are traditionally translated into Spanish; however, we're not sure at this point whether translations of the HCAHPS instrument will be available for use in the pilot due to time constraints. Nevertheless, it is expected that the surveys will be translated into Spanish as well as other languages for national implementation.
Question 48: Unfortunately, many of our Nation's people are functionally illiterate. For example, Washington's Mayor Williams recently stated that over 50 percent of the city's residents are functionally illiterate. What plans to do you have in place so that all patients, illiterate or not, are given an equal opportunity to respond to the survey?
Answer: Since CAHPS® began in 1995, we've been testing proposed instruments and reports with low literacy populations and we will continue to use those methods for HCAHPS. For HCAHPS or any consumer assessment instrument to be useful, people must understand the survey questions, understand how to respond to them in a manner that reflects their actual opinion or experience, understand why the survey results might be useful to them, and how to apply those results to their selection of a hospital. These requirements pose multiple and formidable challenges to any organization developing quality materials for consumer use—challenges that the CAHPS® team has faced since 1995 when AHRQ began funding consumer surveys and reports.
Of the many lessons we have learned in the past 7 years, two are particularly relevant here. The first thing we learned was that the only way to know whether or not people understood a particular concept, question, or response option was to ask them. We used a variety of cognitive testing techniques—think-aloud interviews, usability testing, individual in-depth interviews, and others—to test iterative versions of survey items, instructions, responses and reports. We will use these techniques in development of the HCAHPS instrument and reports to produce documents that are as understandable and accessible to people as we can make them.
Second, we learned that development of understandable surveys and reports is not just a matter of the respondent's literacy, although it is important to use simple words, phrasing, pictures, information-chunking and other strategies to make a document readable by those with reading skills at the sixth-grade or lower. But this only solves part of the problem. The real issue in preparation of understandable surveys and reports is cognitive complexity.
Completing the typical consumer questionnaire requires a person to:
Tasks that require respondents to perform the above are cognitively challenging ones that only 18 percent of the population can perform correctly, according to the last National Adult Literacy Survey. CAHPS® surveys and reports have been specifically designed to address these respondent demands, and reduce the complexity of the tasks.