Under its CAHPS® II initiative, the Agency for Healthcare Research and Quality (AHRQ) began to develop a national standard for assessing hospital patient experiences. The preliminary survey instrument for Hospital CAHPS and its methodology are designed to measure and publicly report patient experiences with hospital care.
On November 20, 2003, AHRQ and the Centers of Medicare & Medicaid Services (CMS) sponsored a stakeholder meeting in Rockville, MD.
Purpose / Update on HCAHPS® Development / Additional Test Sites / Reports and Reporting Activities Update / The Quality Initiative / Update on HCAHPS® Initiative / General Discussion / Closing Remarks / Action Items / Participants
Marybeth Farquhar of the Center for Quality Improvement and Patient Safety at AHRQ welcomed participants and thanked them for attending the meeting. She introduced Dr. Robert Graham, Acting Deputy Director of AHRQ.
On behalf of AHRQ, Dr. Graham welcomed participants to the meeting, particularly the California stakeholders participating by videoconference and teleconference. Dr. Graham expressed his delight at having the opportunity to host the day, noting that Dr. Carolyn Clancy, AHRQ director, had been called to an out-of-town meeting. The present meeting agenda would include an update on HCAHPS® activities of the last year. Dr. Graham announced that the revised questionnaire had been transmitted to CMS the previous week for review. He also reaffirmed AHRQ's commitment to seeking the input of stakeholders during the meeting and throughout the HCAHPS® development process.
Dr. Thomas Reilly, Deputy Director of the Beneficiary Education and Analysis Group of CMS, formally welcomed participants on behalf of CMS. He noted that during the last few years, CMS has emphasized public reporting activities. This began with its report on health plans in 1999, followed by the dialysis facilities comparison in 2001. CMS began a nursing home initiative last year, rolled out performance reporting for home health agencies this year, and looks forward to rolling out HCAHPS® within the next year or so.
Dr. Reilly emphasized the seriousness with which CMS takes the charge presented by the Institute of Medicine in its Crossing the Quality Chasm report regarding the multidimensional nature of quality. Consequently, CMS incorporated clinical and "patient-centered" measurements for each of the above initiatives it tackled. For health plans, CMS uses a version of CAHPS® for the Medicare program and the fee-for-service sector. CMS is working with AHRQ, which recently completed a feasibility study to look at developing a version of CAHPS® for kidney dialysis facilities. The two agencies are also working together to develop a prototype for a version of CAHPS® for nursing home residents and their families and look forward to continuing their work on the HCAHPS® initiative. Dr. Reilly ended by thanking AHRQ and its grantees and vendors for their dedication and hard work. He also thanked meeting participants for their commitment to helping CMS "get this right."
Ms. Farquhar thanked Drs. Graham and Reilly and briefly reviewed the agenda before introducing Charles Darby, from AHRQ's Center for Quality Improvement and Patient Safety.
Charles Darby, M.A.
Project Officer, AHRQ
After introducing the CAHPS® II consortium members, Mr. Darby reviewed the goals and timeline of the HCAHPS® initiative. He emphasized the amount of input that AHRQ has solicited and received through its summer 2002 Call for Measures in the Federal Register and its fall 2002 Web chat, stakeholder meeting, and vendor meeting. Since last year's stakeholder meeting, AHRQ modified the instrument based on the results of cognitive testing and a comprehensive literature review, delivered the draft survey to CMS, secured emergency Office of Management and Budget (OMB) clearance in preparation for pilot testing, completed and analyzed the results of the pilot test, and delivered the revised instrument to CMS as scheduled during the first week of November. They will continue testing and anticipate receiving OMB clearance for national implementation—in cooperation with the Association of American Medical Colleges (AAMC), the Federation of American Hospitals (FAH), and the American Hospital Association (AHA)—in spring 2004.
Mr. Darby explained that the CMS pilot test was conducted in Arizona, Maryland, and New York among patients discharged from at least an overnight stay at a medical/surgical or obstetrics unit. The pilot secured 16,619 completed questionnaires, with an overall response rate of 33 percent. Approximately 13,000 surveys were mailed and another 3,600 were conducted over the telephone.
The data from the CMS pilot were used to assess reliability and validity, analyze the casemix adjustment, test the survey methodology, and identify measures useful for public reporting.
The pilot testing showed the domains had a relatively high internal consistency, with the communications domains (communications with doctors and nurses and about medications) having the highest ratings (.67 - .88). Most items in the case mix—which included general health, mental health, education, age, and gender—were not significant, accounting for up to 8 percent of the variance.
The Cultural Comparability Group performed cognitive testing of the instrument and suggested revisions based on their review of the questionnaire's language and cultural impact. Cognitive interviews conducted with native Spanish speakers led to additional revisions in both the Spanish and English versions. Ongoing cognitive testing with the English version may lead to additional refinements.
Upon analysis, the psychometrics of the questionnaire was shown to be reasonably strong. Revisions included:
Mr. Darby noted that although the survey currently includes screeners (i.e., asking whether the person has had a particular experience before asking that s/he rate it), further analysis would determine whether the screeners could be removed.
Because CMS wants to complement, not disrupt, quality improvement surveys already being conducted by hospitals, and provide hospitals with incentives for change, the revised questionnaire may include space for about 30 questions that can be inserted by individual hospitals or vendors. These would be placed after the HCAHPS® items and before the demographic questions. The HCAHPS® instrument team used the precedent set by National Committee for Quality Assurance (NCQA), which currently allows health plans to insert up to 15 items in their instruments. Mr. Darby stated that AHRQ hopes to work with vendors after they have added their items and then conduct an analysis to determine the correlation of vendor items with the existing HCAHPS® items.
Dr. Joachim Roski, a representative of NCQA, asked how the HCAHPS® response rate compares with response rates of commercially available quality improvement (QI) instruments. Mr. Darby responded that although AHRQ did not receive many vendor responses to their request for that data, he believed that commercial products tended to have response rates ranging from 35 to 45 percent. Marsha Nelson, of the California Institute for Health Systems Performance, added that the Patients' Evaluation of Performance in California (PEP-C's) response rate is 45.2 percent.
Elaine Power, of the National Quality Forum (NQF), asked whether the development of the long-term care instrument was synergistic with CAHPS®. Mr. Darby replied that the instruments have the same foundation because many of the same people who developed past CAHPS® instruments are developing the long-term survey. Because many of the issues in acute and long-term care are similar, many of the domains are the same. However, the questions in the long-term survey will also need to capture issues related to quality-of-life and the fact that patients actually live in nursing homes.
Dr. Paul Schyve, a representative of the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), wanted to know whether the survey is capturing racial and ethnic disparities. Mr. Darby commented that the surveys are capturing those data and have found differences by race and ethnicity. However, the CAHPS® instrument team does not feel it is appropriate to make these items casemix adjusters.
In response to another question by Dr. Schyve, Mr. Darby acknowledged that researchers continue to conduct cognitive tests to determine whether patients' experiences with physicians and nurses reflect their interactions with those individuals or whether they actually reflect systemic issues that are not controlled by the staff. Other unresolved issues include that patients might experience a quality difference between day and night staff and that patients often cannot distinguish between nurses and nurse aides.
Dr. Shoshanna Sofaer, of Baruch College (Harvard), commented that patients attribute their experiences to human beings, not to the system. Dr. Schyve asked how the hospitals would know whether to modify their staffing or their system based on the data. Dr. Sofaer responded that the team had tried to operationalize questions so "Does the person show up when I beep?" would become an actionable item.
Both Mr. Darby and Dr. Reilly apologized for not distributing the latest survey draft to meeting participants, saying that it was still under internal CMS review. They promised to send the approved draft to meeting participants as soon as possible, and will post it to the HCAHPS® LISTSERV® shortly afterwards. Dr. Reilly added that it would also be available for public comment as part of the OMB clearance process in early December.
Marybeth Farquhar, M.S.N., R.N.
Health Scientist Administrator, AHRQ
Ms. Farquhar explained that AHRQ began the process of soliciting volunteer sites for additional HCAHPS® testing through a notice in the July 2003 Federal Register. Fourteen organizations—including hospital systems, individual hospitals, and vendors—responded. After screening for location, number of completed surveys anticipated, and number of facilities participating, AHRQ sent the remaining proposals to the CAHPS® grantees to evaluate their technical aspects. They selected five sites to participate in the first wave of testing:
The sites will conduct a number of tests including but not limited to:
The five sites are scheduled to conduct testing between January and April 2004. AHRQ anticipates two more rounds of testing with sites and vendors that had not previously been able to participate in the first wave of testing. Based on this additional testing, it is expected that the HCAHPS® instrument may be revised.
Ms. Farquhar concluded by reminding participants that they can stay informed about the HCAHPS® initiative through the AHRQ Web site (www.ahrq.gov), or signing up for the HCAHPS® LISTSERV® (www.ahrq.gov/qual/cahps/hcahplist.htm). Information about other CAHPS projects can be found on the SUN Web site (www.cahps-sun.org).
Mr. Darby noted that although the psychometric soundness of the instrument is always important, it is also critical to determine how well it can be integrated with existing surveys (e.g., how hospitals draw their samples and how they conduct their data collection).
Nancy Foster, a representative of the American Hospital Association, asked whether the impact of diagnosis was being considered. Mr. Darby responded that it was a good suggestion and should be considered.
Dr. Schyve wondered whether the survey would need to be modified for use in Canada. Mr. Darby responded that the Calgary site would be conducting cognitive tests to determine whether modifications were needed, but any changes would not affect the American instrument. He added that researchers in both Germany and the Netherlands have expressed an interest in using the instrument in their countries.
Christine Crofton, Ph.D.
CAHPS® Project Officer, AHRQ
Dr. Crofton emphasized that the most important part of HCAHPS® is developing a reporting tool that displays the data in a way that can be understood and used by the people who need the information.
The reporting team followed CMS's charge: first, to solidify the information they had on what people want to know before they select a hospital. They also knew that CMS wanted a format that would closely resemble that of Medicare Compare. Finally, they needed to develop and test samples of the HCAHPS® data display as well as the language that would accompany those data displays and composites.
To solicit input about the information people want to know before selecting a hospital, the team used the Call for Measures, but received few actual reports. They also conducted a literature search and contacted individuals currently conducting relevant research.
To solidify their findings and narrow the number of questions in the instrument, the team conducted six focus groups in October 2003. Each group included individuals with a range of hospitalization experiences who were also mixed in terms of age, race, ethnicity, and gender.
In response to open-ended questions, focus group participants confirmed the team's findings that the most critical issues for consumers in choosing a hospital were high-quality care from doctors and nurses (e.g., spend enough time with you; explain problems, procedures, and prognosis; and come when needed). Participants also raised the importance of hospital cleanliness without prompting. When prompted, participants also noted that medication issues, hospital location, and other factors influence the choice of hospital.
The team would like to address additional research questions, such as how the report format and language could be used to reduce the "threat level" of this information for providers and other hospital staff. Because hospital choice is often based on discussions between a patient and his/her physician, other critical questions include how doctors currently talk to their patients about hospital quality and how they might talk about it. Ultimately, clinical measures and HCAHPS® data will be reported together. The team needs to develop ways to present the report so users can easily find the most meaningful sections and not become overwhelmed by too many numbers.
Ms. Foster asked whether the team knows if people want to look at scores just to confirm that their hospital of choice is a good facility, or whether they really want to review comparative information so they can make a selection. Dr. Sofaer acknowledged that the team does not know the answer.
Dr. Sofaer commented that the team is changing its methodological approach in focus groups because members do not believe that asking hypothetical questions ("What would you do if you had certain information") produces responses that predict behavior. The team has begun to look at how people use existing data. They might work with PEP-C or the New York Business Group on Health, both of which have posted survey data that are accessible to consumers. Dr. Kristin Carman, of the American Institute for Research, agreed that the team wants to advance to more controlled and real-life situations, and it might develop videotapes for use in focus groups.
Dr. Sofaer added that the team is also determining the actual pathways through which patients get to a hospital. They know, for instance, that doctors often present patients with a "package" in which they refer the patient to a surgeon who has privileges at a particular hospital. The fact that a surgeon who is experienced in treating their condition practices at a particular hospital might satisfy an important consideration for patients (as indicated by focus group participants)—that is, the hospital is experienced in taking care of the patient's particular condition. Patients seem to have an intuitive grasp of volume/outcome data, or in lay words, "practice makes perfect."
Dr. Sofaer remarked that the interaction between the patient and doctor is critical. The focus group participants were virtually unanimous in reporting that their primary source of information about a hospital is their physician. Dr. Crofton noted that this raises a number of questions, such as what kind of information physicians need to judge the quality of a hospital versus the kind of information needed by patients.
Dr. Everett Jones, representing the Department of Veterans Affairs, stated that although patients might be satisfied with a composite index that includes both clinical performance and HCAHPS® indicators, physicians would want to see the actual data and judge their validity.
Dr. Schyve reported that the JCAHO recently decided to add the HCAHPS® data to the clinical indicators currently on its Web site. JCAHO wants to provide the data in a summarized format consumers can "dig down into" if they want more detailed information. JCAHO is leaning toward a model that promotes creating a single "rollup" measure that represents the percentage of times the hospital "did the right thing." The denominator in the rollup measure is the number of times the hospital has the opportunity to do the right thing, while the numerator represents the number of times the right thing was actually done. Dr. Schyve said trust is such a critical part of health care that it is crucial to provide the more detailed data—even if patients never use it—so the information is perceived as trustworthy.
Dr. Schyve raised the issue that HCAHPS® will at times create conflicts of interest for physicians. One potential conflict involves physicians who receive data that lead them to advise patients not to use a hospital at which the physician him/herself practices. HCAHPS® must determine how to help physicians deal with this dilemma.
Dr. Dana Keller, of the Delmarva Foundation for Medical Care, noted that patients would be more likely to use the information if they knew numerical differences among hospitals indicated a significant difference in the quality of care they would receive in a particular hospital. There was general agreement that the HCAHPS® reporting mechanism should help interpret the data for patients.
Dr. Patricia Rowell, a representative of the American Nurses Association, commented that the report should allow consumers to choose the categories and amount of information they want to access. Each composite could contain a list of subheadings so consumers could select more or less information on their particular condition, medication class, or other factor of interest.
Ms. Power expressed concern that focus group participants might not have raised certain issues simply because they were unfamiliar with those issues. Drs. Sofaer and Carman noted that by giving the participants the domains, they were actually raising awareness about the issues within those domains. Dr. Crofton acknowledged that focus groups have both strengths and limitations and AHRQ might also consider interviewing patients after discharge. Audience members added that it would be helpful to ask patients who had used the data whether they would use them again and whether they would recommend them to a friend.
Dr. Schyve asked what kind of information was lost when the team went from nine to seven domains. Mr. Darby replied that although some information on admissions, privacy, and family issues was lost, most of the changes were simply repackaging of the same information.
Nancy Foster
Senior Associate Director, American Hospital Association
Ms. Foster provided background on The Quality Initiative, which was launched by the AAMC, the FAH, and the AHA as a result of discussions with their members. The member hospitals were committed to providing the best possible patient care and to being worthy of the trust placed in them by their communities, but were faced with multiple outside efforts to evaluate them, from multiple sources, with no voice in the efforts. Despite the fact that the hospitals were working to produce the data required by each survey, the results of the surveys were often divergent and therefore difficult for the hospitals to respond to or understand. Additionally, it did not appear that anyone else was paying attention to the results.
In December 2002, the three associations announced The Quality Initiative, an effort to create a more unified approach to collecting hospital performance data and sharing that information with the public. A number of agencies and organizations support the Initiative, including CMS, AHRQ, JCAHO, NQF, the AFL-CIO, AARP, and the National Association of Children's Hospitals and Related Institutions.
The Quality Initiative is initially focused on hospital performance on 10 quality measures for three medical conditions for adult patients:
These 10 measures are common to the JCAHO's ORYX program, CMS' Seventh Scope of Work Statement, and NQF's core measure set.
The partners' commitment is to create a vehicle to disseminate the information to the public, to include HCAHPS® survey results as soon as possible, and to add information on other clinical conditions as it becomes available. They also want to encourage organizations that are currently collecting quality data from hospitals to use this common set of measures to reduce redundant, duplicative data collection efforts.
The Quality Initiative released its first data on a CMS Web site (www.cms.hhs.gov/quality/hospital). Information will be available on a consumer Web site in fall 2004.
So far 2,197 hospitals have pledged to participate in The Quality Initiative, a number that grows daily. AAMC, FAH, and AHA are working to encourage hospitals that have not yet pledged to join the collaborative, but find this is a time-consuming process that often involves talking one-on-one with hospitals. Many of the reasons for not pledging are simple to resolve (e.g., had not gotten around to it, the hospital system and its component hospitals each thought the other had done it, and being unclear about the relationship between The Quality Initiative and HCAHPS®). One of the most difficult issues raised is whether HCAHPS® and clinical data will be applicable to small hospitals, whose modest numbers of cases will not produce statistically meaningful data. Ms. Foster noted that they are trying to identify clinical measures that would be relevant to small hospitals and would like assistance from the HCAHPS® team in the effort. The burden of reporting is another concern of hospitals. Specific burden-of-reporting issues—including adding Health Insurance Portability and Accountability Act (HIPAA) recordkeeping requirements, expanding the number of measures collected, changing the ways hospitals currently survey patients, and increasing vendor costs—must all be considered before a hospital pledges.
Ms. Foster reviewed the intricate process involved from the point of data collection to actually posting reports on the Web. A number of unforeseen technical problems are gradually being resolved, including difficulties in data collection and transmission that slow down the reporting process and prevent data from being posted in a timely fashion. Another issue is that applicable HIPAA regulations added a new requirement to quality improvement efforts. Hospitals involved in such efforts must now maintain copies of individual records in case a patient requests disclosure. (This applies to both clinical and HCAHPS® data.)
Ms. Nelson asked whether hospitals that pledge are obligated to submit both clinical and HCAHPS® data. Ms. Foster replied that although AHA's attorneys say pledges encompass both initiatives, this is a voluntary program and hospitals can submit whatever data they choose. Although it appears from the current data posted on the CMS Web site that participating hospitals are not reporting on all measures requested, in most cases "not available" means that the data have not been getting into the quality improvement data storage facilities consistently.
Dr. Schyve asked whether a hospital that decided to collect only clinical data, for instance, but not patient perception data, would be allowed to continue to participate. Ms. Foster responded that leeway is being given during the startup phase, but that eventually those decisions would need to be made.
Margaret Reagan, a representative of Premier Inc., asked whether The Quality Initiative team could identify the kinds of problems smaller hospitals are experiencing as they try to gather the required data. Ms. Foster reported that The Quality Initiative is aggregating data over a year so smaller hospitals' data can be posted on the Web site as soon as they reach the minimum sample size (25) required. Because some hospitals are so small that they will not reach that minimum number within a year, The Quality Initiative is working to either identify alternative measures that might be more effective or to look at the validity of using a minimum smaller than 25.
To help with the sample size issue confronted by smaller hospitals, Dr. Schyve suggested that The Quality Initiative consider the "rollup" measures previously discussed (denominator = opportunities to do the right thing; numerator = number of times the right thing was done). The result would be that the primary measure for smaller hospitals would be the same as the rollup measures for larger hospitals. Although consumers could "drill down" to get more detailed information for larger facilities, that would not be possible for the smaller hospitals.
Dr. Keller suggested that small, geographically clustered hospitals could be aggregated. Although the process would not provide the public with information about individual facilities, it could help the hospitals with their own quality improvement initiatives.
Joyce Dubow, AARP, asked whether CMS has plans to require hospital participation in HCAHPS®. Dr. Reilly responded that CMS views HCAHPS®as part of the national voluntary initiative described by Ms. Foster. Although CMS is building an infrastructure for HCAHPS® implementation, there are no plans to make participation mandatory.
Dr. Sofaer raised the concern that when data displays show "NA," consumers will assume the hospital is trying to hide something no matter how many technical reasons are given to the contrary. This generated quite a bit of discussion from stakeholders, one of whom suggested that hospitals could, in fact, hide behind "NA," while others suggested nonvalue-laden wording for footnotes. Ms. Nelson said PEP-C provides multiple footnotes when data are missing. They found that not being in a report is perceived more poorly than low scores—particularly among the media. To Michelle Fried's (Federation of American Hospitals) question as to whether consumers read, understand, or believe footnotes, Dr. Sofaer replied that there would always be a subset of skeptical consumers who will not believe explanations. Dr. Schyve added the need to educate the public and monitor how they interpret the data. As consumers become more familiar with this type of data, the reports' messages need to change.
Dr. Schyve also reported on decisions made at the recent JCAHO board meeting as part of their goal to improve the usefulness of information they provide the public. As of July 1, 2004, what JCAHO currently calls "Performance Reports" on its Web site will be renamed "Quality Reports." The reports will include:
Dr. Jones recommended hospitals be given time to prepare and begin collecting the data. Ms. Foster added that hospitals all collect quality data, but HCAHPS®measures are an unknown. Implementation will be complicated by factors such as cost (passed on by vendors) and compensation packages for Chief Executive Officers (CEOs) being tied to survey results.
Ms. Dubow stressed that data on patient perception is critical, and despite the added burden on hospitals, the field should make this information available to consumers as quickly as possible.
Dr. Sofaer recommended the group consider using Rhode Island's process as a model. When that State implemented mandatory reporting, hospitals had a period of time in which they were able to see benchmarks and their own data before the second round of data was collected and released to the public. This raised the hospitals' comfort level, gave them time to work on improvement, and allowed them an opportunity to provide feedback to the survey team.
Dr. Beth Kosiak, a representative of AHRQ, agreed and suggested The Quality Initiative team might want to look at the lessons learned from the Rhode Island experience. Lee Partridge, of the National Partnership for Women and Families, cautioned that the team might want to host a series of "how-to" workshops before instituting this process because the data collected would be accessible through The Freedom of Information Act.
Mr. Darby noted that hospitals will probably want to know why they need to collect HCAHPS®data when they are already conducting patient satisfaction surveys. Dr. Rowell remarked that the ANA has studied patient satisfaction instruments within its National Database for Nursing Quality Indicators to see whether item equation could be done among instruments. They found the major vendor instruments could not be compared because they measure different concepts. Dr. Rowell added that one of the advantages of HCAHPS®is it would be a standard instrument that would allow comparison among facilities.
Ms. Nelson commented that PEP-C found that most hospitals collect patient satisfaction data, which are different from measures of patient experience. In her experience, it takes time to help the hospitals understand why the measures do not compare and how to use both effectively.
Ms. Foster added that the only real improvement in hospital performance would occur when providers and other caregivers within the system take responsibility for the issues raised by data. Several participants agreed, noting that the information needs to be interpreted and filtered down to all hospital staff, along with strategies for how they can change performance.
Dr. Schyve commented that the capacity to collect clinical data is technological, and the data are already collected as a routine part of clinical care. He asked whether the HCAHPS®team has considered how to reduce the costs associated with collecting data on patient experience, which is an addendum to care. Mr. Darby responded that this has been one of the team's prime considerations. They have emphasized integrating HCAHPS® with existing surveys and ensuring that the instrument itself is brief. They are also considering using data collection methodologies beyond telephone calls and mail, such as computers.
1. Dr. Schyve provided additional clarification: "There are two separate potential JCAHO uses of HCAHPS®: (1) As a source of data for organization-specific Quality Reports that will be publicly available on the JCAHO Web site (www.jcaho.org) as of July 1, 2004, or (2) As a requirement for accreditation (i.e., a standard). In the Quality Report, the accreditation status is supplemented with additional information, specifically Performance on National Quality Improvement Goals (i.e., ORYX data); Performance on National Patient Safety Goals; Special recognitions, such as Disease-Specific Care Certification, Baldrige Awards, and Codman Awards. This supplemental information will include HCAHPS® data on patient experiences of care under two circumstances: (1) If CMS collects the data on a voluntary basis and is willing to share the data with the Joint Commission, they will be included in the Quality Reports for those hospitals that volunteer. (2) If CMS requires that the data be collected and is willing to share the data with the Joint Commission, they will be included in the Quality Reports. With regard to accreditation, if the use of HCAHPS® is made a Medicare Condition of Participation (COP), the Joint Commission would likely make its use an accreditation requirement (i.e., a standard). If it were not a COP (e.g., its use was related only to differential payment), the Joint Commission would be unlikely to make it a standard. However, even if its use were not required by CMS, the Joint Commission has the opportunity to encourage HCAHPS use by revising current standards that address collection of data on patient satisfaction/experience of care, for example, by specifying the collection of data on those domains included in HCAHPS®."