To assess the feasibility and value of a CAHPS® survey for end-stage renal disease (ESRD) patients, the CAHPS® team met on June 3, 2003 with a technical experts panel (TEP) representing perspectives of the renal community. Focusing on hemodialysis patients, the meeting was held June 3, 2003, in Arlington, VA.
Purpose
Overview
Meeting Highlights
Issue 1. Input to the Feasibility Study
Issue 2. Purpose of the Survey
Issue 3. Value of a Public Domain Standardized Survey
Issue 4. Content of a Survey
Issue 5. Methodology
Issue 6. Reporting Results
Technical Expert Panel and Staff
On June 3, 2003, the CAHPS® team met with a technical experts panel (TEP) representing the renal community to assess the feasibility and value of a CAHPS® survey for end-stage renal disease (ESRD) patients, focusing on hemodialysis patients. Participants included representatives of ESRD network organizations; dialysis facilities, including representatives of national chains; clinicians/nephrologists; clinical researchers; a patient representative; and government agencies involved in the CAHPS® program (specifically the Agency for Healthcare Research and Quality [AHRQ] and the Centers for Medicare & Medicaid Services [CMS]).
The goal of this meeting was to elicit the opinions of the experts on issues that the CAHPS® team will need to resolve to develop a survey of ESRD (hemodialysis) patients. Specifically, the TEP was asked to comment on the following issues:
The group was also asked to comment on the initial research that the CAHPS® team has been doing as part of an ongoing feasibility study.
The meeting is one element of the preliminary, exploratory work that the team is doing to figure out how to proceed with the survey. This work will result in formal recommendations to CMS in the fall of 2003 in the form of a feasibility report concerning the purpose, scope, content, and administration of the survey. Once CMS has decided how to proceed based on the recommendations, the next step will be to develop a draft questionnaire that will be cognitively tested and pilot tested within the ESRD (hemodialysis) population. As with other CAHPS® development efforts, this questionnaire will go through multiple iterations before it is released as a CAHPS® product. It was agreed that a followup TEP teleconference would be held to provide the panel an opportunity to discuss the feasibility study. After a final draft of the survey is complete, it will be available to all stakeholders for review.
The meeting began with a presentation by Charles Darby, the CAHPS® project officer at AHRQ, who gave an overview of the CAHPS® program and an explanation of the principles underlying development of all CAHPS® products. In noting CMS's interest in developing a CAHPS® survey for ESRD patients that would capture their experiences with care, he reiterated the desire of CMS, AHRQ, and the CAHPS® team to better understand the possible uses for this kind of information as well as the challenges they can expect to face.
Following Mr. Darby's presentation, members of the CAHPS® team presented preliminary findings from their research to date:
Finally, Brady Augustine, senior advisor to the administrator at CMS, discussed the agency's interest in improving its oversight of dialysis facilities and using payment incentives to reward high quality. He noted that the agency feels a special responsibility to protect ESRD patients because of their vulnerability and the constraints they face. Through the use of this survey tool, CMS hopes to fill in gaps in its knowledge of patients' experiences and promote improvements in the quality of care they receive. Because the agency wants the survey results to be useful for quality improvement (QI) at the dialysis facility and the ESRD network organizational level, it is particularly interested in learning what would meet the needs of these stakeholders.
In light of this introduction to CAHPS®, CMS, and the research process to date, the participants launched into a full discussion of the issues outlined above. Many of the technical experts also shared information about their own experiences with patient surveys at their dialysis facilities and national companies.
This section summarizes the key lessons about the dialysis market and dialysis patients that came out of this meeting and are likely to influence the development of an ESRD-CAHPS® survey.
ESRD network organizations are different from dialysis facilities. ESRD network organizations are responsible for monitoring quality through data analysis, providing data to CMS in a liaison role between the dialysis facilities and CMS, and providing general information on ESRD care. These organizations do not give direct patient care and do not have the responsibility of measuring patient satisfaction. The dialysis facilities, largely owned by proprietary chains, do systematically collect patients' perceptions of the care that they receive.
CMS requested that the CAHPS® team conduct a feasibility study for ESRD-CAHPS®. This study involves:
AIR had indicated that the focus groups were selected to be heterogeneous. However, several members of the TEP expressed concern that the participants in the focus groups conducted to date may not be representative of ESRD (dialysis) patients. For example, they questioned the assertion that patients may be using more than one facility because they are traveling, noting that ESRD patients are unlikely to travel and rarely have a choice of facility when they do. These concerns made them question the applicability of the feedback to the total ESRD population.
Noting that this approach may not be feasible, some members of the TEP suggested that the CAHPS® team strive for "representativeness" rather than heterogeneity. They were particularly concerned that focus groups would not be able to capture the perspective of hard-to-reach segments of the ESRD population.
A few participants expressed interest in seeing "hard" information on the statistical significance of effects noted in the summary that was presented and evidence that relationships were truly causative rather than just associations. They perceived that the literature review themes contained more conjecture than fact; one person noted that the review was "not convincing."
The TEP had no suggestions related to the interviews conducted by Harvard.
There was a great deal of debate about the purpose of a survey of ESRD patients. The TEP members were particularly concerned that they could not comment on the content or other aspects of the instrument without a better understanding of how the results would be used—which was precisely what CMS, AHRQ, and the CAHPS® team were hoping to learn from the TEP. CMS explained that it has multiple goals in mind but that the survey is not meant to be just for the agencies' use. This is why the agencies want to make sure that all stakeholders get what they need from the survey.
During this discussion, TEP participants raised the following general issues:
Will this survey try to be all things to all people? A few people expressed concern that one survey instrument could not address everyone's needs. For example, they were not sure that the data that CMS would need to report on its oversight activities to Congress would meet the needs of the dialysis facilities that would use it for QI. One participant suggested setting priorities for the instrument rather than trying to do it all.
Is patient satisfaction data relevant? A few people argued that patient satisfaction measures may not be useful for the purposes of either choice or QI. They seemed particularly concerned that information on patient satisfaction would not help them improve the technical quality of care (as measured by intermediate outcomes).
This debate appeared to reflect competing points of view on the purpose of patient satisfaction data: While some of the participants regarded patient experiences as an end in itself, others were more concerned about the links between satisfaction and better clinical outcomes. In response, CMS and AHRQ staff explained that they regard patient satisfaction as one component of quality, on par with mortality, morbidity, and quality of life as components of a balanced scorecard. And while it is a legitimate outcome on its own, they do have evidence that patient satisfaction correlates with clinical outcomes.
In addition, they clarified the distinction between patient satisfaction and patients' experiences with and perceptions of care, which is what CAHPS® instruments measure. Data on patients' experiences have been found to be actionable, by helping people see what is actually happening to the patient and where they might be able to intervene to improve the experience.
QI seen as the most likely use. Several TEP members commented that they currently use their patient satisfaction surveys to improve the patient's experience of care at the facility level. For example, one participant explained that her facilities use the data as a starting point for better understanding patients' concerns; based on the data, her dialysis facilities hold meetings with patients to elicit more information and also to educate patients on why certain things are the way they are. Another noted that, in addition to looking at experiences at the facility level, her survey asked about staff and the vendor used for home-based patients.
However, they were not sure about using data that would come from CMS. To help clarify how this could happen, Amy Heller, who manages the Medicare CAHPS® Health Plan Survey at CMS, discussed how CMS uses CAHPS® survey results as one component of a large toolkit for improving quality. She reviewed how CMS provides the health plans with different cuts and levels of the data, helps them analyze their results, and promotes improvement planning through the Quality Improvement Organizations (QIOs).
Patient choice regarded as not likely. There was broad agreement that ESRD (dialysis) patients were not likely to use this kind of data to choose a facility, primarily because most are constrained in their choices by:
One participant noted that patients do not change centers even when the owners sell the facility or it becomes part of a different business or proprietary chain.
One participant asked whether there is any evidence that people are using CAHPS® data to choose among health plans. Mr. Darby replied that the evidence is not clear, primarily because—for those people who have a choice—the decision is typically a complicated one with many issues to weigh (e.g., cost, convenience, availability of personal physician).
Giving patients a voice. There was general agreement that ESRD patients have a compelling need to be heard. A few participants noted that patients consistently write comments on their surveys—even if there is no space for them to do so. Others commented that these patients find ways to make their needs known, whether by communicating with clinical staff, administrators, or caregivers. Members of the TEP concurred that, regardless of its value for choice, one key benefit of a survey is that it would let patients know that someone is paying attention to them.
Investigating links between patients' experiences and clinical outcomes. Some TEP members appeared to use their data for research purposes to explore potential connections between patient perceptions/satisfaction and objective clinical findings.
Rewarding good quality. CMS indicated that it considered this project to be part of a larger initiative to identify and reward institutions that provide high-quality care. One TEP member noted that the results of a disease-management survey are used for incentive payments to nurses. Another indicated that her company has used patient survey results to determine excellence awards for individual dialysis facilities. However, another participant expressed concern about linking the results to payment incentives. Specifically, this person was concerned that the questions might drive providers to change care processes in ways that are not consistent with good quality. For example, if providers are rewarded for making needle sticks easier on the patient, they will have an incentive to use a catheter rather than a fistula, which may be the better choice from a clinical perspective. This may suggest a need for casemix adjustment to reflect differences in patients' clinical status. It also reinforces the importance of using a balanced scorecard for incentives, so that survey results are only one of several dimensions of performance.
Educating patients. While participants agreed that the survey would have little value for choice, several suggested that it could have value for patient education and patient care. First, the results could be presented in the context of what the dialysis facilities are doing to address patients' concerns and improve their care. Second, the results could help the dialysis facilities identify areas where they may not be doing a good job of educating patients about their disease and their treatment.
CMS noted that standardization of the instrument and reporting of results would provide comparability across dialysis facilities, providers, ESRD network organizations, and geographic regions. A standardized tool would also make it easier for the research community to build upon the data to generate a better understanding of the quality of care for ESRD patients.
At one level, the TEP recognized the value of a scientifically validated and well-tested survey that would produce reliable, uniform data they could all use. One participant commented that she appreciated CMS's ability to call on experts and pay for the development of a survey, because she knows well how much it has cost her in the past to develop a survey. She also respects the knowledge and expertise that the CAHPS® team brings to the task, something she could not have access to on her own.
However, the TEP generally wanted a better understanding of the value to the dialysis facilities and proprietary chains of adopting a different instrument (in contrast to the value to CMS). Specifically, they expressed concerns about the following:
Replacing what they are already doing. Most of the dialysis facilities and proprietary chains represented at the meeting already have their own tools for surveying patients and have established processes for administering their surveys. They expressed concerns that the costs involved in adopting new processes or changing existing ones may be prohibitive.
However, one participant speculated that there may not be any big differences across the proprietary survey, because many of the dialysis facilities and proprietary chains have relied on the same researchers and survey experts to help them develop their instruments.
Losing the ability to look at longitudinal data on performance. These dialysis facilities and proprietary chains have years of data based on their own tools. They are concerned about being able to trend results of a standardized survey against the results of their current and past surveys.
Being able to get answers to their own questions. Some participants pointed out that their companies may still want answers to questions that may not make it onto the standardized CAHPS® survey. Mr. Darby explained that CAHPS® instruments are structured to be flexible in the sense that users can add items to meet their own needs.
Being able to use results for their own purposes. Some participants noted that they currently use the results of their surveys for marketing and branding purposes and expressed concern that a CAHPS® survey would not meet that need.
Having to administer an instrument with irrelevant items. Some participants were concerned that the CAHPS® survey may contain questions that are not relevant to their needs or applicable to their circumstances. (For example, a facility in a rural community may not operate the same way as one in an urban community.) They wanted assurance that the data would be usable and actionable.
When asked to describe their experiences with patient surveys, most TEP members noted the topics or domains that their instruments covered. One participant, who uses a 70-item instrument developed by Dr. Hays, indicated that it covers four domains considered important:
Relationships, or interpersonal care, seemed to be the most important factor for patients. One participant reported that her company used a stepwise regression analysis of survey responses to determine that interpersonal care matters. She also indicated that items about the skills of the staff (specifically about needle sticks) received the lowest scores.
A study conducted by one of the participants found that the following factors were the most important to patients:
Other topics that participants mentioned included:
Create a grid to identify which topics would address potential objectives of a CAHPS® survey of ESRD (hemodialysis) patients. Building on the debate about the purpose of a patient survey, one TEP member suggested that the CAHPS® team create a table that would list the various purposes of a survey across the top and the item topics down the left side. The task would be to fill out the table by checking off the topics that are best suited for a given goal. The intent of this exercise would be to help the team see which items are really necessary, given the priorities of CMS and its stakeholders.
Use the patients' language. Expanding on the idea that patients need to be heard, a few TEP members insisted that the survey instrument use the patients' language and reflect the patients' concerns rather than those of providers and administrators. They also recommended that the instrument be written at a fourth-grade reading level.
Provide a comment field. An open-ended comment field gives patients a way to elaborate on their concerns and raise issues that the survey may not have addressed. For example, one facility was puzzled by patients' discontent with parking, which was right in front of the facility. The comments revealed that patients experienced problems in the winter when the snow prevented them from getting past the high curbs. Comment fields also make it easier for patients to say something about the staff (both to criticize and to praise).
Do research to better understand the expectations of patients and families. One participant noted that the dialysis facilities are not always aware of what patients and families need. She offered the example of a family that was upset that no one from the center had acknowledged the death of a patient. Staff are not required to do that, but they learned that it made a difference to patients and their families to know that the patient would be missed.
Distinguish between physicians and staff. Noting that one item in the CAHPS® Health Plan Survey asks whether the respondent has a personal physician or nurse but then does not differentiate between the two in the remaining items, a few participants recommended that the ESRD survey make a distinction between experiences with physicians versus those with staff. Care at a dialysis facility is very much a multidisciplinary activity, with nurses, social workers, dieticians, and even administrative staff all playing a significant role. However, the TEP members recognized that it may be hard to distinguish among different kinds of staff in the questionnaire, especially if one of the goals is to keep the questionnaire fairly brief.
Keep it short. There seemed to be a consensus that a lengthy survey would not be well received by this population. When asked to comment on the CAHPS® model of core items with optional supplemental items that dialysis facilities or proprietary chains could choose to add, the only negative response from the TEP was concern that the survey may end up with too many items. When asked how long it takes for patients to fill out the existing surveys, most participants indicated an average of 10 to 15 minutes, with some patients taking longer just because they have time on their hands.
CMS asked whether it might be possible to identify a subset of all the questions being asked today (e.g., those on interpersonal relationships) that would be both important to the patients and sufficient to meet the needs of CMS for oversight and public reporting. (The meeting binder included a matrix of the topics covered in the ESRD patient surveys that Westat has been able to collect to date.) This approach might create a "win-win" situation, in which everyone gets what he or she needs. One response to this suggestion was that a subset of items may not meet the needs of State regulators.
Another participant suggested conducting a short (one- to two-page) survey with everyone and a more comprehensive survey with a representative sample (similar to the National Health and Nutrition Examination Survey [NHANES] approach). Then the CAHPS® team could analyze the responses to see if there is a correlation between the results of the two questionnaires and whether the shorter survey is sufficient to give them the information needed. Also, to minimize disruption to facility activities, the CAHPS® team could consider a parallel approach, in which CMS fields a short form with only a few key items that become embedded in the dialysis facilities' long forms.
Offer the survey in Spanish as well as English. Several TEP members indicated that they conduct their surveys in both languages.
Probe to learn about the level of trust. Referring to studies that suggest that problems with trust arise when patients and providers come from different ethnic groups, one participant suggested that the CAHPS® survey ask about trust in the context of the patient-physician relationship.
Don't re-create the wheel. One participant noted that the CAHPS® team should learn from what corporations already do. For example, the pilot instrument should draw from existing surveys.
Test-drive the instrument. TEP participants suggested that the CAHPS® team conduct psychometric testing of the instrument during the development process. They also wanted to know whether the instrument would be field-tested for validity and use.
Are we missing something? As noted, a few participants noted that patients almost always write in comments on their surveys. Most TEP members agreed that this reflected a need to be heard, but another interpretation of this phenomenon was that the surveys did not get at what the patients wanted to say about their facility, which raises the possibility that existing surveys are missing an important topic. One participant noted that she had reviewed the comments to detect any patterns but found none.
Do patients understand the questions? Several TEP members noted that many ESRD patients have low literacy levels, cognitive impairment, and language barriers. They seemed concerned that a CAHPS® survey would be beyond the ability of these patients.