HCAHPS Sources

Diverse sources were drawn on in development of the preliminary Hospital CAHPS® (HCAHPS) CMS Pilot Test Questionnaire. The sources and the processes involved are summarized below.

Developing the Preliminary HCAHPS Questionnaire
Responses to the Federal Register Notice
Identifying Submissions that Influenced the Preliminary Questionnaire
Results

Developing the Preliminary HCAHPS Questionnaire

The HCAHPS Instrument Team reviewed items from three sources:

CAHPS® core survey items.
Items submitted by organizations that responded to the Federal Register notice Call for Measures published July 24, 2002.
Literature review conducted by the HCAHPS Team of relevant English language literature on hospital questionnaires and survey methods.

Return to Contents

Responses to the Federal Register Notice

We are grateful to organizations that responded to our request for measures. Seven sets of questionnaire items were submitted by nine organizations (three organizations submitted together). By submitting items or item sets, organizations agreed to these conditions:

The CAHPS® team could consider an item for use "as is" or with modifications in the HCAHPS questionnaire.
For all HCAHPS users to have free and open access to the final HCAHPS questionnaire, submitters must relinquish any copyrights in items used.

The submissions were from:

Avatar International Inc.
Edge Health Care Research Inc.
Healthcare Financial Management Association (HFMA).
Jackson Organization.*
National Research Corporation (NRC).
Peace Health.
Press Ganey Associates.*
Professional Research Consultants (PRC).*
SSM Health Care.

*Submitted together.

Return to Contents

Identifying Submissions that Influenced the Preliminary Questionnaire

To identify items submitted in response to the Federal Register notice that influenced the preliminary Hospital CAHPS CMS Pilot Test Questionnaire, the following process was developed:

Contract staff members created a data base of all items submitted in response to the Federal Register notice.
The HCAHPS instrument team reviewed all submitted items and determined which items might be useful for the task of writing items for the preliminary Hospital CAHPS CMS Pilot Test Questionnaire. As the review proceeded, items were either eliminated or retained for further consideration.
A contract staff member (Coder 1) reviewed discussions on each item and coded it as either influencing the preliminary HCAHPS Questionnaire or not.
A copy of the original (not annotated) data base of items submitted in response to the Federal Register notice was provided to a member of the CAHPS® Team (Coder 2) who had not been involved in the review performed by the HCAHPS Instrument Team and did not know the results of the review. Coder 2 was asked to assign each submitted item to one of the following five categories:
1. Source item used as is.
2. Source item used with minor changes to response scale, items stem or both.
3. Source item used with major changes to response scale, items stem or both.
4. Source item was similar in content or concept to item in the preliminary HCAHPS questionnaire.
5. Source item not used at all.
If an item was coded a 1, 2, 3 or 4, it was considered to have influenced the preliminary HCAHPS questionnaire.

A team comprised of Coder 1 (who originally assessed the influence of submitted items), Coder 2 (who assessed them independently), and staff from AHRQ, RAND, and Westat reviewed a comparison between the original and second assessments to reconcile differences between the two.

Agreement between coders was defined as one of the following:

Coder 1 coding that the item influenced the preliminary questionnaire and Coder 2 coding the items as 1, 2, 3, or 4 (and influencing the preliminary questionnaire).
Coder 1 coding that the item did not influence the preliminary questionnaire and Coder 2 coding the item as 5 (and not influencing the questionnaire).

Each item in which the coders did not agree was reviewed. Coders gave their rationales for their coding choices. The team arrived at a consensus about a final coding of the items.

Return to Contents

Results

In this analysis, the team did not include screeners, demographic, and descriptive items. These are standard in the field and do not reflect the patient assessment of health care (reports and ratings) that will be used to compare hospitals.

Of the 294 items submitted that dealt with assessment of health care, 78 items influenced the preliminary HCAHPS CMS Pilot Test Questionnaire. In the preliminary questionnaire, there are 38 items on patient assessment of health care. Of these, 22 were influenced by items from more than one submitter.

Of the 294 items, the two independent coders initially differed on 40 items regarding whether they influenced the preliminary questionnaire. The process of determining the influence of items on the preliminary HCAHPS Questionnaire was subject to some coder judgment and the resulting attributions were imperfect. However, the reasonably high agreement between the two coders (86.4 percent) indicates a high degree of objectivity in the assessment process.

The adjudication resolved these differences. Using the final adjudicated data base, items that reflect the reports and ratings of respondents in the preliminary HCAHPS questionnaire were identified (Table 1).

Table 1. Influence of Submitters' Items on Preliminary HCAHPS Questionnaire

Vendor	Total Number of Patient Assessment Items Submitted	Number of items which influenced the draft HCAHPS instrument
Avatar	59	19
Edge Health	69¹	19
HFMA	9²	0
Press Ganey/Jackson Organization/PRC	9	3
NRC	66³	20
Peace Health	28	6
SSM	54	11

CAHPS 3.0 Core		11⁴
¹Edge Health submitted two surveys for consideration. The total number of items submitted is the number of unique items across both surveys. ²All items from HFMA deal with billing issues not in the preliminary HCAHPS questionnaire. ³Only items from the NRC Adult Inpatient Survey were counted. ⁴This count does not include demographics questions. Total items pulled from CAHPS 3.0 including demographics are 21. Total number of items in draft HCAHPS Instrument: 66. Total number of items submitted by vendors: 294. Total number of items submitted and coded either 1,2,3 or 4: 78.

In summary, the HCAHPS survey consolidates a wide range of items submitted by various organizations for possible inclusion in the preliminary instrument and the CAHPS® Core items.

Return to Contents

The analysis of the items in HCAHPS is not meant to endorse or should it be construed to mean that one instrument, based on the number of items accepted for HCAHPS use, is better, or is worse than any other instrument submitted or any that were not submitted. Likewise, inclusion of items in the final HCAHPS questionnaire is not a reflection of the utility or quality of unused items for purposes other than set out for HCAHPS, nor is the use or adaptation of particular items for the CMS and HCAHPS objectives, meant to endorse or otherwise promote a specific organization. Inclusion of selected validated items in the final HCAHPS questionnaire will be based on the results of the CMS pilot test and of additional testing by AHRQ as well as the current literature available.

These items are currently being evaluated for validity and reliability through various mechanisms such as the CMS pilot test and additional testing conducted by AHRQ.

Current as of July 2003

Internet Citation:

HCAHPS Sources. July 2003. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/qual/cahps/hcahpitem.htm

Return to CAHPS®—Consumer Assessment of Health Plans
Quality Assessment
AHRQ Home Page
Department of Health and Human Services