Physicians' Perceived Usefulness of and Satisfaction with Test Reports for Cystic Fibrosis (ΔF508) and Factor V Leiden
Print version

Physician survey

We developed a one-page survey containing a total of 22 items, including demographic information (age, ethnic origin, and sex). There were Likert-type questions asking physicians to rate perceived usefulness of specific report characteristics or components on a scale ranging from 1 (poor) to 5 (excellent), with options also available for "not applicable" and "no information provided." Two additional Likert-style questions asked about satisfaction with the report (response options were "strongly agree" to "strongly disagree") and physician test-ordering frequency for CF (ΔF508) or fVL (response options: 0 times per year, 1-2 times per year, 3-5 times per year, and more than 5 times per year). There were three open-ended questions included to identify recommendations for more effective test reporting:

• "What additional information would you like to have included on the report?"

• "What information do you think should be omitted from the report?"

• "What modifications can you suggest to improve the report?"

Survey packet

The survey packet mailed to physicians consisted of an introductory letter, one randomly assigned mock report (for either CF or fVL), and the one-page survey. The "introductory" letter described a brief clinical scenario for either CF or fVL and provided instructions for review of the enclosed report and response to the statements regarding the "usefulness of the information in the care and management of the patient presented in the case scenario." The scenario for CF was as follows: "You requested DNA testing for CF from a 2 month old male who presented with failure to thrive and recurrent pneumonia." The scenario for fVL was as follows: "You requested DNA-testing for fVL from a 22 year old female who came to you with a recent history of an unexplained vascular accident." The survey was developed with input from a focus group of physicians and public health practitioners. Administration of the survey packet was piloted by 30 local physicians from the specialties included in this study. Nonresponders were sent up to two additional packets. The data collection period extended from April to October 2001.

Analysis of data

Analyses were performed using SAS (Cary, NC) software. CF and fVL data were analyzed separately. "Not applicable" responses were treated as missing data and were not scored; "no information provided" was scored as 1 (1 = poor). Using two-factor analysis of variance (ANOVA), we examined report version by specialty interaction. Because there were no version-by-specialty interactions identified, we were able to examine version differences averaged over specialties. Newman Keuls post hoc procedure was used to explore significant F statistics. The association between satisfaction and report version was examined using two-factor ANOVA; Newman Keuls post hoc procedure was used to explore significant F statistics. The association between frequency of test ordering (frequent users defined as ≥3 times per year vs. infrequent users defined as 0-2 times per year) and perceived usefulness of report items was determined using two-factor ANOVA. Report version by frequency-of-test-ordering interactions were explored. Because there were no report version by frequency-of-test-ordering interactions identified, we were able to examine frequency-of-test-ordering differences averaged over report version.

RESULTS

Cystic fibrosis

The response rate for all physicians was 36% (148/411). The response rate differed by specialty: 27% by FP, 34% by pediatricians, 42% by pediatricians with a secondary specialty of pulmonology, and 44% by pediatric pulmonologists (P = 0.035). There were no significant age, sex, or US regional differences detected between responders and non-responders.

Table 1 presents the survey-response mean scores, P values, and Newman Keuls analyses for each of the survey items assessed. The correlation of each item with satisfaction with the survey is also presented in Table 1. With the exception of "test report format" and "contact information provided," there were significant differences by report version for all of the items. For the "test limitation information," "clinical history," and "information on linkage between ethnicity and mutation panel used," the most comprehensive report (i.e., mock report A) was scored higher than the other mock reports (indicating greater usefulness). For the remaining components, the most comprehensive and the intermediate reports (i.e., mock reports A and B) had higher scores than the least comprehensive report (i.e., mock report C). There was a significant association between satisfaction and report version (P < 0.0001), demonstrating greater satisfaction among responding physicians who were sent the more comprehensive reports. A significant difference was detected by specialty regarding frequency of CF tests ordered per year [χ² = 105.1, 3 degrees of freedom (df), P < 0.0001]; pediatricians with a secondary specialty in pulmonology (68%) and the pediatric pulmonologists (98%) were more likely to order the test more than three times per year compared with the FP (3%) and pediatricians (3%). Overall, 33% of the FP and 42% of the pediatricians responding indicated that they ordered the test at least once. No significant difference in survey responses was found between the pediatricians with a secondary specialty of pulmonology and pediatric pulmonologists. Significant differences were found between frequency of test ordering by physician and perceived usefulness of selected report items [i.e., test performed (P < 0.0001), test limitations (P = 0.01), report format (P < 0.003), linkage between ethnicity and mutation panel (P < 0.0001), recommendation for genetic counseling (P = 0.002), and implications for family members (P = 0.025)]. These results reveal that physicians who order the CF test more frequently reported less perceived usefulness of these items.

We determined correlations between each of the survey items and overall satisfaction with the report (Table 1). The correlations ranged between 0.40 and 0.77; only one correlation fell below 0.5 (i.e., contact information).

In an effort to assess recommendations for effective test reporting, we did a qualitative assessment of the responses from the open-ended questions listed at the end of the survey (see Methods). One hundred one (68%) of the physicians responding to the CF survey included answers to one or more of the questions; some comments differed by specialty and report version. Several comments that were common to all specialties and report versions are noteworthy. A summary of the general recommendations reflected in the responses are as follows: (1) provide sufficient information for clinical interpretation and clinical decision-making, (2) include an indication of the value of a sweat chloride test in light of the results provided, (3) provide additional information regarding patient and (patient's parent) ethnicity and how it relates to the disease, (4) use a format in which the test result and interpretation are obvious and, (5) make clear that mutation panels applied do not cover all mutations that may contribute to disease.

Factor V Leiden

The overall response rate was 25% (89/353). The response rate did not differ significantly by specialty: 28.6% for FP, 19.5% for internists, and 27.6% for hematologists (P = 0.21). There were no significant age, sex, or US regional differences detected between responders and non-responders. Table 2 presents the results of the survey response mean scores, P values, and Newman Keuls analyses for each of the survey items. The correlation of each item with satisfaction with the report is also included. With the exception of "test limitation information provided," there were significant differences by report version for all of the items. For the "clinical history," the most comprehensive report (i.e., mock report D) was scored higher than the other mock reports. For the remaining components, the most comprehensive and the intermediate reports (i.e., mock reports D and E) had significantly higher scores when compared with the least comprehensive report (i.e., mock report F). The higher scores indicate greater usefulness.

A significant association was detected between satisfaction and report version (P < 0.0001) in which physicians responding to the survey reported greater satisfaction with the more comprehensive reports. A significant difference by specialty was detected regarding frequency of fVL tests ordered per year (χ² = 35.6, 2 df, P < 0.0001); internists (41%) and hematologists (76%) were more likely to order the test more than three times per year compared with the FP (3%). No significant differences were detected between frequency of test ordering by physician and perceived usefulness of report items.

The correlation between usefulness of the items and overall satisfaction with the fVL reports are provided in Table 2. All of the correlations are above 0.50.

For fVL, 62 (70%) of the responding physicians included answers to one or more of the open-ended questions; some comments differed by specialty and report version. Several comments were common to all specialties and report versions that are noteworthy. A summary of the general recommendations reflected in the responses are as follows: (1) provide sufficient information for clinical interpretation and clinical decision-making (e.g., sensitivity, specificity, positive and negative predictive values, correlation with deep venous thrombosis, residual risk); (2) include sufficient clinical history; (3) provide additional information regarding other risk factors for the disease; and (4) use a format in which the test result and interpretation are obvious; the use of the "pos-neg" nomenclature on the report was confusing.

DISCUSSION

Concerns exist that medical genetic test-result reports fail to meet the clinical decision-making needs of physicians from various specialties. ^{7, 8} In a previous study, we identified substantial variation in medical genetic test-result report content for both CF and fVL test reports obtained from North American laboratories listed in the GeneTests database. ¹ Given the variable test-report content found in these two genetic tests, we sought to study the physician-perceived usefulness in clinical decision making of medical genetic test reports that differed in content. From the results in this study, it is evident that physician-perceived usefulness of and satisfaction with the medical genetic test reports for CF and fVL differed by report content (most comprehensive versus intermediate versus least comprehensive). The physicians participating in this study, regardless of specialty, reported greater usefulness of and satisfaction with more comprehensive reports. In addition, recommendations for improving usefulness of CF and fVL genetic test reports were suggested by physicians.

Physicians generally perceived the intermediate and most comprehensive reports to be significantly more useful than the least comprehensive report, irrespective of physician specialty. The more comprehensive reports (mock reports A, B, D, and E) contained additional information regarding clinical history, ethnicity, mutation panel, methodology, recommendations for follow-up testing, genetic counseling, and implications for family members (Fig. 1). The least comprehensive report included only test performed, test limitations, test result, and contact information. Specifically, the three content items for both CF and fVL mock test reports in which physician-perceived usefulness was most highly correlated with satisfaction were (1) information for clinical decision making, (2) genetic counseling information, and (3) implications for family members. These report items facilitate the action-oriented plan for physicians and help guide their clinical care. Each of these items is recommended by CLIAC and NCCLS for inclusion in medical genetic-test reports. Our previous study ¹ identified only 61% and 52% of CF and fVL laboratory reports, respectively, including recommendations for genetic counseling. This study reinforces the need for this information, which is based on the physicians' perceived usefulness of these items.

We also analyzed perceived usefulness of report items by frequency of test ordering. We found that, for CF, physicians who ordered the test at least three times a year responded that information on the report such as what test was performed, limitations of the test, linkage between ethnicity and mutation panel, recommendations for genetic counseling, and implications for family members was perceived to be less useful when compared with physicians who order the test between zero and two times per year. No such differences were identified for fVL reports. The differences for CF reports and not for fVL reports may have been identified because there was a higher specialist response rate for CF than for fVL (see Results). In addition, CF testing has greater complexities than fVL testing. The specialist physicians may be more familiar with these complexities and the interpretation of these test results and thus perceive selected report items as less useful. However, it is important to remember that the ordering physician is not the only recipient of the genetic test-result report and that other health care providers may find these items useful in caring for patients who undergo genetic testing for CF.

Other studies have identified deficiencies in the knowledge of the ordering physician regarding genetic services. ^7-9 These studies suggest a significant risk among primary care physicians and non-genetic specialists for ordering genetic tests inappropriately and for interpreting results incorrectly. In a study specifically designed to assess the physician's ability to correctly interpret genetic test results, 32% of physicians incorrectly interpreted a test result for familial adenomatous polyposis. ⁸ Further, another study by Sandhaus and coworkers reported that many physicians were unprepared to interpret genetic risk information presented in a hypothetical breast cancer report. ⁹ A factor influencing appropriate interpretation of test results is information in the report that is deemed useful in clinical decision-making. Our results suggest that physicians, regardless of specialty, prefer additional information in the test report that will facilitate understanding and interpretation of the result. The test report serves as a result form for the ordering health care professional, a resource for information pertinent to the disease, and a record of result for future use and reference. Since health care workers utilizing the test report may not always be the ordering provider, a comprehensive report which includes all pertinent patient information (such as clinical history, age, ethnicity) and the test result offers a more useful tool in clinical decision-making. Factors, which mitigate the usefulness of the report, include the lack of relevant clinical information and lack of information regarding relevance of the test result to disease state and other family members. For laboratories to provide more comprehensive reports, all information requested by the laboratory must be provided by the referring health care provider. Laboratories need to develop a requisition and reporting processes that satisfy the needs of both the physicians and the laboratories and are consistent with practices recommended by professional organizations. Ideally, report forms, which solicit pertinent patient information, will optimize health care professionals inclusion of such information. Future efforts should be directed toward evaluating how different test reports impact action-oriented plans in the clinical setting.

The limitations of this study include a relatively low response rate. However, this rate is consistent with other response rates reported for other studies using physician surveys. ¹⁰ Nevertheless, we had a sufficiently large sample to detect statistically significant differences from a geographically diverse population of physicians with a variety of specialty backgrounds that potentially order these genetic tests. Since the initiation of this study, several guidelines and consensus statements have been published regarding CF and fVL testing and results reporting. ^2-6 Another potential limitation of this study is the "halo effect" of a longer report, resulting in higher perceived usefulness on more comprehensive mock reports of items that were included in all mock reports (e.g., test performed, test result, test limitations). Ideally, there should be identical perceived usefulness for these items between the three mock reports. However, for these report items, it is possible that the additional information provided in the intermediate and most comprehensive reports (e.g., test methodology, implications for family members, recommendation for follow-up testing) further defined the most basic elements included in the least comprehensive reports, thus resulting in greater perceived usefulness of these items in more comprehensive reports. We believe that a potential halo effect had minimal impact on these study results because the Newman Keuls analysis (Table 1, Table 2) revealed differences in perceived usefulness by mock report version for items that were only included in the most comprehensive report (clinical history, CF, and fVL) and linkage between ethnicity and mutation panel (CF only); for items included in the intermediate and comprehensive reports, the Newman Keuls analysis revealed the appropriate differences.

This study assessed physician-perceived usefulness of and satisfaction with medical genetic test-result reports for both CF and fVL in a geographically diverse, multispecialty sample of US physicians. We determined that physicians, regardless of specialty, reported greater perceived usefulness of and satisfaction with more comprehensive reports. The content items of these reports are consistent with those recommended by ACMG and The American College of Obstetricians and Gynecologists (ACOG). Future studies are necessary to evaluate whether greater satisfaction and enhanced perceived usefulness are associated with more accurate interpretation of the test result and appropriate clinical management. In addition, the effect of physician specialty and training background should be assessed in the use of medical genetic tests.