Robert Trowbridge, M.D.
University of California, San Francisco School of Medicine
Scott Weingarten, M.D., M.P.H.
University of California, Los Angeles School of Medicine
Background
A number of techniques have been used to modify the behavior of practicing physicians.1 Continuing medical education, practice guidelines and critical pathways represent a major thrust of these efforts. The relative effectiveness of each is largely dependent on the particular strategy employed in their implementation.2 Traditionally these strategies have focused on lectures and printed materials but other techniques have also been utilized, including audit and feedback, academic detailing, local opinion leaders and reminder systems. In addition, some have championed the use of sentinel event reporting and root cause analysis in graduate medical education programs.3 Only recently have these various techniques been critically evaluated for their effectiveness at changing physician behavior.
This chapter reviews the evidence regarding the utility of educational-oriented techniques to improve provider behavior, particularly as they do or might pertain to patient safety practices. Incident reporting (Chapter 4), root cause analysis (Chapter 5), guidelines (Chapter 51), pathways (Chapter 52), and decision support systems (Chapter 53) are reviewed elsewhere in the Compendium.
Practice Description
The passive dissemination of information through the use of lectures, conferences, mailings and printed materials remains the primary method to alter physician behavior. This primacy has not been substantially challenged in practice, although more interactive techniques have increasingly been utilized. Academic detailing, for example, involves the process of having invested and well-informed agents for change interacting with individual physicians to promote certain tenets of practice. Alternatively, audit and feedback entails the review and return to the clinician of their own process of care and patient outcomes (often compared with local or national benchmarks or evidence-based standards) in the hopes it will result in more appropriate medical care.3 Reminder systems, which may be computerized and embedded in the electronic medical record, prompt physicians to provide certain healthcare measures. They differ from clinical decision support systems (Chapter 53) in that they may not provide information tailored to the specific patient. Finally, opinion leaders, usually respected local physicians, may improve healthcare by championing "best practices" on a regional basis.4, 5
Prevalence and Severity of Target Safety Problem/Opportunities for Impact
It is well established that physicians are unable to keep abreast of the staggering volume of published medical literature. This is reflected by the many studies that demonstrate the glacial pace at which many beneficial advances are incorporated into medical practice. Practice guidelines, clinical decision support systems and programs for physician education are potential solutions to this problem, but their effectiveness is greatly dependent on the methods used in their implementation.2 Despite the presence of comprehensive guidelines on the treatment of reactive airways disease, for example, a substantial percentage of asthmatic patients do not receive appropriate care.6, 7 Physician education techniques that reliably impact practice patterns may yield substantial improvements in patient care and safety.
Study Designs
The Cochrane Group completed a series of systematic reviews of physician education based on the Research and Development Base in Continuing Medical Education, a central database compiled from an extensive search of electronic databases and bibliographies and supplemented by contact with experts in the field. Although the initial review was completed in 1997, reviews are regularly updated as more pertinent data are published. One such study evaluated the role of audit and feedback and found 37 randomized controlled studies comparing this technique to non-interventional control groups.3 An ancillary study by the Cochrane Group compared audit and feedback with other educational strategies and located 12 randomized controlled studies for analysis.10 A second study of the effectiveness of audit and feedback was completed by a separate group that searched MEDLINE and selected bibliographies for trials investigating the strategy's utility in improving immunization rates. Fifteen studies were identified for inclusion, 5 of which were randomized controlled studies with 6 interrupted time series evaluations and 4 before-after trials.11 A third meta-analysis of peer-comparison feedback systems used an extensive electronic database and bibliography search to locate 12 randomized controlled studies.12
The Cochrane Group also investigated the utility of academic detailing and found 18 randomized controlled studies.13 A similar evaluation of local opinion leaders yielded 8 randomized controlled trials.5
A separate Cochrane review was completed on the utility of printed educational materials using the Cochrane Effective Practice and Organization of Care Group database. The search of this database, which was compiled in the same manner as the Research and Development Base in Continuing Medical Education, found 10 randomized controlled trials and one interrupted time series study fulfilling criteria for analysis.14
Study Outcomes
Few of the studies report outcomes specific to the field of patient safety. The vast majority are concerned with process of care rather than the outcomes of care. Although clinical outcomes are reported in at least one of the studies evaluated in each of the systematic reviews (with one exception), the majority relate outcomes pertaining to physician performance. Some of the more commonly described variables include the rates of appropriate provision of preventive care measures and of adherence to appropriate treatment or diagnostic protocols.
Evidence for Effectiveness of Practice
Much of the evidence for the effectiveness of educational and implementation techniques is of fair quality and the results are generally consistent across the various systematic reviews. However, methodologic concerns prevented the completion of quantitative data synthesis in the majority of the reviews. The studies are summarized in Table 54.1.
The initial comprehensive review found overall beneficial effect for 62% of interventions. In investigations of effect on patient outcomes, 48% had favorable results. Academic detailing and the use of local opinion leaders were the most effective techniques evaluated. Physician reminder systems were also effective, as 22 of the 26 evaluated studies revealed some benefit. The technique of audit and feedback was of marginal effectiveness and conferences and printed materials were found to be relatively ineffective. Of note, multifaceted interventions with at least 3 components were associated with a 71% success rate.8 The second comprehensive review of 102 randomized controlled studies supported these conclusions. Yet it emphasized that the degree of effect with even the most consistently effective techniques was moderate at best, and that the process of care rather than the outcome of care was the most readily influenced variable.9
The Cochrane reviews reported similar results. Audit and feedback was found to be effective in 62% of the studies in which it was compared with non-interventional controls, but the effect was typically small. The results were not substantially different when audit and feedback was augmented by conferences or educational materials or was part of a multifaceted intervention.3 In the review of comparative trials, however, this technique was found to be inferior to reminder systems in 2 of the 3 trials where a direct comparison was made.10 The second review of audit and feedback, which focused on improving immunization rates, found beneficial results in 4 of 5 randomized controlled trials evaluated. Statistically significant changes were present in at least 2 of these evaluations. However, the marginal effect was small and likely was overwhelmed by the relatively high cost of the intervention.11 Finally the meta-analysis of the 12 randomized controlled trials investigating peer-comparison feedback systems did establish a modest benefit for the use of audit and feedback (p<0.05), but the magnitude of benefit was again noted to be small.12
The Cochrane review of academic detailing was somewhat more optimistic. All of the evaluated studies showed some degree of a beneficial effect on physician performance although only one of these studies reported patient outcomes. Most combined detailing with other techniques and there was insufficient evidence to make direct comparisons between detailing and the other techniques.13
The use of local opinion leaders was also found to be effective by the Cochrane group, although to a much less convincing degree than academic detailing. Two of 7 trials showed a statistically significant beneficial effect with a trend toward effectiveness in all 7 studies. One of 3 trials investigating patient outcomes demonstrated a significant benefit.5
Finally, the Cochrane review of the use of printed educational materials supported the findings of the previous overviews. None of the 9 studies showed a statistically significant effect when compared with controls and only one of 6 trials that included printed materials in a multifaceted approach demonstrated benefit. Of note, all of the evaluated trials were plagued by methodologic shortcomings.14
Potential for Harm
These educational techniques are unlikely to cause significant patient harm.
Costs and Implementation
Although the cost-effectiveness of the various educational techniques has not been explicitly studied, it is clear that several may require substantial outlay in terms of financial resources and personnel. It also appears that the forms of education that are most effective, including academic detailing and local opinion leaders, are also the most expensive to design and support. Programs of printed materials and lectures, although dramatically less effective, are substantially less expensive to implement. It is unclear whether the integration of Internet technology and computer-based education initiatives will result in substantial changes in efficacy or cost. Finally, the relative cost-effectiveness of the various techniques remains unclear.
Comment
From studies of randomized controlled trials, it appears that academic detailing and local opinion leaders are frequently associated with at least some benefit. Reminder systems are also effective in specific situations and the utility of audit and feedback has been established, although unimpressively. Traditional programs of conferences, lectures and printed materials are ineffective at inducing changes in physician behavior. None of the current techniques, however, have demonstrated a consistent ability to induce substantial and durable changes in physician behavior. The relative cost-effectiveness of the various techniques is uncertain; it remains unclear if the added cost of the more effective strategies (i.e., academic detailing and local opinion leaders) is justified given their relatively small marginal increase in effectiveness. Finally there are few data regarding the specific utility of these techniques in increasing patient safety and/or the prevention of medical errors. However, techniques effective in other areas of medicine are likely to be equally effective in inducing practices changes to improve patient safety.
Table 54.1. Studies of techniques for changing physician behavior*
| Study Setting | Study Design | Results |
| Review of 99 trials assessing the effect of educational techniques on physician performance in all clinical settings8 | Level 1A | 62% of the interventions were associated with beneficial results; academic detailing, local opinion leaders and reminder systems were the most effective while audit and feedback was less so; traditional CME programs were ineffective |
| Review of 102 trials assessing the effect of educational techniques on physician performance in all clinical settings9 | Level 1A | Academic detailing and local opinion leaders were the most effective techniques; audit and feedback and reminder systems were less effective; multifaceted approaches were effective, especially at influencing the process of care |
| Review of 37 randomized controlled trials of the utility of audit and feedback in all clinical settings in the US, Europe and Australia3 | Level 1A | Eight of 13 studies showed a moderate beneficial effect with audit and feedback with little change noted when other interventions were added or a multifaceted approach was used |
| Review of 12 trials comparing the effect of audit and feedback with other educational techniques on 2194 physicians in all clinical settings10 | Level 1A | Two of 3 trials showed reminder systems outperformed audit and feedback; 4 studies demonstrated little benefit to adding other modalities to audit and feedback |
| Review of fifteen studies, of which five were randomized controlled trials, investigating the use of audit and feedback in improving immunization rates in adults and children in the outpatient setting in the US and the UK11 | Level 1A | Twelve of the 15 studies showed a benefit with audit and feedback and of the 5 RCTs, 4 showed a beneficial trend that was significant in at least 2 of the trials |
| Review of 18 trials investigating the effects of academic detailing on 1896 physicians in the US, Canada, Europe, Indonesia and Australia13 | Level 1A | All of the evaluated studies showed some degree of benefit although only one looked specifically at patient outcomes |
Review of 8 randomized controlled trials investigating the effect of local opinion leaders on 296 physicians in the US, Canada and Hong Kong5 |
Level 1A | Six of the 7 studies evaluating effects on physician performance showed a beneficial effect with 2 of these being statistically significant; one of the 3 trials evaluating patient outcomes showed a significantly positive effect |
Review of eleven studies evaluating the effect of printed education materials on over 1848 physicians in a variety of clinical settings14 |
evel 1A | None of the studies reported significantly improved outcomes with the use of printed educational materials |
* CME indicates continuing medical education; RCT, randomized controlled trial.
References
1. Smith WR. Evidence for the effectiveness of techniques To change physician behavior. Chest 2000; 118:8S-17S.
2. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317-22.
3. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Audit and feedback: effects on professional practice and health care outcomes. In: The Cochrane Library, Issue 2, 2000:Oxford: Update Software.
4. Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, et al. Effect of patient reminder/recall interventions on immunization rates: A review. JAMA 2000;284:1820-7.
5. Thomson O'Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL. Local opinion leaders: effects on professional practice and health care outcomes. In: The Cochrane Library, Issue 2, 2000. Oxford: Update Software..
6. Apter AJ, Van Hoof TJ, Sherwin TE, Casey BA, Petrillo MK, Meehan TP. Assessing the quality of asthma care provided to Medicaid patients enrolled in managed care organizations in Connecticut. Ann Allergy Asthma Immunol 2001;86:211-8.
7. Rabe KF, Vermeire PA, Soriano JB, Maier WC. Clinical management of asthma in 1999: the Asthma Insights and Reality in Europe (AIRE) study. Eur Respir J 2000;16:802-7.
8. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274:700-5.
9. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995;153:1423-31.
10. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Audit and feedback versus alternative strategies: effects on professional practice and health care outcomes. In: The Cochrane Library, Issue 2, 2000. Oxford: Update Software.
11. Bordley WC, Chelminski A, Margolis PA, Kraus R, Szilagyi PG, Vann JJ. The effect of audit and feedback on immunization delivery: a systematic review. Am J Prev Med 2000;18:343-50.
12. Balas EA, Boren SA, Brown GD, Ewigman BG, Mitchell JA, Perkoff GT. Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials. J Gen Intern Med 1996;11:584-90.
13. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Educational outreach visits: effects on professional practice and health care outcomes. In: The Cochrane Library, Issue 2, 2000. Oxford: Update Software.
14. Freemantle N, Harvey EL, Wolf F, Grimshaw JM, Grilli R, Bero LA. Printed educational materials: effects on professional practice and health care outcomes. In: The Cochrane Library, Issue 2, 2000. Oxford: Update Software.