Ways to reduce inappropriate Antibiotic Prescription Rates—Part II

In the previous post, I summarized a recent RCT study in JAMA that compared ways to reduce rates of inappropriate antibiotic prescription.

I was particularly struck by one intervention, the peer comparison. This intervention appears to be an example of “if you measure performance (and report it), you can improve performance.”

We just have to be careful not to think that the intervention supports a belief that “If you measure performance (and report it), you will improve performance.”

So let’s revisit the Prius Effect to explain why the peer comparison could reduce inappropriate prescriptions but the observed impact is not due to measurement alone.

Understanding the Peer Comparison: The Prius Effect at Work

I presented the Prius Effect in this post.

Briefly, the Prius Effect is the belief that feedback about mileage performance, as enabled in many hybrid cars like the second-generation Prius, will improve mileage performance by drivers.

The Prius Effect generalizes to a belief in the general power of feedback to change performance in any system..

Note that the selection of a relevant system measure to be used for feedback—e.g. miles per gallon for the actual Prius—is not trivial for arbitrary systems.

In fact, you can often measure aspects that don’t matter that much; that’s at the heart of criticism of much of measurement in healthcare, e.g. here.

Here’s my interpretation of the peer comparison intervention using the Prius Effect conditions.

Prius Effect Condition	RCT example, peer comparison
(1) Sense data relevant to the system’s performance.	(1) Email of performance table prompts physician to see the difference between self and “best” (top 10% of practitioners in clinic group) on inappropriate prescription rates, the measure at the heart of the study. The system here is a clinic group of physicians, with their patients who present with antibiotic-inappropriate acute respiratory tract infections. The rate of inappropriate antibiotic prescription is an obvious performance measure and so passes the relevancy hurdle.
(2) Interpret the data.	(2) Interpretation is simple: (a) “my numbers are lower than the ‘best’” or (b) “my numbers are considered the best.” No advanced course work in data analysis required.
(3) Connect the interpretation to one or more suitable actions (changes) that will affect performance.	(3) Actions based on the interpretation: (a) “I can change my prescription practice by some method(s)” or (b) “I should keep doing what I am already doing.”
(4) have the power or potential to act, based on step (3)--“can do”	(4) Physician controls whether or not to write the prescription—no intermediary, subordinate or boss is involved.
(5) actually act to change system performance--“actually do”	(5) Each physician either prescribes an antibiotic or doesn’t, case by case. In the study group experiencing the peer comparison intervention, the researchers reported a “statistically significant” average reduction beyond the reduction of the control group.

Conditions 3 and 5 deserve scrutiny.

Condition 3 is the bridge between data and potential action. Often the bridge doesn’t exist and has to be built by hard work. When the bridge is missing, performance feedback by itself can’t lead to improvement.

In the antibiotic prescription study, physicians evidently had the requisite knowledge to align their prescribing practice with vetted clinical guidelines, which they were reminded about at the start of the study.

The suitable action in this study is direct and obvious: given a diagnosis that is inappropriate for antibiotic treatment, the action is just say no—don’t prescribe an antibiotic.

In other situations, the path between improved performance and suitable action is uncertain or unknown.   Then you need to work like a detective/scientist to find some actions that help.  You might use control charts to help generate clues or prevent you from promoting ineffective practice, as illustrated here.  

Condition 5 is the bridge between potential action and actual action.

In the antibiotic prescription study, it looks like (re)education and awareness yielded some improvement, as demonstrated by the control group’s performance in changing prescription practice. Nonetheless, the peer comparison intervention is associated with improvement beyond the control performance. On average, more of the peer comparison physicians aligned prescription practice more tightly with guidelines than the control physicians—on average, more physicians in the peer comparison group crossed the bridge from potential to actual action. The study authors infer that the psychological impact of seeing a physician’s own performance relative to the top 10% motivated physicians to align their own practice more closely with the clinical guidelines and thus decrease the rate of inappropriate prescriptions.

Conclusion

I believe that a feedback system that leads to changes in system performance needs to meet the five Prius Effect conditions; that is, these conditions appear necessary.

As the Prius Effect conditions show, the measurement part of the feedback system is not the whole story. You can’t just measure performance and report it to improve performance, though that might be our initial impression. You also need potentially effective changes, authority to act and then make actual changes to the system to get better performance.