If You Can't Measure Performance, Can You Improve It?

In a Journal of the American Medical Association forum article earlier this winter “If You Can’t Measure Performance, Can You Improve it?” Dr. Robert Berenson examined the measurement burden on physicians (article available on-line here)

Physicians and their organizations spend a lot of time and effort collecting and reporting performance measures. For example, a recent article by Casalino et al. in Health Affairs (March 2016 vol. 35 no. 3, 401-406) estimated that the average physician in four specialties in the U.S.—cardiology, orthopedics, primary care (family medicine and general internal medicine), and multispecialty practices that included primary care--needed almost 800 hours of staff and physician time per year to report required performance measures.

Berenson raises the question whether measurement of physician quality as it now exists in the U.S. is what it takes to make things better. He’s skeptical.

Berenson’s title links to a typically provocative observation from W.E. Deming about measurement and management: “It is wrong to suppose that if you can’t measure it, you can’t manage it—a costly myth.” (The New Economics for Industry, Government, Education, 2nd edition, MIT CAES, Cambridge, MA, p. 35).

As Berenson states, Deming’s observation often gets expressed with an opposite meaning, yet still attributed to Deming: “if you can’t measure it, you can’t manage it.”

Of course Deming understood the advantages and necessity of measurement of shop-floor processes. I think that Deming chose his words to provoke organization leaders—the bosses—to understand their role and obligations in Deming’s theory of management. Bosses face planning and design challenges that are typically more difficult to measure than operating and control challenges that you can touch and see over multiple cycles.

The sentences immediately preceding the management-measurement provocation provide Deming's context:

“The magnitudes of the most important losses from action or inaction by management are unknowable ….We must nevertheless learn how to manage these losses.”

For instance, what is the lost profit from a decision to disinvest in a service that could have been salvaged?  

Berenson offered accuracy of diagnoses, a critical aspect of physician quality, as an example of a something that needs managing but does not now have a simple and universally accepted measurement.

Berenson then points out that because accuracy of diagnosis has no agreed-to measurement, it is not included in the Merit Based Incentive Payment System (MIPS) portion of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA).   This means that MACRA's payment stick will necessarily ignore accuracy of diagnosis.

Berenson notes, however, that plausible change ideas exist to improve diagnostic accuracy, citing several recommendations from the Institute of Medicine: “These and other IOM recommendations represent better practices that might dramatically improve diagnostic accuracy, relying not on performance measures but on adopting better work processes and focused education. Measures would help, but substantial progress can be made regardless.” (p. 646, print edition).

I interpret Berenson to mean we should not rely solely on performance measures to drive improvement.

Exclusive focus on measures and their cousin, numerical goals, can and has led to bad consequences (e.g. see this blog post).

We need to develop and apply methods to actually make improvements.  

That's at the core of Deming's view:  "A numerical goal accomplishes nothing.   Only the method is important, not the goal.  By what method?" (The New Economics, p. 31).

Nonetheless, as Berenson says, "measures would help":   if you have some ideas to improve performance and you try them, you’ll be better off if you can figure out some way to measure whether you’re making progress. Measures will help you manage both tangible work processes and complex systems.