Plot Dots, Add Notes, Look at Multiple Instances
Last week, I worked with an audience of dentists, hygienists, dental assistants and clinic administrators at the second learning session of an improvement collaborative.
The overall aim of the collaborative is to improve oral health for kids (see post 1 and post 2 for discussion of one key measure, sealants.). To help us reach the aim, we’ve introduced quality improvement and control methods including the Model for Improvement, ‘Go See’, use of daily huddles, process mapping, and run chart analysis.
While I have presented run chart analysis many times to audiences, preparation for this week’s session got me thinking about the relative importance of plots, notes, and analytic rules. What comes first?
Run Chart Analysis
Here’s the aim of run chart analysis: to increase your understanding of the system represented by the data.
The analysis starts with construction of a plot that shows values in time order, with a median reference line like that shown at the top of this post. The run chart shown comes from an oral health improvement collaborative last year with the same aim as this year’s project.
With the run chart in hand,
- You can apply a set of rules to detect unusual patterns, which may be signals that your system has improved or deteriorated.
- You match events that may affect the system to the patterns you observe, labeling or annotating the plot.
Run chart rules derive from calculations rooted in probability (see here and here) and can help us interpret performance. While the rules are easy to explain, the emphasis on this simple statistical technology runs the risk of obscuring three fundamental points for my oral health colleagues: plot the dots, add notes as you go and look across the teams in the collaborative for evidence of impact.
Plot the Dots
My teacher George Box, a major 20th century contributor to structured analysis of time series, advised that graphing a time series is the first step of any investigation of time-based system performance: plot the dots. A basic plot is often enough to provide sufficient insight to support reasoned action.
Increasingly sophisticated methods, from run chart analysis to Shewhart control charts and tools that use models related to the time series approaches developed by Box and others (Box et al. (2009) here) all start with plotting the dots.
Add notes about your system as you go
The run chart example shows a note to indicate when the center began to test a structured approach to caries risk assessment. You see a text comment and a vertical line between the September and October 2016 dots. The team planned changes to the way they provided care, with the intention of improving results; after the introduction of the change, the next two dots jump in the good direction.
Of course, when an action is followed by a pattern of dots in the good direction, we don't have conclusive evidence that the action caused better performance. However, you can begin to distinguish happenstance good dots from improvement caused by intentional actions if you add notes, dot by dot. To build your understanding, notes added dot by dot are more likely to help you understand your system than looking back at patterns in the distant past and trying to recall events or fabricate an explanation.
This document-in-real-time approach resembles the ‘FBI’ approach when presented by a possible on-going crime: agents take notes about a case day by day and event by event, writing file memos as they go, rather than attempting to build their case documentation long after the fact. They do this to reduce the likelihood that agents will invent a story to fit observed patterns.
Building degree of belief, through multiple Plan-Do-Study-Act cycles, can be formalized using ‘Bayesian’ methods but I’ll save that connection for another post.
Parallel Changes and Common Patterns Increase Degree of Belief
The display here is a ‘small multiples’ display that suggests Health Center O’s experience does not appear to be a fluke.
While not every health center improved (look at center P), many of the participating health centers improved caries risk assessment after the beginning of the collaborative as they applied a common set of changes to clinic operations.
We can see a consistent connection between the timing of the collaborative project and improvements in performance. When we see improvement in a range of settings that apply a common set of changes, we've got stronger evidence that the changes actually are causing the improvement.
The place of run chart rules in practice
Everyone can learn to plot dots, annotate charts and look at similar systems in search of evidence that changes are causing improvement. These skills often provide individuals and teams enough insight into system performance to take appropriate action.
Specialists in quality improvement, supervisors and managers can learn and use probability-based rules to identify unusual patterns of runs , building on the 'plot, note, and look at similar systems' skills.
Technical Notes on Run Chart Analysis
1. 'Equal information assumption'
- Run chart rules for trends, shifts and too few or too many runs assume each 'dot' contains about the same amount of information as every other 'dot'.
- When you apply the run chart rules to per cent data, if the denominators of the dots vary more than about 20%, the rules give less clear guidance--dot information varies.
- You can still plot dots in time order and add notes when the information content of the dots varies.
- You can plot the denominators in time order as well as the per cent data. Line up the denominators with the same time scale as the per cents to see patterns that match in the two charts.
In the small multiples plot, the health centers are ordered by volume of patients top to bottom. The health centers in the bottom two rows have very small numbers of patients (less than 10 each month) and consequently we see much more variation.
2. 'Median a good summary' assumption
- For per cent data: If more than half of the points are zero (or 100), the median will be zero (or 100). The run chart rules break down in these cases.
- You can plot time or number of cases between events of interest when you have many zeros or many perfect 100's.
3. Building on small multiples: An intervention or set of interventions across multiple settings gets formalized in the methods of experimental design. I sketched benefits and limitations of these methods in this series of posts (here, here, and here.)
