Tale of Two Measures

At our Oral Health learning session three weeks ago in Chicago, we explored the relationship between a measure that health centers must report to their federal funding agency and a measure specific to our collaborative. 

I discussed the details of these two measures in a 2017 post; here’s a quick summary.

UDS Measure (reported to the feds): “Percentage of children, age 6-9 years, at moderate to high risk for caries who received a sealant on a first permanent molar during the annual measurement period.”

NNOHA Collaborative measure: “Percentage of 6-9-year-old children who receive a sealant on all eligible permanent first molar teeth during the measurement month.”

The photo at the top of this post shows what happened when 19 team representatives each placed a dot to represent their clinic’s UDS and NNOHA measures.   We asked the teams to compute the UDS number for the first three months of 2019, overlapping with our collaborative work.  Other than one obviously unusual dot at roughly (55,1), the plot shows a roughly linear and positive relationship between the two measures.

As the NNOHA measure requires all eligible molars to be sealed but UDS requires only one molar, collaborative faculty initially predicted that we would see a UDS number at least as high as the NNOHA number for each clinic.  Colleagues at CHAS Health showed that this was in fact the case for each of seven dental clinics in their system.

Contrary to faculty expectations, we see some dots below the 45-degree diagonal purple line in the scatter plot.  The NNOHA sealant % in March 2019 was HIGHER than the Q1 2019 UDS sealant % for six health centers in addition to the unusual (55,1) point.  Why?

Risk Might Matter

As one of the session attendees pointed out, the UDS measure focuses on kids with medium and high risk; the NNOHA measure ignores risk.  Could that explain our faculty surprise?

A table with invented numbers illustrates what might happen. Imagine 10 children 6-9 years old who have at least one molar eligible for sealing and visited the dental clinic for any reason in March 2019.

NNOHA Sealant arithmetic for March 2019:  All 10 children are in the denominator.  Children with ID {1,3,4,5,7,10} got all their eligible molars sealed.   Rate:  6/10 = 60%

UDS Sealant arithmetic for March 2019:  Children with ID {6,7,8,9,10} are in the denominator. Two children in the denominator got all eligible teeth sealed so they meet the UDS requirement that at least one molar is sealed.  Assume children {6,8,9} got no sealants.  Rate:  2/5 = 40%.

So, in this example, the UDS measure is lower than the NNOHA measure.

Faculty expected that most kids seen at health centers are at least medium risk so the example’s risk breakdown might not match reality at most centers. 

Nonetheless, high(er) risk kids might get sealants at a lower rate because they might have a higher frequency of acute care/restoration visits relative to low(er) risk kids.  The acute care/restoration visits may be harder to ‘squeeze in’ sealants without requiring a new appointment. 

A relationship between risk level and sealant rate does seem to be the case, according to Morgan Thomas, Quality Improvement Specialist at CHAS Health; she shared this table of NNOHA sealant measure stratified by risk status.

Two Health Centers’ Data:  Repairing UDS calculations

I shared the work of my colleagues at Primary Health Care in a post two weeks ago, which showed their NNOHA sealant rate is currently around 90%.  In Chicago they reported their UDS percentage for Q1 2019 was 45%.  I asked PHC team member Becky Hall to explain.  She replied: “We have found a lot of errors in the automated report [for UDS], and when we do a manual audit, which isn't sustainable, we are at 100% for UDS.” 

Another team, Omni Health, reported in Chicago their NNOHA sealant about 40% and UDS sealant Q1 2019 about 30%.   They were surprised by this, just like the faculty had been.

They now have done some detective work.  They found that some of their medical providers have used an oral health screening code in the medical clinic. The code D0190 is also used in the dental clinic. When a report is run to include all patients with D0190 along with a couple of other visit codes, some children who have never visited the dental clinic get swept into the UDS denominator.  Since sealants can only be applied by providers in the dental clinic, counting the children who only visit the medical clinic lowers the UDS percentage.   Restricting attention to patients seen in the dental clinic, their Q1 2019 UDS rate is 58%, higher than their NNOHA sealant number.

Lessons

Faculty predicted that UDS sealant rates would be higher than NNOHA sealant rates. 

By comparing UDS sealant rates to the NNOHA sealant rates in our flip chart scatter plot, we saw that our prediction failed to hold for all health centers.  The failure provoked us to uncover problems with calculation of the UDS sealant numbers in at least two health centers. 

We wonder about the automated calculation and reporting of UDS numbers from other health centers not in our collaborative.

Exploring the connection between sealants and risk status also yields a more nuanced view of the relationship between the UDS and NNOHA sealant measures.

Our study of two sealant measures connects to a more general rule for thinking:  comparison of two measures, ideas, or objects is a good way to drive understanding.  “You only understand something new relative to something you already understand.” (Wurman, Richard Saul., and Peter Bradford. Information Architects. Graphis Press Corporation, 1996, p. 23)

Practical Note:  Making dots on flip charts

I like to carry sticky dots in my travel bag to make data displays on flip charts.  The dots shown in the picture at the top of this post are ¾ inch diameter dots from Office Depot.  Lots of colors are available in addition to blue; other vendors supply them, too.