Build Your Intuition about Sampling Variation

Healthcare organizations throughout the United States and in many other countries use surveys to learn about patient experience. 

In the U.S., the Center for Medicare and Medicaid Services (CMS) mandates surveys of patient experience. Hospitals are required to participate in a formal patient survey, the Hospital Consumer Assessment of Healthcare Providers and Systems survey, HCAHPS. There are other CAHPS surveys developed and required for primary care, as well as skilled nursing and hospice facilities.

The typical CAHPS surveys are scientifically designed and deployed, with attention to sampling integrity and interpretation. As such, it is possible to characterize the variation in reported results that arises directly from the operation of random sampling. 

Change in the way care is delivered for better or worse, the particular mix of patients who actually respond to the CAHPS suveys, and sampling variation all contribute to differences in survey results month to month.

Because CMS has begun to ask healthcare organizations for evidence of good performance or improvement from baseline to guarantee full payment of charges, CMS has gotten the attention of healthcare organizations.

So how should leaders and managers react to the variation month to month in such survey results?

First, if the reported levels of patient experience are low, you need to actually improve patient experience. You and your team need to change care processes. That path requires identification of change ideas, targeted work to demonstrate how to use the changes in your organization, and integration of changes that show merit into the daily work of your care teams. I recommend understanding and using a disciplined improvement approach like the Model for Improvement.

Second, you need to monitor the survey numbers in a rational way, to avoid chasing misleading signals of improvement or decline in survey results. Control charts are the tools to help you monitor rationally and are increasingly common in third-party reports and management dashboards but not yet universal.

To help build intuition about the variation in sampling numbers and make the case for control charts, I made a Shiny web application for a recent Institute for Healthcare Improvement webinar. The app is available at https://iecodesign.shinyapps.io/survey_simulator.

The simulator allows you to vary three values:

(1) n, the number of observations in your survey sample. n ranges from 5 to 300.

(2) p, an estimate of the probability of a success in your population (e.g. if you could ask every patient who received care in a specific month about an aspect of nursing care, what would be the fraction that assessed the care as “top box”?). p ranges from 50% to 99%.

(3) Sampling fraction, the size of your sample as a fraction of the patient population. Sampling fraction ranges from 1% to 99%. For example, if 1000 patients visited your clinic in September and you sampled 100, the sampling fraction is 100/1000 = 10%.

The app shows histograms of 10,000 realizations from a simple model and an adjusted model.

The simple model uses simple random sampling from a binomial distribution defined by n and p.

The adjusted model uses simple random sampling from a hypergeometric distribution defined by n, p, and the sampling fraction; this model accounts for sampling of a finite population.

The app also shows a table of percentiles for the two models; the table is a numerical summary of the histograms. The 1% and 99% percentiles, shown on the histograms as dashed lines, give you an approximate feel for the lower and upper control limits for an individual sample result from a model with the given parameters.

Finally, the app shows a run chart of 20 simulated values from the adjusted model, to show you what variation you might see in a chart where there are no changes in the underlying assessment of patient experience: all the movement up and down of the points in the chart arises from sampling variation.   If you want to approximate the variation in a run chart of values for the simple model, just slide the sampling fraction all the way to the left (1%)--your eye would not be able to distinguish variation driven by a 1% sampling fraction from variation driven by the simple model.

I discuss details related to HCAHPS surveys in context of the web app here.

Code for the app is available on GitHub, https://github.com/klittle314/survey_simulator.git.