Quincunx Simulations—Part 2

In the previous post I described an exercise that uses Galton’s Quincunx.

The challenge: maximize the number of results in a five-value range, over three rounds of 20 drops of the Quincunx. Based on the original exercise devised by Dr. Rob Stiratelli, the exercise starts with the “aim” of the Quincunx at the low end of the range. And at the beginning of the second and third rounds, the device has a calibration problem that shifts the aim three or four units.

Last week, we ran the exercise with four teams in a workshop. Groups A, B and C used a simulation I wrote in R and Group D used the modern Quincunx shown at left.

The exercise's participant instruction sheet is available here.

Results of the exercise

As I stated in the first post, if you can aim the funnel at the center of the desired range, you usually can get at least 16 of 20 values to fall in that range.

All four teams realized on Round 1 that the system was running on the low side of the range and made adjustments to move the aim closer to the center of the range.

However, the hidden change in the aim at the start of Round 2 caused confusion and uncertainty. Teams that had been getting almost all values in the desired range now were getting values outside the range, even though the nominal position of the meter setting was the same.

With some testing and debate, teams recovered and by the end of the second round were again getting results mostly in the desired range.

The change at the start of Round 3 caused the same challenges—initial hesitation and lack of confidence in the relationship between meter setting and output gave way to better performance, after adjustments to the aim.

Team B had the best results, but this seems to be related to a problem in the simulation (see below.)
 

Team Lessons from the Exercise

1. No teams made a run chart of the results and the meter values. They looked at the table of numbers and did their best to look at average results and the impact of the meter setting. Our management exercise followed a 90-minute presentation and practice with run charts.  The failure to make run charts is a sobering reminder that it takes repetition and presence of mind to apply improvement methods and tools when under pressure to perform, even in a training-room exercise.

2. No team got perfect results because the Quincunx system as designed is not capable of regularly producing values in a five point range. Team best efforts, management incentives, and public scorecards don’t change the underlying structure.

3. No teams systematically tested the relationship between meter setting and output. Experimentation comes at the cost of possibly poor results. I did not hear any clear discussion of how to test in the face of uncertain outcomes.

4. The teams did not cooperate; no one sent any representatives to other tables to try to learn what strategies seemed promising.
 

Quincunx Properties: Variation Concepts

1. The Quincunx device shows that

a. Variation in results arises from variation in funnel position (an input measured by the meter setting) and system structure, represented by the pins.

b. In other words, variation in input and system structure causes the results to vary.

c. If we can study and identify how the changes in inputs and system conditions drive the variation in results, we can work “upstream” to reduce this variation in a cost-effective way.

2. A Model for Common Causes

a. The structure of the pins provides a physical model for what we call “common causes of variation.”

b. The built-in variation of the pins drives variation in results.

c. Each pin contributes a small but meaningful amount of variation to the results.

d. We can describe the variation that results from the pattern of pins--we expect to see a range of plausible values, without systematic patterns.

3. A Model for Special Causes

a. We can assign the specific movement of the funnel to changes in results so this movement will serve as our model for “special causes of variation.” Walter Shewhart, the inventor of control charts, used the term “assignable causes” to indicate that we may be able to assign a specific cause to this class of variation.

b. Movement of the funnel causes variation in results on top of the variation that arises from the pins.

c. The movement of the funnel can be relatively large or small; the size of the movement matches the ease of detecting the movement.

d. As a useful approximation, the total variation in results is composed of variation from movement of the funnel plus the variation contributed by the pins.

4. General Description of “Common Causes of Variation”

Common causes of variation are system conditions and inputs with the following properties:

a. Variation of the common causes drives variation in system results.

b. Variation in the common causes (and hence variation in the results) is built in to the current structure of the system.

c. Each common cause contributes a small portion of the variation in the results.

d. Common causes of variation combine to give a range of plausible values in the results, without systematic patterns or unusual values.

e. In practice, we can mimic “variation without systematic patterns or unusual values” by models of randomness. Also, we define system variation (with respect to a particular type of result) in terms of common causes of variation.

5. General Description of “Special Causes of Variation”

Special causes of variation are system conditions and inputs with the following properties:

a. Variation of special causes leads to variation in system results.

b. The variation in results from special causes is added to the variation that comes from common causes.

c. If the variation in special causes crosses a threshold, we will see patterns of variation or unusually large or small values in the results.

d. In practice, we say we have “evidence of special causes” when we detect patterns of variation or unusual values in system results. Then, using system knowledge, we often can match the variation in the results with variation in system conditions or inputs. In such a case, we say we have identified one or more special causes of variation.

A control chart is the primary tool to help to distinguish common causes of variation from special causes. See for example L.P. Provost and S.K. Murray (2011), The Health Care Data Guide: Learning from Data for Improvement, Jossey-Bass: San Francisco, especially chapters 4-8.

If you don’t have a physical Quincunx device, you can use the quincunx function in the animation package in R, which shows the pin variation:

My Simulation Software

My R simulation is an R shiny web app that mimics the physical Quincunx, available at https://iecodesign.shinyapps.io/Quincunx_shiny/ .

The code for the simulation is available at https://github.com/klittle314/Stiratelli_quincunx
The Admin-T and Admin-P tabs show a table and plot, respectively, for the simulation results. I do not show these tabs to participants during the simulation.

To generate a value for a given Meter Setting, the system manager clicks the “Tell System to Get Ready!” button and then clicks the “Get value” button.

The meter setting of 30 corresponds to an output value of 48, on the low end of the desired range 48-52.

After generation of 20 values, the meter setting “slips”: if the average of the first 20 values less than 50, meter value is offset three units lower. Otherwise, the meter is offset three units higher. Similarly, after the next 20 values, the meter slips again: if the average of the next 20 values is less than 50, the meter value is offset four units lower. Otherwise, the meter is offset four units higher.

Here’s a graph of 60 results from the Admin-P tab of the web app, with no adjustment to the meter (hence, no adjustment to the aim of the Quincunx funnel.) You can see how the system “center” changes during each set of 20.
 

A problem with the simulation software

If you lose connection to the server in the middle of a simulation, the web app restarts—there is no persistent memory in the version I used last week.

This restart phenomenon accounts for Team B’s good performance on rounds 2 and 3: the laptop used with this group repeatedly lost the connection to the server, so they worked with a system that never experienced a “slip” in the meter—once they had learned that a meter value of 32 was about right, they just could keep that setting and get pretty good results.

To avoid the problem of reset, you can run the simulation locally or edit the code to allow for persistent storage, e.g. https://shiny.rstudio.com/articles/persistent-data-storage.html.