Atlanta School Story: A Causal Update

Back in 2014 I wrote about the Atlanta school testing scandal (here).  Teachers and administrators changed student answers to raise student achievement test scores.

There’s an update to this story in The New York Times Magazine, 9 September 2018, which describes the road back from this disheartening event through the experience and efforts of a dedicated elementary school principal, Cynthia Gunner.

Sara Mosle, author of the new article, describes the history of the test scandal, including the policy decisions that set the environment for what happened.

Working my way through Judea Pearl’s books and articles on causal theory led me to translate Sara’s text into a couple of ‘directed acyclic graphs’ (DAGs), the fundamental visual tool in Pearl’s toolbox.

The DAG at the top of the post represents the policy theory driving “No Child Left Behind.”

“This junction is the simplest example of a ‘chain,’ or of mediation. In science, one often thinks of B as the mechanism, or ‘mediator,’ that transmits the effect of A to C.”  (Pearl, Judea. The Book of Why: The New Science of Cause and Effect (p. 113). Basic Books. Kindle Edition.)

In one way, the “No Child Left Behind” theory holds up:  Teachers caught and convicted in the scandal when held individually accountable for better test scores did in fact find means and motivation to act in ways that improved student achievement outcomes so long as better outcomes is defined as higher test scores.  Of course, the path they took was not the one intended by policy-makers and educational leaders.

A Revised Theory and Diagram

One of Pearl’s messages:  make our causal theories visible so that we can be more thoughtful in analysis and interventions.   An edX short course taught by Professor Miguel Hernan on Causal Diagrams has a relevant subtitle:  Draw Your Assumptions Before Your Conclusions.

Mosle wrote:  “While teacher effectiveness may be the most salient in-school factor contributing to student academic outcomes, it contributes a relatively small slice — no more than 14 percent, according to a recent RAND Corporation analysis of teacher effectiveness — to the overall picture. A far bigger wedge is influenced by out-of-school variables over which teachers have little control: family educational background, the effects of poverty or segregation on children, exposure to stress from gun violence or abuse and how often students change schools, owing to homelessness or other upheavals.”

Label the out-of-school variables Z, so Z = {family educational background, poverty, segregation, exposure to stress, frequency of changing schools,…}.

Now I’ve drawn a revised diagram. It is a more complicated theory of better test outcomes. Better outcomes are proposed as caused jointly by social determinants in the Z vector and the teacher coaching that Mosle observed in Atlanta.  The vector Z has causal influence on factors A’ and B’ as well as on C.   That means the social determinants can affect the way in which teachers are coached and the degree to which teachers apply means and motivation. The diagram preserves a causal link between teachers (node B’) and achievement performance.

Even without going through the causal math developed by Pearl and other causal researchers, you might find it plausible that the new theory says we need to account for the levels of social determinants to appropriately assess impact of teacher actions, B’.

You can certainly argue that neither of the diagrams represents the real world adequately.  All models simplify to make them useful.   Nonetheless, drawing causal pictures increases the odds that we can avoid egregious plans and interventions. Causal pictures look like a good starting point for policy-makers and managers seeking to improve outcomes.