Rachel Aviv in the 21 July 2014 issue of The New Yorker (“Wrong Answer: In the era of high-stakes testing a struggling school made a shocking choice”) tells the story of teachers and administrators at Parks Middle School in Atlanta. The teachers, with administrator support, erased wrong answers and then wrote in correct responses to achieve satisfactory test scores for their students and school.
My former boss and colleague Brian Joiner wrote a book 20 years ago, Fourth Generation Management, that gives context for this cheating.
Brian outlined these four generations:
First Generation: do it yourself.
Second Generation: Master craftsperson takes on apprentices but remains the model and arbiter of production.
Third Generation: manage by results—usually by specifying the goals required without detailing the methods.
Fourth Generation: simultaneous focus on three chunks of work: quality (defined by customers); the scientific approach (iterative learning, using data effectively, to build and maintain effective methods); and all one team (collaboration and respect for people).
Brian described the three chunks as interdependent and mutually reinforcing, symbolized by a triangle.
In Brian’s description, management by results focuses on better measures of performance—better figures. Of course, clarity of aims, including appropriate use of numbers to define performance, is critical. But things can go awry.
How do you get better figures? Brian’s experience and reflection revealed only three ways:
Improve the system that generates the performance numbers—that is, actually change the work flow, the tools, and the ingredients so that results are in fact better.
Distort the system—focus on parts of the work and ignore others to achieve the desired numerical goals. E.g. cut spending on maintenance to maintain profit levels for the current period.
Distort the figures—change the data so that the reported results look good.
Depending on how you interpret the actions of the Atlanta teachers and administrators, they actually took all three approaches to better figures—but the falsification of data drives the pace of Aviv’s story and caused the teacher dismissals and convictions.
John Ewing, one of the sources Aviv interviewed for her story connected the Atlanta events to “Campbell’s law”: “a principle that describes the risks of using a single indicator to measure complex social phenomena.”
Donald Campbell’s 1976 article with his law is available here. It’s worth reading to understand the state of evaluation methods 40 years ago and the problems that persist in evaluating projects today.
Campbell stated his law like this: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Distortion and corruption of the system, while not inevitable, appears to thrive in the absence of multiple, transparent measures of performance, attention to effective methods, and skill in the scientific approach.
While Campbell focused on social programs and Joiner focused on management within organizations, they both hit on the situation faced by the teachers at Parks Middle School: pressure to get better figures can overwhelm the effort and intention to actually improve the system being measured.
