Last month, I described three common management myths W. Edwards Deming worked to dispel. These included the myth of best practices, the myth of the hero educator, and the myth of performance appraisal. I cautioned that it is important to keep in mind that you may very well experience some cognitive dissonance as you read. There is a counterintuitive thread to much of the Deming philosophy, which of course makes sense given that he advocated for a change in state from the prevailing system of management to something better. This month, I’ll unpack the myth of merit pay, the myth of accountability, and the myth of extrinsic motivators.

Myth of Merit Pay (and Rating & Ranking Employees)

Merit pay has been proposed as a reform idea in education going back to at least the Reagan administration. Since the 1980s, the idea has been taken up by governors and presidents, most recently during the Obama administration through the Race to the Top legislation. The theory behind these proposals is that merit pay is a motivator; basically, that the chance to make more money will drive improvements in our schools. It is worth noting here that the discussion that follows is solely focused on merit pay in education and not on whether teacher base salary levels are adequate; that is a separate discussion topic altogether.

There are several problems with the merit pay theory. There is the problem of defining a meaningful measure of performance by which to judge individual educators. There have been advancements in statistical measures of academic growth and computing power since the Reagan administration, but Deming’s assessment of the idea is as valid today as when he remarked on merit pay in Out of the Crisis. Upon learning of the merit pay idea proposed by President Reagan, Deming had this to say:

The problem lies in the difficulty to define a meaningful measure of performance. The only verifiable measure is a short-term count of some kind...Where were the President’s economic advisors? He was only doing his best.[1] 

During his four-day seminars, when participants would ask him how much data they would need on an employee before an accurate evaluation could occur, Dr. Deming would say 15-16 years’ worth. His primary point in answering this way was that it is nearly impossible to accurately disentangle the contributions of the employee, the contributions of the system, and the contributions of the interactions between the employee and the system on a short-term time frame. 

Even now, with many states having teacher level value-added data in some grade levels and subject areas, there are significant problems with the disentanglement problem. These Value-Added Models (VAMs) are statistical models that attempt to distinguish a teacher’s causal impact on her students’ learning from other factors. For example, in a 2016 working paper by University of California Berkeley economics professor Jesse Rothstein, he found the following:

Teachers’ VAM scores are evidently inflated or depressed in part due to the students who they teach, who differ in unobserved ways that are stable over time. This bias accounts for as much as one-third of the variation in teachers’ value-added scores, enough to create a great deal of misclassification in VAM-based evaluations of teacher effectiveness.[2]

There are other problems with merit pay beyond the measurement problem. The theory suggests that the additional money will incentivize improved teaching and in turn improved student outcomes. For this to be true, that means that teachers were previously withholding their best efforts. I’ve not encountered a single educator in my nearly 20-year career for whom this would be true. I’m not saying that we as educators don’t have room for improvement; there is always room improvement, no matter the level of experience. Instead, I’m saying that the lure of more money will not likely lead to improvement because it is completely disconnected from developing better educational methods.

More likely is that the environment created by a merit pay system would disincentive behaviors that are important to improving any complex system, such as cooperation and teamwork. The competition of merit pay in general, and especially where the merit pay calculation is viewed as opaque and unfair, could easily lead to undesired behaviors such as the unwillingness to share ideas or take on certain assignments. Returning to Rothstein’s research he in fact finds that:

Insofar as such biases are important, they create obvious problems for VAM-based teacher evaluations. A teacher whose VAM score is biased upward due to her student assignments might thereby qualify for bonuses that she has not earned, while one who specializes in tougher students risks being undeservedly dismissed for poor performance. This creates obvious incentives for teachers to avoid the latter assignments in favor of the former, potentially making it harder to staff certain courses and reducing the overall efficacy of the school.[3]

This is exactly the type of unintended consequence that is likely to flow from the merit pay idea. This problem is not unique to schools; similar issues occur in any work system that employs these practices. As Brian Joiner astutely notes in his work on understanding variation:

When people are pressured to meet a target value there are three ways they can proceed: (1) they can work to improve the system; (2) they can distort the system; or (3) they can distort the data.[4]

Performance appraisal and merit pay lead to suboptimization of the system. Unfortunately, without profound knowledge, these practices are continually recycled by education, policy, and political leaders. The main idea here is that joy in work, intrinsic motivation, and cooperation are key to a healthy organizational culture within schools. Practices like merit pay that rely on extrinsic motivation and competition are more likely to lead to distortions of the system or the data within the system than they are to lead to an improvement of the system. Here we would be wise to use Deming’s theory to optimize the system rather than attempting to incentivize individual educators within those systems. 

Myth of Accountability (Management by Objective)

In the K-12 education sector, one of the most visible uses of data is in state education department accountability systems. All 50 states now issue school district report cards typically based on various performance metrics such as proficiency rates on standardized tests, absenteeism rates, and college and career readiness indicators. In my home state of Ohio, individual schools also receive report cards, and the accountability system extends down from school districts and schools to individual educators. 

Dr. Deming referred to accountability systems and other similar practices as Management by Objective (MBO) or management by the numbers. This is the practice of focusing on outcomes. Grading and rating school systems and individual schools is akin to the inspection of parts in a factory. One problem with MBO is that goals for accountability and goals for improvement are two very different things. Far too often, these two types of goals get conflated when we discuss educational improvement efforts. Inspection and accountability measures come after the fact, and do not improve the processes that produced the defective results. Accountability goals without a method do not improve the system of education much like the inspection of parts in the factory does nothing to improve the quality of the manufactured product.

The conflation of accountability and improvement goals can have unintended consequences. In the last section, I introduced three ways people can react when pressured to meet a target value, like those in an accountability system. The optimal response is to improve the system. However, the other two responses – distorting the system and distorting the data – are negative reactions. Think here of the schools that you’ve worked in across your career and assess which of the three responses are most common in your experience. For example, if you were around for the implementation of the No Child Left Behind (NCLB) legislation, what did you experience? If you are like many educators, you probably experienced a version of the second bucket – distortion of the system. This occurs when people work to achieve demanded results at the expense of other results. I saw this first-hand as a 5th grade teacher in Atlanta Public Schools from 2001-2004. There was an increased focus on reading and math test results; we were told to spend approximately 180 minutes of the day on these two subject areas. Recess was cut from the elementary-level schedule altogether. Little time was left for science, history, art, music, and physical education. My hunch is that most educators would agree that this system distortion was not a net positive for kids.

I would also be remiss if I didn’t mention the third bucket – distorting the data. This occurs when you get creative with the numbers. Think here of the state testing and other similar cheating scandals that have occurred in the last two decades. While it’s true that the vast majority of educators and educational institutions do not engage in this type of behavior, it’s also true that this is not a rare occurrence. The most egregious example may have occurred in Atlanta Public Schools where the superintendent was accused of running a corrupt organization that used test cores to financially reward and punish teachers. Cheating on student standardized tests were reported in most states and the District of Columbia during the Bush and Obama administrations. The common thread of the cheating scandals is that they came during a time when standardized test scores had become the primary metric to evaluate educators as well as K-12 schools and school districts. I’m not excusing the behavior of those that participated in the cheating, but it is important to learn what it is about the accountability system design that led to so many scandals.

The bottom line in the Deming philosophy is to stop holding people accountable in lieu of improving processes. Numerical goals do not produce quality especially when those goals are outside the capability of the system as it is currently designed.  Instead, they result in distortions of the system or distortions of the data or both.

Myth of Extrinsic Motivators

The basic premise of this myth is that employee performance is determined by putting the right extrinsic motivators in place. Under this supposition, the key is to find the right balance between reward and punishment. While it is true that people are differentially motivated by extrinsic and intrinsic factors, it is a false premise that leaders can improve performance using carrots and sticks. For one, extrinsic motivators do not work or improve performance in most contexts within complex systems like schools. These motivators typically only work in the short-term and even then, they only typically work for simple, repetitive tasks. There are also the unintended consequences that stem from practices like teacher evaluations and merit pay that rely on extrinsic motivation and are more likely to lead to distortions of the system or the data within the system than they are to lead to an improvement of the system. This is because they optimize competition instead of cooperation. All of this brings us to the primary issue with extrinsic motivators targeted at individuals. Individual performance only accounts for a tiny fraction of organizational performance. Deming estimated that 94% of the troubles and possibilities for improvement belong to the system and are the responsibility of management, so incentivizing individuals who have little control over the system as a whole is at best pointless. At worst, these incentive systems decrease organizational performance.

Blog Series: Transformation from Mythology to the New Philosophy

The common thread among all the management myths discussed the last few months is that they sub-optimize the system as a whole. This is a key issue with the prevailing, command and control approach to management. All of the myths – best practices, the hero educator, performance appraisals, merit pay, rating & ranking employees, accountability systems, Management by Objective, and extrinsic motivators – fragment the whole into parts and fail to appreciate the organization as a system. The Deming philosophy does the opposite. The four components of the System of Profound Knowledge work in concert to provide us with profound insights about how our organizations operate so that we as leaders can in turn work to optimize the whole of our systems. However, there is a step beyond simply avoiding these management pitfalls. The next step is to be able to think and make decisions using the lens provided by the System of Profound Knowledge. Here a core set of principles will be helpful. In the next several months, I’ll outline fourteen principles that will enable you to move from theory to practice with the Deming philosophy. These principles will provide systems leaders with a strong philosophical foundation from which to make sound decisions.

***

John A. Dues is the Chief Learning Officer for United Schools Network, a nonprofit charter management organization that supports four public charter schools in Columbus, Ohio. Send feedback to jdues@unitedschoolsnetwork.org. 

Notes

[1] W. Edwards Deming, Out of the Crisis (Cambridge, MA: MIT, Center for Advanced Engineering Study, 1986), 103-104.

[2] Jesse Rothstein, “Measuring the Impacts of Teachers: Comment,” American Economic Review 107, no. 6 (June 2017): 1656-84.

[3] Ibid.
[4] This is attributed to Brian Joiner, but I first read it in: Donald J. Wheeler, Understanding Variation: The Key to Managing Chaos, 2nd ed. (Knoxville, TN: SPC Press, 2000), 20.