Getting Better: A Reflection on Intuition vs. Formula
In December 2014, my first child was born. She was ready to enter the world, and my wife was even more ready. Our emotions were similar to most new parents – a combination of exhilaration, anticipation, nervousness, and bliss. Yet in the midst of everything I distinctly remember taking a moment to ask the doctor a question: “What’s my daughter’s Apgar score?” Naturally, I had examined my daughter immediately after she was born. She appeared pink, she was crying, she flexed her arms and legs, and from what I could tell, she had a pulse. Good, right? Nevertheless, I still wanted that number. The response I received was, “Don’t worry. She’s fine. We don’t really use that.” In the moment, I wasn’t going to challenge the doctor. He was a competent professional and we trusted him. If something was wrong, he would have acted. Still, the thought lingered. Why didn’t he give us the score? And why did I care?
In Daniel Kahneman’s seminal work, Thinking, Fast & Slow, he describes a pivotal moment of his career. The year was 1955 and Kahneman, 21 at the time, was serving in the psychology department of the Israeli Defense Forces. Israel was a new country, and young men and women played significant roles in public life. He was assigned an important task. At that time, Israel was screening candidates for specific roles in the military by administering a battery of psychometric tests, followed by an individual personality assessment. The interviewers, also young, were trained to form a general impression of the candidate and determine their best fit among the military branches. This process took tremendous resources and was time-intensive. It was also useless for predicting candidates’ future success.¹
Kahneman discovered that the existing system relied heavily on interviewers’ general impressions. The conversations with candidates went in a variety of directions, and those impressions depended exclusively on the unique nature of the conversation. There was no consistent structure to the interview, so interviewers drew broad conclusions about candidates from limited knowledge.²
“Whenever we can replace human judgment by a formula, we should at least consider it.” ³
-Daniel Kahneman
Consequently, Kahneman developed a new system. He created a list of six characteristics that were relevant to performance in a combat unit, and then composed a series of factual questions about the individual’s life before enlistment (i.e. how many jobs they held, how punctual they were, interest in sports, etc.). Interviewers were told to ask every question, in order, and rate each on a 5-point scale. They were then directed to turn over the data to Kahneman for analysis. There was immediate rebellion, as the interviewers felt they were being treated like robots. Kahneman asked for patience.⁴
Patience paid off. The new system worked, a substantial improvement in predictive value compared to the former protocol. Using the limited time with each candidate to obtain as much specific information as possible led to more consistent and accurate data. Kahneman’s protocol combatted the “halo effect” and addressed his concern that “global” evaluations could not be trusted. Interestingly, to combat the frustration of his interviewers, he added a question. He told his interviewers to close their eyes, try to imagine the recruit as a soldier, and assign him a score of 1-5. Interestingly, this more intuitive rating correlated well to future success as well, leading Kahneman to the conclusion that while we can’t trust intuition, we should certainly not dismiss it. Intuition adds value, but only after collecting objective information using a consistent, disciplined protocol.⁵
In the education world, we consistently fall victim to an overreliance on intuition, just like the Israeli Defense Forces did. We quickly make universal judgements about a student’s future trajectory based on a very limited amount of information. These judgements are often based on gut feeling rather than hard data. We see a student writing actively in class, so we assume she is learning. We see a student arrive to school with tattered clothing, so we conclude that his parents are neglectful. We notice a student talking to another student at recess, so we presume that she has been accepted into a social circle. We meet a student’s heavily-involved parents and take for granted that he will be on track academically. It is easy to think we know more about a student than we actually do.
At School Performance Institute (SPI), Kahneman’s work is informing our 8th Grade On-Track Project. Instead of relying on intuition to determine if our 8th graders are on track for high school readiness, we have adopted an on-track indicator system that requires us to evaluate students in the areas of GPA, class grades, attendance, and behavior. After we assess every student in each category, we assign them an overall designation of “well above,” “on-track,” “off-track,” or “far below.” This designation serves as a starting point from which to work as we endeavor to increase the percentage of our 8th graders that are on track for high school readiness by the end of the year. Adherence to this formula in assembling the initial data, collected over the summer prior to the start of the year, shields our on-track project from the halo effect and an overreliance on intuition. Perhaps an interesting next step for us would be posing a similar “close your eyes” question to our teachers to determine if there is consistency between the formula and our own intuition.
The above graphic shows the scale we utilize for each on-track indicator in the 8th Grade On-Track Project. R=Reading, W=Writing, M=Math.
It’s easy to understand why Kahneman faced pushback upon introducing his new protocol for determining candidates’ best fit. As humans, we want to believe that we can topple a formula; that our intuition knows best. However, research has consistently shown that formulas are more accurate in predicting outcomes. In fact, Kahneman discovered that 60% of studies related to this topic showed significantly better accuracy for the algorithms (vs. humans). The other studies were effectively a “draw,” with no wins for humans. A draw is tantamount to a victory for formulas since they are vastly less expensive to implement on a large scale. Humans are unreliable and inconsistent. Our judgement can sway on simple things like how hungry or tired we are. Whether the topic is pilot training, criminal recidivism, winners of football games, or even wine, algorithms dominate. Remarkably, the simpler the formula, the better.⁶
Thinking back to the delivery room on that memorable night in December 2014, perhaps I simply wanted the data to back up my (and the doctor’s) gut. I valued and trusted the doctor’s intuition, but wanted the number to lead the way. It was late at night. Labor is long, and perhaps the doctor was fatigued. Things can get missed. After all, we’re only human.
Ben Pacht is the Director of Improvement of the School Performance Institute in Columbus, Ohio. The School Performance Institute is the learning and improvement arm of the United Schools Network. Send feedback to bpacht@unitedschoolsnetwork.org.
² Ibid.
³ Ibid.
⁴ Ibid.
⁵ Ibid.
⁶ Ibid.