Noise
A Flaw in Human Judgment
byDaniel Kahneman, Cass R. Sunstein, Olivier Sibony
Book Edition Details
Summary
"Noise (2021) is an exploration into the chaotic and costly role that randomness plays in human judgment. By uncovering the mechanisms behind how our minds and societies work, the authors show how noise – unwanted variability in decisions – is both inescapable and elusive. We can, however, with a few solid strategies, make our judgments less noisy and our world fairer."
Introduction
Why do two equally qualified judges sentence similar criminals to vastly different prison terms? Why do insurance underwriters, given identical case files, quote premiums that vary by hundreds of thousands of dollars? Why do hiring managers, reviewing the same résumé, reach opposite conclusions about a candidate's potential? These puzzling inconsistencies reveal a pervasive but largely invisible problem in human judgment: noise. Unlike bias, which pushes judgments systematically in one direction, noise creates random variability where there should be consistency. This work introduces a comprehensive theoretical framework for understanding judgment error through two distinct components: bias and noise. While bias has dominated discussions of decision-making flaws, noise operates as an equally destructive but underrecognized force. The framework reveals how noise infiltrates every domain of professional judgment, from medicine and law to business and government, creating unfairness, inefficiency, and costly errors. The central questions explored include how psychological mechanisms generate variability in expert decisions, why simple algorithms often outperform human judgment, and how organizations can implement systematic approaches to achieve more reliable decision-making without sacrificing the essential human elements that complex judgments require.
Understanding Noise vs Bias in Decision Systems
The fundamental distinction between noise and bias forms the cornerstone of judgment error analysis. Bias represents systematic deviation from truth, like a rifle with a bent sight that consistently shoots to the left of the target. Every shot misses in the same predictable direction, creating a pattern that, while wrong, is at least consistent. Noise, by contrast, manifests as random scatter around a target, like shots spread widely across a target face with no discernible pattern. This theoretical framework reveals why both components matter equally in overall error calculation. The error equation demonstrates that total error equals bias squared plus noise squared, meaning these two forces contribute independently to judgment failures. A biased system might consistently overestimate risks, while a noisy system produces wildly inconsistent risk assessments for identical cases. Both create problems, but noise often proves more insidious because it remains hidden from casual observation. The framework decomposes system noise into three distinct components that operate independently. Level noise represents stable differences between judges in their average severity or leniency. Pattern noise captures how individual judges react differently to specific features of cases. Occasion noise reflects the random variability in a single judge's decisions over time, influenced by factors like mood, fatigue, or recent experiences that should be irrelevant to the judgment at hand. Consider criminal sentencing, where bias might manifest as systematically harsher penalties for certain demographic groups, while noise appears as inexplicable variations between judges handling similar cases. A defendant's sentence becomes a lottery dependent on which judge happens to be assigned, creating profound unfairness even when no systematic bias exists. Research reveals that judges are more likely to grant parole early in the day or after meal breaks, and that asylum decisions vary with outdoor temperature, demonstrating how arbitrary factors influence supposedly objective legal judgments.
The Psychology and Measurement of Judgment Variability
Human judgment operates like an imperfect measuring instrument, subject to both systematic errors and random fluctuations. The psychological mechanisms underlying noise stem from the inherent variability in how our minds process information. Unlike physical measuring devices that might drift due to temperature changes or mechanical wear, human judgment varies due to countless factors that shift our mental state from moment to moment. The cognitive architecture that produces noise operates through several key mechanisms. Our minds rely heavily on heuristics and mental shortcuts that, while generally useful, introduce inconsistency when applied across different contexts or time periods. The matching operation that translates our impressions into numerical judgments or categorical decisions is inherently imprecise, much like trying to measure temperature with a ruler that changes its markings from day to day. Additionally, our tendency toward excessive coherence makes us overconfident in judgments based on limited information, causing small differences in initial impressions to cascade into dramatically different final conclusions. Measurement of judgment variability requires sophisticated statistical techniques that separate unwanted noise from legitimate differences in professional opinion. A noise audit, the primary diagnostic tool, involves having multiple qualified professionals evaluate identical cases independently, then analyzing the patterns of disagreement to quantify different types of variability. This process often shocks organizations when they discover the true magnitude of inconsistency among their experts. Consider how two equally qualified doctors might diagnose the same patient differently simply because one encountered the symptoms in a different sequence, or how their recent experiences with similar cases unconsciously influence their interpretation of ambiguous test results. These variations reflect not incompetence or bias, but the natural operation of human psychology under uncertainty. Understanding these mechanisms reveals why noise is not a flaw to be eliminated through better training alone, but a systematic feature of human judgment that requires deliberate countermeasures.
Predictive Models and the Limits of Human Accuracy
The comparison between human judgment and algorithmic prediction reveals fundamental limitations in our cognitive architecture. Simple statistical models consistently outperform human experts, not because algorithms possess superior insight, but because they eliminate noise while maintaining whatever signal exists in the available information. A linear regression model applying the same weights to predictors across all cases achieves more accurate predictions than experts who vary their approach from case to case. This superiority stems from the noiseless nature of mechanical prediction rather than algorithmic sophistication. Even randomly weighted linear models often outperform human judges, demonstrating that consistency matters more than subtle insights. The model of the judge phenomenon shows that a simple formula capturing a person's judgment patterns predicts outcomes more accurately than the person themselves. This occurs because the model eliminates the noise inherent in human judgment while preserving the valid signal. Research across fields from medicine to criminal justice demonstrates that basic algorithmic approaches frequently surpass human experts in predictive accuracy. A simple bail decision algorithm using just two factors, age and number of prior arrests, matches or exceeds the performance of experienced judges who consider extensive case information. Similarly, basic statistical models for medical diagnosis often prove more reliable than physician judgment, not because doctors lack knowledge, but because human decision-making introduces variability that degrades overall performance. The theoretical framework reveals why this pattern emerges across domains. Human experts believe they add value through complex reasoning and case-specific insights, but these supposed advantages rarely compensate for the noise they introduce. The illusion of validity leads professionals to confuse their confidence in evaluating cases with their ability to predict outcomes. However, the choice between human and algorithmic judgment need not be binary. Hybrid approaches that combine algorithmic consistency with human insight often achieve optimal results, with algorithms handling routine classifications while humans focus on exceptional cases requiring nuanced interpretation.
Strategies for Reducing Noise in Professional Judgment
Decision hygiene encompasses a systematic approach to reducing noise through preventive measures that improve judgment quality without requiring knowledge of which specific errors are being prevented. Like medical hygiene that protects against various pathogens without targeting specific diseases, decision hygiene implements structural changes to judgment processes that reduce variability across multiple sources simultaneously. The foundation of effective decision hygiene rests on several key strategies that can be adapted to different organizational contexts. Structuring complex judgments involves decomposing decisions into component parts, evaluating each element independently, and delaying holistic integration until all relevant information has been systematically considered. This approach prevents early impressions from contaminating subsequent assessments and ensures that all relevant factors receive appropriate attention. Aggregating multiple independent judgments harnesses the wisdom of crowds principle, mathematically guaranteeing noise reduction while often improving accuracy through the combination of diverse perspectives. The mediating assessments protocol represents a comprehensive framework for noise reduction in high-stakes decisions. Rather than making holistic judgments, decision-makers break complex evaluations into predetermined dimensions, assess each element independently, and then combine these assessments using specified rules. This method preserves the benefits of human insight while imposing the discipline needed to reduce noise. Organizations implementing such approaches often discover that the structured process not only reduces variability but also improves the quality of deliberation by forcing explicit consideration of factors that might otherwise be overlooked. Establishing shared scales grounded in objective reference points helps judges calibrate their assessments consistently, reducing the variability that emerges when different people interpret rating scales differently. Guidelines that specify relevant factors and their relative importance provide structure without eliminating professional judgment, channeling discretion toward the most important considerations while reducing arbitrary variation. The ultimate goal is not to eliminate human judgment but to improve its quality and consistency, preserving the essential role of human expertise while dramatically improving reliability and fairness in professional decision-making.
Summary
The central insight of this theoretical framework is that noise represents an equal partner to bias in creating judgment error, yet remains largely invisible and unaddressed in most decision-making systems. By developing systematic approaches to measure and reduce noise, organizations can achieve substantial improvements in decision quality, fairness, and efficiency without sacrificing the human elements essential for complex judgments. This framework transforms our understanding of human judgment from a narrow focus on systematic biases to a comprehensive view of error that encompasses both predictable and random components, offering practical pathways for creating more reliable and equitable decision-making systems across every domain where human judgment matters.
By Daniel Kahneman
