This episode is also on Bilibili too.
Today I’m going to unpack a major new psychology book published in 2021: Noise: A Flaw in Human Judgment. Its first author is Nobel Prize–winning economist—and a behavioral economist so famous he could be called half of the psychology world—Daniel Kahneman.
Professor Kahneman’s previous book was Thinking, Fast and Slow, published ten years ago. It received countless rave reviews over the decade and has now become essential reading in psychology. I thought that book had already summed up the essence of Kahneman’s life’s work. But ten years later, Kahneman, approaching 90, has somehow raised and argued for a completely new question in this new book—one he had barely touched before: noise in judgment and decision-making. Because noise exists, we often struggle to make accurate judgments and decisions. What is noise? How does it arise? And how can we avoid it? Those are the questions Kahneman explores in Noise.
Usually, when I talk about a book, I pick out the highlights. But Noise has a very complete system of knowledge. So in today’s episode, I plan to take everyone through the book’s entire framework from beginning to end. Consider this episode a complete guide to Noise. If you later read the book with this episode’s overall impression in mind, it may go down a little more smoothly. Noise is seriously hardcore: its theoretical details and examples are abundant. So if you’re interested, you really should find the book and give it a read.
I’ve divided my discussion of Noise into five parts.
First, what is noise? Second, what types of noise are there? Third, what are the amplifiers of noise? Fourth, how can we reduce noise? Fifth, what obstacles might measures to reduce noise encounter? There is quite a lot of material, so I made a mind map and put it in the episode description. Save it if you need it.
https://www.mubucm.com/doc/5LQ2nZpti3Q
Before introducing the book’s content, let me make a little preview. I invited Mr. Wang Zuojun, translator of the simplified-Chinese edition of Noise and a professor in the psychology department at Ningbo University, to join the show. I’ll have a conversation with him about Noise. After watching this episode, if you have any questions about the book, feel free to ask them in the danmaku comments or in the comment section. I’ll collect representative questions and discuss them with Professor Wang. Professor Wang is also an expert who studies human decision-making, so if you have other questions about how to make judgments or decisions, you can leave those too.
All right, enough preamble. Let’s get into Noise.
The first question: what is noise?
Let’s explain it with a shooting-at-a-target example.
Please take a look at this picture on the screen.
There are four targets in the picture, showing the results of teams A, B, C, and D shooting at them. Each team has five people, and they all use the same rifle.
Team A? Every member is a crack shot, and every bullet hits the bull’s-eye. This is the perfect team.
Team B? Everyone misses, but they miss in an extremely concentrated, perfectly uniform way: every shot lands in the lower-left corner of the target. We call Team B the bias team. Bias means a uniform error like this. It’s not hard to analyze why such an error might occur. The likeliest explanation is that something is wrong with the rifle’s sights.
Team C? Again, everyone misses, but they miss in every which way. The shots are random and scattered, with almost no discernible pattern. Team C is the noise team. This kind of wildly scattered error is noise.
Teams B and C represent two kinds of error humans make when judging. The first is bias, like Team B: when different people make judgments, they systematically lean in the same direction. Why do we make these uniform errors? Because all humans share certain psychological tendencies. For example, we are all hypersensitive to negative information. So when we read some alarmist article online, it can trigger fear in all of us, and we may then make very similar irrational decisions in unison.
The second kind of error humans make is the “noise” represented by Team C. Noise is really fluctuation—a lack of consistency. One person’s decision this time may differ from their decision next time, and when a group judges the same matter, their opinions may differ from one another. That’s noise. It’s everywhere. Interviews contain noise: interviewers often disagree sharply; one may decide you’re exactly right for the job while another kicks you out for supposedly not fitting the company culture. Medical diagnosis contains noise: one doctor says you urgently need hospitalization and surgery, while another says a little medication will do. Stock-market forecasts contain noise: half the institutions say prices will rise tomorrow, half say they’ll fall. That’s just everyday life. Grading papers contains noise too: a teacher may be in a good mood in the morning and give high scores, then be tired in the afternoon and grade strictly. Noise is incredibly, incredibly common.
When making judgments, of course we all hope to be crack shots like Team A. In reality, though, we may be biased like Team B or noisy like Team C. We have a hard time hitting the bull’s-eye and making accurate judgments. Bias and noise often appear together—that’s Team D: the overall pattern leans in one direction, showing bias at work, but the individual shots are still widely scattered, showing noise at work.
In one sentence: errors in human judgment = bias + noise.
Kahneman’s previous book, Thinking, Fast and Slow, was essentially about Team B: it dealt with bias. Noise, naturally, is about Team C. With Noise, the problem of “errors in human judgment” can finally be considered complete.
All right, now we can move on to the second question: how many types of noise are there?
There are three.
- Level noise. Stable pattern noise. Situational noise.
Level noise is the difference between people.
Gangzi and Dazhu are two judges. Gangzi is impartial and extremely harsh. Dazhu is relatively lenient. The same criminal might get 30 years from Gangzi but only 5 years from Dazhu. The sentences handed down by Gangzi and Dazhu, these two people, are wildly inconsistent. That’s level noise.
The second type is called stable pattern noise. Stable pattern noise is an individual’s preferences.
For example, Judge Gangzi is generally quite harsh on criminals, but he has always had a soft spot for white-collar workers, so he is more lenient toward white-collar criminals. Thus, two people who committed roughly equally serious crimes may both have their cases assigned to Gangzi, yet one receives a lighter sentence simply because they are a white-collar worker. Personal preferences lead to inconsistent judgments. That’s stable pattern noise.
If there is stable pattern noise, naturally there is also unstable pattern noise. Gangzi likes white-collar workers—that is a relatively stable tendency. Once he leans that way, he keeps leaning that way. But some tendencies are unstable and change frequently.
Take mood. Gangzi is a night owl and is always especially groggy in the morning. When he gets groggy, his mood worsens, so criminals who appear before him in the morning receive heavier sentences. By afternoon, Gangzi has perked up and his mood has improved, so afternoon defendants receive lighter sentences. That’s unstable pattern noise. In fact, unstable pattern noise is noise produced in different situations and contexts, so it is also called “situational noise.”
To sum up: the disagreement between Gangzi and Dazhu is level noise; Gangzi’s difference in treatment of white-collar and non-white-collar defendants is stable pattern noise; the difference between morning Gangzi and afternoon Gangzi is situational noise. These are the three types of noise.
One thing to note: the three types of noise aren’t an either-or situation. They can appear at the same time. Gangzi might encounter a white-collar criminal while in a bad mood, and his sentence might also differ from Dazhu’s.
Now let’s move on to the third question: what amplifies noise?
Differences between individuals always exist. Everyone has particular preferences, and people’s states rise and fall with the situation, so noise is almost everywhere. But to make matters worse, some factors can increase the magnitude of noise. The book doesn’t give these factors a collective name, but I think we can call them “noise amplifiers.”
There are three common noise amplifiers.
The first is called “objective ignorance.” Objective ignorance means that the person making a judgment genuinely has no way of knowing some necessary information. When is objective ignorance most likely to appear? When predicting the future.
Will stocks rise or fall tomorrow? What changes will occur in the international situation over the next three years? What will the real-estate market do over the next ten? Noise is enormous when making predictions like these. Stockbrokers, political commentators, and economists often disagree dramatically, even making completely opposite predictions. But we can’t entirely blame them for being incompetent, because the future is determined jointly by the present and by many events that have not yet happened. Those future events cannot be predicted today; this is where “objective ignorance” comes in. So predicting the future inevitably involves some blind guessing. Different experts guess blindly in different directions at different times. How could the noise not be large?
When predicting the future, we encounter “objective ignorance.” That’s the first noise amplifier.
The second noise amplifier is called the “matching problem.” What is the matching problem? After watching a movie, you have to rate it. Generally, you won’t write a very long review laying out every single one of your feelings. Usually, you’ll open Douban, go to the movie’s page, and give it a rating from one star to five stars. At that point, you’re actually matching: you’re matching your subjective feelings about the movie to five levels, from one star to five stars. We call this one-to-five-star rating system a scale. Whenever we use a scale to make a judgment, we inevitably run into the matching problem.
So how does the matching problem amplify noise? First, it amplifies level noise—the differences between people. Not long ago, I gave the movie Free Guy a five-star rating. My standard for five stars is “Was I pleasantly surprised?” I thought it would be a ridiculous, mindless movie, but it turned out to be unexpectedly brilliant and a lot of fun, so I gave it five stars. But five stars can mean something completely different to someone else. To another movie fan, five stars might be reserved for an all-time classic. So even if that fan’s subjective reaction to Free Guy was actually pretty similar to mine, they might give it only three stars. To them, three stars is already a very high score. The two of us have completely different understandings of the scale from one to five stars, so our ratings produce level noise, just like Gangzi and Dazhu’s disagreement.
Second, the matching problem also amplifies situational noise. For example, when companies conduct performance reviews, a manager has to rate an employee on a scale from 0 to 100. But matching a person’s performance to a number between 0 and 100 is actually very difficult, so managers’ ratings can be wildly inconsistent. Today Dazhu gets 80; tomorrow he might get 90. Dazhu’s performance hasn’t changed in the manager’s subjective evaluation, but once it has to be turned into a score, the score starts drifting. So the matching problem amplifies situational noise. In other words, people’s understanding of a scale can change from one situation to another.
Okay, the matching problem: that’s the second noise amplifier.
The third noise amplifier is the harmful influence of groups.
We often say, “Three cobblers together can outsmart Zhuge Liang.” Pooling everyone’s wisdom seems like it should improve decision quality. But we’ll see in a moment that when a group makes a judgment or decision together, there are actually some demanding conditions for quality to improve. If those conditions aren’t met, the group can instead increase decision noise. Three mediocre cobblers are often worse than one.
So how do groups amplify noise? There are two ways.
The first is that whoever gets the early lead gains an overwhelming advantage. The book mentions an experiment in which scientists gave volunteers a playlist and let them listen to any songs they wanted. If they especially liked a song, they could download it. By counting downloads, the researchers could see which songs were most popular. But they quietly rigged things: when the playlist was given to volunteers, some songs already appeared to have been downloaded many times. The volunteers assumed those downloads came from people who had taken part in the experiment earlier. In fact, the supposedly popular songs had been selected at random by the researchers. Yet after the experiment, the researchers found that the songs randomly labeled popular at the beginning really had become the popular songs. Volunteers downloaded them more often. In other words, if you get even a small advantage at the start, that advantage will automatically snowball.
That’s also the principle behind the way traffic stars manipulate online reviews. Once the comments pinned at the top are praising a star, later viewers’ opinions will be influenced by them.
Why does this effect happen? Because how good most songs are is somewhat ambiguous, so people are easily influenced by other people’s judgments. If someone decides at the beginning that these songs are good, others will start thinking the same song is good too.
That creates a lot of situational noise. Next time, if a different batch of songs happens to get the early lead, they will become the popular songs.
The experiment also turned up an interesting side finding: the best and worst songs weren’t affected by this dirty trick. Even if the best song started with zero downloads, people would eventually discover it. As for the worst songs, even if they topped the charts at first, people would eventually reject them. So manipulation like review control has limits. If the quality is truly awful, the backlash is only a matter of time.
A similar situation occurs in job interviews. After interviewing each candidate, the interviewers sit together and discuss whom to choose. The interviewer Gangzi is especially fond of a candidate called Little Li. He speaks first and talks about how great Little Li is. Then it’s Dazhu’s turn. Dazhu didn’t have any particular fondness for Little Li, but seeing how certain Gangzi sounds, he assumes Gangzi must have noticed some special advantage in Little Li. So he chimes in: “Little Li really is pretty good.” Then Tiedan speaks. Tiedan’s initial impression of Little Li was negative, though he didn’t have much solid reason for it. Now that he sees both Gangzi and Dazhu supporting Little Li, he decides not to say much. And so Little Li, who may not have been especially outstanding to begin with, ends up passing the interview unanimously by a landslide.
In reality, this is entirely the result of Little Li happening to get the early lead. In another interview, if Tiedan—the interviewer who liked a different candidate, Little Gui—had spoken first, Little Li would have had no chance. That’s how situational noise is created.
Whoever happens to get the early lead then expands that advantage and steers everyone else’s choice. That’s the first reason groups increase noise.
The second reason groups increase noise is called “group polarization.” Polarization means becoming more extreme. Group polarization is what happens when people who already hold roughly similar views on an issue gather to discuss it and come away with much more extreme views: what they liked, they like even more; what they disliked, they dislike even more. That’s group polarization.
For example, a bunch of fans spend all day discussing the traffic star they idolize. They may have started out merely liking the star somewhat, but after the discussion, they all become hardcore fans.
You might say, “Wait, doesn’t that reduce noise? Everyone has become equally extreme, so they’re more consistent with one another.” But the problem is that the world contains many different groups. A group of people who initially only somewhat disliked a traffic star may, after group polarization, come to hate that star bitterly. So the disagreement between fans and nonfans becomes even larger. Overall, then, group polarization still increases noise.
In short, whether because someone got the early lead or because of group polarization, groups often amplify noise.
Objective ignorance, the matching problem, and the harmful influence of groups: these are the three common noise amplifiers.
Now we can move on to the fourth question: if noise is so widespread, how can we reduce it? What methods are available?
I’ve distilled the book’s content into four common methods.
1. Hand the decision-making process to a group in which each individual can make an independent judgment; 2. Replace matching with ranking; 3. Hand the decision-making process to a model; 4. Hand the decision-making process to a “decision-making expert.” Let’s take them one at a time.
The first way to reduce noise is to hand the decision-making process to a group in which every individual can make an independent judgment.
The key word here is “independent.”
We just said that groups often lower decision quality. The key reason is that the judgments of people in a group aren’t independent enough—their judgments about songs are influenced by the initial download counts, and interviewers’ judgments are influenced by other interviewers.
But once each individual in a group makes a completely independent judgment, the situation flips 180 degrees. Aggregating many independent judgments will usually reduce noise substantially. Three independent mediocre cobblers really can outsmart Zhuge Liang.
For example, Noise mentions that in the legal system, fingerprint identification is particularly vulnerable to the harmful influence of groups.
I was pretty surprised when I read this part. I had assumed fingerprint identification worked like it does in many movies: investigators feed a crime-scene fingerprint into a computer, the program automatically compares it, and it spits out a matching suspect. But Kahneman tells us that reality is completely different. Fingerprints collected at crime scenes are generally very poor quality—either incomplete or blurry—so comparing fingerprints relies heavily on the experience and subjective judgment of examiners. That makes fingerprint identification prone to exactly the kind of meeting-room scenario we just saw with the interviewers.
The examiner Gangzi compared a fingerprint collected at the crime scene with the fingerprint of a suspect called Little Gui and concluded that the print belonged to Little Gui.
The detective handling the case wasn’t satisfied, so he took Gangzi’s report to another examiner, Dazhu, and said, “Dazhu, this is Gangzi’s report. Could you take another look?” After examining it, Dazhu also said it was Little Gui’s fingerprint. But this wasn’t genuine agreement between Dazhu and Gangzi. Dazhu already knew Gangzi’s conclusion, so he was very likely to agree with it. Dazhu wasn’t making an independent judgment.
Yet this was apparently the standard procedure used in American forensic identification for a long time: each examiner performed the examination after receiving the results from the examiner before them. The result was a lot of wrongful convictions and other miscarriages of justice, because in practice this procedure amounted to the first examiner having the final say.
So how do we avoid this error? It’s actually quite simple: give the fingerprint to different examiners at the same time and have them make independent judgments. If 10 examiners independently examine the print and most of them—for example, 7—say it belongs to Little Gui, then the detective has good reason to strongly suspect that Little Gui is the criminal.
Make independent judgments, then aggregate the results. This principle can be applied to all kinds of decision-making. For example, when making forecasts, you can use a procedure called the “Delphi method.” It works like this: if you want a group of experts to forecast economic trends, first have them make their forecasts independently. Then average their forecasts of the economic data and send the average back to each expert so they can revise their forecast on that basis, repeating the process for several rounds. In other words, experts can receive statistical information about other experts’ forecasts, but they cannot discuss things with one another. The average forecast produced this way will be much more accurate than one reached by having the experts sit together and discuss freely. The core of the procedure is independent judgment.
Putting the decision process in the hands of a group whose members can each make an independent judgment—that’s the first way to reduce noise.
The second way to reduce noise is to replace matching with ranking.
The independent judgments I just mentioned are a countermeasure to the harmful influence of groups. Replacing matching with ranking is a countermeasure to the other noise amplifier we discussed earlier—the matching problem.
As I said, if a manager has to match an employee’s performance to a score from 0 to 100, that matching is difficult, so the manager’s standards are bound to be pretty erratic. How do we make it easier? First, put employees into simpler categories—for example, Outstanding, Good, Average, Poor, and Very Poor. That’s much easier than assigning a percentage score. Then rank the employees within each category. Xiao Li and Xiao Gui are both in Outstanding, the 81-100 range. If Xiao Li performs better than Xiao Gui, Xiao Li ranks ahead. Once the entire ranking is complete, assign scores according to the ranking.
The principle behind this is that when we compare two things at a time, we can often judge more accurately. It’s hard to say exactly how many points Xiao Li and Xiao Gui deserve for their work, but it’s easier to tell which of them performed a little better. So replacing matching with ranking can reduce noise.
The two methods I just mentioned are both targeted approaches. The next two are general-purpose methods.
The third way to reduce noise is to hand the decision process over to a model.
A model is a formula, a set of rules, or an algorithm.
For example, in a job interview, you don’t decide whom to hire based on the interviewer’s subjective judgment. You use a formula instead: add up the candidate’s scores for responsibility, job ability, and teamwork, and hire whoever gets the highest total.
Or take economic forecasting. Instead of bringing in a group of economists, you feed all kinds of current economic data into an artificial intelligence algorithm, and let the algorithm produce the forecast.
The gaokao, China’s national college entrance examination, is also a kind of model. It uses the exam as a model to select students in place of human judgment.
Human judgment is noisy because, to put it bluntly, people are too flexible and too changeable. Models are rigid, so they have no noise. At bottom, models are just fixed mathematical formulas. As long as the inputs are the same each time, the output will be exactly the same. When it comes to reducing noise, models have an overwhelming advantage over people.
So, if we’re looking only at noise reduction, humanity should hand as much judgment as possible over to models.
Even if humans can’t withdraw completely, they should let the model go first and intervene as late as possible. We should follow the principle of “model first, humans second.” Google does this when hiring. Its interviews have two major stages. First, candidates go through a series of completely standardized assessments and interviews. Each assessment is scored and evaluated independently, every detail is fixed, and even which questions may be asked in the interview is strictly specified. The standardized results are then compiled into a candidate dossier. That dossier is essentially produced by a completely rigid interview model.
But Google doesn’t eliminate human judgment altogether. In the second stage, the dossier is handed to a hiring committee, whose members read it and give the final hiring recommendation.
Google’s hiring process doesn’t eliminate those subtle human intuitions and judgments. It simply delays human involvement as much as possible: let the model judge first, and hand the decision to humans only at the end.
One point needs emphasis here: using a model to make decisions does eliminate noise, but that doesn’t mean the model’s judgment is more accurate. Remember, error in judgment equals bias plus noise. Often, a model merely turns Team D on the target into Team B. The noise is gone, but the bias may remain—the guns are still firing in perfect unison, just at the wrong spot. So handing decisions to a model is not a cure-all. Humans are responsible for continually improving the model, bringing its judgments as close as possible from Team B toward Team A.
The fourth way to reduce noise is to hand the decision process over to “decision makers.”
The final method for reducing noise is simple and crude: find the sharpshooters on Team A in that target diagram. Find the “decision makers” who almost never miss the bull’s-eye, and hand the decisions to them. Wouldn’t that solve the noise problem?
So how do we find these “decision makers”?
Decision makers generally have three distinctive characteristics.
First, they may be experts in a particular field. Lawyers at top law firms and doctors at top hospitals make judgments far more accurately than ordinary people, so we can give their judgments priority.
But be very careful. There are two kinds of experts: genuine experts and honorary experts.
Lawyers and doctors are generally genuine experts, because they earned their status through real results. A famous lawyer became famous by actually winning cases; a famous doctor became famous through genuinely outstanding medical skill. Their performance can be objectively verified. These are the experts whose judgments deserve our trust.
But there is another kind: the “honorary expert.” An honorary expert’s performance can’t be verified. Think of some management consultants or political analysts. Their track records are hard to test against objective standards. If a company prospers, it was the consultant’s doing; if it goes bankrupt, there were other objective causes—in short, never the consultant’s fault. Experts like these build their status on the respect of clients and peers. To put it bluntly, there’s something a little hollow about it. The judgments of honorary experts aren’t better than those of ordinary people, so you should take them with a grain of salt.
Apart from being experts, the second distinctive characteristic of “decision makers” is that they are generally intelligent.
In almost every field, intelligence is associated with better performance. More intelligent people are also more likely to have good judgment. So if we have to choose between two opinions and have no other information to go on, we should give priority to the more intelligent person.
The third distinctive characteristic of “decision makers” is that they have exceptionally open minds.
Intelligence is only one side of the story; the way you think matters too. Some intelligent people are stubbornly self-righteous. They are convinced their own judgments are right, and their judgment is usually not very good.
The truly impressive people are the ones with exceptionally open minds. They are not only intelligent but also humble and practical. They don’t mind hearing opposing opinions. They may even actively seek out new information that could conflict with their original views. They don’t reject integrating new information with their current views; they may even hope that new knowledge will change their minds. Judgments made by people with this kind of openness tend to be more accurate.
So if we have to choose between the advice of two intelligent people, we should prefer the one with the more open mind.
Listen to experts, listen to intelligent people, listen to people with open minds—that’s the fourth way to reduce noise.
We’ve said a lot. We now know that noise is widespread, and we know many ways to reduce it. So why not have every field and industry take action right away, using these methods to reduce noise as much as possible, and call it a day? In reality, measures to reduce noise face a lot of resistance.
That brings us to the final, fifth question: what obstacles do noise-reduction measures face?
The first obstacle is that people don’t trust algorithms. As I said, handing decisions to models can reduce noise dramatically, and the most popular models today are of course algorithms—especially artificial intelligence algorithms based on deep neural networks. The trouble is that people generally don’t trust algorithms. We have a subtle psychological tendency to expect algorithms to be perfect. Making mistakes is a human privilege; machines aren’t allowed to make mistakes. When humans cause a car accident, we find it understandable. But self-driving cars must never have an accident, or they become untrustworthy. If people don’t trust algorithms, they’ll naturally resist measures that use algorithms to reduce noise.
The second obstacle is that people worry models will crush motivation and creativity. If the entire decision process is handed to a rigid model, will people feel like nothing more than a cog in a machine, with no room for initiative? If employees in a company feel they can’t decide anything, won’t their morale take a major hit? And although human decisions can be erratic, they also generate a lot of creative ideas. If the decision process is handed to a rigid model, won’t that crush human creativity?
This actually concerns a major question: in exactly what situations do we need to reduce noise? We want doctors to show no creativity or personal discretion when diagnosing whether someone has high blood pressure; in that situation, the less noise, the better. But in a company, if we want employees to be happier and more inspired, should we allow some noise? Or could we learn from Google’s hiring method? On one hand, standardize the creative process and follow the model’s steps at the broad structural level; on the other hand, give people plenty of flexibility at every step so they can exercise their creativity. Could that work?
Which situations require noise reduction? And how can we reduce noise without damaging people’s motivation and creativity? These are also questions worth thinking about.
The two obstacles I just mentioned are really both aimed at models. This last obstacle is the general problem.
The final obstacle to reducing noise is cost. Reducing noise is obviously good, but is the cost-benefit ratio high enough? For example, if one teacher grades a set of papers, there will inevitably be noise. The most suitable solution would be to have five teachers grade the papers independently and then average their scores. But who pays for all that extra work? Even if someone does pay, is the investment worth it? How do we measure the cost-benefit ratio here? These are also questions that deserve further thought and research.
Although Kahneman repeatedly emphasizes in the book that measures to reduce noise are necessary because noise creates a great deal of unfairness, after reading this section, I came away feeling that questions like how algorithms can earn human trust and how to assess the cost-effectiveness of reducing noise still lack sufficiently in-depth research at this stage. Compared with the earlier material, this part is more open-ended, and there are still plenty of details here worth thinking about and discussing.
And with that, we’ve finished mapping out the knowledge framework of Noise: A Flaw in Human Judgment. We covered the definition and types of noise, noise amplifiers, and the measures and obstacles involved in reducing noise. If you have any thoughts or questions about these issues, feel free to tell me in the comments or the danmu—the scrolling comments on the video. After I collect them, I’ll discuss them with Mr. Wang Zuojun, the translator of Noise, on the next episode or the one after that.
I’m Zhichao Wei. If you enjoy my programs, please follow, like, save, and comment. I’ll keep sharing new insights from psychology on this channel. That’s all for today. See you next time.
Leave a Reply