In this week’s New Findings in Psychology class, I introduced Annie Duke’s book Thinking in Bets (thinking like a poker player). One of its themes is:
Betting forces people to take a hard look at their own beliefs, and thus make more reliable decisions.
Does this trick really work?
At least one study has shown that, when it comes to scientists reviewing papers, it really does!
For more than a decade, the field of psychology has been embroiled in a debate over the reproducibility of psychological experiments.
Many researchers have found that the results of quite a few psychological experiments—even some famous studies published in top journals—cannot be replicated. And the ability to replicate results is a cornerstone of science; psychology’s scientific credibility has therefore come under serious doubt. The debate reached its climax when a group of scholars launched a team called the Open Science Collaboration (OSC) and conducted replication experiments on 100 psychological studies. They found that fewer than 40% were successfully replicated.
When these findings were published in Science in 2015, they caused an enormous uproar.
This large-scale effort to replicate psychological experiments also gave rise to another fascinating study:
Anna Dreber, a behavioral economist at the Stockholm School of Economics, had a sudden brainstorm: she set up a betting market for the replication effort and opened a betting pool. She recruited a group of psychology experts, gave them some money, and asked them to bet on which experiments could be replicated and which could not. There was also a control condition: no betting, no money involved. The experts simply used their professional knowledge to judge which experiments could be replicated and which could not.
The results showed that under the no-betting control condition, the experts’ predictions were correct 58% of the time. Under the betting condition, their accuracy shot up to 71%.
Apparently, once real money is on the line, people’s judgment on professional matters suddenly becomes much sharper! Money really is a wonderful thing!
In fact, the current system of academic peer review already contains an implicit betting element, because the quality of the peer reviews scientists provide affects their own academic reputations and interests. But Anna Dreber’s study suggests that this betting effect may not be strong enough. Perhaps adding a little real money to the reputational stakes would very likely improve the quality of scientists’ gatekeeping.
Let’s take the speculation one step further: what would happen if we applied the betting mechanism to the people publishing the papers? Suppose every psychologist had to bet on the reproducibility of the experiments in their paper when they published it. If the experiments were successfully replicated later, they would win money; if not, they would lose money. Would the reproducibility of psychological experiments rise to a whole new level?
My guess is that whether this brutal trick works depends on whether the minimum bet is high enough. After all, as long as you get a paper published—even a paper whose findings cannot be replicated—that paper still adds a fairly substantial chip to your promotion-title betting pot. Once you factor in that account, calculating the gains and losses gets awfully tricky.
References:
1. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
2.Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., … & Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343-15347.
3. Annie Duke (2018). Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts.
Notes
- This series of short popular-science essays and miscellanies, New Findings in Psychology, was written at the time to accompany the related content of Zhichao Wei: New Findings in Psychology. They were published on the now-closed platform Fatuan, and the original link no longer exists.
Leave a Reply