Famous studies that didn’t replicate
Lesson 13 of 31 in the course “Brain Hacks”. Free, with animations and a quiz.
The replication crisis
Science’s main test is simple: other people repeat an experiment and get the same result. In 2015 a large team of scientists ran this check on 100 studies from leading psychology journals. In the originals, 97% had a “significant” result. In the repeats, only 36%. That doesn’t mean two thirds of psychology is lies: some effects are just smaller than they seemed, others depend on conditions. But it became clear that a splashy journal study is not yet a fact. The hardest hit were experiments in which a single word or picture supposedly changed behaviour. That’s what this lesson is about.
Does thinking about money make you selfish?
In 2006 a series of experiments came out: people were subtly reminded of money, with phrases, play money on a table, a screensaver of dollar bills. Afterwards they supposedly worked longer on a hard problem alone, helped others less and even set chairs for a conversation further apart: 118 cm versus 80. The story spread through books: “money makes us individualists”. Then the replications began. A large 2015 series found no effect. A 2019 analysis showed that in studies with plans announced in advance, the effect is close to zero, and that “successful” experiments seem to have been published more often. Label: failed to replicate.
Picturing a professor doesn’t add points
A 1998 study: some students were asked to spend five minutes imagining a typical professor, others a football hooligan. Then a quiz. The “professors” got 55.6% of questions right, the “hooligans” 42.6%. Lovely: think of someone smart and you get smarter. Popular books loved it. In 2018, dozens of labs repeated it following a single plan announced in advance, with thousands of participants. The difference between the groups was practically zero. Label: failed to replicate. Thinking about a professor before an exam does no harm, but sleep, revision and practice problems earn the points, not the mindset.
Washing away guilt
Another lovely story: the “Macbeth effect”. In a 2006 study, people recalled a bad deed of their own and then more often chose antiseptic wipes as a gift and were quicker to complete words as “soap” and “wash”. The conclusion sounded poetic: a guilty conscience craves physical cleanliness. But several independent replications, including in different countries, found no effect. Label: failed to replicate. It’s a good example of a trap: the more literary a finding sounds, the more eagerly it’s retold and the less people ask whether it was ever repeated.
Where false discoveries come from
Almost nobody lied on purpose. Three things were at work. First: small samples. With 30 people per group, chance easily produces a “difference”. Second: flexible analysis. You can crunch the data many ways: drop “odd” participants, compare by gender, by time of day. Test 20 options at the usual 5% threshold and the chance of hitting at least one “finding” by luck is about 64%. Third: journals preferred to print “wow”, while experiments with no effect stayed in a drawer. So the luckiest flukes made it into print. This isn’t about scientists being dumb; it’s the autopilot: everyone wants to see a pattern.
Science fixes itself
In 2012, the author of one of the best-known bestsellers on thinking wrote an open letter to colleagues studying behavioural priming: he “sees a train wreck looming” and asked them to replicate their own experiments. Later he admitted: “I placed too much faith in underpowered studies.” The priming chapter in his book leaned heavily on such experiments. That’s no disgrace; it’s science working normally: the error was found and the rules changed. Now good studies announce their plan in advance, recruit hundreds or thousands of participants and replicate across several labs at once. The “disputed” and “failed to replicate” labels in this course come from exactly this.
Hack: 4 questions for “scientists proved”
When you see “scientists proved”, ask four questions. First: have other teams repeated it? One study is a reason to be curious, not a fact. Second: how many people took part? Dozens is few; hundreds or thousands is serious. Third: was the analysis plan set in advance? That gives researchers less chance to “find” something by luck. Fourth: does it sound too good? If one word supposedly changes behaviour by 40%, it’s probably not true: real effects in psychology are usually modest. The answers often take a couple of minutes: search the study’s name plus the word “replication”.
Try it: break down a video
Find a video or post in your feed that says “scientists proved”, “a study showed” or “science-backed”. For example, Danya came across a reel: “Scientists proved a morning ritual gives +40% success.” Try three steps. Find the original: if there’s no link and the study isn’t named, that’s already a red flag. Count the people: how many took part and who they were. Look for replications: the study’s name plus “replication”. For Danya, no original existed at all, and the “40%” came from a course ad. Five minutes of checking saves months spent on “proven” life hacks.
Check yourself
- In 2015, 100 psychology studies were repeated. In what share of the repeats was the result “significant” again?
Correct answer: About a third: 36%. Yes. 97% of originals were “significant”, 36% of repeats. That doesn’t make the rest lies, but a splashy result isn’t yet a fact. - Why can a small study show a “wow effect” by chance?
Correct answer: Few people mean big random swings, and “wow” gets published more. Yes. With 30 people per group, chance easily paints a “difference”, and no-effect papers often stayed in the drawer. - Which sign best suggests an effect can be trusted?
Correct answer: Independent labs replicated it using a plan announced in advance. Yes. Independent replications with a pre-announced plan are the best defence against chance “discoveries”. - The “think of a professor” study gave 55.6% vs 42.6%. What did the big 2018 replication show?
Correct answer: The difference was practically zero. Yes. Dozens of labs and thousands of participants on one plan, and almost zero. Label: failed to replicate.