When Science Fails to Reproduce Itself
Open Science Collaboration (2015): 270 researchers attempted to replicate 100 psychology studies from top journals. Of 97 original studies claiming statistically significant results, only 36 replicated. Effect sizes in replications were half the magnitude of originals. The foundation of psychological science was shakier than anyone realized.
Replication effects were HALF the size of original claims
Journals prefer "positive" results. Studies that find nothing get filed away. Only exciting (often exaggerated) findings get published.
Researchers analyze data many ways until p < 0.05 appears. With enough tries, random noise looks significant. "Torture the data until it confesses."
Small sample sizes can detect effects only if they're huge—or lucky. Real small effects get missed; flukes get magnified.
"Hypothesizing After Results are Known." Present exploratory findings as if you predicted them all along. Distorts true discovery rates.
Contextual differences matter. Effects that work in one culture, time, or setting may not transfer. Science assumes universality that doesn't exist.
Publish or perish. Novel findings get jobs; replications don't. The entire career system rewards flash over rigor.
The crisis sparked a "credibility revolution"—positive structural changes to make science more reliable.
The replication crisis isn't just psychology. Cancer biology: Only 11% of 53 "landmark" studies replicated. Economics: 40% of 18 experimental economics papers failed. Medicine: Most clinical trials don't replicate in practice. We've been treating p < 0.05 as truth when it's just "probably not random noise." Science is self-correcting—but it took a crisis to trigger the correction.