# Replication Crisis The replication crisis refers to the widespread failure of scientific studies—particularly in psychology, medicine, and social sciences—to reproduce their original findings when repeated by other researchers. The crisis gained prominence after the 2015 Reproducibility Project found that only 36% of 100 psychology studies could be replicated. High-profile casualties include [[Roy Baumeister]]'s ego depletion effect, social priming studies (elderly walking slower after seeing old-age words), and power pose research. The crisis has forced a fundamental rethinking of research practices, statistical methods, and publication incentives. The causes are structural: journals prefer novel, positive findings ("publication bias"), researchers face pressure to publish frequently ("publish or perish"), small sample sizes produce unreliable results, and "p-hacking" (manipulating analysis to achieve statistical significance) is common. Solutions emerging from the "credibility revolution" include pre-registration of studies, larger sample sizes, open data/materials, registered reports (peer review before data collection), and adversarial collaborations. The crisis has been painful but ultimately healthy—science is self-correcting, and research practices are improving. Fields affected include social psychology, behavioral economics, cancer biology, and neuroscience. ## The Crisis Visualized ``` ┌─────────────────────────────────────────────────────────────┐ │ THE REPLICATION CRISIS │ ├─────────────────────────────────────────────────────────────┤ │ │ │ REPRODUCIBILITY PROJECT: PSYCHOLOGY (2015) │ │ │ │ 100 published studies tested │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │████████████████████████████████████░░░░░░░░░░░░░░░░░│ │ │ │ 36% replicated │ 64% failed │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ HIGH-PROFILE FAILURES: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ • Ego depletion (Baumeister) - effect not found │ │ │ │ • Social priming (Bargh) - elderly walking study │ │ │ │ • Power poses (Cuddy) - hormonal effects contested │ │ │ │ • Facial feedback (Strack) - smiling makes happy │ │ │ │ • Stereotype threat - mixed replications │ │ │ │ • Many more... │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ## Causes of the Crisis | Cause | Description | |-------|-------------| | **Publication bias** | Journals prefer positive, novel findings | | **Small samples** | Underpowered studies produce false positives | | **P-hacking** | Manipulating analysis to get p < 0.05 | | **HARKing** | Hypothesizing After Results are Known | | **Publish or perish** | Career pressure to produce findings | | **Lack of transparency** | Hidden data, unreported analyses | | **Flexibility in analysis** | Many "researcher degrees of freedom" | ## Failed Replications (Examples) | Finding | Original Claim | Replication Result | |---------|---------------|-------------------| | **Ego depletion** | Willpower depletes with use | Large studies found no effect | | **Social priming** | Old-age words → walk slower | Failed to replicate | | **Power poses** | Posture affects hormones | No hormonal changes found | | **Facial feedback** | Holding pen in teeth → happier | Failed multi-lab replication | | **Red enhances attraction** | Red clothing increases attractiveness | Mixed/failed replications | | **Money priming** | Money images → selfish behavior | Failed to replicate | ## Statistical Issues | Problem | Description | |---------|-------------| | **P-value misuse** | p < 0.05 misunderstood as "truth" | | **Low power** | Small samples miss real effects, find false ones | | **Multiple comparisons** | Testing many outcomes inflates false positives | | **Questionable research practices** | Selective reporting, data exclusion | | **File drawer problem** | Null results unpublished | ## Solutions: The Credibility Revolution | Solution | How It Helps | |----------|--------------| | **Pre-registration** | Commit to analysis plan before data collection | | **Registered reports** | Peer review before results known | | **Open data** | Share data for verification | | **Open materials** | Share methods for exact replication | | **Larger samples** | More statistical power | | **Multi-lab studies** | Same study across many labs | | **Adversarial collaboration** | Critics and proponents work together | | **Effect size focus** | Report magnitude, not just significance | ## Timeline | Year | Event | |------|-------| | 2005 | Ioannidis: "Why Most Published Research Findings Are False" | | 2011 | Bem's "Feeling the Future" (precognition) published, sparks debate | | 2011 | Diederik Stapel fraud exposed | | 2012 | Failed Bargh priming replication | | 2015 | Reproducibility Project: 36% replication rate | | 2016 | Ego depletion fails large replication | | 2018 | "Many Labs 2" - 14 of 28 effects replicated | | Ongoing | Reforms spreading through science | ## Fields Affected | Field | Issues | |-------|--------| | **Social psychology** | Priming, implicit effects | | **Behavioral economics** | Nudges, decision-making | | **Cancer biology** | Drug target studies | | **Neuroscience** | Brain imaging studies | | **Medicine** | Clinical trial results | | **Education** | Learning interventions | ## Impact on Specific Theories | Theory/Finding | Status | | ------------------------------------------ | ----------------------------------- | | **[[Roy Baumeister]]'s [[Ego depletion]]** | Contested, may depend on beliefs | | **[[Growth mindset]] (Dweck)** | Effects smaller than claimed | | **[[Grit]] (Duckworth)** | Overlaps with conscientiousness | | **[[Nudge theory]]** | Some replicate, some don't | | **Marshmallow test** | Confounded by socioeconomic factors | ## Positive Outcomes | Outcome | Description | |---------|-------------| | **Better practices** | Pre-registration becoming standard | | **Larger studies** | More statistical power | | **Open science** | Data sharing increasing | | **Healthy skepticism** | Replications valued | | **Reform incentives** | Some journals reward rigor | ## References - Open Science Collaboration. "Estimating the reproducibility of psychological science" (2015) - Ioannidis, John. "Why Most Published Research Findings Are False" (2005) - Nosek et al. "Scientific Utopia" papers - https://en.wikipedia.org/wiki/Replication_crisis ## Related - [[Roy Baumeister]] - [[Ego Depletion]] - [[Scientific Method]] - [[Grit]] - [[Nudge theory]]