Half of Social Science Didn't Survive a Seven-Year Retest. 274 Claims Redone, and the Survivors Came Back Half as Strong

A finding in a top journal carries an implicit promise. Peer review is the hardest filter in science, so a study that clears it earns a special status: readers cite it, lecturers teach it, policymakers build on it. The working assumption, rarely stated and almost never tested at scale, is that most of these findings would hold up if someone ran them again. Three papers in Nature just tested that assumption at a scale nobody had attempted, and the assumption did not survive the encounter, because when 865 researchers spent seven years redoing 274 published claims drawn from 54 leading journals, roughly half of those claims failed outright and the survivors came back dramatically smaller than advertised.

The project is SCORE, Systematizing Confidence in Open Research and Evidence, funded by the US Defense Advanced Research Projects Agency and coordinated by the Center for Open Science. Over seven years, 865 collaborators began with 3,900 papers across 62 journals and selected 274 specific claims from 164 quantitative papers published between 2009 and 2018 in 54 journals, spanning twelve fields from economics and criminology to psychology and sociology.

The retest was built to be fair to the originals. Each replication carried a median 99.6% power to detect the original effect size, so a miss cannot be dismissed as an underpowered shrug, and teams used the original materials when available, preregistered their protocols, and had those protocols peer-reviewed in advance through a standardized internal process. Andrew H. Tyner led the replication effort with 291 co-authors. If the reported effects were real at the reported sizes, this machinery would find them.

It found about half, with replications returning statistically significant results in the original pattern for 151 of 274 claims: 55.1%, with a 95% confidence interval of 49.2 to 60.9, and 49.3% at the paper level when weighted for papers contributing multiple claims. Rates varied modestly across disciplines, from 42.5% to 63.1%, with high uncertainty on some field estimates. The plain reading is the uncomfortable one: the odds that a published claim in these journals survives a serious retest are roughly a coin flip.

The misses are only half the story, because the hits shrank. Across claims measured with Pearson’s r, the median effect size was 0.25 in the originals and 0.10 in the replications, an 82.4% reduction in shared variance. Here is the arithmetic the paper invites but never runs in plain language: an r of 0.25 explains about 6% of behavioral variance, while an r of 0.10 explains about 1%, which means the typical published effect accounted for roughly six cents on the dollar of the behavior it described and, after replication, about one cent. The literature was not only half wrong. Its surviving half was about one-sixth as strong as advertised.

The headline number is softer than it looks, and the authors say so themselves. They applied thirteen different methods for judging replication success, and the answers ranged from 28.6% to 74.8%, with 49.3% as the median. They also sampled only positive results, so some shrinkage toward zero is expected from regression to the mean, a caveat they state explicitly. The literature’s defenders have real material here, but every one of the thirteen methods found substantial non-replication, so the direction is not negotiable.

The replication paper did not travel alone. Miske and colleagues tested reproducibility in a second Nature paper: given the original data and code, can you recompute the result? From a stratified sample of 600 papers, only 144 authors, 24%, provided usable data at all, and of the 143 studies that could be evaluated, 53.6% reproduced exactly and 73.5% approximately. Sharing decided it: when data and code were shared in advance, reproduction reached 88%; when reconstructed from partial information, it collapsed to 11%. Aczel and colleagues tested analytical robustness by handing the same data to different analysts: for each of 100 studies, at least five independent reanalysts worked the same claim, with only 34% landing within a narrow tolerance of the original effect, 57% within a fourfold-wider one, and 74% agreeing on the conclusion. Three separate gauntlets, and three middling survival rates.

Do the reader arithmetic: of the 274 claims retested, 123 failed outright, so reading a hundred social-science findings a year from these journals means roughly forty-five will be the kind that would not survive a retest, while the survivors explain a fraction of what they claim. The funder’s motive makes it concrete, since SCORE began because defense planners wanted to know which social science to trust, and the answer is to assume half and verify the rest. The team also tried to train AI models to predict which papers would replicate, hoping for a cheap credibility filter, but the models were not reliable enough to use, so no shortcut exists.

The strongest defense starts with the number itself: thirteen methods, thirteen answers, 28.6% to 74.8%, where the binary criterion behind the headline is a harsh judge. A replication returning p = 0.06 on a real but smaller effect counts as a failure, though the effect may exist, and the authors acknowledge the regression problem directly. Second, there is no exact replication in the social sciences: different participants and contexts produce different outcomes, a point Nature’s own commentary stressed, and no field was completely broken. Third, the sample is a decade old, because papers from 2009 to 2018 predate the open-science reforms, preregistration badges, reporting guidelines, and data-sharing mandates that the 2015 crisis triggered, so this audit photographs the literature before the response, a response that may have improved the picture and has not yet been audited. Fourth, some misses are context rather than error: a finding that depends on a cultural moment or a hidden moderator is a boundary condition the original authors never knew they had, not fraud, and the critics who made this case against the 2015 psychology project were never fully answered.

Related Reading

Fact-Checking Doesn’t Backfire. 52 Issues, 10,100 People, Zero Boomerangs

One of the findings SCORE would have sampled: for a decade, the standard advice was that correcting misinformation only entrenches it. Five experiments with more than 10,100 subjects found the boomerang was a phantom, and facts mostly worked.

Growth Mindset Was Supposed to Revolutionize Education. Two Meta-Analyses of 423,000 Students Put the Intervention Effect at 0.08 Standard Deviations

The shrinkage SCORE documented in miniature: a famous education finding that survived replication technically, but at an effect size so small it barely registers.

The Marshmallow Test Convinced the World That a Four-Year-Old’s Willpower Predicts Adult Success. A 918-Child Replication Cut the Effect by Two Thirds

The original finding shrank but survived, exactly the pattern SCORE found across the literature: real effects, reported at two to three times their true size.

Sources

1. Tyner AH, Abatayo AL, Daley M, et al. Investigating the replicability of the social and behavioural sciences. Nature. 2026;652:143-150. doi:10.1038/s41586-025-10078-y
2. Miske O, et al. Investigating the reproducibility of the social and behavioral sciences. Nature. 2026;652:126-134. doi:10.1038/s41586-026-10203-5
3. Aczel B, et al. Investigating the analytical robustness of the social and behavioural sciences. Nature. 2026;652:135-142. doi:10.1038/s41586-025-09844-9
4. Jones N. Half of social-science studies fail replication test in years-long project. Nature (news). April 1, 2026. doi:10.1038/d41586-026-00955-5
5. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716. doi:10.1126/science.aac4716
6. Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2018;2:637-644. doi:10.1038/s41562-018-0399-z