Few phrases have travelled further than “replication crisis.”
It is dramatic.
Memorable.
Clickable.
And the National Academies report is notably uncomfortable with it.
The reason is not that reproducibility problems are imaginary.
They are real.
The problem is the word crisis.
How broken is science?
The frustrating answer is:
We do not know precisely.
Replication projects have often selected particular sets of studies rather than random samples of entire disciplines.
Different projects define successful replication differently.
Some compare statistical significance.
Others compare effect sizes.
Others examine direction.
Others consider uncertainty.
Some replication attempts also differ methodologically from the original study.
The committee therefore concluded that existing evidence cannot provide one defensible science-wide replication rate.
Brian Nosek, whose work has been central to large-scale replication research, is quoted in the report as saying:
“I don’t like the term crisis.”
His reason is revealing.
The word implies knowledge about the scale and abnormality of the problem that we simply do not possess.
What replication rate should healthy science have?
This question is rarely asked.
Should 100% of results replicate?
Probably not.
A field producing nearly perfect replication could achieve that by studying only extremely predictable phenomena.
Innovative research necessarily enters regions of greater uncertainty.
Some bold hypotheses will be wrong.
Some apparent discoveries will disappear.
Others will reveal hidden variables.
The report explicitly emphasizes that some failures to replicate are part of normal scientific progress.
That does not excuse bad science
There is an equally important distinction.
Non-replication caused by intrinsic complexity is scientifically informative.
Non-replication caused by:
poor design,
inappropriate statistics,
errors,
selective reporting,
incomplete methods,
or misconduct
is avoidable and costly.
The report therefore separates potentially informative sources of non-replicability from those that merely waste scientific effort.
Perhaps science is not a collapsing building
The book contains a lovely analogy borrowed from philosopher Massimo Pigliucci.
Scientific knowledge is better imagined as a web than as a tower.
A tower can collapse if its foundation fails.
A web can lose individual strands and remain intact while damaged connections are repaired and strengthened.
That may be the best image with which to end this series.
Individual papers can fail.
Famous results can disappear.
Methods can be corrected.
Entire subfields can revise their practices.
Yet the larger scientific network can become stronger precisely because those weaknesses are exposed.
The real danger is not that science sometimes fails to replicate.
The danger would be a scientific culture in which nobody tried.
