Scientific publishing loves a dramatic result.
A treatment works.
A gene has an effect.
A relationship is statistically significant.
A surprising mechanism appears.
But what happens to the experiment in which nothing happens?
Historically, quite often, very little.
The National Academies report identifies “publication bias” as an important source of distortion in the scientific literature.
Researchers and journals face incentives favouring novel, statistically significant and apparently important results. Null findings, replications and inconclusive results can be much harder to publish.
That creates a peculiar problem.
Suppose 20 laboratories test a weak effect.
Eighteen find nothing interesting and the studies disappear into drawers or abandoned folders.
Two obtain positive results and publish them.
A reader searching the literature encounters two papers supporting the effect and zero contradicting it.
The literature is technically real.
But it is not representative.
The report provides a powerful example involving tamoxifen. Twenty-two of 23 individual trials discussed in a systematic review did not individually cross the conventional p < 0.05 threshold, yet the cumulative analysis demonstrated an important treatment effect.
That example reveals the opposite danger too.
Scientific understanding cannot be reduced to counting individually significant papers.
The entire distribution of evidence matters.
This is why negative results are scientifically valuable.
They constrain hypotheses.
They improve estimates.
They prevent future researchers from unknowingly repeating dead ends.
And they make systematic reviews more representative of reality.
The report discusses emerging attempts to publish null results, replications and registered reports, while acknowledging that changing journals alone will not solve the problem if academic incentives continue rewarding novelty more strongly than reliability.
The scientific literature should be a record of what we learned.
Not merely a highlight reel.
########################################################################
Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.
No comments:
Post a Comment