What Happened When a Journal Actually Tried to Run the Code

Scientific peer review normally asks whether a paper appears scientifically sound.

But what happens if a journal asks a more mechanical question:

Does the submitted code actually reproduce the submitted results?

The American Journal of Political Science introduced external reproducibility checks for accepted manuscripts.

The results are striking.

Among the first 116 papers examined, only 8 reproduced successfully on the first attempt.

In another description in the report, 108 of 116 required authors to provide additional information or make some change.

That is roughly 93%.

These were papers that had already survived scholarly review

This is what makes the case interesting.

The papers were not random student assignments.

They were publication-ready research.

Yet when someone attempted to reconstruct the calculations, missing information or other issues commonly surfaced.

This reveals a gap between two questions:

Does the analysis seem plausible?

and

Can someone actually execute it?

Peer review often answers the first.

Reproducibility auditing attempts the second.

It is not cheap

The report estimates that the external reproducibility check added about 52 days to the publication process and required approximately eight person-hours on average.

That is not trivial.

Someone must obtain the code.

Understand the data.

Reconstruct the environment.

Run the analysis.

Compare outputs.

Communicate discrepancies.

Wait for revisions.

Then try again.

But the audit changes behaviour before it happens

There is also a subtler benefit.

If researchers know that another person will actually run their scripts, they have a powerful incentive to organize code, document dependencies and make analytical steps explicit.

The audit does not simply catch problems.

Its existence may alter research behaviour upstream.

Think of it as the scientific equivalent of knowing someone else will eventually open your kitchen cupboards. Suddenly, everything gets labelled.

Should every journal do this?

Perhaps not immediately.

The costs could be substantial, especially for complex analyses.

But the experiment raises an uncomfortable question.

If a major conclusion depends entirely on computation, should verifying that the computation can actually be reconstructed remain optional?

Peer review asks whether we believe the recipe.

A reproducibility audit tries cooking the dish.

Those are not the same test.

Your Paper May Survive. Your Software Might Not.

Imagine finding a fascinating computational paper from 2012.

The methods sound clear.

The authors even shared their code.

Excellent.

You click the repository.

Gone.

Perhaps the repository disappeared.

Perhaps a dependency no longer exists.

Perhaps the code requires an obsolete version of Python.

Perhaps the dataset survives in a format that modern software can no longer read.

The article remains pristine in the journal archive.

Its computational ecosystem has become archaeological material.

Computational research has dependencies

Modern analyses are rarely just code + data.

They can depend on:

operating systems,

library versions,

reference databases,

compiler versions,

hardware,

workflow order,

parameter files,

and external software.

The National Academies therefore recommends documenting not only data and analytical methods but also the computational environment, including software dependencies.

Version control records the scientific history

The report discusses version-control systems such as Git because the precise version of code matters.

If collaborators modify the same program over months, which version generated Figure 4?

Version control can preserve that history automatically.

The report also describes workflow-management systems that record the path from input data through transformations to final output.

That is important because computational provenance answers a deceptively simple question:

Where did this number come from?

A GitHub link is not necessarily an archive

The report makes another important distinction.

Commercial hosting platforms can change priorities or disappear.

A researcher who deposits code only in a mutable online repository risks losing the exact scientific object associated with publication.

Archival repositories and persistent identifiers such as DOIs provide stronger preservation.

And sometimes even preservation is not enough.

The report notes that old datasets may eventually exist in formats that modern machines cannot read. Yale's library, for example, had begun exploring emulation infrastructure to provide access to obsolete software.

The scientific record now includes software

We once thought of scientific archiving largely in terms of papers.

Then datasets.

Now software environments must increasingly be treated as research objects too.

The future historian of science may therefore need more than PDFs.

They may need containers, virtual machines, archived libraries, reference-database snapshots and executable workflows.

A modern scientific result is sometimes less like a paragraph and more like a small ecosystem.

If we want the result to survive, we have to preserve more than its photograph.

The File Drawer: Science's Invisible Literature

Imagine conducting an experiment and finding nothing.

No statistically significant difference.

No dramatic effect.

No exciting mechanism.

You write up the result and send it to a journal.

Would it receive the same enthusiasm as a surprising positive finding?

Historically, often not.

This creates what has become known as the file-drawer problem: studies with null, negative or inconclusive results quietly disappear while positive findings travel into the published literature.

The National Academies report states the consequence plainly: publication bias can produce a literature that “does not reflect the full range of evidence.”

Imagine 20 experiments

Suppose 20 laboratories independently test an effect that does not actually exist.

By chance, one or two may produce p < 0.05.

If the 18 null results remain unpublished while the two positive findings appear in journals, anyone conducting a literature search receives a distorted picture.

The published literature appears to contain two successful demonstrations.

The invisible literature contains the rest of the story.

This creates a strange situation when another laboratory later publishes a null result.

It may be described as “failing to replicate” the positive papers.

Yet the National Academies points out that this supposed failure might actually replicate numerous negative experiments that were never published.

The problem is not necessarily the new experiment.

The problem may be the window through which we are looking at the evidence.

Funnel plots reveal the missing half of the landscape

One method for detecting possible publication bias is the funnel plot.

Studies are plotted according to estimated effect size and precision.

If all results are represented, the scatter should have a broadly expected pattern.

If small studies showing weak or null effects mysteriously vanish from the published literature, the pattern can become asymmetric.

The report's Figure 5-3 illustrates exactly how selective publication can distort what an evidence base looks like.

Why publishing “nothing happened” matters

A good null result can:

prevent duplication of failed approaches,

constrain theories,

improve meta-analysis,

reveal boundary conditions,

and provide a more realistic estimate of an effect.

The report therefore discusses efforts to create outlets for negative, null and replication studies, as well as preprint repositories that allow results to circulate even if conventional journals do not publish them.

Science does not only need spectacular discoveries.

It also needs a reliable record of all the places researchers looked and found less than they expected.

Sometimes “nothing happened” is exactly the missing piece.

Twenty-Two “Negative” Trials and a Treatment That Worked

Here is a statistical puzzle.

Twenty-three clinical trials examine a treatment.

Twenty-two fail to achieve the conventional threshold of p < 0.05.

Would you conclude that the treatment does not work?

That conclusion would have been badly wrong in the example discussed by the National Academies.

The treatment was tamoxifen, used in breast cancer.

When researchers examined the trials collectively, the accumulated evidence indicated a substantial benefit, corresponding to about a 16% reduction in the odds of death, with the uncertainty reported as ±3%.

The example exposes one of the strangest habits in scientific reasoning:

treating each study as an isolated referendum in which p < 0.05 means YES and p ≥ 0.05 means NO.

A non-significant study is not necessarily evidence of no effect

A study can fail to reach statistical significance for many reasons.

The true effect may be modest.

The sample may be small.

Measurements may be noisy.

The estimate may point in the expected direction but remain uncertain.

If many studies produce similar but individually imprecise estimates, the collective evidence can become convincing.

This is why scientific understanding often comes from research synthesis, not from asking which individual paper crossed a numerical finish line.

The tamoxifen example flips the usual replication anxiety

Much discussion about reproducibility worries that positive studies may turn out to be false positives.

That is important.

But the opposite problem exists too.

An obsession with statistical significance can cause useful evidence to be dismissed simply because individual studies lack sufficient power.

The report therefore cautions against judging replication solely by whether two studies independently cross the same significance threshold.

Imagine:

Study A: p = 0.049.

Study B: p = 0.051.

One is declared “significant.”

The other is declared “not significant.”

Yet the actual estimates may be almost indistinguishable.

Nature did not suddenly change at the third decimal place.

Look at the evidence, not merely the label

A better comparison considers:

effect sizes,

uncertainty,

distributions,

sample sizes,

consistency of direction,

and the full body of available evidence.

The report repeatedly argues that scientific confidence should emerge from cumulative evidence rather than a duel between individual studies.

The tamoxifen story therefore contains a beautiful statistical warning.

Twenty-two studies can each fail to shout loudly enough on their own.

Together, they may form a chorus.

Thirty Teams, One Dataset, Many Answers

Give scientists the same data and the same question.

Surely they will obtain the same answer?

Not necessarily.

One of the most revealing examples in the National Academies report concerns an experiment in which nearly 30 independent research teams were given the same dataset.

They were asked whether football referees were more likely to give red cards to darker-skinned players and whether this relationship was associated with measures of bias in the referees' countries.

The scientists did not simply converge on one result.

Sixty-nine percent of the teams reported a statistically significant positive relationship. Thirty-one percent did not.

Same data.

Same broad question.

Different answers.

Welcome to the analytical multiverse. 🧭

Data do not analyse themselves

Between a raw dataset and a final graph lies a chain of decisions.

Should certain observations be excluded?

How should variables be grouped?

Which statistical model should be fitted?

Should particular variables be transformed?

Which covariates belong in the model?

How should missing data be treated?

Which observations count as outliers?

These decisions are often perfectly defensible individually.

But different combinations can lead researchers down different analytical paths.

The football study found that not only statistical modelling but also decisions about grouping the data contributed to the variation in outcomes.

There may be hundreds of invisible methodological decisions

The report makes the broader point that a scientific experiment can contain hundreds or thousands of choices, many so routine that researchers may not even think to report them.

Was a reagent shaken or stirred?

At exactly what temperature was it stored?

How were survey questions worded?

In what order were they asked?

Which samples were discarded?

Which software parameters were left at defaults?

Such details can sometimes turn out to matter enormously.

This has a direct parallel in genomics.

Consider how many choices occur between a FASTQ file and a biological conclusion:

reference genome,

read trimming,

mapping quality threshold,

duplicate handling,

variant filtering,

annotation database,

missingness threshold,

population filtering,

statistical model.

A pipeline may look deterministic once it is written.

The process by which that pipeline was chosen often was not.

Transparency means showing the forks in the road

This is why reproducibility cannot mean merely uploading one final script.

Ideally, a scientific report should make clear how data were prepared, which analyses were planned, which emerged later, and how uncertainty was assessed.

The goal is not to eliminate researcher judgment.

That would be impossible.

The goal is to make consequential judgment visible.

The bigger lesson

When two researchers disagree, we instinctively ask:

Which one made the mistake?

Sometimes neither did.

They may have made different reasonable analytical decisions.

That possibility changes how we interpret scientific disagreement.

Instead of treating variation among analysts as noise that should be hidden, perhaps we should sometimes measure it.

Because if a scientific conclusion changes dramatically when one reasonable analytical choice is swapped for another, that fragility is itself part of the result.

Cold Fusion and the Experiment That Science Refused to Believe

In March 1989, electrochemists Martin Fleischmann and Stanley Pons announced an extraordinary result.

They claimed that they had achieved fusion of hydrogen into helium at room temperature.

If correct, “cold fusion” would have challenged conventional expectations about the conditions necessary for nuclear fusion. The finding was sufficiently extraordinary that laboratories around the world immediately began trying to reproduce it.

And that is where this story becomes more important than cold fusion itself.

The National Academies report uses the episode to explain one of the most basic assumptions underlying all scientific inquiry:

“Nature is not capricious.”

If the laws governing an experiment are real, another laboratory reproducing the relevant conditions should not discover that physics has suddenly decided to behave differently.

Dozens of laboratories attempted to replicate the cold-fusion result during the following months. A scientific consensus eventually emerged that the original researchers had not demonstrated cold fusion.

This is sometimes narrated as a story of scientific failure.

It is more interesting as a story of scientific self-correction.

An extraordinary result creates an extraordinary scientific opportunity

Imagine that the original experiment had simply been published and accepted.

Cold fusion would have entered textbooks.

Research programmes would have been constructed around it.

Funding would have followed.

Technologies might have been proposed.

Instead, the scientific community did something fundamental: it asked whether the phenomenon could be made to happen again.

That simple question protects science from becoming a museum of impressive claims.

A surprising observation is not automatically rejected. But neither is it granted diplomatic immunity because it appeared in a prestigious publication.

The result has to survive contact with other laboratories.

Replication rests on a philosophical assumption

The report makes an unusually deep point here.

Science assumes that nature follows rules that are sufficiently consistent to be discovered.

A chemical reaction cannot work only because the universe happened to be feeling cooperative in Utah on one afternoon in 1989.

Of course, contexts matter. Biological effects can depend on age, temperature, genotype or environment. Human behaviour can change across societies. But those contextual dependencies themselves must ultimately form part of the explanation.

A failed replication therefore asks a productive question:

What changed?

Perhaps the original claim was wrong.

Perhaps the replication was poorly performed.

Perhaps an unknown variable differed.

Perhaps the phenomenon exists only under a narrow set of conditions.

All four possibilities are scientifically interesting.

The lesson of cold fusion

Science does not become reliable because scientists are unusually resistant to error.

Scientists are human.

Science becomes reliable because important claims can be challenged by people who were not involved in making them.

Cold fusion did not demonstrate that science was broken.

The rapid effort to test it demonstrated precisely why science can recover from dramatic claims that do not hold up.

Sometimes the scientific machine makes progress by discovering something spectacular.

Sometimes it makes progress by discovering that something spectacular probably did not happen.

Both are discoveries.

Stop Asking Whether One Paper Is “True”

Scientific news is often written as a sequence of reversals.

Coffee prevents disease.

Coffee causes disease.

A gene increases risk.

No, it doesn't.

A diet extends life.

Another study disagrees.

To anyone following science through headlines, the whole enterprise can begin to look like a weather vane in a thunderstorm.

The National Academies report offers a better way to think.

Confidence in scientific knowledge generally comes from multiple channels of evidence, not from one paper defeating another in intellectual combat. The report argues that the aim is to understand the overall inference from a body of studies rather than obsessively asking whether one study exactly replicated another.

This is where systematic reviews, meta-analysis, converging evidence and independent methods become powerful.

A single study can be wrong.

A replication can also be wrong.

But a network of evidence gathered using different populations, methods, laboratories and assumptions can gradually constrain the plausible explanations.

The report's discussion of public communication is especially important here.

Scientific journalism tends to favour surprising single studies and apparent contradictions over slow accumulation of evidence. It also notes that uncertainty present in scientific papers frequently disappears on the journey into news coverage.

Scientists are not innocent passengers in this process.

Press releases and researchers themselves can exaggerate implications.

The report therefore recommends caution both from scientists describing their results and journalists communicating them.

For readers, there is a remarkably useful rule:

Do not rebuild your worldview because of one paper.

And do not destroy an established body of scientific knowledge because one contrary paper appears.

The report explicitly advises against serious decisions based on one promising study, while also warning that one contrary study should not be treated as refuting conclusions supported by multiple lines of evidence.

Science is strongest when evidence accumulates.

A single study is a thread.

Knowledge is the fabric.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.