Reproducibility and Replicability Are Not the Same Thing

Scientists frequently use reproducibility and replicability as though they mean the same thing.

The National Academies report deliberately separates them.

That distinction is extremely useful, particularly now that so much research depends on computation.

Suppose I publish a genomic analysis.

I provide the sequencing data, scripts, software versions and parameter settings. You download exactly those materials, run exactly my analysis and obtain the same results.

That is reproducibility.

Now imagine that you collect a completely new biological dataset to test the same scientific question. You analyse those new data and obtain results consistent with mine.

That is replicability.

The report defines reproducibility as obtaining consistent results using the same input data and computational procedures, whereas replication involves another study addressing the same scientific question using newly obtained data.

There is a third concept too:

generalizability.

A finding may reproduce perfectly and replicate in another experiment, but does it hold in another population, species, laboratory, climate, time period or experimental context?

That is a different question.

These distinctions matter because each tests a different layer of scientific reliability.

Reproducibility asks:

Can I reconstruct what you actually did?

Replicability asks:

Does the phenomenon appear again when we gather new evidence?

Generalizability asks:

How far beyond the original circumstances does the finding extend?

And crucially, they need not succeed or fail together.

A computational result can be perfectly reproducible yet scientifically wrong. If a programming error exists in the original code, rerunning the same erroneous code can faithfully reproduce the same erroneous answer.

Conversely, a meticulously performed study may fail to replicate because the biological or social system being studied contains previously unidentified variation.

So asking simply, Did it replicate? can be far too crude.

The better questions are:

What exactly was tested?

What changed?

How large was the uncertainty?

And what does the disagreement teach us?

Those questions lead to better science than a binary stamp reading PASS or FAIL.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.