A Failed Replication Is Not Automatically Bad Science

Few phrases sound more fatal today than:

The result failed to replicate.

It sounds like a scientific obituary.

The National Academies report strongly resists that simplistic interpretation.

Some failures to replicate result from poor design, inadequate reporting, mistakes or misconduct.

But others can be scientifically productive.

Complex natural systems contain variation. A relationship may depend on environmental conditions, population structure, developmental stage, cultural context, measurement precision or variables nobody thought to record.

The report therefore distinguishes potentially useful sources of non-replicability from avoidable ones.

Imagine that Study A reports an effect.

Study B finds none.

That discrepancy could eventually reveal that the effect occurs only within a certain age range.

Study C might then demonstrate the mechanism responsible for that age dependence.

What initially looked like failure has uncovered biology.

This is why the report argues that the overall frequency of non-replication is a poor measure of the health of science.

The important task is diagnosing why results differed.

The report identifies characteristics that make some phenomena intrinsically more difficult to replicate: complex systems, large numbers of interacting variables, noisy measurements, poor control of variables, changing conditions across space or time, and imperfect measurement of the underlying phenomenon.

There is therefore an important distinction between:

productive disagreement, which exposes previously unknown complexity,

and

avoidable disagreement, which results from weak design, inadequate reporting, mistakes or bias.

Science benefits from the first.

It should work aggressively to reduce the second.

The difference cannot be discovered by attaching the label failed replication and walking away.

Investigation begins precisely where the disagreement starts.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.