Thirty Teams, One Dataset, Many Answers

Give scientists the same data and the same question.

Surely they will obtain the same answer?

Not necessarily.

One of the most revealing examples in the National Academies report concerns an experiment in which nearly 30 independent research teams were given the same dataset.

They were asked whether football referees were more likely to give red cards to darker-skinned players and whether this relationship was associated with measures of bias in the referees' countries.

The scientists did not simply converge on one result.

Sixty-nine percent of the teams reported a statistically significant positive relationship. Thirty-one percent did not.

Same data.

Same broad question.

Different answers.

Welcome to the analytical multiverse. 🧭

Data do not analyse themselves

Between a raw dataset and a final graph lies a chain of decisions.

Should certain observations be excluded?

How should variables be grouped?

Which statistical model should be fitted?

Should particular variables be transformed?

Which covariates belong in the model?

How should missing data be treated?

Which observations count as outliers?

These decisions are often perfectly defensible individually.

But different combinations can lead researchers down different analytical paths.

The football study found that not only statistical modelling but also decisions about grouping the data contributed to the variation in outcomes.

There may be hundreds of invisible methodological decisions

The report makes the broader point that a scientific experiment can contain hundreds or thousands of choices, many so routine that researchers may not even think to report them.

Was a reagent shaken or stirred?

At exactly what temperature was it stored?

How were survey questions worded?

In what order were they asked?

Which samples were discarded?

Which software parameters were left at defaults?

Such details can sometimes turn out to matter enormously.

This has a direct parallel in genomics.

Consider how many choices occur between a FASTQ file and a biological conclusion:

reference genome,

read trimming,

mapping quality threshold,

duplicate handling,

variant filtering,

annotation database,

missingness threshold,

population filtering,

statistical model.

A pipeline may look deterministic once it is written.

The process by which that pipeline was chosen often was not.

Transparency means showing the forks in the road

This is why reproducibility cannot mean merely uploading one final script.

Ideally, a scientific report should make clear how data were prepared, which analyses were planned, which emerged later, and how uncertainty was assessed.

The goal is not to eliminate researcher judgment.

That would be impossible.

The goal is to make consequential judgment visible.

The bigger lesson

When two researchers disagree, we instinctively ask:

Which one made the mistake?

Sometimes neither did.

They may have made different reasonable analytical decisions.

That possibility changes how we interpret scientific disagreement.

Instead of treating variation among analysts as noise that should be hidden, perhaps we should sometimes measure it.

Because if a scientific conclusion changes dramatically when one reasonable analytical choice is swapped for another, that fragility is itself part of the result.

Cold Fusion and the Experiment That Science Refused to Believe

In March 1989, electrochemists Martin Fleischmann and Stanley Pons announced an extraordinary result.

They claimed that they had achieved fusion of hydrogen into helium at room temperature.

If correct, “cold fusion” would have challenged conventional expectations about the conditions necessary for nuclear fusion. The finding was sufficiently extraordinary that laboratories around the world immediately began trying to reproduce it.

And that is where this story becomes more important than cold fusion itself.

The National Academies report uses the episode to explain one of the most basic assumptions underlying all scientific inquiry:

“Nature is not capricious.”

If the laws governing an experiment are real, another laboratory reproducing the relevant conditions should not discover that physics has suddenly decided to behave differently.

Dozens of laboratories attempted to replicate the cold-fusion result during the following months. A scientific consensus eventually emerged that the original researchers had not demonstrated cold fusion.

This is sometimes narrated as a story of scientific failure.

It is more interesting as a story of scientific self-correction.

An extraordinary result creates an extraordinary scientific opportunity

Imagine that the original experiment had simply been published and accepted.

Cold fusion would have entered textbooks.

Research programmes would have been constructed around it.

Funding would have followed.

Technologies might have been proposed.

Instead, the scientific community did something fundamental: it asked whether the phenomenon could be made to happen again.

That simple question protects science from becoming a museum of impressive claims.

A surprising observation is not automatically rejected. But neither is it granted diplomatic immunity because it appeared in a prestigious publication.

The result has to survive contact with other laboratories.

Replication rests on a philosophical assumption

The report makes an unusually deep point here.

Science assumes that nature follows rules that are sufficiently consistent to be discovered.

A chemical reaction cannot work only because the universe happened to be feeling cooperative in Utah on one afternoon in 1989.

Of course, contexts matter. Biological effects can depend on age, temperature, genotype or environment. Human behaviour can change across societies. But those contextual dependencies themselves must ultimately form part of the explanation.

A failed replication therefore asks a productive question:

What changed?

Perhaps the original claim was wrong.

Perhaps the replication was poorly performed.

Perhaps an unknown variable differed.

Perhaps the phenomenon exists only under a narrow set of conditions.

All four possibilities are scientifically interesting.

The lesson of cold fusion

Science does not become reliable because scientists are unusually resistant to error.

Scientists are human.

Science becomes reliable because important claims can be challenged by people who were not involved in making them.

Cold fusion did not demonstrate that science was broken.

The rapid effort to test it demonstrated precisely why science can recover from dramatic claims that do not hold up.

Sometimes the scientific machine makes progress by discovering something spectacular.

Sometimes it makes progress by discovering that something spectacular probably did not happen.

Both are discoveries.

Stop Asking Whether One Paper Is “True”

Scientific news is often written as a sequence of reversals.

Coffee prevents disease.

Coffee causes disease.

A gene increases risk.

No, it doesn't.

A diet extends life.

Another study disagrees.

To anyone following science through headlines, the whole enterprise can begin to look like a weather vane in a thunderstorm.

The National Academies report offers a better way to think.

Confidence in scientific knowledge generally comes from multiple channels of evidence, not from one paper defeating another in intellectual combat. The report argues that the aim is to understand the overall inference from a body of studies rather than obsessively asking whether one study exactly replicated another.

This is where systematic reviews, meta-analysis, converging evidence and independent methods become powerful.

A single study can be wrong.

A replication can also be wrong.

But a network of evidence gathered using different populations, methods, laboratories and assumptions can gradually constrain the plausible explanations.

The report's discussion of public communication is especially important here.

Scientific journalism tends to favour surprising single studies and apparent contradictions over slow accumulation of evidence. It also notes that uncertainty present in scientific papers frequently disappears on the journey into news coverage.

Scientists are not innocent passengers in this process.

Press releases and researchers themselves can exaggerate implications.

The report therefore recommends caution both from scientists describing their results and journalists communicating them.

For readers, there is a remarkably useful rule:

Do not rebuild your worldview because of one paper.

And do not destroy an established body of scientific knowledge because one contrary paper appears.

The report explicitly advises against serious decisions based on one promising study, while also warning that one contrary study should not be treated as refuting conclusions supported by multiple lines of evidence.

Science is strongest when evidence accumulates.

A single study is a thread.

Knowledge is the fabric.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

Can We Redesign the Scientific Paper Before We Know the Answer?

Imagine submitting a scientific paper before doing the experiment.

The reviewers evaluate the question.

They examine the experimental design.

They inspect the planned statistical analysis.

And the journal agrees in advance to publish the study if it is conducted properly, regardless of whether the final result is positive, negative or inconvenient.

That is the logic behind registered reports.

The National Academies report discusses this approach as one possible response to publication bias. By judging the importance of the question and the quality of the methodology before the result is known, registered reports attempt to separate scientific quality from whether the experiment produces an exciting outcome.

The attraction is obvious.

Researchers have less incentive to reshape an analysis around the result.

Null findings remain publishable.

Unexpected outcomes cannot quietly transform a confirmatory study into an apparently preplanned discovery.

But the report does not present preregistration as a magic wand.

Critics worry about administrative burden and about discouraging exploratory research. And a badly designed study remains badly designed even if its bad design was meticulously preregistered.

This reveals a broader lesson running throughout the report.

No single reform can manufacture reliable science.

Open data do not guarantee good analysis.

Shared code can faithfully reproduce a programming mistake.

Preregistration cannot rescue a poor hypothesis.

Replication cannot automatically tell us which of two conflicting studies is correct.

Peer review cannot detect every error.

Each reform addresses a different failure mode.

The goal should therefore be an ecosystem of practices: better documentation, transparent analysis, appropriate statistics, publication of negative results, data and code sharing where possible, replication, training and incentives that reward careful work.

Scientific reliability is unlikely to arrive through one magnificent reform.

It is more likely to be assembled from many rather unglamorous habits done consistently well.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

The P-Value Was Never Supposed to Be a Truth Machine

Few numbers in science have acquired as much psychological power as 0.05.

A result at p = 0.049 can feel publishable.

A result at p = 0.051 can feel disappointing.

Yet nature does not know that somebody drew a line at 0.05.

The National Academies report devotes considerable attention to the misuse of statistical significance.

A p-value does not tell us the probability that a scientific hypothesis is true.

It is calculated under an assumption about the null hypothesis and describes the compatibility of the observed data with that framework.

Reducing an entire study to significant versus non-significant therefore throws away enormous amounts of information.

What was the effect size?

How precise was the estimate?

How plausible was the hypothesis beforehand?

Were multiple hypotheses tested?

How large was the sample?

What assumptions were built into the model?

What uncertainty accompanies the result?

The report notes that judging replication simply by asking whether both studies cross the same statistical-significance threshold can be unreliable. Researchers should instead examine distributions, estimated effects, variability and uncertainty.

There is another fascinating tension.

A scientific field could achieve excellent replication rates by studying only extremely predictable phenomena.

But science also exists to discover things we do not already know.

The report points out that bold, low-prior-probability hypotheses are naturally less likely to replicate than safe hypotheses. Yet some transformative discoveries necessarily begin as improbable ideas.

The goal, therefore, cannot be to engineer science so that every experiment succeeds.

A perfectly predictable scientific enterprise would be reliable.

It might also be spectacularly boring.

Good science needs both exploration and confirmation.

The challenge is knowing which mode we are in.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

The File Drawer Has a Distorting Mirror

Scientific publishing loves a dramatic result.

A treatment works.

A gene has an effect.

A relationship is statistically significant.

A surprising mechanism appears.

But what happens to the experiment in which nothing happens?

Historically, quite often, very little.

The National Academies report identifies “publication bias” as an important source of distortion in the scientific literature.

Researchers and journals face incentives favouring novel, statistically significant and apparently important results. Null findings, replications and inconclusive results can be much harder to publish.

That creates a peculiar problem.

Suppose 20 laboratories test a weak effect.

Eighteen find nothing interesting and the studies disappear into drawers or abandoned folders.

Two obtain positive results and publish them.

A reader searching the literature encounters two papers supporting the effect and zero contradicting it.

The literature is technically real.

But it is not representative.

The report provides a powerful example involving tamoxifen. Twenty-two of 23 individual trials discussed in a systematic review did not individually cross the conventional p < 0.05 threshold, yet the cumulative analysis demonstrated an important treatment effect.

That example reveals the opposite danger too.

Scientific understanding cannot be reduced to counting individually significant papers.

The entire distribution of evidence matters.

This is why negative results are scientifically valuable.

They constrain hypotheses.

They improve estimates.

They prevent future researchers from unknowingly repeating dead ends.

And they make systematic reviews more representative of reality.

The report discusses emerging attempts to publish null results, replications and registered reports, while acknowledging that changing journals alone will not solve the problem if academic incentives continue rewarding novelty more strongly than reliability.

The scientific literature should be a record of what we learned.

Not merely a highlight reel.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

A Failed Replication Is Not Automatically Bad Science

Few phrases sound more fatal today than:

The result failed to replicate.

It sounds like a scientific obituary.

The National Academies report strongly resists that simplistic interpretation.

Some failures to replicate result from poor design, inadequate reporting, mistakes or misconduct.

But others can be scientifically productive.

Complex natural systems contain variation. A relationship may depend on environmental conditions, population structure, developmental stage, cultural context, measurement precision or variables nobody thought to record.

The report therefore distinguishes potentially useful sources of non-replicability from avoidable ones.

Imagine that Study A reports an effect.

Study B finds none.

That discrepancy could eventually reveal that the effect occurs only within a certain age range.

Study C might then demonstrate the mechanism responsible for that age dependence.

What initially looked like failure has uncovered biology.

This is why the report argues that the overall frequency of non-replication is a poor measure of the health of science.

The important task is diagnosing why results differed.

The report identifies characteristics that make some phenomena intrinsically more difficult to replicate: complex systems, large numbers of interacting variables, noisy measurements, poor control of variables, changing conditions across space or time, and imperfect measurement of the underlying phenomenon.

There is therefore an important distinction between:

productive disagreement, which exposes previously unknown complexity,

and

avoidable disagreement, which results from weak design, inadequate reporting, mistakes or bias.

Science benefits from the first.

It should work aggressively to reduce the second.

The difference cannot be discovered by attaching the label failed replication and walking away.

Investigation begins precisely where the disagreement starts.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.