Thirty Teams, One Dataset, Many Answers

Give scientists the same data and the same question.

Surely they will obtain the same answer?

Not necessarily.

One of the most revealing examples in the National Academies report concerns an experiment in which nearly 30 independent research teams were given the same dataset.

They were asked whether football referees were more likely to give red cards to darker-skinned players and whether this relationship was associated with measures of bias in the referees' countries.

The scientists did not simply converge on one result.

Sixty-nine percent of the teams reported a statistically significant positive relationship. Thirty-one percent did not.

Same data.

Same broad question.

Different answers.

Welcome to the analytical multiverse. 🧭

Data do not analyse themselves

Between a raw dataset and a final graph lies a chain of decisions.

Should certain observations be excluded?

How should variables be grouped?

Which statistical model should be fitted?

Should particular variables be transformed?

Which covariates belong in the model?

How should missing data be treated?

Which observations count as outliers?

These decisions are often perfectly defensible individually.

But different combinations can lead researchers down different analytical paths.

The football study found that not only statistical modelling but also decisions about grouping the data contributed to the variation in outcomes.

There may be hundreds of invisible methodological decisions

The report makes the broader point that a scientific experiment can contain hundreds or thousands of choices, many so routine that researchers may not even think to report them.

Was a reagent shaken or stirred?

At exactly what temperature was it stored?

How were survey questions worded?

In what order were they asked?

Which samples were discarded?

Which software parameters were left at defaults?

Such details can sometimes turn out to matter enormously.

This has a direct parallel in genomics.

Consider how many choices occur between a FASTQ file and a biological conclusion:

reference genome,

read trimming,

mapping quality threshold,

duplicate handling,

variant filtering,

annotation database,

missingness threshold,

population filtering,

statistical model.

A pipeline may look deterministic once it is written.

The process by which that pipeline was chosen often was not.

Transparency means showing the forks in the road

This is why reproducibility cannot mean merely uploading one final script.

Ideally, a scientific report should make clear how data were prepared, which analyses were planned, which emerged later, and how uncertainty was assessed.

The goal is not to eliminate researcher judgment.

That would be impossible.

The goal is to make consequential judgment visible.

The bigger lesson

When two researchers disagree, we instinctively ask:

Which one made the mistake?

Sometimes neither did.

They may have made different reasonable analytical decisions.

That possibility changes how we interpret scientific disagreement.

Instead of treating variation among analysts as noise that should be hidden, perhaps we should sometimes measure it.

Because if a scientific conclusion changes dramatically when one reasonable analytical choice is swapped for another, that fragility is itself part of the result.