Why Two Competent Scientists Can Analyse the Same Data and Disagree

There is a comforting fantasy that data speak for themselves.

They do not.

Researchers make decisions before the data ever reach the final graph.

Which observations should be excluded?

How should missing values be handled?

How should categories be defined?

Which statistical model is appropriate?

Should an outlier remain?

Should variables be transformed?

Where should a threshold be placed?

The report describes a remarkable study in which almost 30 independent research teams received the same raw dataset and the same research question concerning red cards in football.

The outcomes differed substantially.

Sixty-nine percent of teams reported a significant positive relationship, while 31 percent did not. Differences in analytical approaches and data-grouping choices contributed to the divergent conclusions.

Nobody had secretly swapped the dataset.

The analytical pathways diverged.

This exposes an underappreciated feature of modern research: a dataset may contain a branching tree of defensible analytical choices.

The report similarly notes that scientific experiments contain hundreds or thousands of seemingly minor decisions. Something as mundane as reagent storage temperature, mixing procedure, survey wording or data-cleaning choices may alter outcomes.

Some decisions matter.

Others do not.

The difficult part is that we may not know which are consequential beforehand.

This is one reason transparency matters so much.

A researcher does not merely need to tell readers what result emerged. They need to reveal enough of the analytical route that somebody else can understand how the result emerged.

That idea becomes particularly important in genomics and other data-rich sciences.

A pipeline is not just a neutral pipe carrying truth from raw reads to final figure.

Every filter, threshold, reference database and parameter is a junction.

Reproducible science leaves signposts at those junctions.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.