Why Your Methods Section Is No Longer Enough

Modern science has quietly undergone a technological revolution.

Sequencing machines generate terabytes of data.

Telescopes produce enormous digital archives.

Climate scientists run simulations on supercomputers.

Machine-learning models may contain billions of parameters.

And researchers routinely construct analytical pipelines containing dozens of programs, scripts, transformations and intermediate datasets.

Yet we often try to describe all of this in a few paragraphs labelled Methods.

The National Academies report argues that this model is increasingly inadequate. When scientific results arise from complex computations and large datasets, a conventional methods section may not contain enough information to reproduce the analysis.

A computational experiment contains many hidden ingredients.

Which version of the program was used?

Which reference genome?

Which parameters?

Which libraries?

Which operating system?

How were missing values treated?

Which samples were discarded?

Was a random seed fixed?

What intermediate filtering occurred?

The final figure may look wonderfully clean while sitting on top of a computational jungle.

One passage cited in the report makes the point sharply, referring to “all details of the computation” being made available.

That means reproducibility increasingly requires more than prose.

It may require data.

Code.

Exact software versions.

Dependency information.

Workflow descriptions.

Intermediate files.

Parameter configurations.

And sometimes the computational environment itself.

This is particularly important because the report notes that many scientists use software without formal training in practices such as version control, workflow management and software testing.

The transformation required is conceptual.

Code is not merely machinery operating behind the research.

In computational science, the code is part of the scientific method.

And if an important part of the method disappears when a student's laptop dies, then part of the scientific record disappears with it.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.