When Science News Loses the Error Bars

A scientific paper usually comes with uncertainty attached.

Confidence intervals.

Error bars.

Alternative explanations.

Methodological limitations.

Cautious language.

Then the finding begins travelling.

Paper → press release → headline → social media.

And somewhere along the journey, the uncertainty can fall off the vehicle.

Headlines prefer discoveries to accumulated evidence

The National Academies report notes that journalism is event-driven.

A surprising new study is news.

The fact that 37 earlier studies broadly agree with one another is less likely to become tomorrow's headline.

Scientific news therefore tends to emphasize single studies, contradictions and apparent breakthroughs rather than slow accumulation of evidence.

This naturally amplifies instability.

“Study confirms existing understanding with slightly improved estimate” is scientifically useful.

It is not headline catnip.

Exaggeration can start before journalists arrive

Researchers sometimes blame journalists for overselling science.

The report complicates that story.

One study comparing scientific papers, university press releases and subsequent news coverage found that more than one-third of press releases contained exaggerated advice, causal claims or inappropriate extrapolation.

When the press release exaggerated, subsequent news stories were far more likely to exaggerate too.

So the distortion pipeline may begin surprisingly close to the laboratory.

Uncertainty is part of the result

Imagine a study reports:

Effect estimate = 20%

95% confidence interval = 2% to 38%

A headline may compress that into:

“Scientists discover 20% effect.”

A great deal of information has disappeared.

The uncertainty is not decorative statistical parsley.

It tells us how much confidence to place in the estimate.

The public notices the problem

The report cites survey data in which 73% of Americans said that a major problem with scientific news concerned how reporters covered research, while 43% described the media as too quick to report findings that might not hold up.

Scientists face a similar concern: surveys found substantial numbers of respondents believed researchers sometimes overstate the implications of their own work.

The remedy is not to make science communication dull.

It is to make uncertainty intelligible.

“Promising but preliminary” is not a weakness.

“Consistent with” is not cowardice.

“Evidence suggests” is not an admission that scientists know nothing.

Good science communication preserves the shape of the evidence even after removing the equations.

Keep the story.

But keep the error bars too.

Genetics and Psychology Took Different Roads Toward Reliable Science

Talk about reproducibility often collapses science into one giant category.

But the replication problem faced by a psychologist studying human behaviour is not identical to the problem faced by a geneticist conducting a genome-wide association study.

Different fields have different sources of noise, different instruments and different traditions.

The National Academies uses genetics and psychology to make exactly this point.

Genetics built a large-scale replication machinery

Genome-wide association studies scan enormous numbers of genetic variants for associations with disease or other traits.

That creates obvious statistical dangers.

With huge numbers of tests, chance associations are inevitable unless stringent safeguards are used.

The GWAS community responded by developing norms around:

large sample sizes,

international consortia,

quality control,

phenotype and genotype harmonization,

control of population stratification,

stringent multiple-testing thresholds,

independent replication,

and meta-analysis across cohorts.

This was not merely a statistical reform.

It was sociological.

Reliability increasingly depended on collaboration, infrastructure, data sharing and agreement about analytical standards.

Psychology studies a very different kind of system

Human behaviour is extraordinarily context-sensitive.

Researchers cannot control people with the same precision with which a physicist controls a laboratory instrument.

Psychology has consequently been at the centre of public discussion about a supposed “replication crisis.”

But the report is deliberately cautious.

There is no consensus that the field should be characterized as being in crisis.

Some researchers point to low power, p-hacking and inadequate distinctions between prior and post hoc hypotheses.

Others warn that framing psychology as broken may exaggerate the evidence, discourage risky research and overemphasize false positives at the cost of missing genuine discoveries.

Replication rates cannot simply be compared like cricket scores

The report reviewed replication efforts across fields ranging from genetics and chemistry to psychology and economics and found reported rates ranging from below one in five to above three in four.

But it explicitly warns that these numbers are not directly comparable because studies were selected differently and success was defined differently.

That is a crucial lesson.

The question should not be:

Which discipline has the highest replication percentage?

It should be:

What kind of system is being studied?

How much can its variables be controlled?

How noisy are the measurements?

What counts as successful replication?

And what practices would meaningfully improve reliability in that particular field?

Reproducibility has universal principles.

Its implementation is local.

What If a Journal Accepted Your Experiment Before You Knew the Result?

Scientific publishing usually happens backwards.

Researchers conduct a study.

They discover the result.

Then journals decide whether the study is interesting enough to publish.

That arrangement creates an obvious temptation.

Interesting outcomes become valuable.

Null outcomes become disappointing.

What if we reversed the chronology?

Enter the registered report.

Review the science before seeing the answer

In the model described by the National Academies, researchers prepare the parts of the manuscript that can be written before data collection:

the scientific rationale,

research question,

methods,

and planned analyses.

The journal peer-reviews that proposal.

If it is accepted, the researchers conduct the study and later submit the results.

The publication decision is therefore less dependent on whether the final answer looks exciting.

This attacks publication bias at its root.

The question becomes: Was this worth testing?

Conventional publishing can accidentally encourage:

“Interesting result, therefore interesting paper.”

Registered reports attempt to replace this with:

“Important question + rigorous design, therefore worthwhile study.”

That distinction is subtle but profound.

If the hypothesis is scientifically important, then discovering that it is unsupported is also scientifically useful.

But preregistration is not a halo

The National Academies carefully avoids treating preregistration as a universal cure.

Critics worry that mandatory preregistration could favour easily testable hypotheses and discourage open-ended exploration.

And the report makes an especially important point:

Poor ideas and weak methods can be preregistered too.

Preregistration records a plan.

It does not certify that the plan is intelligent.

Exploration and confirmation need different rules

Many important discoveries arise because scientists notice something unexpected.

Exploratory science needs room to wander.

Confirmatory science needs stronger protection against rewriting the story after seeing the outcome.

The real value of preregistration may therefore lie not in banning exploration, but in distinguishing the two.

There is nothing wrong with saying:

“We discovered this pattern unexpectedly.”

The problem begins when an unexpected pattern is written as though it had been predicted from the beginning.

Registered reports are one attempt to keep those narratives honest.

What Happened When a Journal Actually Tried to Run the Code

Scientific peer review normally asks whether a paper appears scientifically sound.

But what happens if a journal asks a more mechanical question:

Does the submitted code actually reproduce the submitted results?

The American Journal of Political Science introduced external reproducibility checks for accepted manuscripts.

The results are striking.

Among the first 116 papers examined, only 8 reproduced successfully on the first attempt.

In another description in the report, 108 of 116 required authors to provide additional information or make some change.

That is roughly 93%.

These were papers that had already survived scholarly review

This is what makes the case interesting.

The papers were not random student assignments.

They were publication-ready research.

Yet when someone attempted to reconstruct the calculations, missing information or other issues commonly surfaced.

This reveals a gap between two questions:

Does the analysis seem plausible?

and

Can someone actually execute it?

Peer review often answers the first.

Reproducibility auditing attempts the second.

It is not cheap

The report estimates that the external reproducibility check added about 52 days to the publication process and required approximately eight person-hours on average.

That is not trivial.

Someone must obtain the code.

Understand the data.

Reconstruct the environment.

Run the analysis.

Compare outputs.

Communicate discrepancies.

Wait for revisions.

Then try again.

But the audit changes behaviour before it happens

There is also a subtler benefit.

If researchers know that another person will actually run their scripts, they have a powerful incentive to organize code, document dependencies and make analytical steps explicit.

The audit does not simply catch problems.

Its existence may alter research behaviour upstream.

Think of it as the scientific equivalent of knowing someone else will eventually open your kitchen cupboards. Suddenly, everything gets labelled.

Should every journal do this?

Perhaps not immediately.

The costs could be substantial, especially for complex analyses.

But the experiment raises an uncomfortable question.

If a major conclusion depends entirely on computation, should verifying that the computation can actually be reconstructed remain optional?

Peer review asks whether we believe the recipe.

A reproducibility audit tries cooking the dish.

Those are not the same test.

Your Paper May Survive. Your Software Might Not.

Imagine finding a fascinating computational paper from 2012.

The methods sound clear.

The authors even shared their code.

Excellent.

You click the repository.

Gone.

Perhaps the repository disappeared.

Perhaps a dependency no longer exists.

Perhaps the code requires an obsolete version of Python.

Perhaps the dataset survives in a format that modern software can no longer read.

The article remains pristine in the journal archive.

Its computational ecosystem has become archaeological material.

Computational research has dependencies

Modern analyses are rarely just code + data.

They can depend on:

operating systems,

library versions,

reference databases,

compiler versions,

hardware,

workflow order,

parameter files,

and external software.

The National Academies therefore recommends documenting not only data and analytical methods but also the computational environment, including software dependencies.

Version control records the scientific history

The report discusses version-control systems such as Git because the precise version of code matters.

If collaborators modify the same program over months, which version generated Figure 4?

Version control can preserve that history automatically.

The report also describes workflow-management systems that record the path from input data through transformations to final output.

That is important because computational provenance answers a deceptively simple question:

Where did this number come from?

A GitHub link is not necessarily an archive

The report makes another important distinction.

Commercial hosting platforms can change priorities or disappear.

A researcher who deposits code only in a mutable online repository risks losing the exact scientific object associated with publication.

Archival repositories and persistent identifiers such as DOIs provide stronger preservation.

And sometimes even preservation is not enough.

The report notes that old datasets may eventually exist in formats that modern machines cannot read. Yale's library, for example, had begun exploring emulation infrastructure to provide access to obsolete software.

The scientific record now includes software

We once thought of scientific archiving largely in terms of papers.

Then datasets.

Now software environments must increasingly be treated as research objects too.

The future historian of science may therefore need more than PDFs.

They may need containers, virtual machines, archived libraries, reference-database snapshots and executable workflows.

A modern scientific result is sometimes less like a paragraph and more like a small ecosystem.

If we want the result to survive, we have to preserve more than its photograph.

The File Drawer: Science's Invisible Literature

Imagine conducting an experiment and finding nothing.

No statistically significant difference.

No dramatic effect.

No exciting mechanism.

You write up the result and send it to a journal.

Would it receive the same enthusiasm as a surprising positive finding?

Historically, often not.

This creates what has become known as the file-drawer problem: studies with null, negative or inconclusive results quietly disappear while positive findings travel into the published literature.

The National Academies report states the consequence plainly: publication bias can produce a literature that “does not reflect the full range of evidence.”

Imagine 20 experiments

Suppose 20 laboratories independently test an effect that does not actually exist.

By chance, one or two may produce p < 0.05.

If the 18 null results remain unpublished while the two positive findings appear in journals, anyone conducting a literature search receives a distorted picture.

The published literature appears to contain two successful demonstrations.

The invisible literature contains the rest of the story.

This creates a strange situation when another laboratory later publishes a null result.

It may be described as “failing to replicate” the positive papers.

Yet the National Academies points out that this supposed failure might actually replicate numerous negative experiments that were never published.

The problem is not necessarily the new experiment.

The problem may be the window through which we are looking at the evidence.

Funnel plots reveal the missing half of the landscape

One method for detecting possible publication bias is the funnel plot.

Studies are plotted according to estimated effect size and precision.

If all results are represented, the scatter should have a broadly expected pattern.

If small studies showing weak or null effects mysteriously vanish from the published literature, the pattern can become asymmetric.

The report's Figure 5-3 illustrates exactly how selective publication can distort what an evidence base looks like.

Why publishing “nothing happened” matters

A good null result can:

prevent duplication of failed approaches,

constrain theories,

improve meta-analysis,

reveal boundary conditions,

and provide a more realistic estimate of an effect.

The report therefore discusses efforts to create outlets for negative, null and replication studies, as well as preprint repositories that allow results to circulate even if conventional journals do not publish them.

Science does not only need spectacular discoveries.

It also needs a reliable record of all the places researchers looked and found less than they expected.

Sometimes “nothing happened” is exactly the missing piece.

Twenty-Two “Negative” Trials and a Treatment That Worked

Here is a statistical puzzle.

Twenty-three clinical trials examine a treatment.

Twenty-two fail to achieve the conventional threshold of p < 0.05.

Would you conclude that the treatment does not work?

That conclusion would have been badly wrong in the example discussed by the National Academies.

The treatment was tamoxifen, used in breast cancer.

When researchers examined the trials collectively, the accumulated evidence indicated a substantial benefit, corresponding to about a 16% reduction in the odds of death, with the uncertainty reported as ±3%.

The example exposes one of the strangest habits in scientific reasoning:

treating each study as an isolated referendum in which p < 0.05 means YES and p ≥ 0.05 means NO.

A non-significant study is not necessarily evidence of no effect

A study can fail to reach statistical significance for many reasons.

The true effect may be modest.

The sample may be small.

Measurements may be noisy.

The estimate may point in the expected direction but remain uncertain.

If many studies produce similar but individually imprecise estimates, the collective evidence can become convincing.

This is why scientific understanding often comes from research synthesis, not from asking which individual paper crossed a numerical finish line.

The tamoxifen example flips the usual replication anxiety

Much discussion about reproducibility worries that positive studies may turn out to be false positives.

That is important.

But the opposite problem exists too.

An obsession with statistical significance can cause useful evidence to be dismissed simply because individual studies lack sufficient power.

The report therefore cautions against judging replication solely by whether two studies independently cross the same significance threshold.

Imagine:

Study A: p = 0.049.

Study B: p = 0.051.

One is declared “significant.”

The other is declared “not significant.”

Yet the actual estimates may be almost indistinguishable.

Nature did not suddenly change at the third decimal place.

Look at the evidence, not merely the label

A better comparison considers:

effect sizes,

uncertainty,

distributions,

sample sizes,

consistency of direction,

and the full body of available evidence.

The report repeatedly argues that scientific confidence should emerge from cumulative evidence rather than a duel between individual studies.

The tamoxifen story therefore contains a beautiful statistical warning.

Twenty-two studies can each fail to shout loudly enough on their own.

Together, they may form a chorus.

Thirty Teams, One Dataset, Many Answers

Give scientists the same data and the same question.

Surely they will obtain the same answer?

Not necessarily.

One of the most revealing examples in the National Academies report concerns an experiment in which nearly 30 independent research teams were given the same dataset.

They were asked whether football referees were more likely to give red cards to darker-skinned players and whether this relationship was associated with measures of bias in the referees' countries.

The scientists did not simply converge on one result.

Sixty-nine percent of the teams reported a statistically significant positive relationship. Thirty-one percent did not.

Same data.

Same broad question.

Different answers.

Welcome to the analytical multiverse. 🧭

Data do not analyse themselves

Between a raw dataset and a final graph lies a chain of decisions.

Should certain observations be excluded?

How should variables be grouped?

Which statistical model should be fitted?

Should particular variables be transformed?

Which covariates belong in the model?

How should missing data be treated?

Which observations count as outliers?

These decisions are often perfectly defensible individually.

But different combinations can lead researchers down different analytical paths.

The football study found that not only statistical modelling but also decisions about grouping the data contributed to the variation in outcomes.

There may be hundreds of invisible methodological decisions

The report makes the broader point that a scientific experiment can contain hundreds or thousands of choices, many so routine that researchers may not even think to report them.

Was a reagent shaken or stirred?

At exactly what temperature was it stored?

How were survey questions worded?

In what order were they asked?

Which samples were discarded?

Which software parameters were left at defaults?

Such details can sometimes turn out to matter enormously.

This has a direct parallel in genomics.

Consider how many choices occur between a FASTQ file and a biological conclusion:

reference genome,

read trimming,

mapping quality threshold,

duplicate handling,

variant filtering,

annotation database,

missingness threshold,

population filtering,

statistical model.

A pipeline may look deterministic once it is written.

The process by which that pipeline was chosen often was not.

Transparency means showing the forks in the road

This is why reproducibility cannot mean merely uploading one final script.

Ideally, a scientific report should make clear how data were prepared, which analyses were planned, which emerged later, and how uncertainty was assessed.

The goal is not to eliminate researcher judgment.

That would be impossible.

The goal is to make consequential judgment visible.

The bigger lesson

When two researchers disagree, we instinctively ask:

Which one made the mistake?

Sometimes neither did.

They may have made different reasonable analytical decisions.

That possibility changes how we interpret scientific disagreement.

Instead of treating variation among analysts as noise that should be hidden, perhaps we should sometimes measure it.

Because if a scientific conclusion changes dramatically when one reasonable analytical choice is swapped for another, that fragility is itself part of the result.

Cold Fusion and the Experiment That Science Refused to Believe

In March 1989, electrochemists Martin Fleischmann and Stanley Pons announced an extraordinary result.

They claimed that they had achieved fusion of hydrogen into helium at room temperature.

If correct, “cold fusion” would have challenged conventional expectations about the conditions necessary for nuclear fusion. The finding was sufficiently extraordinary that laboratories around the world immediately began trying to reproduce it.

And that is where this story becomes more important than cold fusion itself.

The National Academies report uses the episode to explain one of the most basic assumptions underlying all scientific inquiry:

“Nature is not capricious.”

If the laws governing an experiment are real, another laboratory reproducing the relevant conditions should not discover that physics has suddenly decided to behave differently.

Dozens of laboratories attempted to replicate the cold-fusion result during the following months. A scientific consensus eventually emerged that the original researchers had not demonstrated cold fusion.

This is sometimes narrated as a story of scientific failure.

It is more interesting as a story of scientific self-correction.

An extraordinary result creates an extraordinary scientific opportunity

Imagine that the original experiment had simply been published and accepted.

Cold fusion would have entered textbooks.

Research programmes would have been constructed around it.

Funding would have followed.

Technologies might have been proposed.

Instead, the scientific community did something fundamental: it asked whether the phenomenon could be made to happen again.

That simple question protects science from becoming a museum of impressive claims.

A surprising observation is not automatically rejected. But neither is it granted diplomatic immunity because it appeared in a prestigious publication.

The result has to survive contact with other laboratories.

Replication rests on a philosophical assumption

The report makes an unusually deep point here.

Science assumes that nature follows rules that are sufficiently consistent to be discovered.

A chemical reaction cannot work only because the universe happened to be feeling cooperative in Utah on one afternoon in 1989.

Of course, contexts matter. Biological effects can depend on age, temperature, genotype or environment. Human behaviour can change across societies. But those contextual dependencies themselves must ultimately form part of the explanation.

A failed replication therefore asks a productive question:

What changed?

Perhaps the original claim was wrong.

Perhaps the replication was poorly performed.

Perhaps an unknown variable differed.

Perhaps the phenomenon exists only under a narrow set of conditions.

All four possibilities are scientifically interesting.

The lesson of cold fusion

Science does not become reliable because scientists are unusually resistant to error.

Scientists are human.

Science becomes reliable because important claims can be challenged by people who were not involved in making them.

Cold fusion did not demonstrate that science was broken.

The rapid effort to test it demonstrated precisely why science can recover from dramatic claims that do not hold up.

Sometimes the scientific machine makes progress by discovering something spectacular.

Sometimes it makes progress by discovering that something spectacular probably did not happen.

Both are discoveries.

Stop Asking Whether One Paper Is “True”

Scientific news is often written as a sequence of reversals.

Coffee prevents disease.

Coffee causes disease.

A gene increases risk.

No, it doesn't.

A diet extends life.

Another study disagrees.

To anyone following science through headlines, the whole enterprise can begin to look like a weather vane in a thunderstorm.

The National Academies report offers a better way to think.

Confidence in scientific knowledge generally comes from multiple channels of evidence, not from one paper defeating another in intellectual combat. The report argues that the aim is to understand the overall inference from a body of studies rather than obsessively asking whether one study exactly replicated another.

This is where systematic reviews, meta-analysis, converging evidence and independent methods become powerful.

A single study can be wrong.

A replication can also be wrong.

But a network of evidence gathered using different populations, methods, laboratories and assumptions can gradually constrain the plausible explanations.

The report's discussion of public communication is especially important here.

Scientific journalism tends to favour surprising single studies and apparent contradictions over slow accumulation of evidence. It also notes that uncertainty present in scientific papers frequently disappears on the journey into news coverage.

Scientists are not innocent passengers in this process.

Press releases and researchers themselves can exaggerate implications.

The report therefore recommends caution both from scientists describing their results and journalists communicating them.

For readers, there is a remarkably useful rule:

Do not rebuild your worldview because of one paper.

And do not destroy an established body of scientific knowledge because one contrary paper appears.

The report explicitly advises against serious decisions based on one promising study, while also warning that one contrary study should not be treated as refuting conclusions supported by multiple lines of evidence.

Science is strongest when evidence accumulates.

A single study is a thread.

Knowledge is the fabric.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

Can We Redesign the Scientific Paper Before We Know the Answer?

Imagine submitting a scientific paper before doing the experiment.

The reviewers evaluate the question.

They examine the experimental design.

They inspect the planned statistical analysis.

And the journal agrees in advance to publish the study if it is conducted properly, regardless of whether the final result is positive, negative or inconvenient.

That is the logic behind registered reports.

The National Academies report discusses this approach as one possible response to publication bias. By judging the importance of the question and the quality of the methodology before the result is known, registered reports attempt to separate scientific quality from whether the experiment produces an exciting outcome.

The attraction is obvious.

Researchers have less incentive to reshape an analysis around the result.

Null findings remain publishable.

Unexpected outcomes cannot quietly transform a confirmatory study into an apparently preplanned discovery.

But the report does not present preregistration as a magic wand.

Critics worry about administrative burden and about discouraging exploratory research. And a badly designed study remains badly designed even if its bad design was meticulously preregistered.

This reveals a broader lesson running throughout the report.

No single reform can manufacture reliable science.

Open data do not guarantee good analysis.

Shared code can faithfully reproduce a programming mistake.

Preregistration cannot rescue a poor hypothesis.

Replication cannot automatically tell us which of two conflicting studies is correct.

Peer review cannot detect every error.

Each reform addresses a different failure mode.

The goal should therefore be an ecosystem of practices: better documentation, transparent analysis, appropriate statistics, publication of negative results, data and code sharing where possible, replication, training and incentives that reward careful work.

Scientific reliability is unlikely to arrive through one magnificent reform.

It is more likely to be assembled from many rather unglamorous habits done consistently well.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

The P-Value Was Never Supposed to Be a Truth Machine

Few numbers in science have acquired as much psychological power as 0.05.

A result at p = 0.049 can feel publishable.

A result at p = 0.051 can feel disappointing.

Yet nature does not know that somebody drew a line at 0.05.

The National Academies report devotes considerable attention to the misuse of statistical significance.

A p-value does not tell us the probability that a scientific hypothesis is true.

It is calculated under an assumption about the null hypothesis and describes the compatibility of the observed data with that framework.

Reducing an entire study to significant versus non-significant therefore throws away enormous amounts of information.

What was the effect size?

How precise was the estimate?

How plausible was the hypothesis beforehand?

Were multiple hypotheses tested?

How large was the sample?

What assumptions were built into the model?

What uncertainty accompanies the result?

The report notes that judging replication simply by asking whether both studies cross the same statistical-significance threshold can be unreliable. Researchers should instead examine distributions, estimated effects, variability and uncertainty.

There is another fascinating tension.

A scientific field could achieve excellent replication rates by studying only extremely predictable phenomena.

But science also exists to discover things we do not already know.

The report points out that bold, low-prior-probability hypotheses are naturally less likely to replicate than safe hypotheses. Yet some transformative discoveries necessarily begin as improbable ideas.

The goal, therefore, cannot be to engineer science so that every experiment succeeds.

A perfectly predictable scientific enterprise would be reliable.

It might also be spectacularly boring.

Good science needs both exploration and confirmation.

The challenge is knowing which mode we are in.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

The File Drawer Has a Distorting Mirror

Scientific publishing loves a dramatic result.

A treatment works.

A gene has an effect.

A relationship is statistically significant.

A surprising mechanism appears.

But what happens to the experiment in which nothing happens?

Historically, quite often, very little.

The National Academies report identifies “publication bias” as an important source of distortion in the scientific literature.

Researchers and journals face incentives favouring novel, statistically significant and apparently important results. Null findings, replications and inconclusive results can be much harder to publish.

That creates a peculiar problem.

Suppose 20 laboratories test a weak effect.

Eighteen find nothing interesting and the studies disappear into drawers or abandoned folders.

Two obtain positive results and publish them.

A reader searching the literature encounters two papers supporting the effect and zero contradicting it.

The literature is technically real.

But it is not representative.

The report provides a powerful example involving tamoxifen. Twenty-two of 23 individual trials discussed in a systematic review did not individually cross the conventional p < 0.05 threshold, yet the cumulative analysis demonstrated an important treatment effect.

That example reveals the opposite danger too.

Scientific understanding cannot be reduced to counting individually significant papers.

The entire distribution of evidence matters.

This is why negative results are scientifically valuable.

They constrain hypotheses.

They improve estimates.

They prevent future researchers from unknowingly repeating dead ends.

And they make systematic reviews more representative of reality.

The report discusses emerging attempts to publish null results, replications and registered reports, while acknowledging that changing journals alone will not solve the problem if academic incentives continue rewarding novelty more strongly than reliability.

The scientific literature should be a record of what we learned.

Not merely a highlight reel.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

A Failed Replication Is Not Automatically Bad Science

Few phrases sound more fatal today than:

The result failed to replicate.

It sounds like a scientific obituary.

The National Academies report strongly resists that simplistic interpretation.

Some failures to replicate result from poor design, inadequate reporting, mistakes or misconduct.

But others can be scientifically productive.

Complex natural systems contain variation. A relationship may depend on environmental conditions, population structure, developmental stage, cultural context, measurement precision or variables nobody thought to record.

The report therefore distinguishes potentially useful sources of non-replicability from avoidable ones.

Imagine that Study A reports an effect.

Study B finds none.

That discrepancy could eventually reveal that the effect occurs only within a certain age range.

Study C might then demonstrate the mechanism responsible for that age dependence.

What initially looked like failure has uncovered biology.

This is why the report argues that the overall frequency of non-replication is a poor measure of the health of science.

The important task is diagnosing why results differed.

The report identifies characteristics that make some phenomena intrinsically more difficult to replicate: complex systems, large numbers of interacting variables, noisy measurements, poor control of variables, changing conditions across space or time, and imperfect measurement of the underlying phenomenon.

There is therefore an important distinction between:

productive disagreement, which exposes previously unknown complexity,

and

avoidable disagreement, which results from weak design, inadequate reporting, mistakes or bias.

Science benefits from the first.

It should work aggressively to reduce the second.

The difference cannot be discovered by attaching the label failed replication and walking away.

Investigation begins precisely where the disagreement starts.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

Why Two Competent Scientists Can Analyse the Same Data and Disagree

There is a comforting fantasy that data speak for themselves.

They do not.

Researchers make decisions before the data ever reach the final graph.

Which observations should be excluded?

How should missing values be handled?

How should categories be defined?

Which statistical model is appropriate?

Should an outlier remain?

Should variables be transformed?

Where should a threshold be placed?

The report describes a remarkable study in which almost 30 independent research teams received the same raw dataset and the same research question concerning red cards in football.

The outcomes differed substantially.

Sixty-nine percent of teams reported a significant positive relationship, while 31 percent did not. Differences in analytical approaches and data-grouping choices contributed to the divergent conclusions.

Nobody had secretly swapped the dataset.

The analytical pathways diverged.

This exposes an underappreciated feature of modern research: a dataset may contain a branching tree of defensible analytical choices.

The report similarly notes that scientific experiments contain hundreds or thousands of seemingly minor decisions. Something as mundane as reagent storage temperature, mixing procedure, survey wording or data-cleaning choices may alter outcomes.

Some decisions matter.

Others do not.

The difficult part is that we may not know which are consequential beforehand.

This is one reason transparency matters so much.

A researcher does not merely need to tell readers what result emerged. They need to reveal enough of the analytical route that somebody else can understand how the result emerged.

That idea becomes particularly important in genomics and other data-rich sciences.

A pipeline is not just a neutral pipe carrying truth from raw reads to final figure.

Every filter, threshold, reference database and parameter is a junction.

Reproducible science leaves signposts at those junctions.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.

Can Chromosomes Create New Species? How Genome Rearrangements Shape Evolution

From microscopic chromosome changes to reproductive isolation, a century of research reveals that evolution is not only about changes in genes, but also about how genomes are organized.

When we think about evolution, we often imagine small genetic mutations accumulating over thousands or millions of generations. These mutations alter proteins, change observable traits, and sometimes help organisms adapt to new environments. Eventually, populations become sufficiently different to form new species.

But what if evolution can also proceed through much larger changes? What if the rearrangement of entire chromosomes could alter how genes interact, affect fertility, and even help establish reproductive barriers between populations?

This is the question explored by I. Yu. Bakloushinskaya in the 2016 review “Chromosomal Rearrangements, Genome Reorganization, and Speciation,” published in Biology Bulletin.

The article brings together more than a century of discoveries in cytogenetics, evolutionary biology, and genome organization. Its central argument is that chromosomes are not passive containers of genes. Their structure, arrangement, spatial organization, and patterns of inheritance can influence evolutionary processes, sometimes contributing directly to speciation.

The review also emphasizes an important qualification: chromosomal differences do not automatically create species, and demonstrating that a rearrangement initiated speciation is often difficult.

1. Beyond genes: Why chromosome organization matters

Every organism inherits genetic information arranged into chromosomes. Yet the significance of chromosomes extends beyond the DNA sequences they contain.

Two organisms can possess largely similar collections of genes while differing considerably in chromosome number, chromosome shape, or the relative positions of genetic regions.

Such differences arise through chromosomal rearrangements, including inversions, translocations, fusions, and fissions.

An inversion reverses the orientation of a chromosomal segment. A translocation relocates genetic material between chromosomal regions or chromosomes. A chromosome fusion joins formerly separate chromosomes, while a fission divides one chromosome into two.

These structural changes need not alter the total DNA content dramatically. Nevertheless, they can reorganize linkage relationships, modify recombination, affect gene expression, and interfere with chromosome pairing during meiosis.

Bakloushinskaya highlights a longstanding debate: are chromosomal rearrangements merely consequences of evolutionary divergence, or can they help initiate the formation of new species?

The distinction is fundamental.

Under one interpretation, populations first diverge through conventional genetic processes, and their chromosomes subsequently accumulate differences. Under the alternative, chromosomal restructuring itself can help establish reproductive isolation, enabling further evolutionary divergence.

The review argues that the second pathway is biologically possible and supported by certain natural systems.

2. A historical journey from chromosomes to evolutionary theory

The article begins by tracing how the study of chromosomes transformed biological thinking.

During the nineteenth century, chromosomes were recognized primarily as structures visible during cell division. Their role in heredity became clearer through the work of researchers including Sutton, Stevens, Wilson, Morgan, and their contemporaries.

The chromosomal theory of inheritance established that genes are carried by chromosomes and transmitted through the processes of cell division and reproduction.

The concept of the karyotype, developed in the early twentieth century, added another dimension. Instead of examining individual genes alone, researchers could characterize organisms by the number, size, shape, and organization of their chromosomes.

Chromosome sets became useful tools for distinguishing species, identifying hybrids, and investigating evolutionary relationships.

A major early demonstration came from plant hybridization studies. The artificial hybrid Raphanobrassica, produced through intergeneric hybridization followed by chromosome doubling, helped establish that changes in chromosome complements could contribute to the origin of new reproductive lineages.

The implications were especially clear for plants, where polyploidy can establish reproductive differences relatively rapidly.

In animals, however, the relationship between chromosome evolution and speciation proved more complicated.

The twentieth century therefore witnessed the development of several competing theories, seeking to explain how chromosome changes might arise, persist, and ultimately influence the formation of species.

3. Chromosomal speciation: Can a structural mutation start evolutionary divergence?

One of the review's central themes is chromosomal speciation, the hypothesis that chromosome rearrangements can contribute to the establishment of reproductive isolation.

Traditional population genetics emphasized changes in allele frequencies, natural selection, migration, genetic drift, and geographic separation. Chromosomal rearrangements were often treated as secondary events.

But several evolutionary biologists proposed a more active role for chromosome restructuring.

Wallace's work in the 1950s emphasized inversions and their potential consequences when geographically differentiated populations come into secondary contact. Mayr's ideas about changes in the genetic environment of isolated populations provided another framework for understanding rapid evolutionary divergence.

The review also discusses Vorontsov's distinction between gradualistic speciation and a more abrupt pathway initiated by changes in the chromosome complement.

In a gradualistic scenario, populations become geographically or ecologically separated, accumulate genetic differences, and eventually develop reproductive barriers.

In a chromosome-initiated scenario, structural changes can precede or help establish reproductive isolation, after which additional genetic and phenotypic divergence accumulates.

These are not necessarily mutually exclusive alternatives.

Chromosome restructuring may interact with geographic isolation, ecological divergence, demographic history, and conventional genetic changes.

The difficult question is determining causality: did the chromosome rearrangement help create the reproductive barrier, or did it accumulate after reproductive isolation had already developed?

4. Robertsonian fusions and the remarkable role of chromosome arms

One of the most compelling mechanisms discussed in the review involves Robertsonian, or centric, fusions.

These rearrangements typically join two acrocentric chromosomes, whose centromeres lie near one end, to form a metacentric chromosome.

An organism may therefore undergo a reduction in chromosome number without losing much of its major gene-containing sequence.

A simple fusion need not immediately cause severe reproductive problems. In some heterozygotes, the chromosomes can pair and segregate sufficiently well for fertility to remain relatively high.

The evolutionary consequences become more dramatic when independent populations acquire different fusions.

Consider three ancestral chromosomes, A, B, and C.

One population develops a fusion between A and B. Another develops a fusion between A and C.

Each new arrangement may function effectively within its own population. However, hybrids between the two populations inherit chromosomes that share only one homologous arm in common.

This condition is known as monobrachial homology.

During meiosis, the chromosomes may form complex pairing configurations rather than ordinary bivalents. These configurations can interfere with proper segregation, generate unbalanced gametes, and reduce hybrid fertility.

The result can be reproductive isolation even when each chromosomal arrangement was relatively harmless within its original population.

Bakloushinskaya discusses examples involving house mice, common shrews, mole voles, bats, rats, and rock-wallabies.

The broader evolutionary lesson is striking: two chromosome arrangements that are individually compatible with reproduction can become incompatible when brought together in hybrids.

This offers a plausible mechanism through which chromosome evolution contributes to speciation.

Importantly, the review also discusses cases in which hybrid fertility is retained despite substantial chromosomal differences. Such examples demonstrate that the consequences of rearrangements depend on chromosome structure, meiotic behavior, and genetic background.

5. Why chromosomal evolution may be nonrandom

Chromosomal rearrangements are not distributed uniformly across evolutionary lineages or genomic regions.

Why should certain rearrangements repeatedly occur in particular groups while others remain rare?

The review revisits the concept of karyotypic orthoselection, associated with M. J. D. White.

This concept addresses the tendency for particular chromosome configurations or rearrangement types to recur or become established in certain evolutionary groups.

Several explanations are considered.

Chromosome size and shape may influence which rearrangements are mechanically possible. The distribution of heterochromatin, satellite DNA, and ribosomal DNA may make particular regions more susceptible to structural changes. The locations of meiotic exchanges may affect which rearrangements can arise or persist.

Another important factor is the three-dimensional organization of chromosomes inside the nucleus.

In this interpretation, chromosome evolution is constrained by the physical and functional architecture of the genome.

Mutations may arise through stochastic processes, but the probability that a given rearrangement occurs, survives, or becomes fixed need not be uniform.

This provides a bridge between cytogenetics and modern genome biology: chromosome structure can influence evolutionary outcomes without requiring every rearrangement to be directly adaptive.

6. The nucleus is not a bag of chromosomes

One of the most forward-looking sections of the review concerns the spatial organization of the genome.

During interphase, chromosomes occupy distinct but interacting regions called chromosome territories.

Their arrangement is not completely random. Chromosome size, gene density, transcriptional activity, and cell type can influence their positions within the nucleus.

Bakloushinskaya discusses evidence that gene-rich chromosomal regions may preferentially occupy more central nuclear positions in certain cell types, while other regions are found closer to the nuclear periphery.

These patterns matter because genes are regulated within a three-dimensional nuclear environment.

Chromosomal rearrangements can change the spatial relationships among genes and regulatory sequences. A fusion may bring previously separated regions into closer proximity, while a translocation may relocate a gene into a different chromatin environment.

Such changes could influence transcription, replication timing, or recombination.

The article therefore encourages readers to move beyond the idea of chromosomes as one-dimensional strings of genes.

Their arrangement inside the nucleus may itself be evolutionarily consequential.

The author connects this perspective with evolutionary breakpoint regions, genomic locations at which chromosome rearrangements are inferred to have occurred during evolution.

These regions are not uniformly distributed. Their relationship with chromatin organization suggests that nuclear architecture may help influence the genomic locations of structural change.

The review presents this as an important research direction rather than a universal explanation for all evolutionary breakpoints.

7. Centromeres: Small chromosome regions with enormous evolutionary consequences

The centromere is essential for chromosome segregation. It provides the chromosomal site at which the kinetochore assembles, allowing the spindle apparatus to move chromosomes during mitosis and meiosis.

Traditionally, centromeres might appear to be stable structural features.

However, their evolutionary history is remarkably dynamic.

The review emphasizes that there is no single universal DNA sequence specifying centromere identity in all organisms. Centromere function depends on a combination of sequence context, chromatin organization, and epigenetic mechanisms.

A particularly intriguing phenomenon is the emergence of evolutionarily new centromeres, or neocentromeres.

These occur when centromere activity becomes established at a new genomic location.

Such changes can alter chromosome morphology and contribute to differences between species, sometimes without an obvious rearrangement of the surrounding gene order.

Bakloushinskaya discusses neocentromere evolution in mammals, including primates and mole voles.

The implication is that chromosome evolution does not depend exclusively on DNA breakage and reunion. Changes in the functional identity of chromosomal regions can also contribute to karyotype evolution.

This adds another layer of complexity to speciation research: differences in chromosome function may matter even when their gene content remains conserved.

8. Heterochromatin: From supposedly inert DNA to an evolutionary participant

Another major theme involves heterochromatin.

Historically, heterochromatic regions were often characterized as transcriptionally inactive and dominated by repetitive sequences. They were frequently considered less important than the gene-rich euchromatic portions of chromosomes.

The review challenges that simplistic view.

Heterochromatin contributes to chromosome structure and stability, while differences in its organization may influence interactions between chromosomes and the processes of cell division.

Evidence from Drosophila is discussed in connection with hybrid incompatibility, illustrating how species-specific heterochromatic differences can interfere with normal chromosome behavior.

The article also examines chromosomal races of the Ural field mouse, Sylvaemus uralensis.

These populations differ in their pericentromeric heterochromatin. Cytogenetic analyses indicate that variation in these regions is associated with differences in repetitive DNA content.

Such observations suggest that evolutionary divergence can involve changes in highly repetitive parts of the genome, not only conventional protein-coding genes.

The central message is that genomic regions once dismissed as relatively unimportant can influence chromosome organization and potentially reproductive compatibility.

However, the author acknowledges that the role of heterochromatin variation in speciation remains incompletely understood.

9. Polyploidy and the duplication of entire genomes

The review next turns to one of the most dramatic forms of genome reorganization: polyploidy.

Polyploid organisms possess additional complete sets of chromosomes.

Whole-genome duplication has played a major role in evolutionary history, particularly in plants and several vertebrate lineages.

The article discusses ancient genome duplication events in vertebrates and additional duplication histories in certain fishes.

Polyploidy can provide extra genetic material, allowing duplicated genes to acquire new functions, divide ancestral functions, or undergo loss.

Yet doubling an entire genome also introduces problems.

It changes gene dosage, complicates chromosome pairing, and may disrupt sex determination or other regulatory systems.

The author discusses unusual mammals with large chromosome complements, including the red viscacha rat, Tympanoctomys barrerae, and the golden viscacha rat, Pipanacoctomys aureus, whose genome sizes and chromosome characteristics have generated debate about polyploid evolution.

These examples illustrate that genome duplication in animals is a complex phenomenon requiring evidence from DNA content, chromosome organization, and meiotic behavior.

Polyploidy is thus both an important evolutionary mechanism and a challenge for maintaining a functional reproductive system.

10. Transposable elements and noncoding RNA: Additional forces shaping chromosomes

The paper also considers mobile genetic elements and noncoding RNAs.

Transposable elements are DNA sequences capable of changing their positions or producing additional copies within genomes.

Their activity can generate mutations, promote recombination between repetitive sequences, and contribute to structural rearrangements.

Building on Barbara McClintock's work, the review discusses the possibility that environmental or genetic stress can influence transposable element activity.

Hybrid dysgenesis in certain Drosophila groups provides an example in which mobile elements, genome instability, and reproductive abnormalities are connected.

The review also considers noncoding RNA as a potential participant in chromosome regulation and rearrangement.

These mechanisms broaden the picture of chromosome evolution.

Instead of viewing structural changes only as occasional accidental chromosome breaks, the article points to interactions among repetitive DNA, transposable elements, RNA-mediated regulation, chromatin structure, and cellular stress.

Nevertheless, it does not establish that transposable elements or noncoding RNAs initiate most speciation events. Their possible roles are presented as components of a complex evolutionary process.

11. How do chromosome rearrangements become established in populations?

A chromosome rearrangement cannot influence species evolution unless it survives and is transmitted across generations.

This raises a fundamental population-genetic problem.

Many rearrangements can reduce fertility in heterozygotes. If so, why are they not immediately eliminated by natural selection?

Bakloushinskaya explores several mechanisms that can facilitate their persistence.

Meiotic drive

Under ordinary Mendelian inheritance, the two alternatives of a genetic variant are expected to have equal transmission probabilities.

Meiotic drive occurs when transmission becomes biased, allowing one variant to be inherited more frequently than expected.

The review discusses how chromosomal variants may benefit from preferential segregation into functional gametes, particularly under circumstances involving asymmetric female meiosis.

Even a rearrangement with some negative fitness consequences might spread if its transmission advantage is sufficiently strong.

Genetic drift

In small populations, random changes in variant frequencies can have strong evolutionary effects.

A chromosomal rearrangement may therefore become established without providing an adaptive advantage.

Inbreeding and population subdivision

Small, partially isolated groups may experience elevated inbreeding and stronger drift.

These conditions can sometimes facilitate the fixation of chromosome arrangements that would struggle to become common in a large, freely mixing population.

The review discusses the potential significance of small demes and restricted dispersal in chromosome evolution.

The overarching message is that chromosome rearrangements become evolutionarily consequential through interactions between their biological effects and the demographic structure of populations.

12. Are there special genes that cause speciation?

The review also addresses the possibility of identifying individual genes whose divergence contributes directly to reproductive isolation.

A central example is PRDM9, a gene involved in directing meiotic recombination in mammals.

PRDM9 influences the locations at which recombination-associated chromatin marks are established. Changes in its function can alter recombination patterns and contribute to meiotic abnormalities.

The author highlights studies connecting PRDM9 divergence with hybrid sterility in mice.

This example is particularly significant because it connects a specific molecular mechanism with chromosome behavior during meiosis.

However, the review resists reducing speciation to the action of one or a few special genes.

Instead, it emphasizes that chromosome organization, recombination, genomic conflict, gene regulation, and reproductive isolation are interconnected.

The evolutionary emergence of species may result from changes to an entire genetic system rather than from isolated mutations acting independently.

PRDM9 provides a strong example of a gene contributing to reproductive incompatibility in a particular system, not proof that one universal speciation gene exists.

13. Speciation with gene flow: Chromosomal barriers need not be absolute

An important subtlety in the article concerns the relationship between reproductive isolation and gene flow.

It is tempting to imagine speciation as a clean separation in which two populations stop exchanging genes entirely.

In reality, evolutionary divergence can be more complicated.

Some chromosome regions may resist introgression while others continue to move between populations through hybridization.

Consequently, gene flow does not automatically imply that two populations have failed to diverge.

The review discusses examples involving chromosome races of shrews, house mice, and wallabies, where reproductive barriers and genetic exchange may coexist.

This leads to a more flexible understanding of speciation.

Chromosomal rearrangements may reduce effective gene flow in particular genomic regions, facilitate the maintenance of divergent genetic combinations, or interact with other incompatibilities.

Complete hybrid sterility is only one possible outcome.

14. What does this mean for modern evolutionary genomics?

Although published in 2016, the article provides an important conceptual foundation for contemporary comparative genome research.

Whole-genome sequencing allows researchers to identify conserved syntenic blocks, ancestral chromosome arrangements, fusion and fission events, inversion boundaries, segmental deletions, and differences in gene repertoires.

But detecting a genomic difference is only the beginning.

Understanding its evolutionary significance requires determining when it arose, whether it was fixed before or after population divergence, and whether it influenced fertility, recombination, gene regulation, or adaptation.

The review suggests that several levels of genome organization should be considered together: the sequence of individual genes, their arrangement along chromosomes, the architecture of chromatin, and the spatial organization of chromosomes within nuclei.

This perspective is especially relevant when evolutionary studies identify nonrandom genomic patterns, such as concentrations of deletions near rearrangement boundaries or repeated losses of genes within particular chromosomal regions.

Such patterns may reflect structural constraints, recurrent mutation mechanisms, selection, drift, or combinations of these processes.

Importantly, the article does not demonstrate that genes near evolutionary breakpoint regions are necessarily more likely to be lost. Nor does it establish that chromosome rearrangements inevitably produce gene deletions.

Those relationships require direct comparative genomic evidence.

15. The larger message: Evolution can reorganize genetic systems

The deepest insight of Bakloushinskaya's review is that evolution operates on multiple organizational levels.

Individual DNA mutations matter, but so do chromosome number, chromosome structure, chromatin composition, nuclear positioning, and the mechanisms governing inheritance.

A chromosome fusion may change meiotic compatibility. An inversion may alter recombination. A translocation may change a gene's regulatory environment. A neocentromere may modify chromosome behavior. Differences in heterochromatin may contribute to reproductive incompatibility.

These processes can interact with natural selection, genetic drift, inbreeding, and genomic conflict.

None provides a universal explanation for speciation.

The author ultimately presents chromosomal speciation as one biologically credible pathway among several, while recognizing that proving its initiating role in particular evolutionary events remains challenging.

Conclusion: Sometimes evolution rewrites the genome's architecture

A genome is more than its list of genes.

Its chromosomes determine how genetic information is packaged, arranged, recombined, regulated, and inherited. Changes to that architecture can have evolutionary consequences extending far beyond the individual rearranged segment.

Chromosomal Rearrangements, Genome Reorganization, and Speciation reminds us that the emergence of species cannot always be understood simply by counting mutations or measuring differences in DNA sequence.

To understand how species arise, we must also ask how their genomes are organized, how those organizations change, and what happens when distinct chromosomal systems meet during reproduction.

The review's most enduring message is not that every chromosome rearrangement creates a new species. It is that the organization of the genome can itself be a participant in evolutionary change, rather than merely a record of it.

---

Article discussed

Bakloushinskaya, I. Yu. (2016). Chromosomal Rearrangements, Genome Reorganization, and Speciation. Biology Bulletin, 43(8). DOI: 10.1134/S1062359016080057.

This blog post summarizes and interprets the attached 2016 review. It is not an independent systematic review of subsequent research, and the scientific examples reflect the evidence and historical framing presented in that article.

Why Your Methods Section Is No Longer Enough

Modern science has quietly undergone a technological revolution.

Sequencing machines generate terabytes of data.

Telescopes produce enormous digital archives.

Climate scientists run simulations on supercomputers.

Machine-learning models may contain billions of parameters.

And researchers routinely construct analytical pipelines containing dozens of programs, scripts, transformations and intermediate datasets.

Yet we often try to describe all of this in a few paragraphs labelled Methods.

The National Academies report argues that this model is increasingly inadequate. When scientific results arise from complex computations and large datasets, a conventional methods section may not contain enough information to reproduce the analysis.

A computational experiment contains many hidden ingredients.

Which version of the program was used?

Which reference genome?

Which parameters?

Which libraries?

Which operating system?

How were missing values treated?

Which samples were discarded?

Was a random seed fixed?

What intermediate filtering occurred?

The final figure may look wonderfully clean while sitting on top of a computational jungle.

One passage cited in the report makes the point sharply, referring to “all details of the computation” being made available.

That means reproducibility increasingly requires more than prose.

It may require data.

Code.

Exact software versions.

Dependency information.

Workflow descriptions.

Intermediate files.

Parameter configurations.

And sometimes the computational environment itself.

This is particularly important because the report notes that many scientists use software without formal training in practices such as version control, workflow management and software testing.

The transformation required is conceptual.

Code is not merely machinery operating behind the research.

In computational science, the code is part of the scientific method.

And if an important part of the method disappears when a student's laptop dies, then part of the scientific record disappears with it.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.