Genetics and Psychology Took Different Roads Toward Reliable Science

Talk about reproducibility often collapses science into one giant category.

But the replication problem faced by a psychologist studying human behaviour is not identical to the problem faced by a geneticist conducting a genome-wide association study.

Different fields have different sources of noise, different instruments and different traditions.

The National Academies uses genetics and psychology to make exactly this point.

Genetics built a large-scale replication machinery

Genome-wide association studies scan enormous numbers of genetic variants for associations with disease or other traits.

That creates obvious statistical dangers.

With huge numbers of tests, chance associations are inevitable unless stringent safeguards are used.

The GWAS community responded by developing norms around:

large sample sizes,

international consortia,

quality control,

phenotype and genotype harmonization,

control of population stratification,

stringent multiple-testing thresholds,

independent replication,

and meta-analysis across cohorts.

This was not merely a statistical reform.

It was sociological.

Reliability increasingly depended on collaboration, infrastructure, data sharing and agreement about analytical standards.

Psychology studies a very different kind of system

Human behaviour is extraordinarily context-sensitive.

Researchers cannot control people with the same precision with which a physicist controls a laboratory instrument.

Psychology has consequently been at the centre of public discussion about a supposed “replication crisis.”

But the report is deliberately cautious.

There is no consensus that the field should be characterized as being in crisis.

Some researchers point to low power, p-hacking and inadequate distinctions between prior and post hoc hypotheses.

Others warn that framing psychology as broken may exaggerate the evidence, discourage risky research and overemphasize false positives at the cost of missing genuine discoveries.

Replication rates cannot simply be compared like cricket scores

The report reviewed replication efforts across fields ranging from genetics and chemistry to psychology and economics and found reported rates ranging from below one in five to above three in four.

But it explicitly warns that these numbers are not directly comparable because studies were selected differently and success was defined differently.

That is a crucial lesson.

The question should not be:

Which discipline has the highest replication percentage?

It should be:

What kind of system is being studied?

How much can its variables be controlled?

How noisy are the measurements?

What counts as successful replication?

And what practices would meaningfully improve reliability in that particular field?

Reproducibility has universal principles.

Its implementation is local.

What If a Journal Accepted Your Experiment Before You Knew the Result?

Scientific publishing usually happens backwards.

Researchers conduct a study.

They discover the result.

Then journals decide whether the study is interesting enough to publish.

That arrangement creates an obvious temptation.

Interesting outcomes become valuable.

Null outcomes become disappointing.

What if we reversed the chronology?

Enter the registered report.

Review the science before seeing the answer

In the model described by the National Academies, researchers prepare the parts of the manuscript that can be written before data collection:

the scientific rationale,

research question,

methods,

and planned analyses.

The journal peer-reviews that proposal.

If it is accepted, the researchers conduct the study and later submit the results.

The publication decision is therefore less dependent on whether the final answer looks exciting.

This attacks publication bias at its root.

The question becomes: Was this worth testing?

Conventional publishing can accidentally encourage:

“Interesting result, therefore interesting paper.”

Registered reports attempt to replace this with:

“Important question + rigorous design, therefore worthwhile study.”

That distinction is subtle but profound.

If the hypothesis is scientifically important, then discovering that it is unsupported is also scientifically useful.

But preregistration is not a halo

The National Academies carefully avoids treating preregistration as a universal cure.

Critics worry that mandatory preregistration could favour easily testable hypotheses and discourage open-ended exploration.

And the report makes an especially important point:

Poor ideas and weak methods can be preregistered too.

Preregistration records a plan.

It does not certify that the plan is intelligent.

Exploration and confirmation need different rules

Many important discoveries arise because scientists notice something unexpected.

Exploratory science needs room to wander.

Confirmatory science needs stronger protection against rewriting the story after seeing the outcome.

The real value of preregistration may therefore lie not in banning exploration, but in distinguishing the two.

There is nothing wrong with saying:

“We discovered this pattern unexpectedly.”

The problem begins when an unexpected pattern is written as though it had been predicted from the beginning.

Registered reports are one attempt to keep those narratives honest.

What Happened When a Journal Actually Tried to Run the Code

Scientific peer review normally asks whether a paper appears scientifically sound.

But what happens if a journal asks a more mechanical question:

Does the submitted code actually reproduce the submitted results?

The American Journal of Political Science introduced external reproducibility checks for accepted manuscripts.

The results are striking.

Among the first 116 papers examined, only 8 reproduced successfully on the first attempt.

In another description in the report, 108 of 116 required authors to provide additional information or make some change.

That is roughly 93%.

These were papers that had already survived scholarly review

This is what makes the case interesting.

The papers were not random student assignments.

They were publication-ready research.

Yet when someone attempted to reconstruct the calculations, missing information or other issues commonly surfaced.

This reveals a gap between two questions:

Does the analysis seem plausible?

and

Can someone actually execute it?

Peer review often answers the first.

Reproducibility auditing attempts the second.

It is not cheap

The report estimates that the external reproducibility check added about 52 days to the publication process and required approximately eight person-hours on average.

That is not trivial.

Someone must obtain the code.

Understand the data.

Reconstruct the environment.

Run the analysis.

Compare outputs.

Communicate discrepancies.

Wait for revisions.

Then try again.

But the audit changes behaviour before it happens

There is also a subtler benefit.

If researchers know that another person will actually run their scripts, they have a powerful incentive to organize code, document dependencies and make analytical steps explicit.

The audit does not simply catch problems.

Its existence may alter research behaviour upstream.

Think of it as the scientific equivalent of knowing someone else will eventually open your kitchen cupboards. Suddenly, everything gets labelled.

Should every journal do this?

Perhaps not immediately.

The costs could be substantial, especially for complex analyses.

But the experiment raises an uncomfortable question.

If a major conclusion depends entirely on computation, should verifying that the computation can actually be reconstructed remain optional?

Peer review asks whether we believe the recipe.

A reproducibility audit tries cooking the dish.

Those are not the same test.

Your Paper May Survive. Your Software Might Not.

Imagine finding a fascinating computational paper from 2012.

The methods sound clear.

The authors even shared their code.

Excellent.

You click the repository.

Gone.

Perhaps the repository disappeared.

Perhaps a dependency no longer exists.

Perhaps the code requires an obsolete version of Python.

Perhaps the dataset survives in a format that modern software can no longer read.

The article remains pristine in the journal archive.

Its computational ecosystem has become archaeological material.

Computational research has dependencies

Modern analyses are rarely just code + data.

They can depend on:

operating systems,

library versions,

reference databases,

compiler versions,

hardware,

workflow order,

parameter files,

and external software.

The National Academies therefore recommends documenting not only data and analytical methods but also the computational environment, including software dependencies.

Version control records the scientific history

The report discusses version-control systems such as Git because the precise version of code matters.

If collaborators modify the same program over months, which version generated Figure 4?

Version control can preserve that history automatically.

The report also describes workflow-management systems that record the path from input data through transformations to final output.

That is important because computational provenance answers a deceptively simple question:

Where did this number come from?

A GitHub link is not necessarily an archive

The report makes another important distinction.

Commercial hosting platforms can change priorities or disappear.

A researcher who deposits code only in a mutable online repository risks losing the exact scientific object associated with publication.

Archival repositories and persistent identifiers such as DOIs provide stronger preservation.

And sometimes even preservation is not enough.

The report notes that old datasets may eventually exist in formats that modern machines cannot read. Yale's library, for example, had begun exploring emulation infrastructure to provide access to obsolete software.

The scientific record now includes software

We once thought of scientific archiving largely in terms of papers.

Then datasets.

Now software environments must increasingly be treated as research objects too.

The future historian of science may therefore need more than PDFs.

They may need containers, virtual machines, archived libraries, reference-database snapshots and executable workflows.

A modern scientific result is sometimes less like a paragraph and more like a small ecosystem.

If we want the result to survive, we have to preserve more than its photograph.

The File Drawer: Science's Invisible Literature

Imagine conducting an experiment and finding nothing.

No statistically significant difference.

No dramatic effect.

No exciting mechanism.

You write up the result and send it to a journal.

Would it receive the same enthusiasm as a surprising positive finding?

Historically, often not.

This creates what has become known as the file-drawer problem: studies with null, negative or inconclusive results quietly disappear while positive findings travel into the published literature.

The National Academies report states the consequence plainly: publication bias can produce a literature that “does not reflect the full range of evidence.”

Imagine 20 experiments

Suppose 20 laboratories independently test an effect that does not actually exist.

By chance, one or two may produce p < 0.05.

If the 18 null results remain unpublished while the two positive findings appear in journals, anyone conducting a literature search receives a distorted picture.

The published literature appears to contain two successful demonstrations.

The invisible literature contains the rest of the story.

This creates a strange situation when another laboratory later publishes a null result.

It may be described as “failing to replicate” the positive papers.

Yet the National Academies points out that this supposed failure might actually replicate numerous negative experiments that were never published.

The problem is not necessarily the new experiment.

The problem may be the window through which we are looking at the evidence.

Funnel plots reveal the missing half of the landscape

One method for detecting possible publication bias is the funnel plot.

Studies are plotted according to estimated effect size and precision.

If all results are represented, the scatter should have a broadly expected pattern.

If small studies showing weak or null effects mysteriously vanish from the published literature, the pattern can become asymmetric.

The report's Figure 5-3 illustrates exactly how selective publication can distort what an evidence base looks like.

Why publishing “nothing happened” matters

A good null result can:

prevent duplication of failed approaches,

constrain theories,

improve meta-analysis,

reveal boundary conditions,

and provide a more realistic estimate of an effect.

The report therefore discusses efforts to create outlets for negative, null and replication studies, as well as preprint repositories that allow results to circulate even if conventional journals do not publish them.

Science does not only need spectacular discoveries.

It also needs a reliable record of all the places researchers looked and found less than they expected.

Sometimes “nothing happened” is exactly the missing piece.

Twenty-Two “Negative” Trials and a Treatment That Worked

Here is a statistical puzzle.

Twenty-three clinical trials examine a treatment.

Twenty-two fail to achieve the conventional threshold of p < 0.05.

Would you conclude that the treatment does not work?

That conclusion would have been badly wrong in the example discussed by the National Academies.

The treatment was tamoxifen, used in breast cancer.

When researchers examined the trials collectively, the accumulated evidence indicated a substantial benefit, corresponding to about a 16% reduction in the odds of death, with the uncertainty reported as ±3%.

The example exposes one of the strangest habits in scientific reasoning:

treating each study as an isolated referendum in which p < 0.05 means YES and p ≥ 0.05 means NO.

A non-significant study is not necessarily evidence of no effect

A study can fail to reach statistical significance for many reasons.

The true effect may be modest.

The sample may be small.

Measurements may be noisy.

The estimate may point in the expected direction but remain uncertain.

If many studies produce similar but individually imprecise estimates, the collective evidence can become convincing.

This is why scientific understanding often comes from research synthesis, not from asking which individual paper crossed a numerical finish line.

The tamoxifen example flips the usual replication anxiety

Much discussion about reproducibility worries that positive studies may turn out to be false positives.

That is important.

But the opposite problem exists too.

An obsession with statistical significance can cause useful evidence to be dismissed simply because individual studies lack sufficient power.

The report therefore cautions against judging replication solely by whether two studies independently cross the same significance threshold.

Imagine:

Study A: p = 0.049.

Study B: p = 0.051.

One is declared “significant.”

The other is declared “not significant.”

Yet the actual estimates may be almost indistinguishable.

Nature did not suddenly change at the third decimal place.

Look at the evidence, not merely the label

A better comparison considers:

effect sizes,

uncertainty,

distributions,

sample sizes,

consistency of direction,

and the full body of available evidence.

The report repeatedly argues that scientific confidence should emerge from cumulative evidence rather than a duel between individual studies.

The tamoxifen story therefore contains a beautiful statistical warning.

Twenty-two studies can each fail to shout loudly enough on their own.

Together, they may form a chorus.

Thirty Teams, One Dataset, Many Answers

Give scientists the same data and the same question.

Surely they will obtain the same answer?

Not necessarily.

One of the most revealing examples in the National Academies report concerns an experiment in which nearly 30 independent research teams were given the same dataset.

They were asked whether football referees were more likely to give red cards to darker-skinned players and whether this relationship was associated with measures of bias in the referees' countries.

The scientists did not simply converge on one result.

Sixty-nine percent of the teams reported a statistically significant positive relationship. Thirty-one percent did not.

Same data.

Same broad question.

Different answers.

Welcome to the analytical multiverse. 🧭

Data do not analyse themselves

Between a raw dataset and a final graph lies a chain of decisions.

Should certain observations be excluded?

How should variables be grouped?

Which statistical model should be fitted?

Should particular variables be transformed?

Which covariates belong in the model?

How should missing data be treated?

Which observations count as outliers?

These decisions are often perfectly defensible individually.

But different combinations can lead researchers down different analytical paths.

The football study found that not only statistical modelling but also decisions about grouping the data contributed to the variation in outcomes.

There may be hundreds of invisible methodological decisions

The report makes the broader point that a scientific experiment can contain hundreds or thousands of choices, many so routine that researchers may not even think to report them.

Was a reagent shaken or stirred?

At exactly what temperature was it stored?

How were survey questions worded?

In what order were they asked?

Which samples were discarded?

Which software parameters were left at defaults?

Such details can sometimes turn out to matter enormously.

This has a direct parallel in genomics.

Consider how many choices occur between a FASTQ file and a biological conclusion:

reference genome,

read trimming,

mapping quality threshold,

duplicate handling,

variant filtering,

annotation database,

missingness threshold,

population filtering,

statistical model.

A pipeline may look deterministic once it is written.

The process by which that pipeline was chosen often was not.

Transparency means showing the forks in the road

This is why reproducibility cannot mean merely uploading one final script.

Ideally, a scientific report should make clear how data were prepared, which analyses were planned, which emerged later, and how uncertainty was assessed.

The goal is not to eliminate researcher judgment.

That would be impossible.

The goal is to make consequential judgment visible.

The bigger lesson

When two researchers disagree, we instinctively ask:

Which one made the mistake?

Sometimes neither did.

They may have made different reasonable analytical decisions.

That possibility changes how we interpret scientific disagreement.

Instead of treating variation among analysts as noise that should be hidden, perhaps we should sometimes measure it.

Because if a scientific conclusion changes dramatically when one reasonable analytical choice is swapped for another, that fragility is itself part of the result.