Tuesday, November 3, 2026

The Anna Karenina Principle: Why Success Looks Similar but Failure Comes in Many Forms


“All happy families are alike; each unhappy family is unhappy in its own way.”
— Leo Tolstoy, Anna Karenina

Tolstoy was talking about families. But the underlying idea has turned out to be useful far beyond literature.

The Anna Karenina principle describes a peculiar feature of complex systems:

For something to succeed, many conditions may have to be satisfied simultaneously. But for it to fail, only one important condition may need to go wrong.

This simple asymmetry appears everywhere—from families and businesses to animal domestication, evolution, scientific experiments, and even learning.

Success requires many things to go right

Imagine that you want to keep an animal and eventually establish a domesticated population.

It is not enough for the animal simply to be useful.

It might need to:

  • eat food that humans can provide economically;

  • grow quickly enough to make raising it worthwhile;

  • reproduce reliably in captivity;

  • tolerate confinement;

  • have a manageable temperament;

  • not be excessively dangerous to humans;

  • live in social groups that humans can control;

  • have a breeding system compatible with captive management.

In other words, domestication is not controlled by a single characteristic.

It is more like a checklist:

Diet + growth + reproduction + temperament + social behavior + captivity tolerance + manageable breeding

If all of these conditions are favorable, domestication may be possible.

But suppose an animal fails badly on just one of them.

Perhaps it does not reproduce in captivity.

Or perhaps it becomes extremely aggressive when confined.

Or perhaps it has a diet that makes raising it prohibitively expensive.

That single problem may be sufficient to prevent domestication—even if everything else about the animal is favorable.

This is the Anna Karenina principle in action.

Why failures are much more diverse than successes

Consider two hypothetical species.

Species A fails because it cannot reproduce in captivity.

Species B fails because it is too aggressive.

Species C fails because it has an extremely specialized diet.

Species D fails because it panics when confined.

Species E fails because its social structure makes captive breeding difficult.

They have almost nothing in common except the fact that they failed.

By contrast, successful domesticated species tend to share a recognizable collection of characteristics.

This produces an interesting asymmetry:

There may be many different ways to fail, but relatively few ways to satisfy all the requirements for success.

This is why looking only at successful examples can sometimes be misleading.

The zebra problem

Jared Diamond famously used this idea when discussing why relatively few large mammals were domesticated.

Take the zebra.

At first glance, one might ask:

Why didn't Africans domesticate zebras in the same way that humans domesticated horses?

The obvious answer might seem to be that zebras are simply "not domesticable."

But the Anna Karenina principle encourages a more useful question:

Which particular requirements for domestication does the zebra fail to satisfy?

That shifts the problem from a vague statement about the animal to a series of testable biological characteristics.

Perhaps the animal has an unsuitable temperament. Perhaps its reproductive biology presents difficulties. Perhaps its social behavior is problematic. Perhaps several factors interact.

The important point is methodological:

Instead of looking for one magical characteristic that explains success, examine the collection of constraints that must simultaneously be overcome.

The principle extends to evolution

The same logic can be applied to evolutionary biology.

Suppose a lineage survives for millions of years.

Its continued existence may depend on many interacting features:

  • reproductive success;

  • sufficient genetic variation;

  • ability to obtain food;

  • resistance to pathogens;

  • tolerance of environmental conditions;

  • suitable developmental biology;

  • successful interactions with other organisms.

Failure of any sufficiently important component can contribute to extinction.

And extinction itself can therefore have many different explanations.

One species might disappear because of climatic change.

Another because of a pathogen.

Another because of competition.

Another because of a demographic bottleneck.

Another because several relatively small disadvantages accumulate.

There is therefore no contradiction in saying:

Successful lineages may share important characteristics, while extinct lineages can have very different histories.

It also applies to science

There is an interesting version of the Anna Karenina principle in scientific research.

A successful experiment often requires many things to work:

good experimental design + appropriate measurements + adequate controls + reliable samples + suitable statistical analysis + reproducibility + correct interpretation

A failure, however, can happen because any one of these breaks down.

A study may have an excellent hypothesis but poor measurements.

Another may have excellent measurements but inadequate controls.

Another may have a perfectly reproducible experiment but an incorrect interpretation.

This is particularly important because reproducibility does not automatically guarantee correctness.

An experiment can be performed exactly as described and produce the same result every time, while the explanation attached to that result is nevertheless wrong.

Thus, "the experiment worked" and "the scientific conclusion is correct" are separate questions.

The Anna Karenina principle helps us see why.

And it applies to people

Learning provides another familiar example.

Becoming very good at something can depend on many factors:

cognitive ability + prior knowledge + learning strategy + attention + motivation + practice + domain-specific aptitude + circumstances

Someone who becomes highly proficient may have had a favorable combination of many of these.

But someone can fail to learn for many different reasons.

One person may lack foundational knowledge.

Another may use an ineffective learning strategy.

Another may have insufficient practice.

Another may understand the material but lack opportunities to apply it.

Another may have the necessary ability and motivation but simply encounter unfavorable circumstances.

Therefore, comparing a successful learner with an unsuccessful learner and searching for one single explanation can be misleading.

The hidden lesson: don't confuse necessary conditions with sufficient conditions

The Anna Karenina principle is particularly useful because it forces us to distinguish between two kinds of explanations.

A characteristic can be necessary without being sufficient.

For example, suppose a particular trait is found in almost every successful case.

That does not necessarily mean:

"This trait causes success."

It may simply be one item on a larger list of requirements.

Likewise, the absence of one characteristic may explain a particular failure without explaining all failures.

This is a common problem in biological reasoning.

We find a trait associated with successful species and are tempted to declare:

"That's why they succeeded."

But success may actually require a combination of traits.

The real explanation may be closer to:

A∧B∧C∧D∧EA \land B \land C \land D \land E

rather than:

A→successA \rightarrow \text{success}

A useful way of thinking about complex systems

The principle therefore suggests a general strategy for investigating complicated outcomes.

Instead of asking only:

Why did this succeed?

ask:

What are all the conditions that had to be satisfied for success?

And instead of asking:

Why did this fail?

ask:

Which one or more of those conditions failed?

That changes the nature of the investigation.

Success becomes a problem of identifying constraints that must all be satisfied.

Failure becomes a problem of identifying which constraint was violated.

This is particularly powerful in evolutionary biology, where organisms are shaped simultaneously by development, ecology, genetics, behavior, physiology, and history.

But there is an important limitation

The Anna Karenina principle should not be interpreted as a universal law saying that every success has exactly the same causes.

Real systems are messier.

Sometimes there are multiple routes to success. Sometimes one factor dominates. Sometimes different combinations of characteristics can compensate for one another.

For example:

(A+B+C)or(A+D+E)(A+B+C) \quad \text{or} \quad (A+D+E)

might both produce success.

In such cases, successful systems need not be identical.

The principle is therefore best understood as a heuristic—a way of structuring questions about complex systems, rather than a mathematical law that applies without exception.

The broader lesson

Perhaps the most valuable part of the Anna Karenina principle is not the claim that all successful systems look alike.

It is the recognition of an asymmetry between success and failure.

When success requires many things to work together, success can be relatively rare and clustered around combinations of favorable conditions.

Failure, meanwhile, has an enormous number of possible causes.

That means:

The absence of success does not imply the presence of a single common failure mechanism.

And conversely:

The presence of a characteristic in successful cases does not automatically mean that characteristic alone explains success.

This is why the principle is so useful.

It encourages us to stop looking for one simple explanation for complicated outcomes and instead ask a more powerful question:

What are all the things that have to go right—and which of them went wrong?

Tolstoy was writing about families.

But the same logic can help us understand animals, evolution, science, organizations, learning, and perhaps almost any complex system in which success requires several pieces to fall into place at the same time.

Sunday, November 1, 2026

Is There Really a “Replication Crisis”?

Few phrases have travelled further than “replication crisis.”

It is dramatic.

Memorable.

Clickable.

And the National Academies report is notably uncomfortable with it.

The reason is not that reproducibility problems are imaginary.

They are real.

The problem is the word crisis.

How broken is science?

The frustrating answer is:

We do not know precisely.

Replication projects have often selected particular sets of studies rather than random samples of entire disciplines.

Different projects define successful replication differently.

Some compare statistical significance.

Others compare effect sizes.

Others examine direction.

Others consider uncertainty.

Some replication attempts also differ methodologically from the original study.

The committee therefore concluded that existing evidence cannot provide one defensible science-wide replication rate.

Brian Nosek, whose work has been central to large-scale replication research, is quoted in the report as saying:

“I don’t like the term crisis.”

His reason is revealing.

The word implies knowledge about the scale and abnormality of the problem that we simply do not possess.

What replication rate should healthy science have?

This question is rarely asked.

Should 100% of results replicate?

Probably not.

A field producing nearly perfect replication could achieve that by studying only extremely predictable phenomena.

Innovative research necessarily enters regions of greater uncertainty.

Some bold hypotheses will be wrong.

Some apparent discoveries will disappear.

Others will reveal hidden variables.

The report explicitly emphasizes that some failures to replicate are part of normal scientific progress.

That does not excuse bad science

There is an equally important distinction.

Non-replication caused by intrinsic complexity is scientifically informative.

Non-replication caused by:

poor design,

inappropriate statistics,

errors,

selective reporting,

incomplete methods,

or misconduct

is avoidable and costly.

The report therefore separates potentially informative sources of non-replicability from those that merely waste scientific effort.

Perhaps science is not a collapsing building

The book contains a lovely analogy borrowed from philosopher Massimo Pigliucci.

Scientific knowledge is better imagined as a web than as a tower.

A tower can collapse if its foundation fails.

A web can lose individual strands and remain intact while damaged connections are repaired and strengthened.

That may be the best image with which to end this series.

Individual papers can fail.

Famous results can disappear.

Methods can be corrected.

Entire subfields can revise their practices.

Yet the larger scientific network can become stronger precisely because those weaknesses are exposed.

The real danger is not that science sometimes fails to replicate.

The danger would be a scientific culture in which nobody tried.

Saturday, October 31, 2026

When Science News Loses the Error Bars

A scientific paper usually comes with uncertainty attached.

Confidence intervals.

Error bars.

Alternative explanations.

Methodological limitations.

Cautious language.

Then the finding begins travelling.

Paper → press release → headline → social media.

And somewhere along the journey, the uncertainty can fall off the vehicle.

Headlines prefer discoveries to accumulated evidence

The National Academies report notes that journalism is event-driven.

A surprising new study is news.

The fact that 37 earlier studies broadly agree with one another is less likely to become tomorrow's headline.

Scientific news therefore tends to emphasize single studies, contradictions and apparent breakthroughs rather than slow accumulation of evidence.

This naturally amplifies instability.

“Study confirms existing understanding with slightly improved estimate” is scientifically useful.

It is not headline catnip.

Exaggeration can start before journalists arrive

Researchers sometimes blame journalists for overselling science.

The report complicates that story.

One study comparing scientific papers, university press releases and subsequent news coverage found that more than one-third of press releases contained exaggerated advice, causal claims or inappropriate extrapolation.

When the press release exaggerated, subsequent news stories were far more likely to exaggerate too.

So the distortion pipeline may begin surprisingly close to the laboratory.

Uncertainty is part of the result

Imagine a study reports:

Effect estimate = 20%

95% confidence interval = 2% to 38%

A headline may compress that into:

“Scientists discover 20% effect.”

A great deal of information has disappeared.

The uncertainty is not decorative statistical parsley.

It tells us how much confidence to place in the estimate.

The public notices the problem

The report cites survey data in which 73% of Americans said that a major problem with scientific news concerned how reporters covered research, while 43% described the media as too quick to report findings that might not hold up.

Scientists face a similar concern: surveys found substantial numbers of respondents believed researchers sometimes overstate the implications of their own work.

The remedy is not to make science communication dull.

It is to make uncertainty intelligible.

“Promising but preliminary” is not a weakness.

“Consistent with” is not cowardice.

“Evidence suggests” is not an admission that scientists know nothing.

Good science communication preserves the shape of the evidence even after removing the equations.

Keep the story.

But keep the error bars too.

Friday, October 30, 2026

Genetics and Psychology Took Different Roads Toward Reliable Science

Talk about reproducibility often collapses science into one giant category.

But the replication problem faced by a psychologist studying human behaviour is not identical to the problem faced by a geneticist conducting a genome-wide association study.

Different fields have different sources of noise, different instruments and different traditions.

The National Academies uses genetics and psychology to make exactly this point.

Genetics built a large-scale replication machinery

Genome-wide association studies scan enormous numbers of genetic variants for associations with disease or other traits.

That creates obvious statistical dangers.

With huge numbers of tests, chance associations are inevitable unless stringent safeguards are used.

The GWAS community responded by developing norms around:

large sample sizes,

international consortia,

quality control,

phenotype and genotype harmonization,

control of population stratification,

stringent multiple-testing thresholds,

independent replication,

and meta-analysis across cohorts.

This was not merely a statistical reform.

It was sociological.

Reliability increasingly depended on collaboration, infrastructure, data sharing and agreement about analytical standards.

Psychology studies a very different kind of system

Human behaviour is extraordinarily context-sensitive.

Researchers cannot control people with the same precision with which a physicist controls a laboratory instrument.

Psychology has consequently been at the centre of public discussion about a supposed “replication crisis.”

But the report is deliberately cautious.

There is no consensus that the field should be characterized as being in crisis.

Some researchers point to low power, p-hacking and inadequate distinctions between prior and post hoc hypotheses.

Others warn that framing psychology as broken may exaggerate the evidence, discourage risky research and overemphasize false positives at the cost of missing genuine discoveries.

Replication rates cannot simply be compared like cricket scores

The report reviewed replication efforts across fields ranging from genetics and chemistry to psychology and economics and found reported rates ranging from below one in five to above three in four.

But it explicitly warns that these numbers are not directly comparable because studies were selected differently and success was defined differently.

That is a crucial lesson.

The question should not be:

Which discipline has the highest replication percentage?

It should be:

What kind of system is being studied?

How much can its variables be controlled?

How noisy are the measurements?

What counts as successful replication?

And what practices would meaningfully improve reliability in that particular field?

Reproducibility has universal principles.

Its implementation is local.

Thursday, October 29, 2026

What If a Journal Accepted Your Experiment Before You Knew the Result?

Scientific publishing usually happens backwards.

Researchers conduct a study.

They discover the result.

Then journals decide whether the study is interesting enough to publish.

That arrangement creates an obvious temptation.

Interesting outcomes become valuable.

Null outcomes become disappointing.

What if we reversed the chronology?

Enter the registered report.

Review the science before seeing the answer

In the model described by the National Academies, researchers prepare the parts of the manuscript that can be written before data collection:

the scientific rationale,

research question,

methods,

and planned analyses.

The journal peer-reviews that proposal.

If it is accepted, the researchers conduct the study and later submit the results.

The publication decision is therefore less dependent on whether the final answer looks exciting.

This attacks publication bias at its root.

The question becomes: Was this worth testing?

Conventional publishing can accidentally encourage:

“Interesting result, therefore interesting paper.”

Registered reports attempt to replace this with:

“Important question + rigorous design, therefore worthwhile study.”

That distinction is subtle but profound.

If the hypothesis is scientifically important, then discovering that it is unsupported is also scientifically useful.

But preregistration is not a halo

The National Academies carefully avoids treating preregistration as a universal cure.

Critics worry that mandatory preregistration could favour easily testable hypotheses and discourage open-ended exploration.

And the report makes an especially important point:

Poor ideas and weak methods can be preregistered too.

Preregistration records a plan.

It does not certify that the plan is intelligent.

Exploration and confirmation need different rules

Many important discoveries arise because scientists notice something unexpected.

Exploratory science needs room to wander.

Confirmatory science needs stronger protection against rewriting the story after seeing the outcome.

The real value of preregistration may therefore lie not in banning exploration, but in distinguishing the two.

There is nothing wrong with saying:

“We discovered this pattern unexpectedly.”

The problem begins when an unexpected pattern is written as though it had been predicted from the beginning.

Registered reports are one attempt to keep those narratives honest.

Wednesday, October 28, 2026

What Happened When a Journal Actually Tried to Run the Code

Scientific peer review normally asks whether a paper appears scientifically sound.

But what happens if a journal asks a more mechanical question:

Does the submitted code actually reproduce the submitted results?

The American Journal of Political Science introduced external reproducibility checks for accepted manuscripts.

The results are striking.

Among the first 116 papers examined, only 8 reproduced successfully on the first attempt.

In another description in the report, 108 of 116 required authors to provide additional information or make some change.

That is roughly 93%.

These were papers that had already survived scholarly review

This is what makes the case interesting.

The papers were not random student assignments.

They were publication-ready research.

Yet when someone attempted to reconstruct the calculations, missing information or other issues commonly surfaced.

This reveals a gap between two questions:

Does the analysis seem plausible?

and

Can someone actually execute it?

Peer review often answers the first.

Reproducibility auditing attempts the second.

It is not cheap

The report estimates that the external reproducibility check added about 52 days to the publication process and required approximately eight person-hours on average.

That is not trivial.

Someone must obtain the code.

Understand the data.

Reconstruct the environment.

Run the analysis.

Compare outputs.

Communicate discrepancies.

Wait for revisions.

Then try again.

But the audit changes behaviour before it happens

There is also a subtler benefit.

If researchers know that another person will actually run their scripts, they have a powerful incentive to organize code, document dependencies and make analytical steps explicit.

The audit does not simply catch problems.

Its existence may alter research behaviour upstream.

Think of it as the scientific equivalent of knowing someone else will eventually open your kitchen cupboards. Suddenly, everything gets labelled.

Should every journal do this?

Perhaps not immediately.

The costs could be substantial, especially for complex analyses.

But the experiment raises an uncomfortable question.

If a major conclusion depends entirely on computation, should verifying that the computation can actually be reconstructed remain optional?

Peer review asks whether we believe the recipe.

A reproducibility audit tries cooking the dish.

Those are not the same test.

Tuesday, October 27, 2026

Your Paper May Survive. Your Software Might Not.

Imagine finding a fascinating computational paper from 2012.

The methods sound clear.

The authors even shared their code.

Excellent.

You click the repository.

Gone.

Perhaps the repository disappeared.

Perhaps a dependency no longer exists.

Perhaps the code requires an obsolete version of Python.

Perhaps the dataset survives in a format that modern software can no longer read.

The article remains pristine in the journal archive.

Its computational ecosystem has become archaeological material.

Computational research has dependencies

Modern analyses are rarely just code + data.

They can depend on:

operating systems,

library versions,

reference databases,

compiler versions,

hardware,

workflow order,

parameter files,

and external software.

The National Academies therefore recommends documenting not only data and analytical methods but also the computational environment, including software dependencies.

Version control records the scientific history

The report discusses version-control systems such as Git because the precise version of code matters.

If collaborators modify the same program over months, which version generated Figure 4?

Version control can preserve that history automatically.

The report also describes workflow-management systems that record the path from input data through transformations to final output.

That is important because computational provenance answers a deceptively simple question:

Where did this number come from?

A GitHub link is not necessarily an archive

The report makes another important distinction.

Commercial hosting platforms can change priorities or disappear.

A researcher who deposits code only in a mutable online repository risks losing the exact scientific object associated with publication.

Archival repositories and persistent identifiers such as DOIs provide stronger preservation.

And sometimes even preservation is not enough.

The report notes that old datasets may eventually exist in formats that modern machines cannot read. Yale's library, for example, had begun exploring emulation infrastructure to provide access to obsolete software.

The scientific record now includes software

We once thought of scientific archiving largely in terms of papers.

Then datasets.

Now software environments must increasingly be treated as research objects too.

The future historian of science may therefore need more than PDFs.

They may need containers, virtual machines, archived libraries, reference-database snapshots and executable workflows.

A modern scientific result is sometimes less like a paragraph and more like a small ecosystem.

If we want the result to survive, we have to preserve more than its photograph.