When AI Audits Evolutionary Genomics: Deep Review Takes on a 261-Genome Gene-Loss Study

What happens when an AI reviewer is given a published evolutionary-genomics paper built from hundreds of genomes, multiple orthology pipelines, raw sequencing reads, phylogenetics, synteny, selection analyses, and comparative statistics?

We recently put our 2026 BMC Biology paper, “Chromosomal rearrangements and segmental deletions contribute to gene loss in squamates,” through Google Deep Review.

The study screened 261 squamate genome assemblies and combined genome annotations, synteny, BLAST searches, raw short- and long-read data, transcriptomes, pairwise genome alignments, paralog phylogenetics and comparative genomic analyses to investigate genes apparently missing from lizards and snakes.

Deep Review was broadly positive about the scientific ambition of the study. It highlighted the scale of the genomic survey, the multi-evidence validation framework, and particularly the detailed IL34-CSF1R case study as strengths.

But then it produced an unusually detailed list of criticisms.

Some are straightforward typographical errors.

Some are surprisingly sharp internal-consistency checks.

Some identify genuine statistical questions.

And, as with our previous experience using Deep Review, a few take a reasonable concern and push it further than the evidence allows.

So once again, the useful question is not:

“Was Deep Review right?”

It is:

“Which parts of the review survive a review of the review?”

First, what was the paper actually claiming?

This distinction turns out to be central.

The study began with 53 genes whose orthologs could not be retrieved from squamate genomes. It then attempted to distinguish genuine evolutionary loss from annotation failure, assembly problems, sequence divergence and paralog confusion.

The final classification reported:

14 genes: ortholog loss with no retained paralog
15 genes: ortholog loss with retained paralogs
24 genes: unresolved or unretrieved orthologs

Thus, the strongest evolutionary-loss inference applies to 29 genes, while another 24 remain deliberately unresolved.

That hierarchy of evidence is scientifically important.

Deep Review's first major criticism was that the paper occasionally forgets its own hierarchy.

And on that point, it is right.

Auditing the criticisms one by one

Deep Review criticism	Assessment	My evaluation
The Abstract sometimes frames all 53 unretrieved orthologs as gene losses, although only 29 are classified as supported losses.	✅ Valid and important	The Abstract first refers to evidence for the “putative loss of 53 genes,” but immediately later divides them into 14 + 15 supported losses and 24 unretrieved orthologs. The latter wording is the scientifically precise interpretation. “53 candidate unretrieved orthologs, of which 29 have stronger evidence for loss” would be clearer.
Five genes occur simultaneously in the retained-paralog loss category and the unresolved category.	✅ Unambiguously valid	This is an excellent catch. FREM2, LCP1, LRCH1, SIAH3, and STOML3 appear in the 15-gene supported retained-paralog category and then reappear immediately in the “remaining 24” unresolved set. The published text confirms this duplication.
The “remaining 24” paragraph actually contains only 23 names and omits six low-identity genes.	✅ Valid	Reconstructing the lists confirms Deep Review's arithmetic. The omitted genes are DNM3, HPGDS, MATK, MOB1A, RBBP7, and ZNF438. Replacing the five duplicated names with these six restores the intended 24 unique genes and the total 14 + 15 + 24 = 53. This is exactly the kind of tedious bookkeeping at which automated review excels.
The manuscript says 19 genes show segmental deletion but lists 20.	✅ Valid	The sentence says “19 out of 36 genes” but the enumerated list contains 20 genes. This is a straightforward numerical error.
The Discussion refers to 14 genes lost with retained paralogs although the reported category contains 15.	✅ Valid	The paper repeatedly reports 15 retained-paralog losses, but page 9 calls the category 14. Another simple consistency error.
Some Figure 2A structural classifications conflict with the textual gene classifications.	✅/⚠️ Strong concern requiring a source-table audit	Figure 2A separates 36 segmental-deletion cases and 17 rearrangement cases, yet several genes described as rearrangement-driven in the text appear under the segmental-deletion panel and vice versa. The discrepancy is visible enough to merit correction, although the safest fix is to regenerate both prose and figure directly from one definitive classification table rather than adjudicate individual cases manually.
Text says 8 intrachromosomal + 9 interchromosomal rearrangements, whereas Figure 2 appears to indicate 10 + 7.	✅ Likely valid	The published text explicitly gives 8 and 9. If Figure 2's set-size bars encode 10 and 7, one representation must be updated.
Figure 3 appears to contradict the statement that 14 genes lack paralogs.	⚠️ Partly valid, but the review needs nuance	Figure 3 shows human Ensembl paralogs, whereas “no retained paralog” appears to refer to paralogs recovered and retained in the relevant squamate orthology analysis. Those are not necessarily the same statement. Deep Review is right that the terminology is confusing, but this need not be a biological contradiction. The manuscript should distinguish “has a paralog in the vertebrate/human gene family” from “has a retained squamate paralog capable of complicating ortholog-loss inference.”
Sequence-identity thresholds shift between 55%, 65% and 70%.	✅/⚠️ Valid reporting problem, but not necessarily one universal cutoff	Category (i) uses >55% as part of one decision rule, the unresolved set is discussed using <65%, and the Discussion later describes many genes as <70%. These thresholds may have different descriptive roles, but that distinction is not clear enough. Deep Review is right that the classification algorithm should be stated formally.
Coverage boundaries change between ≤25%, 25–74%, ≥75% and >75%.	✅ Valid minor inconsistency	This is largely boundary notation rather than a scientific flaw, but exact thresholds matter in a classification pipeline.
The phylogenetic logistic regression treats a single ancestral loss as though many squamate species provide independent evidence.	✅ The most important statistical criticism	IL34 loss appears to have occurred once on the ancestral squamate branch. Descendant squamate species are therefore repeated observations of one evolutionary transition rather than dozens of independent loss events. The comparative-methods literature explicitly warns that when a discrete character has only one evolutionary origin, even hundreds of sampled species cannot provide replicated evolutionary tests of that trait.
Replacing the logistic model with PGLS would solve the problem.	⚠️ Only partly	A PGLS model treating relative intergenic distance as the continuous response could be more natural for quantifying lineage-associated genomic contraction. But PGLS cannot create independent evolutionary origins where only one exists. The statistical limitation should therefore be acknowledged even after changing models.
The Wilcoxon analyses also risk phylogenetic pseudoreplication.	✅ Valid caution	Pooling species observations assumes more independence than the phylogeny actually supplies. This matters especially when all squamates inherit a shared ancestral genomic event. It does not mean the observed contraction is unreal, only that the nominal p-value may overstate the number of independent evolutionary observations.
The 6 × 6 intergenic-distance ratio matrix is insufficiently defined.	✅ Valid, and confirmed by the public code	The released R script calculates every ordered ratio of six pairwise genomic distances and performs one-tailed Wilcoxon tests. Because A/B and B/A have opposite interpretations, numerator and denominator order must be stated explicitly. Deep Review correctly identified a reproducibility gap here.
TOGA should have been considered as an orthogonal loss-validation framework.	⚠️ Good suggestion, not an error	TOGA integrates gene annotation and orthology inference from whole-genome alignments and is specifically capable of detecting gene losses. It would be a particularly valuable additional check for the 24 unresolved genes. But the absence of TOGA does not invalidate a pipeline already combining synteny, raw reads, genome alignments, HybPiper and phylogenetics.
BLASTn alone may lack sensitivity across ~250–280 million years of divergence.	✅ Valid methodological concern	The Methods do not specify the BLASTn task. This matters because NCBI recommends megablast for highly similar sequences, whereas traditional blastn or discontiguous megablast are more suitable for interspecies comparisons and use substantially shorter seeds. Translated searches or alignment-chain approaches would provide useful safeguards against false negatives in highly divergent genes.
The tBLASTx workflow is mentioned in Results but insufficiently described in Methods.	✅ Valid	If translated searches contribute to the evidence score, parameters and scope should be reproducible.
Cross-species RNA-seq mapping with minimap2 -ax splice -uf -k14 may generate false negatives.	✅/⚠️ Valid concern, dependent on library design	Minimap2 documentation states that -uf forces alignment to the forward transcript strand and is appropriate for genuinely stranded direct-RNA or Iso-Seq data. It warns that using it for data that are not appropriately stranded can cause errors. Mapping Pogona RNA reads directly to chicken across deep divergence further reduces sensitivity. Deep Review is right to request clarification, but this does not prove that all negative RNA evidence is invalid.
Custom LASTZ alignments lack sufficient parameter detail.	✅ Valid reproducibility criticism	For seven newly generated whole-genome alignments, software version, masking strategy, scoring, seeding, chaining and netting should ideally be documented.
Exon overlap divided by CDS length could inflate values if UTRs or overlapping alignment blocks are included.	⚠️ Valid clarification request, not a demonstrated error	The Methods should state whether only CDS features were used and whether overlapping alignment blocks were merged. Without inspecting the exact intermediate files, Deep Review cannot establish that inflation actually occurred.
The paper incorrectly uses the RELAX parameter k when discussing codeml.	✅ Strongly valid	This is an important methodological wording error. HyPhy RELAX specifically estimates the selection-intensity parameter k, where significant k < 1 indicates relaxation and k > 1 intensification. PAML codeml instead models ω = dN/dS and branch-specific ω values. The two frameworks cannot be described using the same k criterion.
FEL and MEME species-specific analyses are under-described.	✅/⚠️ Valid request for clarification	The paper depicts site-level results for focal species, but the exact branch sets, significance thresholds and procedure used to generate species-specific FEL/MEME profiles should be specified. Deep Review is right to request this. Whether the analysis itself is invalid cannot be determined from the manuscript alone.
TNIP2 is called unresolved in one section and subsequently treated as a confirmed loss.	✅ Unambiguously valid	Page 10 explicitly says TNIP2 remains an unretrieved ortholog because true loss cannot yet be distinguished from methodological limitations. Page 12 then states that the study “reports the loss” of IL34, STAP1, LAPTM5, and TNIP2. Those two statements cannot both stand without qualification.
RBBP7 and FREM2 are also discussed functionally as though their loss is established.	✅ Valid	Discussing their biological roles is entirely reasonable, but unresolved candidates should remain labelled as unresolved candidates throughout the biological interpretation.
Retention and transcription of a paralog do not demonstrate functional compensation.	✅ Valid and conceptually important	Figure 8 provides good evidence that CAV3-like exists and is transcribed. That establishes a prerequisite for compensation, not compensation itself. Demonstrating compensation would require expression shifts in relevant tissues, altered selection, functional assays or other evidence.
Paralog buffering is not automatically evidence for the “less-is-more” hypothesis.	✅ Valid conceptual distinction	Adaptive gene loss and neutral loss permitted by functional redundancy are different evolutionary scenarios. If a pre-existing paralog simply makes one duplicate dispensable, this can fit birth-and-death gene-family evolution without requiring adaptive loss. The paper should present compensation and adaptive advantage as hypotheses rather than equivalents.
Using mouse knockout phenotypes to conclude that squamates are immunologically compromised is too strong.	✅ Strongly valid	This is perhaps Deep Review's best biological-interpretation criticism. The paper states that squamates may be “functionally compromised” and more susceptible to pathogens because IL34 is absent. But squamates have evolved without IL34 for hundreds of millions of years, while the study itself reports possible compensatory evolution of CSF1R. Mouse knockouts reveal what IL34 does in mice, not the immune competence of an ancient reptilian lineage.
The pharmaceutical extrapolation is too speculative.	✅ Valid	A comparative-genomics study can motivate hypotheses about immune-system remodeling. Claiming that these findings may directly enable new macrophage-modulating pharmaceutical strategies moves several experimental steps beyond the presented evidence.
The Conclusion says all unretrieved genes are GC-neutral and members of multigene families.	✅ Valid overgeneralization	The paper itself reports that 14 lack retained paralogs and that 19 genes exceed 55% GC. What the data more convincingly show is that these genes do not resemble the extreme GC-rich “dark genes” responsible for many historical avian assembly failures. “Most show moderate rather than extreme GC content” would be more precise than universally “GC-neutral.”
The cumulative evidence score is not mathematically defined.	✅ Important reproducibility issue	Figure 2 ranks genes by a cumulative evidence score, but the Methods do not clearly state the weighting formula and exact decision logic translating individual evidence layers into the three final categories. This should be explicit in a computational genomics paper.
The GC-stretch metric is shown but not adequately defined.	✅ Valid	If a quantity appears in Figure 2B and contributes to interpretation, its computation should be reproducible.
Upstream and downstream distance labels are swapped in the Methods.	✅ Valid but computationally harmless	The left and right flanks are labelled opposite to standard upstream/downstream terminology. Because those quantities are added in the denominator, the swap does not change the calculated relative-distance value. This is a nomenclature problem, not a numerical one.
Supplementary Figure S5 calls TNIP2 intrachromosomal although the main text describes an interchromosomal rearrangement.	✅ Valid	The public repository itself contains a directory explicitly named TNIP2-Interchromosomal_rearrangement, strengthening the interpretation that the Supplementary Information description is the incorrect one.
Reference numbering, duplicate references and corrupted bibliographic entries require correction.	✅ Valid	These include duplicate platypus references, incorrect citations for HyPhy/PAML, a missing journal title, a corrupted GUIDANCE2 citation, duplicated journal/year strings, “Science (80-)”, and “Nielsent R.” Deep Review's reference audit is impressively thorough.
Numerous small figure-label and gene-name errors.	✅/⚠️ Mostly valid	Errors such as “Closet paralog,” “LAPTM5S,” “EVOVL3,” “Coelacanthimormes,” aBSERL, and inconsistent IL34 naming are straightforward. Suggestions such as changing IONP2 to LONP2 or ADAP1L to ADAP1 should still be checked against the underlying annotation rather than accepted automatically.

The biggest thing Deep Review found was not a typo

The duplicated gene names are eye-catching, but the deeper issue is the distinction between 53 unretrieved genes and 29 reasonably supported gene losses.

That distinction already exists in the paper.

The problem is that the wording sometimes collapses it.

This matters because “we could not retrieve an ortholog” and “the lineage lost the gene” are fundamentally different statements.

A gene can become difficult to retrieve because of:

high sequence divergence,

assembly gaps,

annotation failure,

rapid evolution,

duplication followed by paralog confusion,

chromosomal relocation,

or genuine deletion.

The entire purpose of the paper's multi-evidence framework was to distinguish these possibilities.

Therefore, the strongest wording should preserve its own hierarchy:

53 initial unretrieved orthologs → 29 stronger gene-loss candidates → 24 unresolved candidates.

Deep Review is correct that treating all 53 interchangeably as “lost genes” weakens one of the paper's conceptual strengths.

This is a good example of AI review detecting something that is not a flaw in the underlying experiment, but a flaw in how the experiment tells its own story.

The category-counting errors are almost tailor-made for Deep Review

The most striking part of this report is its reconstruction of the gene lists.

A human reviewer reads:

14 + 15 + 24 = 53.

Everything looks fine.

Deep Review reads the actual names.

It notices that five genes occur in both mutually exclusive categories.

It counts the “24” names and finds 23.

It then compares the list against the earlier 24-gene low-identity set and identifies exactly six names that disappeared.

That is not sophisticated evolutionary theory.

It is relentless bookkeeping.

And that is precisely why it is useful.

Scientific papers contain hundreds of little relational constraints:

this table must equal that paragraph,

this gene must occur in one category,

this figure must contain the same number as the Methods,

this p-value must refer to the same test described elsewhere.

Humans are excellent at understanding the biological narrative. We are not equally good at remembering every noun encountered six pages earlier.

Deep Review has a different weakness profile.

It does not get tired.

The phylogenetic logistic regression criticism is more serious

The IL34 analysis is biologically compelling.

The gene is retained in outgroups.

Its syntenic interval contracts in squamates.

Long PacBio and Nanopore reads span the expected locus without revealing an assembly gap.

BLAST searches fail to recover the gene.

The surrounding genes remain.

There is additional evolutionary signal in its receptor, CSF1R.

That remains a strong comparative-genomic case.

But Deep Review raises a legitimate question about the reported phylogenetic logistic regression.

The paper assigns:

squamates = 0
non-squamates = 1

and asks whether relative intergenic distance predicts this status across 179 species.

The issue is evolutionary replication.

If IL34 disappeared once in the common ancestor of living squamates, then a gecko, cobra, python, anole and bearded dragon are not five independent gene-loss experiments.

They inherited one ancestral event.

Recent reviews of phylogenetic comparative methods make this point starkly: a discrete trait with a single evolutionary origin cannot support an ordinary replicated test of that trait's evolution merely by sampling hundreds of descendant species.

This does not weaken the physical evidence for the deletion.

It weakens the interpretation of the very small regression p-value as an independent statistical confirmation of that deletion.

The regression is best viewed as describing a powerful clade-associated association, not 179 replicated evolutionary tests.

Would PGLS solve everything?

No.

This is where Deep Review itself deserves reviewing.

Its suggestion to model intergenic distance using PGLS is sensible. A continuous genomic-distance response is arguably more natural than predicting gene status from distance.

But PGLS cannot manufacture independent gene-loss origins.

If the question is:

“Did independent evolutionary losses repeatedly accompany reduced distance?”

the dataset contains essentially one squamate loss event.

If the question is:

“Is the squamate lineage characterized by a phylogenetically structured contraction of this interval?”

then a phylogenetic continuous-trait model is more appropriate.

Those are different questions.

Deep Review spots the problem correctly, but its proposed remedy is not a magic eraser.

Deep Review is also right about BLAST sensitivity

Another important criticism concerns the search strategy.

The study is deliberately conservative about highly divergent genes. It explicitly leaves 24 genes unresolved partly because nucleotide divergence could make them hard to retrieve.

That is good.

But the Methods do not specify the blastn task used.

That detail matters.

NCBI documentation distinguishes megablast, optimized for very similar nucleotide sequences, from traditional blastn and discontiguous megablast, which are more appropriate for interspecies comparisons. The default seed sizes differ substantially.

Across roughly 250 to 280 million years of divergence, a highly divergent coding gene may become invisible to a relatively strict nucleotide search while remaining clearly detectable at the amino-acid or syntenic level.

So Deep Review's suggestion to use or discuss translated searches and TOGA is scientifically sensible.

TOGA is particularly relevant because it integrates structural gene annotation with orthology inference from whole-genome alignments and explicitly detects gene-loss events.

But this is where wording matters again.

“TOGA would strengthen the analysis” is justified.

“The analysis is invalid without TOGA” would not be.

The study already uses far more evidence than a simple NCBI annotation screen, including long reads, synteny, raw sequencing data, HybPiper, IQ-TREE and genome alignments.

TOGA would be an excellent independent judge, particularly for the 24 unresolved defendants.

The RNA-seq criticism is technically sharp

Deep Review also examines this command:

minimap2 -ax splice -uf -k14

and asks whether -uf was appropriate.

This is not nitpicking.

The minimap2 documentation says -uf forces mapping to the forward transcript strand. It is specifically recommended for stranded direct-RNA and suitable Iso-Seq data, and the documentation warns that using it on reads without the expected orientation can increase errors.

So if the cross-species datasets include unstranded or differently stranded libraries, clarification is needed.

There is an additional biological issue.

The analysis maps RNA-seq from Pogona vitticeps to the chicken genome across enormous evolutionary divergence.

A lack of mapping in such an experiment is not equivalent to absence of transcription.

This makes negative RNA evidence weaker than positive RNA evidence.

That does not erase the gene-loss signal when long-read genomic and syntenic evidence independently support deletion.

But it should affect the weight assigned to transcriptomic non-detection.

Again, Deep Review is most useful when it forces each piece of evidence to carry exactly the weight it deserves.

The RELAX versus codeml criticism is definitely correct

This is another excellent technical catch.

The Methods state that RELAX and codeml were used to investigate relaxed selection and then describe:

k > 1 = intensified selection
k < 1 = relaxed selection

That description belongs specifically to HyPhy RELAX.

RELAX estimates the selection-intensity parameter k. Its documentation explicitly states that significant k > 1 indicates intensified selection and significant k < 1 indicates relaxation.

PAML codeml does something different.

Its codon models estimate ω = dN/dS, including branch-specific ω values.

An increased foreground ω can arise from relaxed purifying selection, positive selection, or a mixture of processes depending on the model and context.

So Deep Review is right.

The Methods should specify separately:

what RELAX tested,

what codeml models were compared,

what parameter was examined,

which likelihood-ratio tests were performed,

and how significance was corrected across multiple foreground branches.

This is not merely grammar. It is methodological interpretation.

But Deep Review occasionally asks for more than the study can provide

One place where the review becomes somewhat demanding is its treatment of paralog compensation.

The paper says genes such as CAV3, KDM3A, ADAP2, SUV39H2, and TBC1D14 could be compensated by retained paralogs.

That conditional word matters.

Figure 8 demonstrates that a CAV3-like paralog is retained and transcribed in Pogona.

Deep Review correctly points out that this does not demonstrate functional compensation.

That is true.

But neither does the manuscript claim a biochemical rescue experiment was performed.

The reasonable revision is therefore not to remove the hypothesis.

It is to state it more explicitly as a hypothesis:

Retention and expression of paralogs provide a plausible substrate for functional buffering, but direct compensation remains to be experimentally demonstrated.

That distinction preserves the evolutionary idea without upgrading correlation into mechanism.

The “less-is-more” criticism is conceptually useful

There is also an elegant evolutionary point hiding in this review.

Suppose gene A and gene B are partially redundant.

Gene A disappears.

Gene B continues performing enough of the shared function that the organism suffers little cost.

Was losing gene A adaptive?

Not necessarily.

Its loss may simply have become selectively tolerable because redundancy reduced the cost.

That is classic birth-and-death gene-family evolution.

The stronger “less-is-more” hypothesis argues that losing a gene can itself be advantageous under changed selective conditions.

Those are different claims.

Deep Review is therefore right that the study should distinguish:

paralog buffering permits gene loss

from

gene loss itself was adaptive.

The first can occur without the second.

Evolution has many ways to throw away a spare key without deciding that keylessness is beneficial.

The immune-compromise claim deserves to be softened

This is the biological criticism I agree with most strongly.

The paper discusses mouse knockout studies showing impaired immune phenotypes after loss of IL34 and then suggests that squamates may be functionally compromised and potentially more susceptible to pathogens.

That extrapolation is too direct.

An acute knockout in a laboratory mouse is not equivalent to an ancient gene loss followed by roughly a quarter-billion years of evolutionary accommodation.

Squamates have had enormous evolutionary time for:

changes in CSF1R,

altered CSF1 signaling,

changes in myeloid development,

regulatory compensation,

paralog recruitment,

or completely different immune-network architecture.

Indeed, the paper's own CSF1R analysis suggests evolutionary modification after IL34 loss.

So a better conclusion is:

The absence of IL34 indicates substantial evolutionary remodeling of the macrophage regulatory network in squamates and raises the question of how alternative pathways compensate for its ancestral function.

That is arguably more interesting than declaring squamates compromised.

A lineage that has successfully produced more than ten thousand living species after losing a gene 250 million years ago may have rewritten the circuitry rather than limped along with a broken component.

The pharmaceutical claim travels even further

The paper concludes that these findings may help develop novel pharmaceutical approaches for monocyte and macrophage immune modulation.

Deep Review objects.

I think correctly.

The evolutionary result may eventually inspire biomedical hypotheses.

But between:

“squamates evolved without IL34”

and

“this informs a therapeutic macrophage-modulation strategy”

lie several missing bridges:

mechanism,

cell biology,

functional compensation,

receptor signaling,

experimental perturbation,

and translational validation.

The safest and stronger scientific framing is that squamates provide a natural evolutionary experiment for studying long-term remodeling of the IL34-CSF1-CSF1R axis.

That is already a compelling result.

There is no need to bolt a pharmaceutical rocket onto it.

Deep Review finds another genuine contradiction in TNIP2

This is a small sentence-level inconsistency with large interpretive consequences.

Page 10 explicitly says TNIP2 remains an unretrieved ortholog because distinguishing true evolutionary loss from methodological limitations remains difficult.

Two pages later:

the study reports the loss of four genes, including TNIP2.

Those are different confidence levels.

Deep Review correctly catches this.

The same principle applies to RBBP7 and FREM2.

It is perfectly legitimate to discuss what the loss of an unresolved gene would mean.

But scientific grammar should preserve uncertainty:

“candidate loss,”

“unretrieved ortholog,”

“potential absence,”

and

“confirmed/high-confidence loss”

should not slowly melt into synonyms as the paper approaches the Discussion.

The GC criticism is mostly about language

The paper's central point regarding GC is reasonable.

Previous apparent gene losses in birds often turned out to be artifacts involving extremely GC-rich regions that were difficult to assemble.

The squamate candidates do not show that same extreme pattern, and high-quality assemblies plus long-read evidence reduce the likelihood of a similar systematic artifact.

But calling all 53 genes “GC-neutral” is stronger than necessary.

Nineteen genes exceed 55% GC, and the observed means extend to about 64.7%.

A more accurate conclusion would be:

The candidate genes do not show the extreme GC enrichment characteristic of several previously reported avian “missing gene” artifacts.

That states exactly what the comparison demonstrates.

Deep Review is right to push for that precision.

The public repository makes several criticisms much easier to judge

This is another repeat of what we learned from the LCR-paper audit.

Open code does not merely allow reproduction of results.

It allows reproduction of criticism.

For the IL34 distance analysis, the public R code confirms that the heatmap is built by calculating ordered ratios among six genomic distances and applying one-tailed Wilcoxon tests.

That means Deep Review's request to explicitly specify numerator and denominator orientation is justified.

The repository also explicitly contains a TNIP2-Interchromosomal_rearrangement analysis directory, which supports Deep Review's suspicion that the Supplementary Information's description of the event as “intrachromosomal” is likely a labeling error.

Open science creates an unusual but healthy situation:

the paper can be audited,

the AI reviewer can be audited,

and then the audit of the AI can itself be audited.

Scientific criticism becomes executable. 🧬

Where Deep Review slightly overreaches

Despite an impressive performance, I would not accept every line of this review without qualification.

Three examples stand out.

First, Figure 3 versus “no retained paralog.”

Deep Review treats these as directly contradictory. But the figure refers to human/Ensembl paralog annotations, whereas the evolutionary classification concerns whether an appropriate paralog is retained in squamates after orthology analysis. This needs clearer terminology, but the two statements are not necessarily biologically incompatible.

Second, PGLS as the solution to the single-transition problem.

Deep Review correctly recognizes phylogenetic pseudoreplication. But PGLS cannot turn one ancestral loss into multiple independent losses. It can improve the modeling of continuous genomic distance, not solve the fundamental lack of replicated loss events.

Third, TOGA.

TOGA would be a valuable additional orthology check, especially for the 24 unresolved genes. But calling its absence a weakness of the existing evidence risks hindsight bias. A multi-evidence pipeline can be rigorous without including every available orthology program.

This distinction between:

“this would improve the paper”

and

“the paper is wrong because it lacks this”

is one that AI reviewers still do not always handle gracefully.

This review strongly resembles Deep Review's broader behavior

Google describes Deep Review as an agentic system designed specifically to catch long-range logical dependencies, mathematical inconsistencies, unsupported claims and factual problems across complex documents. The system grew out of the Paper Assistant Tool used on more than 16,000 conference submissions.

The published PAT evaluation is especially informative.

At STOC and ICML:

92.7% and 90.7% of respondents respectively rated the feedback very or mostly helpful.

But only 55.8% and 64.8% regarded the feedback as mostly or completely grounded.

And 35.4% of ICML respondents said PAT found substantive theoretical gaps, while 31% ran new experiments because of its review.

Those numbers perfectly capture the experience here.

The review is extremely useful without being uniformly correct.

Google's own PAT paper explicitly acknowledges that hallucinations and mistaken criticisms remain a limitation and that humans must verify the AI's findings.

That is exactly the right model.

Compared with our previous Deep Review, this one has a different personality

Our earlier Deep Review of the low-complexity-region benchmarking paper was particularly strong at:

figure-to-text contradictions,

algorithm labels,

coordinate conventions,

Jaccard values,

and computational definitions.

This review is strongest at something slightly different:

scientific bookkeeping across a complex evolutionary argument.

It reconstructs gene sets across pages.

It remembers which genes were unresolved.

It notices when the Discussion silently promotes them to confirmed losses.

It tracks thresholds across Results and Methods.

It checks that 19 really means 19.

It compares the meaning of RELAX with the meaning of codeml.

It distinguishes human paralog annotations from loss classifications.

It asks whether 179 species really mean 179 evolutionary observations.

That is not merely proofreading.

It is closer to database integrity checking applied to scientific prose.

What should actually be corrected?

If I were preparing a formal correction, revised version or future follow-up based on this Deep Review, I would prioritize five things.

First, rebuild the 53-gene master classification table and automatically generate every gene list, count, category and figure label from that single source. That would remove the duplicate/omitted-gene problem at its root.

Second, clearly separate 53 unretrieved orthologs, 29 supported losses, and 24 unresolved candidates everywhere, especially in the Abstract, Discussion and Conclusions.

Third, revise the IL34 statistical language. The physical and genomic evidence for IL34 loss is strong; the phylogenetic logistic regression should be presented as supporting association rather than an independent replicated test of gene-loss evolution.

Fourth, clarify the search and evolutionary-modeling Methods: BLAST task, translated searches, RNA-seq strandedness, custom LASTZ parameters, evidence-score computation, RELAX versus codeml, FEL/MEME branch definitions and multiple-testing procedures.

Fifth, soften the biological extrapolations from “functional compromise” and pharmaceutical relevance toward “evolutionary remodeling, compensatory mechanisms and experimentally testable hypotheses.”

Those changes would strengthen the paper without changing its central comparative-genomic contribution.

Does Deep Review undermine the central result?

I do not think so.

The most compelling individual case, IL34, does not depend on one p-value or one annotation database.

It is supported by multiple convergent observations:

the gene is conserved outside squamates,

the expected syntenic locus is identifiable,

long reads span the locus,

the interval is contracted,

the gene cannot be recovered from genomic searches,

and its receptor shows unusual evolutionary signatures.

Even if one removed the phylogenetic logistic regression completely, that evidence would remain.

Likewise, the broad observation that squamate unretrieved orthologs occur across macro- and microchromosomes and are frequently associated with rearranged genomic regions remains interesting.

What Deep Review does challenge is the precision of the classification boundaries and the strength of some downstream interpretations.

Those are important distinctions.

A paper can contain real discoveries and also contain errors.

Those propositions are not opposites.

Perhaps this is where AI review is genuinely transformative

A conventional human reviewer may spend most of their attention asking:

Is this gene-loss story biologically plausible?

Are 261 genomes enough?

Is IL34 interesting?

Are the figures convincing?

Does the work advance the field?

Those are exactly the questions humans should ask.

Deep Review is unusually willing to ask a different set:

You said 19. Why are there 20 names?

Where did these six genes go?

Why is FREM2 in two mutually exclusive categories?

Why does page 10 say TNIP2 is unresolved while page 12 says it is lost?

Why is RELAX's k parameter being applied to codeml?

Why is a single ancestral loss being represented by 179 tip observations?

Why does the Supplementary Information call an event intrachromosomal when the main text calls it interchromosomal?

None of those questions is glamorous.

Collectively, however, they tighten the bolts holding the scientific argument together.

The verdict

This is another high-value Deep Review.

I would classify its major criticisms roughly as follows:

Most of the internal-consistency findings are fully valid.

The 53-versus-29 distinction is scientifically important.

The category duplication and missing-gene bookkeeping errors are demonstrably real.

The RELAX/codeml issue is technically correct.

The phylogenetic single-transition criticism is substantive and should be acknowledged.

The concerns about BLAST sensitivity and cross-species RNA mapping are well founded but do not prove false negatives occurred.

TOGA is a valuable additional analysis, not a prerequisite that the original study failed to meet.

Paralog compensation is a plausible hypothesis, not yet a demonstrated mechanism.

The claims of immune compromise and pharmaceutical relevance are too strong for the current evidence.

And several of the smaller comments are exactly what they appear to be: typos, mislabeled figures and bibliography debris that survived human review and publication.

That last point is perhaps the most fascinating.

This is already a peer-reviewed paper in BMC Biology.

It went through authors, co-authors, reviewers, editors, copyediting and publication.

Deep Review still found a surprisingly large collection of internally checkable errors.

That does not mean human peer review failed.

It means human peer review and machine review appear to have different error spectra.

Humans remain better positioned to judge biological significance, novelty, plausibility and conceptual importance.

AI is becoming exceptionally good at checking whether every number, name, threshold and logical dependency survives a 19-page journey intact.

The two capabilities are complementary.

The lesson from two papers

After examining Deep Review on two of our published papers, a pattern is beginning to emerge.

Deep Review is at its best when the question can be phrased as:

“Does statement A agree with statement B somewhere else in the document or with an externally verifiable technical definition?”

It is less reliable when the question becomes:

“Is this unconventional scientific choice actually wrong, or merely debatable?”

That boundary is important.

A duplicated gene in two mutually exclusive categories is an error.

Calling one threshold 65% and another 70% may be an error, or it may reflect two different analytical purposes that were inadequately explained.

A k parameter attributed to codeml is an error.

Not using TOGA is a methodological choice.

Calling TNIP2 both unresolved and lost is an error.

Whether paralog retention supports adaptive “less-is-more” evolution is an evolutionary interpretation.

An AI reviewer should have enormous authority over the first kind of question.

It should have considerably less authority over the second.

And that may be the most useful way to integrate these systems into science.

Use AI aggressively to find inconsistencies. Use scientists to decide what those inconsistencies mean.

Deep Review is proving remarkably good at finding the loose threads.

We should still decide which ones actually unravel the sweater.

Signs, Science, and the Scientist Who Couldn't Tell Them Apart

An imaginary scientist, a linguistic confusion and a rather serious lesson about the importance of science for humanity.

Imagine a scientist who has spent decades studying the natural world. He has published papers, attended conferences, supervised doctoral students and acquired a formidable reputation in his field. There is just one small problem: he cannot distinguish between signs and science. To make matters worse, he believes the two words mean essentially the same thing.

When asked about the distinction, he confidently explains that science is the art of interpreting signs, and that the purpose of scientific research is to identify signs that reveal the hidden intentions of the universe.

At first, this sounds amusing. After all, English has plenty of confusing words. But suppose our scientist takes this confusion seriously. Suppose he begins treating every coincidence as evidence, every unusual event as a message and every unexplained phenomenon as a sign that confirms what he already believes.

Suddenly, the confusion becomes considerably more consequential.

The world of signs

Signs are everywhere. Dark clouds may be a sign of approaching rain. A fever may be a sign of infection. Smoke may be a sign of fire. A decline in a bird population may be a sign of environmental degradation.

Human beings are remarkably good at recognising patterns and interpreting signs. Indeed, this ability has been essential to our survival. Our ancestors learned to associate animal tracks with predators, changes in vegetation with seasonal transitions and certain sounds with impending danger.

But recognising a sign is not the same as understanding what caused it.

Dark clouds do not necessarily mean it will rain at a particular location. A fever does not identify a specific pathogen. A declining bird population does not, by itself, establish whether the cause is habitat destruction, pesticides, disease or climate change.

A sign is an observation that may carry information. Its interpretation requires evidence, context and, often, further investigation.

This is where science enters the picture.

The world of science

Science begins with curiosity, but it does not end with curiosity. It is a systematic way of investigating the natural world, testing explanations and correcting errors.

A scientist observing dark clouds might ask several questions. What atmospheric conditions produced them? How much moisture is present? What are the wind patterns? How likely is precipitation? The scientist collects observations, develops hypotheses, examines evidence and evaluates predictions.

The essential distinction is that science does not merely ask what something might signify. It asks what can be demonstrated, what alternative explanations exist and what observations would prove a proposed explanation wrong.

This distinction is fundamental. Anyone can identify a pattern. The scientific challenge is to establish whether that pattern is real, whether it is reproducible and whether the proposed explanation is justified.

Our imaginary scientist, unfortunately, has a different approach. If it rains after he sees a black cat, he records the event as confirmation. If it doesn't rain, he considers it an exception. Eventually, he develops an elaborate theory of meteorology based on feline movements.

His theory may be internally consistent. It may even have an impressive vocabulary. But it has one fatal weakness: it has no reliable connection to the actual mechanisms of rainfall.

When signs replace science

The human mind has a remarkable capacity to find patterns, including patterns that do not exist. We associate events, remember coincidences and construct narratives that make the world feel predictable.

Consider a person who believes that a particular ritual brings good fortune. They perform the ritual and subsequently receive a promotion. The sequence of events may be interpreted as a sign that the ritual worked.

But what happened to all the people who performed the ritual and did not receive a promotion? What about those who received promotions without performing it? What role did qualifications, opportunities, institutional decisions and chance play?

Without comparisons, measurements and appropriate controls, a coincidence can easily be mistaken for causation.

This is precisely why science places so much emphasis on experimental design, statistical reasoning, reproducibility and independent verification.

The problem is not that signs are useless. The problem arises when an observation is mistaken for an explanation, or when an interpretation is protected from scrutiny.

Science is not the rejection of intuition. It is a means of testing intuition.

Science is not just about discoveries

Our scientist is now beginning to understand the difference between the two words. However, he makes another mistake. He assumes that science is important only when it produces spectacular discoveries: a new species, a revolutionary technology, a cure for a previously incurable disease or a theory that transforms our understanding of the universe.

These achievements are certainly important. But science also has an equally valuable, less glamorous responsibility: establishing when we are wrong.

Imagine a physician who discovers that a widely accepted treatment is ineffective. The discovery might not make headlines, produce a new medicine or win a prize. Yet preventing thousands of patients from receiving an ineffective treatment could be an extraordinary contribution to humanity.

Similarly, an ecologist demonstrating that a seemingly beneficial conservation intervention has unintended consequences may prevent considerable environmental damage. A geneticist discovering that an apparent association is caused by a technical artefact may save other researchers years of misguided investigation.

Science is valuable not only because of what it discovers, but also because of what it prevents us from mistakenly believing.

The ability to correct an error is one of the most consequential achievements of scientific thinking.

How science has changed human civilisation

From understanding the invisible to transforming everyday life

Medicine and public health

Understanding microorganisms, developing vaccines, discovering antibiotics and improving sanitation have transformed human survival. These advances emerged from investigations into mechanisms of disease, rather than interpretations of symptoms alone.

Agriculture and food security

Plant genetics, soil science, microbiology and agricultural engineering have helped increase food production, improve crop resilience and understand the ecological costs of agricultural practices.

Energy and engineering

Discoveries in physics, chemistry and materials science have enabled electricity generation, telecommunications, modern transport and increasingly efficient renewable energy systems.

Understanding our planet

Climate science, geology, ecology and satellite observations help us understand natural hazards, monitor environmental change and make better-informed decisions about resource management.

None of these advances depended simply on observing signs. They required investigating underlying processes, developing explanations, testing predictions and accumulating reliable knowledge.

And yet, science alone does not guarantee that its discoveries will benefit humanity. The development of a technology does not determine how it will be used. Nuclear physics, for example, has contributed both to nuclear power and to nuclear weapons.

The betterment of humanity therefore requires scientific knowledge alongside ethical judgement, responsible governance and thoughtful social choices.

Science is a process, not a collection of beliefs

Our scientist now faces a more difficult question. If science is so powerful, does that mean scientists are always right?

Certainly not.

Scientists are human beings. They have assumptions, ambitions, biases, professional incentives and intellectual blind spots. Scientific publications can contain errors. Experiments can be poorly designed. Statistical analyses can be inappropriate. Entire scientific communities can sometimes accept explanations that later prove inadequate.

What distinguishes science is not the perfection of scientists but the availability of methods for identifying and correcting mistakes.

A scientific explanation must remain open to examination. Evidence can strengthen it, contradict it or reveal the need for revision. An explanation that cannot be meaningfully challenged is not made scientific simply because someone with a scientific title supports it.

This is also why disagreement is not inherently a weakness in science. Competing hypotheses can be examined against evidence. Interpretations can be challenged. Independent researchers can attempt to reproduce findings.

Scientific consensus, when supported by substantial convergent evidence, is meaningful. But it is not equivalent to unquestionable authority.

Science is a cumulative enterprise in which today's knowledge becomes the starting point for tomorrow's questions.

The real danger of confusing signs with science

Let us return to our scientist.

He has now learnt to pronounce both words correctly. Unfortunately, he still confuses their meanings.

Whenever he encounters a complex problem, he searches for a convenient explanation. He prefers evidence that confirms his existing ideas and dismisses observations that contradict them. He mistakes confidence for certainty, authority for evidence and correlation for causation.

He may even publish papers, supervise students and participate in important academic discussions.

His difficulty is no longer linguistic. It is epistemological: he has misunderstood how knowledge is established.

And this is not merely a fictional problem confined to laboratories. The same confusion can influence decisions about public health, education, agriculture, environmental protection and public policy.

When people mistake anecdotal experiences for general evidence, unsupported claims can spread. When decision-makers interpret correlations as causal relationships, resources can be wasted. When inconvenient evidence is dismissed rather than investigated, preventable mistakes can persist.

Scientific literacy, therefore, is not simply about knowing scientific facts. It is about being able to distinguish observations from interpretations, evidence from assertions and uncertainty from ignorance.

The importance of asking better questions

Perhaps the greatest contribution of science is not any single discovery, but a particular intellectual habit: the willingness to ask questions whose answers might contradict our expectations.

Why did this happen? How do we know? What else could explain it? What evidence would change our minds? Can someone else independently obtain the same result?

These questions are useful far beyond the natural sciences. They help us evaluate claims, recognise uncertainty, make informed decisions and resist the temptation to accept convenient explanations.

Importantly, science does not answer every question that matters to humanity. It can investigate the consequences of an action, but deciding what is morally right also requires ethical reasoning. It can measure economic outcomes, but determining what constitutes a just society involves values and public deliberation.

Recognising the boundaries of science is itself an essential part of scientific thinking.

A final lesson for our scientist

After years of confusing signs with science, our scientist finally attends a lecture on scientific methodology.

The speaker begins with a simple statement:

"A sign tells you that something may be happening. Science helps you investigate what is happening, why it is happening and how confident you should be in your explanation."

For the first time, the scientist realises that the distinction between the two words is much more than a spelling exercise.

Signs may inspire questions. Science provides a disciplined approach to investigating them.

Signs may alert us to danger. Science helps us identify its causes and develop interventions.

Signs may suggest patterns. Science helps us establish whether those patterns are genuine, meaningful and reproducible.

And while signs can be interpreted in countless ways, scientific explanations are constrained by evidence.

Signs help us notice the world. Science helps us understand it.

And understanding the world gives humanity a better opportunity to change it for the better.

Perhaps, then, the most important thing our scientist needs to learn is not how to spell science. It is how to practise it.

Because humanity does not progress simply by noticing more signs, collecting more observations or accumulating more information. It progresses when we develop better explanations, test them rigorously, acknowledge our mistakes and use the knowledge we gain responsibly.

That is the enduring importance of science: not that it gives us all the answers, but that it offers us a systematic way to find better ones.

When an AI Referee Reviews Your Published Paper: Auditing Google Deep Review

What happens when a published scientific paper is handed to an AI system designed specifically to behave like an unusually persistent reviewer?

We recently tried this with our paper, A benchmarking framework for comparative evaluation of low-complexity region detection tools in the human proteome, published in Scientific Reports. The study compared eight low-complexity region, or LCR, detection approaches under 13 configurations across 20,610 representative human proteins. It examined length, protein coverage, amino-acid composition, entropy, purity, method consensus, pairwise overlap, sequence-complexity landscapes, and operational sensitivity and specificity.

We submitted the paper to Google's Deep Review.

What came back was not the familiar generic AI review consisting of “clarify the novelty,” “improve the discussion,” and “add more validation.” Deep Review produced a long technical audit containing specific mathematical, computational, graphical, methodological, and textual criticisms. Its high-level assessment was actually rather positive about the scale of the benchmark, the consensus analysis, probabilistic treatment of LCR sequence space, and the availability of code and processed data. But it then identified several potentially serious weaknesses.

The interesting question is not whether Deep Review found many things to criticize. A sufficiently energetic reviewer can always do that.

The interesting question is: how many of those criticisms survive scrutiny?

I therefore audited the audit. I compared the Deep Review comments with the article, inspected the figures, checked the publicly available analysis scripts, and compared some of the methodological claims with the original literature.

The conclusion is more interesting than either “AI reviewing is amazing” or “AI reviewers hallucinate.”

Deep Review is remarkably good at forensic consistency checking. It is less reliable when it moves from detecting an inconsistency to declaring what that inconsistency necessarily means biologically or methodologically.

That pattern is also strikingly consistent with what has been reported from earlier deployments of Deep Review's predecessor, Google's Paper Assistant Tool.

Deep Review is not just another chatbot prompt

Deep Review grew out of Google's Paper Assistant Tool, or PAT, which was deployed as an optional pre-submission reviewer at STOC, ICML, NeurIPS, and ICLR. Google reports that the system has now been used on more than 16,000 manuscripts. The stated aim is not to replace scientific judgment but to perform labor-intensive checks of logical consistency, technical claims, mathematical statements, and supporting evidence.

The author feedback from the earlier PAT experiments is illuminating. More than 90% of surveyed authors at STOC and ICML described the feedback as very or mostly helpful, and more than 90% said they would use it again. Yet only 55.8% of STOC respondents and 64.8% of ICML respondents said the feedback was mostly or completely grounded.

Those numbers describe our experience surprisingly well.

The review contains some excellent catches. It also contains several comments that are better regarded as hypotheses to investigate rather than verdicts.

Google itself explicitly acknowledges this limitation. Earlier users reported occasional hallucinations, incorrect challenges to valid arguments, and difficulties interpreting figures and complex notation. The ICLR instructions similarly warn authors that PAT may flag correct statements as erroneous or miss real flaws, and that authors remain responsible for verifying every criticism.

With that in mind, how did it perform on our paper?

Auditing the individual criticisms

The table below gives my assessment of each substantive criticism.

Deep Review criticism	Assessment	My evaluation
1. The Abstract says high-consensus LCRs are “longer,” although consensus analysis examines motif composition, entropy and purity, not length.	✅ Valid	This is one of the cleanest catches. Figure 2 analyses motif composition, Shannon entropy and purity as consensus increases. It does not show length versus consensus. The Abstract and Discussion therefore extend the result beyond the analysis actually presented. The paper does indeed describe consensus regions as “longer” despite the corresponding consensus analysis not establishing this.
2. “Residue-centric” and “protein-centric” metrics are partly misclassified in the Abstract.	✅ Valid	Coverage is calculated as the fraction of each protein covered by LCRs and is naturally a protein-level summary. LCR count per protein is explicitly protein-centric. Amino-acid composition and entropy are calculated from LCR sequences themselves. Deep Review is correct that the terminology should be reorganized.
3. CAST, SIMPLE, GBSC and LCD-Composer were omitted.	⚠️ Fair criticism, not a methodological error	No benchmark can contain every available program. Eight approaches and 13 modes already constitute a substantial comparison. However, because the paper invokes PlaToLoCo and broad algorithmic diversity, explaining why these other tools were excluded would improve the study. The criticism becomes too strong if interpreted as meaning the benchmark is invalid because they are absent.
4. The Introduction does not make the gap relative to PlaToLoCo and existing databases sufficiently explicit.	✅ Valid presentation criticism	The paper describes previous resources and then moves rapidly to the new benchmark. Explaining explicitly that aggregation servers are not quantitative proteome-wide benchmarks would make the novelty sharper. This does not change the results, but it would improve the argument.
5. The paper says coordinates were standardized to a 1-based BED-like format and then processed by BEDTools, which expects 0-based starts.	✅ Valid and important	This was Deep Review's strongest technical catch. The manuscript explicitly says the common files are 1-based. BEDTools documentation states that BED starts are zero-based and ends are one-based. More importantly, inspection of our public code confirms the concern. seq_parser.py deliberately outputs start + 1 as a 1-based inclusive coordinate, while downstream scripts pass these coordinates into BEDTools. complexity_plot.py also performs Python slicing as sequence[start:end], despite receiving those 1-based coordinates. Thus, a nominal 20-residue window can become a 19-residue extracted sequence. This is a real coordinate-convention problem and should be corrected and the affected analyses regenerated.
6. The operational TPR/FPR reference favors homorepeat-like LCRs and disadvantages multi-residue repeat detectors.	✅/⚠️ Core criticism valid; wording somewhat overstated	The reference definition strongly favors low mutation and high single-residue dominance. The paper itself clearly says this is an internal operational reference rather than biological ground truth. That caveat is important. Deep Review is nevertheless correct that such a definition is inherently better aligned with some classes of LCR than others. This matters especially because T-REKS is specifically designed for tandem repeats, including degenerate multi-residue repeats. However, the review says a perfect dipeptide with exactly 50% representation is necessarily classified as negative. That is true under the manuscript's written Y > 50% rule, but not under the public code, which uses Y >= 50%. Thus the bias is real, but one of Deep Review's illustrative examples overstates it.
7. Calling Dotplot the best-performing detector may partly reflect similarity between the detector and the operational reference.	✅ Valid caution	When “truth” is operationally defined using particular compositional and repetitiveness features, performance means agreement with that definition. The paper acknowledges this in the Methods and Figure 6 caption, but later language such as “best-performing,” “real LCRs,” and “correct detection” occasionally slides back toward biological accuracy. The AlcoR paper itself notes that there is no computable universal gold standard for LCR detection, illustrating why this distinction matters.
8. DisProt and missing PDB residues are problematic as a negative/background class for LCRs.	⚠️ Substantively important, but Deep Review goes further than can be proved from the manuscript alone	LCRs and intrinsic disorder overlap but are not identical. Experimental work has explicitly shown differing structural and amino-acid preferences between LCRs and disordered regions. Therefore, treating disorder as the simple biological complement of low complexity would indeed be inappropriate. In our paper, these datasets were intended as biological anchors/backgrounds rather than a claim that all IDRs are non-LCRs. The review is right to ask whether overlapping LCR coordinates were removed and how class membership was handled. But its claim that label contamination definitely occurred requires inspection of the exact boundary-generation data workflow. It is a valid question, not yet a demonstrated fatal error.
9. Changing the number of positive consensus regions with increasing k changes class priors in the GAM/posterior analysis.	✅ Valid statistical concern	If the positive class shrinks as consensus becomes more stringent while the background remains fixed, raw posterior probabilities are affected by prevalence as well as changes in feature-space geometry. Comparing probability thresholds across k therefore requires either class balancing, weighting, calibration, or an explicit statement that these are prevalence-dependent empirical posteriors. Deep Review is right here.
10. The held-out ROC description is ambiguous.	✅ Valid	The paper says ROC analysis was performed on held-out data and then refers specifically to coordinates of IDRs and missing residues. But TPR requires positive as well as negative cases. The train/test construction needs to be stated much more precisely. This is a reporting problem even if the underlying implementation used an appropriate split.
11. Tool versions and exact parameter configurations are under-specified in the manuscript.	✅ Mostly valid	The repository contains considerably more information than the article itself, and many scripts reveal exact settings. That mitigates the reproducibility problem. But a benchmarking paper should ideally summarize all 13 configurations, software versions, command-line flags, and key thresholds in one table. Deep Review's recommendation is sensible.
12. The example ACDEFEGEIE should have 40% rather than 60% minimal mutation if dipeptide periodicity is considered.	⚠️ Deep Review identifies a real conceptual problem, but its exact correction is not fully justified	Inspection of the public implementation is revealing. The code does not actually compute the mathematically minimal number of substitutions required to transform a sequence into any possible periodic sequence. Instead, it searches exact k-mers already occurring at least twice and estimates the uncovered fraction using their non-overlapping counts. Under that implementation, the reported 60% can arise. So Deep Review's assertion that the software must return 40% assumes a more rigorous definition than the code implements. The deeper criticism is actually stronger: the manuscript's phrase “minimal mutational distance to a perfect repeat” does not accurately describe what the current algorithm computes.
13. Exact 50% boundaries are unassigned because the text uses strict inequalities.	✅ Valid for the manuscript, not for the implementation	The written definitions leave X = 50 and Y = 50 ambiguous. Deep Review correctly catches that. But the public classification.py uses mutation <= 50 and dominance >= 50, so the implementation does assign boundary values. This is therefore primarily a methods-text/code inconsistency, rather than an unclassified-data bug.
14. fLPS and fLPS 2.0 default and strict modes are repeatedly confused in the Results.	✅ Strongly valid	This is probably the most impressive manuscript-level catch. Figure 1 visually shows broad, high-coverage behavior for the default fLPS modes, while the strict modes are substantially narrower. Figure 7 similarly shows the highest TPR/FPR combination for the default modes, not the strict modes. Yet the Results repeatedly attribute those patterns to the strict variants. This needs correction throughout.
15. The interpretation of the fLPS purity-retention curves is reversed.	✅ Valid	Figure 3 shows the strict fLPS curves remaining above the default curves over much of the moderate-purity range. A curve that remains high as the purity threshold increases indicates a larger fraction of that method's calls surviving the threshold. The text says the strict versions have lower retention and uses this to infer greater stringency, which is not the correct reading of the plotted relative-retention curves.
16. AlcoR and XSTREAM are described inconsistently as conservative, compositionally sharp, heterogeneous and high-entropy.	✅/⚠️ Mostly valid, with one important nuance	There are genuine narrative inconsistencies. AlcoR is called “less selective” in one place and “conservative” shortly afterward. It is also described as favoring sharply biased LCRs although the entropy and complexity-space analyses emphasize heterogeneous predictions. However, Deep Review slightly overstates the contradiction when comparing pooled amino-acid composition with within-region Shannon entropy. A method can produce a dataset globally enriched for one residue while individual detected regions remain fairly heterogeneous. These metrics are related but not identical.
17. The direction of the consensus trajectory contradicts itself across supplementary analyses.	✅ Unambiguously valid	One paragraph says increasing consensus produces increasing mutation tolerance and decreasing compositional dominance. Another says increasing consensus produces lower mutation and higher dominance. Both cannot describe the same directional trend. Figure 2 supports decreasing entropy and increasing purity with consensus, so the former description almost certainly has its direction reversed.
18. The paper gives opposite descriptions of whether DisProt or PDB-derived boundaries are more restrictive.	✅ Unambiguously valid	The manuscript first says DisProt boundaries extend further into high-mutation space and PDB-derived boundaries are more restrictive, then shortly afterwards says DisProt boundaries are consistently more restrictive than PDB-derived boundaries. Deep Review is correct: this is a direct textual contradiction, not a matter of interpretation.
19. High TPR together with near-unity FPR for default fLPS makes “high sensitivity” potentially misleading.	✅ Valid and important	A detector that labels nearly everything positive can achieve both high TPR and high FPR. Figure 7 makes this trade-off very clear. Reporting TPR and FPR separately is legitimate, but describing such a detector as especially effective without emphasizing discrimination is misleading. A quantity such as Youden's J, balanced accuracy, precision/recall, MCC, or residue-level ROC/PR analysis would make the trade-off clearer.
20. Several Figure 6 trend descriptions do not match the plotted curves.	✅ Valid	Deep Review went beyond broad interpretation here and actually checked directions of individual trends. The figure supports several of its observations, including declines that are described in the text as increases or near-constant behavior.
21. Several Jaccard values are interpreted incorrectly.	✅ Strongly valid	The clearest example is the statement that strict modes overlap less, followed by an example in which 0.1235 is compared with 0.099 even though 0.1235 is larger. The text also describes 0.9462 as involving “strict modes” even though that value corresponds to the two default fLPS generations. Figure 4 confirms the numerical mismatch.
22. Figure legends, scales and some visual elements are difficult to read or incompletely described.	✅ Mostly valid, but editorial rather than fundamental	Several legends and axes are indeed cramped. Figure 5 uses tiny discrete labels rather than intuitively readable percentage axes, and Figure 6 could be scaled more consistently. These issues matter for reproducibility and interpretation but do not themselves invalidate the analyses.
23. The Discussion groups SEG strict, T-REKS and XSTREAM together as choices for “compositionally pure” or tandem-repeat-like sequences.	✅ Valid	This sentence combines two distinct goals. SEG strict and LCRFinder fit the “compositionally pure/homorepetitive” description better, whereas T-REKS and XSTREAM are specifically repeat-oriented and can detect compositionally heterogeneous tandem structures. T-REKS was explicitly developed for tandem repeats, including degenerate repeat architecture. The practical recommendation should separate these use cases.
24. The >50 aa length and >20% protein-coverage thresholds recommended for functional prioritization are not directly demonstrated by the consensus analysis.	✅ Valid	These cutoffs correspond conveniently to bins used elsewhere in the study, but Figure 2 does not demonstrate that consensus LCRs exceeding these thresholds are especially enriched for lower entropy or higher purity. The recommendation is reasonable as a hypothesis, but it should be described as heuristic rather than empirically established by the present analyses.
25. The discussion overstates how closely PlaToLoCo overlaps the benchmarked tool set and incorrectly places LCD-Composer in the context of emerging ML methods.	⚠️ Partly valid	The broader suggestion to discuss omitted methods is useful. The manuscript's wording around PlaToLoCo can be made more precise. LCD-Composer is composition-based rather than a machine-learning method, although the manuscript does actually say this immediately after mentioning emerging ML approaches. So this is more an issue of paragraph construction than a major factual error.
26. Reference 29 is inappropriate support for the claim that experimentally curated disorder represents regulated, evolutionarily maintained architecture.	✅ Valid	Wootton and Federhen's SEG work concerns local sequence complexity, not experimental evidence for the evolutionary maintenance of intrinsic disorder. A reference specifically addressing IDR evolution and functional constraint would be more appropriate.
27. Minor terminology, acronym, typo and reference corrections.	✅ Mostly valid	These are largely good copy-editing catches. In particular, T-REKS is indeed named for “Tandem REpeats in sequences with a K-meanS based algorithm,” not the expansion given in the manuscript. The AlcoR paper is titled alignment-free simulation, mapping, and visualization of low-complexity regions, so the manuscript's invented expansion of “AlcoR” should also be corrected. Other spacing, naming, citation-format and reference-list comments are mostly straightforward editorial corrections.

The most consequential thing Deep Review found

For me, the most important result of this exercise is not one of the many textual contradictions.

It is the coordinate issue.

The paper states that outputs were standardized in a 1-based BED-like format. Deep Review correctly noted that BEDTools interprets BED coordinates with zero-based starts.

Initially, that could have been dismissed as an AI reviewer making an assumption about what the scripts actually did.

But the public code confirms the mismatch.

seq_parser.py explicitly converts windows into 1-based inclusive coordinates. Downstream sequence extraction then uses Python's zero-based slicing convention. The confusion-matrix script subsequently passes interval files directly to BEDTools. This means the issue is not merely terminological. At least some sequences and interval lengths are shifted or shortened by one residue.

For long LCRs this may have a small quantitative effect. For very short motifs it can matter appreciably. More importantly, the operational reference itself is generated from short sliding windows, so a one-residue change can alter amino-acid dominance, entropy, and repeat statistics.

This is exactly the kind of error for which an automated reviewer can be unusually useful. Human reviewers tend to concentrate on whether the conceptual analysis makes sense. A machine can relentlessly notice that “1-based” appears on page 3 and “BEDTools” appears on page 4 and ask whether those two statements are compatible.

That is a real contribution.

Where Deep Review is too confident

The review is less convincing when it moves from:

“This design may favor one class of sequences”

to:

“Therefore this tool is systematically penalized and this ranking is circular.”

The first statement is demonstrable from the operational definition. The second requires quantitative testing.

For example, the positive reference strongly emphasizes high single-amino-acid dominance. That undeniably makes it a particular definition of LCR rather than a neutral biological truth. The paper itself explicitly acknowledges that limitation.

Deep Review is therefore correct to say that TPR and FPR must be interpreted as agreement with the operational reference.

But demonstrating how much that definition changes the ranking of T-REKS, XSTREAM, AlcoR or fLPS would require repeating the benchmark under alternative definitions.

That would actually make an excellent follow-up analysis:

use several operational ground truths,

include the long-tandem-repeat region,

include curated compositional-bias annotations,

include experimentally characterized functional LCRs,

and ask how stable tool rankings are across reference definitions.

That is a better scientific response than simply accepting or rejecting the AI criticism.

The reference-set criticism cuts to the heart of the field

Deep Review's criticism of the operational reference is especially interesting because it touches a fundamental problem in LCR research: there is no universally accepted biological gold standard for low complexity.

Even the AlcoR publication explicitly notes that no computable gold standard for LCR detection is available because the true boundaries are not known.

This means any benchmark must choose some operational definition, simulated data, curated annotations, consensus standard, or biological proxy.

Every choice has a lens.

Our mistake is therefore not that we used an operational definition. Some operational reference is necessary.

The point requiring greater care is what we infer from it.

“Tool A agrees most closely with this operational LCR definition” is defensible.

“Tool A is the most accurate LCR detector” is much harder to establish.

Deep Review is right to insist on that distinction.

Deep Review was exceptionally good at finding contradictions

The review's strongest performance is almost forensic.

It found that one paragraph says increasing consensus moves sequences toward higher mutation and lower dominance, while another says the opposite.

It found that the text says DisProt boundaries are less restrictive than PDB boundaries and later says precisely the opposite.

It found that Figure 7 shows default fLPS variants with the highest TPR/FPR values while the prose attributes this to the strict variants.

It checked numerical values in the Jaccard matrix against sentences interpreting those values and noticed that the stated example literally has the inequality backwards.

These are mundane errors compared with inventing a new biological mechanism, but they matter. They are also exactly the kind of errors that can survive several rounds of human reading because readers understand the intended narrative and unconsciously repair inconsistencies while reading.

An AI has no such courtesy. It can be gloriously literal.

But it also demonstrates the classic AI-review failure mode

The ACDEFEGEIE criticism is a good example.

Deep Review reasons that if “mutation percentage” truly means the minimum number of substitutions required to make the sequence periodic, an alternating repeat can be created with fewer changes than a homopolymer, so the manuscript's 60% example must be wrong.

Mathematically, that interpretation is sensible.

But the public code reveals that the implemented algorithm does something narrower. It searches for exact k-mers already repeated within the sequence and calculates a coverage-like quantity. Under that algorithm the 60% value can arise.

So Deep Review has found a genuine problem, but not exactly the problem it claims to have found.

The problem is that the prose gives the reader a stronger mathematical definition than the software implements.

This distinction is crucial. AI reviewers are very capable of constructing a logically consistent interpretation of ambiguous prose and then criticizing the paper for violating that interpretation. Sometimes the paper is wrong. Sometimes the description is merely ambiguous. Sometimes the model's interpretation is not the intended one.

A human expert still has to separate those cases.

How does this compare with other experiences of Deep Review?

Quite closely.

Google's own PAT study reports an intriguing combination: around 91–93% of authors rated the feedback very or mostly helpful, but only 56–65% judged it mostly or completely grounded.

That is almost the signature of the review we received.

It is highly useful even though not every criticism is correct.

That sounds paradoxical until one recognizes what a tool like this is good for. It functions less like a judge issuing a ruling and more like a very energetic colleague covering the manuscript in sticky notes.

You investigate the sticky notes.

Some reveal errors.

Some reveal ambiguities.

Some reveal that the reviewer misunderstood you, which may itself mean the text needs clarification.

And some can simply be discarded.

There is also some early independent evidence supporting this picture. In one recent informal evaluation, Deep Review correctly identified a serious pseudoreplication problem in a biomedical study and independently found textual inconsistencies in doses and injection sites. But on another paper it missed image-related concerns raised elsewhere and even stated that the images were consistent with their descriptions.

That is only an anecdotal test, but it mirrors the broader pattern.

Deep Review appears strongest when the evidence for an error is encoded in text, equations, numerical tables, methods or internal consistency. It is less trustworthy when the task requires subtle visual interpretation, domain-specific causal judgment or deciding whether an unconventional scientific choice is actually wrong.

Interestingly, Google's earlier STOC deployment reported the same weakness: users could identify useful insights but also occasional hallucinations, and the system sometimes struggled with figures and complex notation.

Does the review mean the paper is fundamentally flawed?

No.

That would itself be an overreading of the Deep Review report.

The central descriptive result of the paper remains valuable: different LCR algorithms detect strikingly different portions of protein sequence space, and consensus increases compositional simplicity. Deep Review itself recognized the scale, systematic framework, consensus analysis and open-science components as strengths.

But several conclusions need to be disentangled from the operational reference used to derive TPR and FPR.

Several figure interpretations need correcting.

The fLPS default/strict labels need a systematic textual audit.

The consensus-direction and DisProt/PDB contradictions need correction.

The coordinate convention deserves a rerun of analyses that depend on interval extraction and BEDTools.

And some claims in the Abstract and Discussion should be narrowed to what was actually measured.

That is not a collapse of the study. It is what a rigorous post-publication technical audit is supposed to accomplish.

Perhaps the most valuable lesson is about reproducibility

One unexpected benefit of the exercise is that the public code made it possible to adjudicate the AI reviewer.

Without the repository, the 1-based/0-based issue would remain an accusation.

With the code, we can inspect it.

Without the implementation of mutation percentage, we would have to choose between the manuscript and Deep Review's mathematical interpretation.

With the code, we can see that both descriptions miss part of the story.

Open science therefore does more than make a study reproducible. It makes criticism reproducible.

That may become increasingly important as AI-generated reviews become common.

The future scientific conversation may involve three layers:

the paper,

the AI critique,

and the evidence needed to audit the critique.

So, how good was Deep Review on this paper?

My assessment is that this was an impressively high-value review, but it should not be treated as an authoritative peer-review verdict.

Most of its internal-consistency criticisms are valid.

The coordinate issue is both valid and important.

The default-versus-strict fLPS confusion is real.

The contradictory descriptions of consensus trajectories and DisProt/PDB boundaries are real.

Several Jaccard interpretations are demonstrably wrong in the prose.

Its concern about reference-set neutrality is scientifically well founded.

Its call for stronger parameter reporting is reasonable.

Where it becomes less reliable is in assigning consequences. It sometimes treats “this benchmark favors a particular operational definition” as equivalent to “this tool has been unfairly penalized.” The former follows logically; the latter needs an alternative analysis. It sometimes interprets ambiguous methodology in one particular mathematical way and then declares the manuscript wrong for not following that interpretation. And some of its requested additional tools and validation datasets are valuable extensions rather than requirements for the original study.

That distinction matters enormously.

AI review may be most useful before human review, not instead of it

Google describes Deep Review as a tool that handles cognitively expensive verification while leaving higher-level judgments to people. Based on this exercise, that seems like the right division of labor.

A human reviewer should spend precious attention asking:

Is the biological question important?

Does the operational definition make sense for the stated purpose?

Are the conclusions interesting?

What would actually change our understanding of low-complexity protein sequence?

Deep Review can spend its silicon patience checking:

Did Figure 7 actually show what paragraph 3 says?

Is 0.1235 really smaller than 0.099?

Does “1-based BED” make sense with BEDTools?

Did we call the same algorithm conservative on page 8 and less selective on page 9?

Did the Abstract claim a length result that was never plotted?

That is not a small contribution.

It is exactly the sort of painstaking second-pass review that humans are notoriously bad at sustaining across a long technical manuscript.

But the final scientific judgment still belongs to someone who understands what the paper is trying to discover.

A final thought

The most interesting outcome of this experiment is that Deep Review did not simply tell us whether our paper was “good” or “bad.”

It produced testable criticisms.

Some survived immediately.

Some became stronger after inspecting the code.

Some became weaker.

Some exposed deficiencies in the writing rather than the underlying analysis.

And a few turned out to depend on assumptions made by the AI reviewer itself.

That is probably the productive way to think about AI scientific review.

Not as an oracle.

Not as a replacement referee.

Not even necessarily as a referee at all.

It is closer to a tireless adversarial reader with an enormous memory and no embarrassment about checking whether one sentence on page 18 contradicts a caption on page 11.

Used that way, it is extremely useful.

The important rule is simple:

Deep Review should generate the questions. Science still has to determine the answers.

The Anna Karenina Principle: Why Success Looks Similar but Failure Comes in Many Forms

“All happy families are alike; each unhappy family is unhappy in its own way.”
— Leo Tolstoy, Anna Karenina

Tolstoy was talking about families. But the underlying idea has turned out to be useful far beyond literature.

The Anna Karenina principle describes a peculiar feature of complex systems:

For something to succeed, many conditions may have to be satisfied simultaneously. But for it to fail, only one important condition may need to go wrong.

This simple asymmetry appears everywhere—from families and businesses to animal domestication, evolution, scientific experiments, and even learning.

Success requires many things to go right

Imagine that you want to keep an animal and eventually establish a domesticated population.

It is not enough for the animal simply to be useful.

It might need to:

• eat food that humans can provide economically;

• grow quickly enough to make raising it worthwhile;

• reproduce reliably in captivity;

• tolerate confinement;

• have a manageable temperament;

• not be excessively dangerous to humans;

• live in social groups that humans can control;

• have a breeding system compatible with captive management.

In other words, domestication is not controlled by a single characteristic.

It is more like a checklist:

Diet + growth + reproduction + temperament + social behavior + captivity tolerance + manageable breeding

If all of these conditions are favorable, domestication may be possible.

But suppose an animal fails badly on just one of them.

Perhaps it does not reproduce in captivity.

Or perhaps it becomes extremely aggressive when confined.

Or perhaps it has a diet that makes raising it prohibitively expensive.

That single problem may be sufficient to prevent domestication—even if everything else about the animal is favorable.

This is the Anna Karenina principle in action.

Why failures are much more diverse than successes

Consider two hypothetical species.

Species A fails because it cannot reproduce in captivity.

Species B fails because it is too aggressive.

Species C fails because it has an extremely specialized diet.

Species D fails because it panics when confined.

Species E fails because its social structure makes captive breeding difficult.

They have almost nothing in common except the fact that they failed.

By contrast, successful domesticated species tend to share a recognizable collection of characteristics.

This produces an interesting asymmetry:

There may be many different ways to fail, but relatively few ways to satisfy all the requirements for success.

This is why looking only at successful examples can sometimes be misleading.

The zebra problem

Jared Diamond famously used this idea when discussing why relatively few large mammals were domesticated.

Take the zebra.

At first glance, one might ask:

Why didn't Africans domesticate zebras in the same way that humans domesticated horses?

The obvious answer might seem to be that zebras are simply "not domesticable."

But the Anna Karenina principle encourages a more useful question:

Which particular requirements for domestication does the zebra fail to satisfy?

That shifts the problem from a vague statement about the animal to a series of testable biological characteristics.

Perhaps the animal has an unsuitable temperament. Perhaps its reproductive biology presents difficulties. Perhaps its social behavior is problematic. Perhaps several factors interact.

The important point is methodological:

Instead of looking for one magical characteristic that explains success, examine the collection of constraints that must simultaneously be overcome.

The principle extends to evolution

The same logic can be applied to evolutionary biology.

Suppose a lineage survives for millions of years.

Its continued existence may depend on many interacting features:

• reproductive success;

• sufficient genetic variation;

• ability to obtain food;

• resistance to pathogens;

• tolerance of environmental conditions;

• suitable developmental biology;

• successful interactions with other organisms.

Failure of any sufficiently important component can contribute to extinction.

And extinction itself can therefore have many different explanations.

One species might disappear because of climatic change.

Another because of a pathogen.

Another because of competition.

Another because of a demographic bottleneck.

Another because several relatively small disadvantages accumulate.

There is therefore no contradiction in saying:

Successful lineages may share important characteristics, while extinct lineages can have very different histories.

It also applies to science

There is an interesting version of the Anna Karenina principle in scientific research.

A successful experiment often requires many things to work:

good experimental design + appropriate measurements + adequate controls + reliable samples + suitable statistical analysis + reproducibility + correct interpretation

A failure, however, can happen because any one of these breaks down.

A study may have an excellent hypothesis but poor measurements.

Another may have excellent measurements but inadequate controls.

Another may have a perfectly reproducible experiment but an incorrect interpretation.

This is particularly important because reproducibility does not automatically guarantee correctness.

An experiment can be performed exactly as described and produce the same result every time, while the explanation attached to that result is nevertheless wrong.

Thus, "the experiment worked" and "the scientific conclusion is correct" are separate questions.

The Anna Karenina principle helps us see why.

And it applies to people

Learning provides another familiar example.

Becoming very good at something can depend on many factors:

cognitive ability + prior knowledge + learning strategy + attention + motivation + practice + domain-specific aptitude + circumstances

Someone who becomes highly proficient may have had a favorable combination of many of these.

But someone can fail to learn for many different reasons.

One person may lack foundational knowledge.

Another may use an ineffective learning strategy.

Another may have insufficient practice.

Another may understand the material but lack opportunities to apply it.

Another may have the necessary ability and motivation but simply encounter unfavorable circumstances.

Therefore, comparing a successful learner with an unsuccessful learner and searching for one single explanation can be misleading.

The hidden lesson: don't confuse necessary conditions with sufficient conditions

The Anna Karenina principle is particularly useful because it forces us to distinguish between two kinds of explanations.

A characteristic can be necessary without being sufficient.

For example, suppose a particular trait is found in almost every successful case.

That does not necessarily mean:

"This trait causes success."

It may simply be one item on a larger list of requirements.

Likewise, the absence of one characteristic may explain a particular failure without explaining all failures.

This is a common problem in biological reasoning.

We find a trait associated with successful species and are tempted to declare:

"That's why they succeeded."

But success may actually require a combination of traits.

The real explanation may be closer to:

$$A \land B \land C \land D \land E$$

rather than:

$$A \rightarrow \text{success}$$

A useful way of thinking about complex systems

The principle therefore suggests a general strategy for investigating complicated outcomes.

Instead of asking only:

Why did this succeed?

ask:

What are all the conditions that had to be satisfied for success?

And instead of asking:

Why did this fail?

ask:

Which one or more of those conditions failed?

That changes the nature of the investigation.

Success becomes a problem of identifying constraints that must all be satisfied.

Failure becomes a problem of identifying which constraint was violated.

This is particularly powerful in evolutionary biology, where organisms are shaped simultaneously by development, ecology, genetics, behavior, physiology, and history.

But there is an important limitation

The Anna Karenina principle should not be interpreted as a universal law saying that every success has exactly the same causes.

Real systems are messier.

Sometimes there are multiple routes to success. Sometimes one factor dominates. Sometimes different combinations of characteristics can compensate for one another.

For example:

$$(A+B+C) \quad \text{or} \quad (A+D+E)$$

might both produce success.

In such cases, successful systems need not be identical.

The principle is therefore best understood as a heuristic—a way of structuring questions about complex systems, rather than a mathematical law that applies without exception.

The broader lesson

Perhaps the most valuable part of the Anna Karenina principle is not the claim that all successful systems look alike.

It is the recognition of an asymmetry between success and failure.

When success requires many things to work together, success can be relatively rare and clustered around combinations of favorable conditions.

Failure, meanwhile, has an enormous number of possible causes.

That means:

The absence of success does not imply the presence of a single common failure mechanism.

And conversely:

The presence of a characteristic in successful cases does not automatically mean that characteristic alone explains success.

This is why the principle is so useful.

It encourages us to stop looking for one simple explanation for complicated outcomes and instead ask a more powerful question:

What are all the things that have to go right—and which of them went wrong?

Tolstoy was writing about families.

But the same logic can help us understand animals, evolution, science, organizations, learning, and perhaps almost any complex system in which success requires several pieces to fall into place at the same time.

Is There Really a “Replication Crisis”?

Few phrases have travelled further than “replication crisis.”

It is dramatic.

Memorable.

Clickable.

And the National Academies report is notably uncomfortable with it.

The reason is not that reproducibility problems are imaginary.

They are real.

The problem is the word crisis.

How broken is science?

The frustrating answer is:

We do not know precisely.

Replication projects have often selected particular sets of studies rather than random samples of entire disciplines.

Different projects define successful replication differently.

Some compare statistical significance.

Others compare effect sizes.

Others examine direction.

Others consider uncertainty.

Some replication attempts also differ methodologically from the original study.

The committee therefore concluded that existing evidence cannot provide one defensible science-wide replication rate.

Brian Nosek, whose work has been central to large-scale replication research, is quoted in the report as saying:

“I don’t like the term crisis.”

His reason is revealing.

The word implies knowledge about the scale and abnormality of the problem that we simply do not possess.

What replication rate should healthy science have?

This question is rarely asked.

Should 100% of results replicate?

Probably not.

A field producing nearly perfect replication could achieve that by studying only extremely predictable phenomena.

Innovative research necessarily enters regions of greater uncertainty.

Some bold hypotheses will be wrong.

Some apparent discoveries will disappear.

Others will reveal hidden variables.

The report explicitly emphasizes that some failures to replicate are part of normal scientific progress.

That does not excuse bad science

There is an equally important distinction.

Non-replication caused by intrinsic complexity is scientifically informative.

Non-replication caused by:

poor design,

inappropriate statistics,

errors,

selective reporting,

incomplete methods,

or misconduct

is avoidable and costly.

The report therefore separates potentially informative sources of non-replicability from those that merely waste scientific effort.

Perhaps science is not a collapsing building

The book contains a lovely analogy borrowed from philosopher Massimo Pigliucci.

Scientific knowledge is better imagined as a web than as a tower.

A tower can collapse if its foundation fails.

A web can lose individual strands and remain intact while damaged connections are repaired and strengthened.

That may be the best image with which to end this series.

Individual papers can fail.

Famous results can disappear.

Methods can be corrected.

Entire subfields can revise their practices.

Yet the larger scientific network can become stronger precisely because those weaknesses are exposed.

The real danger is not that science sometimes fails to replicate.

The danger would be a scientific culture in which nobody tried.