Signs, Science, and the Scientist Who Couldn't Tell Them Apart

An imaginary scientist, a linguistic confusion and a rather serious lesson about the importance of science for humanity.

Imagine a scientist who has spent decades studying the natural world. He has published papers, attended conferences, supervised doctoral students and acquired a formidable reputation in his field. There is just one small problem: he cannot distinguish between signs and science. To make matters worse, he believes the two words mean essentially the same thing.

When asked about the distinction, he confidently explains that science is the art of interpreting signs, and that the purpose of scientific research is to identify signs that reveal the hidden intentions of the universe.

At first, this sounds amusing. After all, English has plenty of confusing words. But suppose our scientist takes this confusion seriously. Suppose he begins treating every coincidence as evidence, every unusual event as a message and every unexplained phenomenon as a sign that confirms what he already believes.

Suddenly, the confusion becomes considerably more consequential.

The world of signs

Signs are everywhere. Dark clouds may be a sign of approaching rain. A fever may be a sign of infection. Smoke may be a sign of fire. A decline in a bird population may be a sign of environmental degradation.

Human beings are remarkably good at recognising patterns and interpreting signs. Indeed, this ability has been essential to our survival. Our ancestors learned to associate animal tracks with predators, changes in vegetation with seasonal transitions and certain sounds with impending danger.

But recognising a sign is not the same as understanding what caused it.

Dark clouds do not necessarily mean it will rain at a particular location. A fever does not identify a specific pathogen. A declining bird population does not, by itself, establish whether the cause is habitat destruction, pesticides, disease or climate change.

A sign is an observation that may carry information. Its interpretation requires evidence, context and, often, further investigation.

This is where science enters the picture.

The world of science

Science begins with curiosity, but it does not end with curiosity. It is a systematic way of investigating the natural world, testing explanations and correcting errors.

A scientist observing dark clouds might ask several questions. What atmospheric conditions produced them? How much moisture is present? What are the wind patterns? How likely is precipitation? The scientist collects observations, develops hypotheses, examines evidence and evaluates predictions.

The essential distinction is that science does not merely ask what something might signify. It asks what can be demonstrated, what alternative explanations exist and what observations would prove a proposed explanation wrong.

This distinction is fundamental. Anyone can identify a pattern. The scientific challenge is to establish whether that pattern is real, whether it is reproducible and whether the proposed explanation is justified.

Our imaginary scientist, unfortunately, has a different approach. If it rains after he sees a black cat, he records the event as confirmation. If it doesn't rain, he considers it an exception. Eventually, he develops an elaborate theory of meteorology based on feline movements.

His theory may be internally consistent. It may even have an impressive vocabulary. But it has one fatal weakness: it has no reliable connection to the actual mechanisms of rainfall.

When signs replace science

The human mind has a remarkable capacity to find patterns, including patterns that do not exist. We associate events, remember coincidences and construct narratives that make the world feel predictable.

Consider a person who believes that a particular ritual brings good fortune. They perform the ritual and subsequently receive a promotion. The sequence of events may be interpreted as a sign that the ritual worked.

But what happened to all the people who performed the ritual and did not receive a promotion? What about those who received promotions without performing it? What role did qualifications, opportunities, institutional decisions and chance play?

Without comparisons, measurements and appropriate controls, a coincidence can easily be mistaken for causation.

This is precisely why science places so much emphasis on experimental design, statistical reasoning, reproducibility and independent verification.

The problem is not that signs are useless. The problem arises when an observation is mistaken for an explanation, or when an interpretation is protected from scrutiny.

Science is not the rejection of intuition. It is a means of testing intuition.

Science is not just about discoveries

Our scientist is now beginning to understand the difference between the two words. However, he makes another mistake. He assumes that science is important only when it produces spectacular discoveries: a new species, a revolutionary technology, a cure for a previously incurable disease or a theory that transforms our understanding of the universe.

These achievements are certainly important. But science also has an equally valuable, less glamorous responsibility: establishing when we are wrong.

Imagine a physician who discovers that a widely accepted treatment is ineffective. The discovery might not make headlines, produce a new medicine or win a prize. Yet preventing thousands of patients from receiving an ineffective treatment could be an extraordinary contribution to humanity.

Similarly, an ecologist demonstrating that a seemingly beneficial conservation intervention has unintended consequences may prevent considerable environmental damage. A geneticist discovering that an apparent association is caused by a technical artefact may save other researchers years of misguided investigation.

Science is valuable not only because of what it discovers, but also because of what it prevents us from mistakenly believing.

The ability to correct an error is one of the most consequential achievements of scientific thinking.

How science has changed human civilisation

From understanding the invisible to transforming everyday life

Medicine and public health

Understanding microorganisms, developing vaccines, discovering antibiotics and improving sanitation have transformed human survival. These advances emerged from investigations into mechanisms of disease, rather than interpretations of symptoms alone.

Agriculture and food security

Plant genetics, soil science, microbiology and agricultural engineering have helped increase food production, improve crop resilience and understand the ecological costs of agricultural practices.

Energy and engineering

Discoveries in physics, chemistry and materials science have enabled electricity generation, telecommunications, modern transport and increasingly efficient renewable energy systems.

Understanding our planet

Climate science, geology, ecology and satellite observations help us understand natural hazards, monitor environmental change and make better-informed decisions about resource management.

None of these advances depended simply on observing signs. They required investigating underlying processes, developing explanations, testing predictions and accumulating reliable knowledge.

And yet, science alone does not guarantee that its discoveries will benefit humanity. The development of a technology does not determine how it will be used. Nuclear physics, for example, has contributed both to nuclear power and to nuclear weapons.

The betterment of humanity therefore requires scientific knowledge alongside ethical judgement, responsible governance and thoughtful social choices.

Science is a process, not a collection of beliefs

Our scientist now faces a more difficult question. If science is so powerful, does that mean scientists are always right?

Certainly not.

Scientists are human beings. They have assumptions, ambitions, biases, professional incentives and intellectual blind spots. Scientific publications can contain errors. Experiments can be poorly designed. Statistical analyses can be inappropriate. Entire scientific communities can sometimes accept explanations that later prove inadequate.

What distinguishes science is not the perfection of scientists but the availability of methods for identifying and correcting mistakes.

A scientific explanation must remain open to examination. Evidence can strengthen it, contradict it or reveal the need for revision. An explanation that cannot be meaningfully challenged is not made scientific simply because someone with a scientific title supports it.

This is also why disagreement is not inherently a weakness in science. Competing hypotheses can be examined against evidence. Interpretations can be challenged. Independent researchers can attempt to reproduce findings.

Scientific consensus, when supported by substantial convergent evidence, is meaningful. But it is not equivalent to unquestionable authority.

Science is a cumulative enterprise in which today's knowledge becomes the starting point for tomorrow's questions.

The real danger of confusing signs with science

Let us return to our scientist.

He has now learnt to pronounce both words correctly. Unfortunately, he still confuses their meanings.

Whenever he encounters a complex problem, he searches for a convenient explanation. He prefers evidence that confirms his existing ideas and dismisses observations that contradict them. He mistakes confidence for certainty, authority for evidence and correlation for causation.

He may even publish papers, supervise students and participate in important academic discussions.

His difficulty is no longer linguistic. It is epistemological: he has misunderstood how knowledge is established.

And this is not merely a fictional problem confined to laboratories. The same confusion can influence decisions about public health, education, agriculture, environmental protection and public policy.

When people mistake anecdotal experiences for general evidence, unsupported claims can spread. When decision-makers interpret correlations as causal relationships, resources can be wasted. When inconvenient evidence is dismissed rather than investigated, preventable mistakes can persist.

Scientific literacy, therefore, is not simply about knowing scientific facts. It is about being able to distinguish observations from interpretations, evidence from assertions and uncertainty from ignorance.

The importance of asking better questions

Perhaps the greatest contribution of science is not any single discovery, but a particular intellectual habit: the willingness to ask questions whose answers might contradict our expectations.

Why did this happen? How do we know? What else could explain it? What evidence would change our minds? Can someone else independently obtain the same result?

These questions are useful far beyond the natural sciences. They help us evaluate claims, recognise uncertainty, make informed decisions and resist the temptation to accept convenient explanations.

Importantly, science does not answer every question that matters to humanity. It can investigate the consequences of an action, but deciding what is morally right also requires ethical reasoning. It can measure economic outcomes, but determining what constitutes a just society involves values and public deliberation.

Recognising the boundaries of science is itself an essential part of scientific thinking.

A final lesson for our scientist

After years of confusing signs with science, our scientist finally attends a lecture on scientific methodology.

The speaker begins with a simple statement:

"A sign tells you that something may be happening. Science helps you investigate what is happening, why it is happening and how confident you should be in your explanation."

For the first time, the scientist realises that the distinction between the two words is much more than a spelling exercise.

Signs may inspire questions. Science provides a disciplined approach to investigating them.

Signs may alert us to danger. Science helps us identify its causes and develop interventions.

Signs may suggest patterns. Science helps us establish whether those patterns are genuine, meaningful and reproducible.

And while signs can be interpreted in countless ways, scientific explanations are constrained by evidence.

Signs help us notice the world. Science helps us understand it.

And understanding the world gives humanity a better opportunity to change it for the better.

Perhaps, then, the most important thing our scientist needs to learn is not how to spell science. It is how to practise it.

Because humanity does not progress simply by noticing more signs, collecting more observations or accumulating more information. It progresses when we develop better explanations, test them rigorously, acknowledge our mistakes and use the knowledge we gain responsibly.

That is the enduring importance of science: not that it gives us all the answers, but that it offers us a systematic way to find better ones.

When an AI Referee Reviews Your Published Paper: Auditing Google Deep Review

What happens when a published scientific paper is handed to an AI system designed specifically to behave like an unusually persistent reviewer?

We recently tried this with our paper, A benchmarking framework for comparative evaluation of low-complexity region detection tools in the human proteome, published in Scientific Reports. The study compared eight low-complexity region, or LCR, detection approaches under 13 configurations across 20,610 representative human proteins. It examined length, protein coverage, amino-acid composition, entropy, purity, method consensus, pairwise overlap, sequence-complexity landscapes, and operational sensitivity and specificity.

We submitted the paper to Google's Deep Review.

What came back was not the familiar generic AI review consisting of “clarify the novelty,” “improve the discussion,” and “add more validation.” Deep Review produced a long technical audit containing specific mathematical, computational, graphical, methodological, and textual criticisms. Its high-level assessment was actually rather positive about the scale of the benchmark, the consensus analysis, probabilistic treatment of LCR sequence space, and the availability of code and processed data. But it then identified several potentially serious weaknesses.

The interesting question is not whether Deep Review found many things to criticize. A sufficiently energetic reviewer can always do that.

The interesting question is: how many of those criticisms survive scrutiny?

I therefore audited the audit. I compared the Deep Review comments with the article, inspected the figures, checked the publicly available analysis scripts, and compared some of the methodological claims with the original literature.

The conclusion is more interesting than either “AI reviewing is amazing” or “AI reviewers hallucinate.”

Deep Review is remarkably good at forensic consistency checking. It is less reliable when it moves from detecting an inconsistency to declaring what that inconsistency necessarily means biologically or methodologically.

That pattern is also strikingly consistent with what has been reported from earlier deployments of Deep Review's predecessor, Google's Paper Assistant Tool.

Deep Review is not just another chatbot prompt

Deep Review grew out of Google's Paper Assistant Tool, or PAT, which was deployed as an optional pre-submission reviewer at STOC, ICML, NeurIPS, and ICLR. Google reports that the system has now been used on more than 16,000 manuscripts. The stated aim is not to replace scientific judgment but to perform labor-intensive checks of logical consistency, technical claims, mathematical statements, and supporting evidence.

The author feedback from the earlier PAT experiments is illuminating. More than 90% of surveyed authors at STOC and ICML described the feedback as very or mostly helpful, and more than 90% said they would use it again. Yet only 55.8% of STOC respondents and 64.8% of ICML respondents said the feedback was mostly or completely grounded.

Those numbers describe our experience surprisingly well.

The review contains some excellent catches. It also contains several comments that are better regarded as hypotheses to investigate rather than verdicts.

Google itself explicitly acknowledges this limitation. Earlier users reported occasional hallucinations, incorrect challenges to valid arguments, and difficulties interpreting figures and complex notation. The ICLR instructions similarly warn authors that PAT may flag correct statements as erroneous or miss real flaws, and that authors remain responsible for verifying every criticism.

With that in mind, how did it perform on our paper?

Auditing the individual criticisms

The table below gives my assessment of each substantive criticism.

Deep Review criticism	Assessment	My evaluation
1. The Abstract says high-consensus LCRs are “longer,” although consensus analysis examines motif composition, entropy and purity, not length.	✅ Valid	This is one of the cleanest catches. Figure 2 analyses motif composition, Shannon entropy and purity as consensus increases. It does not show length versus consensus. The Abstract and Discussion therefore extend the result beyond the analysis actually presented. The paper does indeed describe consensus regions as “longer” despite the corresponding consensus analysis not establishing this.
2. “Residue-centric” and “protein-centric” metrics are partly misclassified in the Abstract.	✅ Valid	Coverage is calculated as the fraction of each protein covered by LCRs and is naturally a protein-level summary. LCR count per protein is explicitly protein-centric. Amino-acid composition and entropy are calculated from LCR sequences themselves. Deep Review is correct that the terminology should be reorganized.
3. CAST, SIMPLE, GBSC and LCD-Composer were omitted.	⚠️ Fair criticism, not a methodological error	No benchmark can contain every available program. Eight approaches and 13 modes already constitute a substantial comparison. However, because the paper invokes PlaToLoCo and broad algorithmic diversity, explaining why these other tools were excluded would improve the study. The criticism becomes too strong if interpreted as meaning the benchmark is invalid because they are absent.
4. The Introduction does not make the gap relative to PlaToLoCo and existing databases sufficiently explicit.	✅ Valid presentation criticism	The paper describes previous resources and then moves rapidly to the new benchmark. Explaining explicitly that aggregation servers are not quantitative proteome-wide benchmarks would make the novelty sharper. This does not change the results, but it would improve the argument.
5. The paper says coordinates were standardized to a 1-based BED-like format and then processed by BEDTools, which expects 0-based starts.	✅ Valid and important	This was Deep Review's strongest technical catch. The manuscript explicitly says the common files are 1-based. BEDTools documentation states that BED starts are zero-based and ends are one-based. More importantly, inspection of our public code confirms the concern. seq_parser.py deliberately outputs start + 1 as a 1-based inclusive coordinate, while downstream scripts pass these coordinates into BEDTools. complexity_plot.py also performs Python slicing as sequence[start:end], despite receiving those 1-based coordinates. Thus, a nominal 20-residue window can become a 19-residue extracted sequence. This is a real coordinate-convention problem and should be corrected and the affected analyses regenerated.
6. The operational TPR/FPR reference favors homorepeat-like LCRs and disadvantages multi-residue repeat detectors.	✅/⚠️ Core criticism valid; wording somewhat overstated	The reference definition strongly favors low mutation and high single-residue dominance. The paper itself clearly says this is an internal operational reference rather than biological ground truth. That caveat is important. Deep Review is nevertheless correct that such a definition is inherently better aligned with some classes of LCR than others. This matters especially because T-REKS is specifically designed for tandem repeats, including degenerate multi-residue repeats. However, the review says a perfect dipeptide with exactly 50% representation is necessarily classified as negative. That is true under the manuscript's written Y > 50% rule, but not under the public code, which uses Y >= 50%. Thus the bias is real, but one of Deep Review's illustrative examples overstates it.
7. Calling Dotplot the best-performing detector may partly reflect similarity between the detector and the operational reference.	✅ Valid caution	When “truth” is operationally defined using particular compositional and repetitiveness features, performance means agreement with that definition. The paper acknowledges this in the Methods and Figure 6 caption, but later language such as “best-performing,” “real LCRs,” and “correct detection” occasionally slides back toward biological accuracy. The AlcoR paper itself notes that there is no computable universal gold standard for LCR detection, illustrating why this distinction matters.
8. DisProt and missing PDB residues are problematic as a negative/background class for LCRs.	⚠️ Substantively important, but Deep Review goes further than can be proved from the manuscript alone	LCRs and intrinsic disorder overlap but are not identical. Experimental work has explicitly shown differing structural and amino-acid preferences between LCRs and disordered regions. Therefore, treating disorder as the simple biological complement of low complexity would indeed be inappropriate. In our paper, these datasets were intended as biological anchors/backgrounds rather than a claim that all IDRs are non-LCRs. The review is right to ask whether overlapping LCR coordinates were removed and how class membership was handled. But its claim that label contamination definitely occurred requires inspection of the exact boundary-generation data workflow. It is a valid question, not yet a demonstrated fatal error.
9. Changing the number of positive consensus regions with increasing k changes class priors in the GAM/posterior analysis.	✅ Valid statistical concern	If the positive class shrinks as consensus becomes more stringent while the background remains fixed, raw posterior probabilities are affected by prevalence as well as changes in feature-space geometry. Comparing probability thresholds across k therefore requires either class balancing, weighting, calibration, or an explicit statement that these are prevalence-dependent empirical posteriors. Deep Review is right here.
10. The held-out ROC description is ambiguous.	✅ Valid	The paper says ROC analysis was performed on held-out data and then refers specifically to coordinates of IDRs and missing residues. But TPR requires positive as well as negative cases. The train/test construction needs to be stated much more precisely. This is a reporting problem even if the underlying implementation used an appropriate split.
11. Tool versions and exact parameter configurations are under-specified in the manuscript.	✅ Mostly valid	The repository contains considerably more information than the article itself, and many scripts reveal exact settings. That mitigates the reproducibility problem. But a benchmarking paper should ideally summarize all 13 configurations, software versions, command-line flags, and key thresholds in one table. Deep Review's recommendation is sensible.
12. The example ACDEFEGEIE should have 40% rather than 60% minimal mutation if dipeptide periodicity is considered.	⚠️ Deep Review identifies a real conceptual problem, but its exact correction is not fully justified	Inspection of the public implementation is revealing. The code does not actually compute the mathematically minimal number of substitutions required to transform a sequence into any possible periodic sequence. Instead, it searches exact k-mers already occurring at least twice and estimates the uncovered fraction using their non-overlapping counts. Under that implementation, the reported 60% can arise. So Deep Review's assertion that the software must return 40% assumes a more rigorous definition than the code implements. The deeper criticism is actually stronger: the manuscript's phrase “minimal mutational distance to a perfect repeat” does not accurately describe what the current algorithm computes.
13. Exact 50% boundaries are unassigned because the text uses strict inequalities.	✅ Valid for the manuscript, not for the implementation	The written definitions leave X = 50 and Y = 50 ambiguous. Deep Review correctly catches that. But the public classification.py uses mutation <= 50 and dominance >= 50, so the implementation does assign boundary values. This is therefore primarily a methods-text/code inconsistency, rather than an unclassified-data bug.
14. fLPS and fLPS 2.0 default and strict modes are repeatedly confused in the Results.	✅ Strongly valid	This is probably the most impressive manuscript-level catch. Figure 1 visually shows broad, high-coverage behavior for the default fLPS modes, while the strict modes are substantially narrower. Figure 7 similarly shows the highest TPR/FPR combination for the default modes, not the strict modes. Yet the Results repeatedly attribute those patterns to the strict variants. This needs correction throughout.
15. The interpretation of the fLPS purity-retention curves is reversed.	✅ Valid	Figure 3 shows the strict fLPS curves remaining above the default curves over much of the moderate-purity range. A curve that remains high as the purity threshold increases indicates a larger fraction of that method's calls surviving the threshold. The text says the strict versions have lower retention and uses this to infer greater stringency, which is not the correct reading of the plotted relative-retention curves.
16. AlcoR and XSTREAM are described inconsistently as conservative, compositionally sharp, heterogeneous and high-entropy.	✅/⚠️ Mostly valid, with one important nuance	There are genuine narrative inconsistencies. AlcoR is called “less selective” in one place and “conservative” shortly afterward. It is also described as favoring sharply biased LCRs although the entropy and complexity-space analyses emphasize heterogeneous predictions. However, Deep Review slightly overstates the contradiction when comparing pooled amino-acid composition with within-region Shannon entropy. A method can produce a dataset globally enriched for one residue while individual detected regions remain fairly heterogeneous. These metrics are related but not identical.
17. The direction of the consensus trajectory contradicts itself across supplementary analyses.	✅ Unambiguously valid	One paragraph says increasing consensus produces increasing mutation tolerance and decreasing compositional dominance. Another says increasing consensus produces lower mutation and higher dominance. Both cannot describe the same directional trend. Figure 2 supports decreasing entropy and increasing purity with consensus, so the former description almost certainly has its direction reversed.
18. The paper gives opposite descriptions of whether DisProt or PDB-derived boundaries are more restrictive.	✅ Unambiguously valid	The manuscript first says DisProt boundaries extend further into high-mutation space and PDB-derived boundaries are more restrictive, then shortly afterwards says DisProt boundaries are consistently more restrictive than PDB-derived boundaries. Deep Review is correct: this is a direct textual contradiction, not a matter of interpretation.
19. High TPR together with near-unity FPR for default fLPS makes “high sensitivity” potentially misleading.	✅ Valid and important	A detector that labels nearly everything positive can achieve both high TPR and high FPR. Figure 7 makes this trade-off very clear. Reporting TPR and FPR separately is legitimate, but describing such a detector as especially effective without emphasizing discrimination is misleading. A quantity such as Youden's J, balanced accuracy, precision/recall, MCC, or residue-level ROC/PR analysis would make the trade-off clearer.
20. Several Figure 6 trend descriptions do not match the plotted curves.	✅ Valid	Deep Review went beyond broad interpretation here and actually checked directions of individual trends. The figure supports several of its observations, including declines that are described in the text as increases or near-constant behavior.
21. Several Jaccard values are interpreted incorrectly.	✅ Strongly valid	The clearest example is the statement that strict modes overlap less, followed by an example in which 0.1235 is compared with 0.099 even though 0.1235 is larger. The text also describes 0.9462 as involving “strict modes” even though that value corresponds to the two default fLPS generations. Figure 4 confirms the numerical mismatch.
22. Figure legends, scales and some visual elements are difficult to read or incompletely described.	✅ Mostly valid, but editorial rather than fundamental	Several legends and axes are indeed cramped. Figure 5 uses tiny discrete labels rather than intuitively readable percentage axes, and Figure 6 could be scaled more consistently. These issues matter for reproducibility and interpretation but do not themselves invalidate the analyses.
23. The Discussion groups SEG strict, T-REKS and XSTREAM together as choices for “compositionally pure” or tandem-repeat-like sequences.	✅ Valid	This sentence combines two distinct goals. SEG strict and LCRFinder fit the “compositionally pure/homorepetitive” description better, whereas T-REKS and XSTREAM are specifically repeat-oriented and can detect compositionally heterogeneous tandem structures. T-REKS was explicitly developed for tandem repeats, including degenerate repeat architecture. The practical recommendation should separate these use cases.
24. The >50 aa length and >20% protein-coverage thresholds recommended for functional prioritization are not directly demonstrated by the consensus analysis.	✅ Valid	These cutoffs correspond conveniently to bins used elsewhere in the study, but Figure 2 does not demonstrate that consensus LCRs exceeding these thresholds are especially enriched for lower entropy or higher purity. The recommendation is reasonable as a hypothesis, but it should be described as heuristic rather than empirically established by the present analyses.
25. The discussion overstates how closely PlaToLoCo overlaps the benchmarked tool set and incorrectly places LCD-Composer in the context of emerging ML methods.	⚠️ Partly valid	The broader suggestion to discuss omitted methods is useful. The manuscript's wording around PlaToLoCo can be made more precise. LCD-Composer is composition-based rather than a machine-learning method, although the manuscript does actually say this immediately after mentioning emerging ML approaches. So this is more an issue of paragraph construction than a major factual error.
26. Reference 29 is inappropriate support for the claim that experimentally curated disorder represents regulated, evolutionarily maintained architecture.	✅ Valid	Wootton and Federhen's SEG work concerns local sequence complexity, not experimental evidence for the evolutionary maintenance of intrinsic disorder. A reference specifically addressing IDR evolution and functional constraint would be more appropriate.
27. Minor terminology, acronym, typo and reference corrections.	✅ Mostly valid	These are largely good copy-editing catches. In particular, T-REKS is indeed named for “Tandem REpeats in sequences with a K-meanS based algorithm,” not the expansion given in the manuscript. The AlcoR paper is titled alignment-free simulation, mapping, and visualization of low-complexity regions, so the manuscript's invented expansion of “AlcoR” should also be corrected. Other spacing, naming, citation-format and reference-list comments are mostly straightforward editorial corrections.

The most consequential thing Deep Review found

For me, the most important result of this exercise is not one of the many textual contradictions.

It is the coordinate issue.

The paper states that outputs were standardized in a 1-based BED-like format. Deep Review correctly noted that BEDTools interprets BED coordinates with zero-based starts.

Initially, that could have been dismissed as an AI reviewer making an assumption about what the scripts actually did.

But the public code confirms the mismatch.

seq_parser.py explicitly converts windows into 1-based inclusive coordinates. Downstream sequence extraction then uses Python's zero-based slicing convention. The confusion-matrix script subsequently passes interval files directly to BEDTools. This means the issue is not merely terminological. At least some sequences and interval lengths are shifted or shortened by one residue.

For long LCRs this may have a small quantitative effect. For very short motifs it can matter appreciably. More importantly, the operational reference itself is generated from short sliding windows, so a one-residue change can alter amino-acid dominance, entropy, and repeat statistics.

This is exactly the kind of error for which an automated reviewer can be unusually useful. Human reviewers tend to concentrate on whether the conceptual analysis makes sense. A machine can relentlessly notice that “1-based” appears on page 3 and “BEDTools” appears on page 4 and ask whether those two statements are compatible.

That is a real contribution.

Where Deep Review is too confident

The review is less convincing when it moves from:

“This design may favor one class of sequences”

to:

“Therefore this tool is systematically penalized and this ranking is circular.”

The first statement is demonstrable from the operational definition. The second requires quantitative testing.

For example, the positive reference strongly emphasizes high single-amino-acid dominance. That undeniably makes it a particular definition of LCR rather than a neutral biological truth. The paper itself explicitly acknowledges that limitation.

Deep Review is therefore correct to say that TPR and FPR must be interpreted as agreement with the operational reference.

But demonstrating how much that definition changes the ranking of T-REKS, XSTREAM, AlcoR or fLPS would require repeating the benchmark under alternative definitions.

That would actually make an excellent follow-up analysis:

use several operational ground truths,

include the long-tandem-repeat region,

include curated compositional-bias annotations,

include experimentally characterized functional LCRs,

and ask how stable tool rankings are across reference definitions.

That is a better scientific response than simply accepting or rejecting the AI criticism.

The reference-set criticism cuts to the heart of the field

Deep Review's criticism of the operational reference is especially interesting because it touches a fundamental problem in LCR research: there is no universally accepted biological gold standard for low complexity.

Even the AlcoR publication explicitly notes that no computable gold standard for LCR detection is available because the true boundaries are not known.

This means any benchmark must choose some operational definition, simulated data, curated annotations, consensus standard, or biological proxy.

Every choice has a lens.

Our mistake is therefore not that we used an operational definition. Some operational reference is necessary.

The point requiring greater care is what we infer from it.

“Tool A agrees most closely with this operational LCR definition” is defensible.

“Tool A is the most accurate LCR detector” is much harder to establish.

Deep Review is right to insist on that distinction.

Deep Review was exceptionally good at finding contradictions

The review's strongest performance is almost forensic.

It found that one paragraph says increasing consensus moves sequences toward higher mutation and lower dominance, while another says the opposite.

It found that the text says DisProt boundaries are less restrictive than PDB boundaries and later says precisely the opposite.

It found that Figure 7 shows default fLPS variants with the highest TPR/FPR values while the prose attributes this to the strict variants.

It checked numerical values in the Jaccard matrix against sentences interpreting those values and noticed that the stated example literally has the inequality backwards.

These are mundane errors compared with inventing a new biological mechanism, but they matter. They are also exactly the kind of errors that can survive several rounds of human reading because readers understand the intended narrative and unconsciously repair inconsistencies while reading.

An AI has no such courtesy. It can be gloriously literal.

But it also demonstrates the classic AI-review failure mode

The ACDEFEGEIE criticism is a good example.

Deep Review reasons that if “mutation percentage” truly means the minimum number of substitutions required to make the sequence periodic, an alternating repeat can be created with fewer changes than a homopolymer, so the manuscript's 60% example must be wrong.

Mathematically, that interpretation is sensible.

But the public code reveals that the implemented algorithm does something narrower. It searches for exact k-mers already repeated within the sequence and calculates a coverage-like quantity. Under that algorithm the 60% value can arise.

So Deep Review has found a genuine problem, but not exactly the problem it claims to have found.

The problem is that the prose gives the reader a stronger mathematical definition than the software implements.

This distinction is crucial. AI reviewers are very capable of constructing a logically consistent interpretation of ambiguous prose and then criticizing the paper for violating that interpretation. Sometimes the paper is wrong. Sometimes the description is merely ambiguous. Sometimes the model's interpretation is not the intended one.

A human expert still has to separate those cases.

How does this compare with other experiences of Deep Review?

Quite closely.

Google's own PAT study reports an intriguing combination: around 91–93% of authors rated the feedback very or mostly helpful, but only 56–65% judged it mostly or completely grounded.

That is almost the signature of the review we received.

It is highly useful even though not every criticism is correct.

That sounds paradoxical until one recognizes what a tool like this is good for. It functions less like a judge issuing a ruling and more like a very energetic colleague covering the manuscript in sticky notes.

You investigate the sticky notes.

Some reveal errors.

Some reveal ambiguities.

Some reveal that the reviewer misunderstood you, which may itself mean the text needs clarification.

And some can simply be discarded.

There is also some early independent evidence supporting this picture. In one recent informal evaluation, Deep Review correctly identified a serious pseudoreplication problem in a biomedical study and independently found textual inconsistencies in doses and injection sites. But on another paper it missed image-related concerns raised elsewhere and even stated that the images were consistent with their descriptions.

That is only an anecdotal test, but it mirrors the broader pattern.

Deep Review appears strongest when the evidence for an error is encoded in text, equations, numerical tables, methods or internal consistency. It is less trustworthy when the task requires subtle visual interpretation, domain-specific causal judgment or deciding whether an unconventional scientific choice is actually wrong.

Interestingly, Google's earlier STOC deployment reported the same weakness: users could identify useful insights but also occasional hallucinations, and the system sometimes struggled with figures and complex notation.

Does the review mean the paper is fundamentally flawed?

No.

That would itself be an overreading of the Deep Review report.

The central descriptive result of the paper remains valuable: different LCR algorithms detect strikingly different portions of protein sequence space, and consensus increases compositional simplicity. Deep Review itself recognized the scale, systematic framework, consensus analysis and open-science components as strengths.

But several conclusions need to be disentangled from the operational reference used to derive TPR and FPR.

Several figure interpretations need correcting.

The fLPS default/strict labels need a systematic textual audit.

The consensus-direction and DisProt/PDB contradictions need correction.

The coordinate convention deserves a rerun of analyses that depend on interval extraction and BEDTools.

And some claims in the Abstract and Discussion should be narrowed to what was actually measured.

That is not a collapse of the study. It is what a rigorous post-publication technical audit is supposed to accomplish.

Perhaps the most valuable lesson is about reproducibility

One unexpected benefit of the exercise is that the public code made it possible to adjudicate the AI reviewer.

Without the repository, the 1-based/0-based issue would remain an accusation.

With the code, we can inspect it.

Without the implementation of mutation percentage, we would have to choose between the manuscript and Deep Review's mathematical interpretation.

With the code, we can see that both descriptions miss part of the story.

Open science therefore does more than make a study reproducible. It makes criticism reproducible.

That may become increasingly important as AI-generated reviews become common.

The future scientific conversation may involve three layers:

the paper,

the AI critique,

and the evidence needed to audit the critique.

So, how good was Deep Review on this paper?

My assessment is that this was an impressively high-value review, but it should not be treated as an authoritative peer-review verdict.

Most of its internal-consistency criticisms are valid.

The coordinate issue is both valid and important.

The default-versus-strict fLPS confusion is real.

The contradictory descriptions of consensus trajectories and DisProt/PDB boundaries are real.

Several Jaccard interpretations are demonstrably wrong in the prose.

Its concern about reference-set neutrality is scientifically well founded.

Its call for stronger parameter reporting is reasonable.

Where it becomes less reliable is in assigning consequences. It sometimes treats “this benchmark favors a particular operational definition” as equivalent to “this tool has been unfairly penalized.” The former follows logically; the latter needs an alternative analysis. It sometimes interprets ambiguous methodology in one particular mathematical way and then declares the manuscript wrong for not following that interpretation. And some of its requested additional tools and validation datasets are valuable extensions rather than requirements for the original study.

That distinction matters enormously.

AI review may be most useful before human review, not instead of it

Google describes Deep Review as a tool that handles cognitively expensive verification while leaving higher-level judgments to people. Based on this exercise, that seems like the right division of labor.

A human reviewer should spend precious attention asking:

Is the biological question important?

Does the operational definition make sense for the stated purpose?

Are the conclusions interesting?

What would actually change our understanding of low-complexity protein sequence?

Deep Review can spend its silicon patience checking:

Did Figure 7 actually show what paragraph 3 says?

Is 0.1235 really smaller than 0.099?

Does “1-based BED” make sense with BEDTools?

Did we call the same algorithm conservative on page 8 and less selective on page 9?

Did the Abstract claim a length result that was never plotted?

That is not a small contribution.

It is exactly the sort of painstaking second-pass review that humans are notoriously bad at sustaining across a long technical manuscript.

But the final scientific judgment still belongs to someone who understands what the paper is trying to discover.

A final thought

The most interesting outcome of this experiment is that Deep Review did not simply tell us whether our paper was “good” or “bad.”

It produced testable criticisms.

Some survived immediately.

Some became stronger after inspecting the code.

Some became weaker.

Some exposed deficiencies in the writing rather than the underlying analysis.

And a few turned out to depend on assumptions made by the AI reviewer itself.

That is probably the productive way to think about AI scientific review.

Not as an oracle.

Not as a replacement referee.

Not even necessarily as a referee at all.

It is closer to a tireless adversarial reader with an enormous memory and no embarrassment about checking whether one sentence on page 18 contradicts a caption on page 11.

Used that way, it is extremely useful.

The important rule is simple:

Deep Review should generate the questions. Science still has to determine the answers.

The Anna Karenina Principle: Why Success Looks Similar but Failure Comes in Many Forms

“All happy families are alike; each unhappy family is unhappy in its own way.”
— Leo Tolstoy, Anna Karenina

Tolstoy was talking about families. But the underlying idea has turned out to be useful far beyond literature.

The Anna Karenina principle describes a peculiar feature of complex systems:

For something to succeed, many conditions may have to be satisfied simultaneously. But for it to fail, only one important condition may need to go wrong.

This simple asymmetry appears everywhere—from families and businesses to animal domestication, evolution, scientific experiments, and even learning.

Success requires many things to go right

Imagine that you want to keep an animal and eventually establish a domesticated population.

It is not enough for the animal simply to be useful.

It might need to:

• eat food that humans can provide economically;

• grow quickly enough to make raising it worthwhile;

• reproduce reliably in captivity;

• tolerate confinement;

• have a manageable temperament;

• not be excessively dangerous to humans;

• live in social groups that humans can control;

• have a breeding system compatible with captive management.

In other words, domestication is not controlled by a single characteristic.

It is more like a checklist:

Diet + growth + reproduction + temperament + social behavior + captivity tolerance + manageable breeding

If all of these conditions are favorable, domestication may be possible.

But suppose an animal fails badly on just one of them.

Perhaps it does not reproduce in captivity.

Or perhaps it becomes extremely aggressive when confined.

Or perhaps it has a diet that makes raising it prohibitively expensive.

That single problem may be sufficient to prevent domestication—even if everything else about the animal is favorable.

This is the Anna Karenina principle in action.

Why failures are much more diverse than successes

Consider two hypothetical species.

Species A fails because it cannot reproduce in captivity.

Species B fails because it is too aggressive.

Species C fails because it has an extremely specialized diet.

Species D fails because it panics when confined.

Species E fails because its social structure makes captive breeding difficult.

They have almost nothing in common except the fact that they failed.

By contrast, successful domesticated species tend to share a recognizable collection of characteristics.

This produces an interesting asymmetry:

There may be many different ways to fail, but relatively few ways to satisfy all the requirements for success.

This is why looking only at successful examples can sometimes be misleading.

The zebra problem

Jared Diamond famously used this idea when discussing why relatively few large mammals were domesticated.

Take the zebra.

At first glance, one might ask:

Why didn't Africans domesticate zebras in the same way that humans domesticated horses?

The obvious answer might seem to be that zebras are simply "not domesticable."

But the Anna Karenina principle encourages a more useful question:

Which particular requirements for domestication does the zebra fail to satisfy?

That shifts the problem from a vague statement about the animal to a series of testable biological characteristics.

Perhaps the animal has an unsuitable temperament. Perhaps its reproductive biology presents difficulties. Perhaps its social behavior is problematic. Perhaps several factors interact.

The important point is methodological:

Instead of looking for one magical characteristic that explains success, examine the collection of constraints that must simultaneously be overcome.

The principle extends to evolution

The same logic can be applied to evolutionary biology.

Suppose a lineage survives for millions of years.

Its continued existence may depend on many interacting features:

• reproductive success;

• sufficient genetic variation;

• ability to obtain food;

• resistance to pathogens;

• tolerance of environmental conditions;

• suitable developmental biology;

• successful interactions with other organisms.

Failure of any sufficiently important component can contribute to extinction.

And extinction itself can therefore have many different explanations.

One species might disappear because of climatic change.

Another because of a pathogen.

Another because of competition.

Another because of a demographic bottleneck.

Another because several relatively small disadvantages accumulate.

There is therefore no contradiction in saying:

Successful lineages may share important characteristics, while extinct lineages can have very different histories.

It also applies to science

There is an interesting version of the Anna Karenina principle in scientific research.

A successful experiment often requires many things to work:

good experimental design + appropriate measurements + adequate controls + reliable samples + suitable statistical analysis + reproducibility + correct interpretation

A failure, however, can happen because any one of these breaks down.

A study may have an excellent hypothesis but poor measurements.

Another may have excellent measurements but inadequate controls.

Another may have a perfectly reproducible experiment but an incorrect interpretation.

This is particularly important because reproducibility does not automatically guarantee correctness.

An experiment can be performed exactly as described and produce the same result every time, while the explanation attached to that result is nevertheless wrong.

Thus, "the experiment worked" and "the scientific conclusion is correct" are separate questions.

The Anna Karenina principle helps us see why.

And it applies to people

Learning provides another familiar example.

Becoming very good at something can depend on many factors:

cognitive ability + prior knowledge + learning strategy + attention + motivation + practice + domain-specific aptitude + circumstances

Someone who becomes highly proficient may have had a favorable combination of many of these.

But someone can fail to learn for many different reasons.

One person may lack foundational knowledge.

Another may use an ineffective learning strategy.

Another may have insufficient practice.

Another may understand the material but lack opportunities to apply it.

Another may have the necessary ability and motivation but simply encounter unfavorable circumstances.

Therefore, comparing a successful learner with an unsuccessful learner and searching for one single explanation can be misleading.

The hidden lesson: don't confuse necessary conditions with sufficient conditions

The Anna Karenina principle is particularly useful because it forces us to distinguish between two kinds of explanations.

A characteristic can be necessary without being sufficient.

For example, suppose a particular trait is found in almost every successful case.

That does not necessarily mean:

"This trait causes success."

It may simply be one item on a larger list of requirements.

Likewise, the absence of one characteristic may explain a particular failure without explaining all failures.

This is a common problem in biological reasoning.

We find a trait associated with successful species and are tempted to declare:

"That's why they succeeded."

But success may actually require a combination of traits.

The real explanation may be closer to:

$$A \land B \land C \land D \land E$$

rather than:

$$A \rightarrow \text{success}$$

A useful way of thinking about complex systems

The principle therefore suggests a general strategy for investigating complicated outcomes.

Instead of asking only:

Why did this succeed?

ask:

What are all the conditions that had to be satisfied for success?

And instead of asking:

Why did this fail?

ask:

Which one or more of those conditions failed?

That changes the nature of the investigation.

Success becomes a problem of identifying constraints that must all be satisfied.

Failure becomes a problem of identifying which constraint was violated.

This is particularly powerful in evolutionary biology, where organisms are shaped simultaneously by development, ecology, genetics, behavior, physiology, and history.

But there is an important limitation

The Anna Karenina principle should not be interpreted as a universal law saying that every success has exactly the same causes.

Real systems are messier.

Sometimes there are multiple routes to success. Sometimes one factor dominates. Sometimes different combinations of characteristics can compensate for one another.

For example:

$$(A+B+C) \quad \text{or} \quad (A+D+E)$$

might both produce success.

In such cases, successful systems need not be identical.

The principle is therefore best understood as a heuristic—a way of structuring questions about complex systems, rather than a mathematical law that applies without exception.

The broader lesson

Perhaps the most valuable part of the Anna Karenina principle is not the claim that all successful systems look alike.

It is the recognition of an asymmetry between success and failure.

When success requires many things to work together, success can be relatively rare and clustered around combinations of favorable conditions.

Failure, meanwhile, has an enormous number of possible causes.

That means:

The absence of success does not imply the presence of a single common failure mechanism.

And conversely:

The presence of a characteristic in successful cases does not automatically mean that characteristic alone explains success.

This is why the principle is so useful.

It encourages us to stop looking for one simple explanation for complicated outcomes and instead ask a more powerful question:

What are all the things that have to go right—and which of them went wrong?

Tolstoy was writing about families.

But the same logic can help us understand animals, evolution, science, organizations, learning, and perhaps almost any complex system in which success requires several pieces to fall into place at the same time.

Is There Really a “Replication Crisis”?

Few phrases have travelled further than “replication crisis.”

It is dramatic.

Memorable.

Clickable.

And the National Academies report is notably uncomfortable with it.

The reason is not that reproducibility problems are imaginary.

They are real.

The problem is the word crisis.

How broken is science?

The frustrating answer is:

We do not know precisely.

Replication projects have often selected particular sets of studies rather than random samples of entire disciplines.

Different projects define successful replication differently.

Some compare statistical significance.

Others compare effect sizes.

Others examine direction.

Others consider uncertainty.

Some replication attempts also differ methodologically from the original study.

The committee therefore concluded that existing evidence cannot provide one defensible science-wide replication rate.

Brian Nosek, whose work has been central to large-scale replication research, is quoted in the report as saying:

“I don’t like the term crisis.”

His reason is revealing.

The word implies knowledge about the scale and abnormality of the problem that we simply do not possess.

What replication rate should healthy science have?

This question is rarely asked.

Should 100% of results replicate?

Probably not.

A field producing nearly perfect replication could achieve that by studying only extremely predictable phenomena.

Innovative research necessarily enters regions of greater uncertainty.

Some bold hypotheses will be wrong.

Some apparent discoveries will disappear.

Others will reveal hidden variables.

The report explicitly emphasizes that some failures to replicate are part of normal scientific progress.

That does not excuse bad science

There is an equally important distinction.

Non-replication caused by intrinsic complexity is scientifically informative.

Non-replication caused by:

poor design,

inappropriate statistics,

errors,

selective reporting,

incomplete methods,

or misconduct

is avoidable and costly.

The report therefore separates potentially informative sources of non-replicability from those that merely waste scientific effort.

Perhaps science is not a collapsing building

The book contains a lovely analogy borrowed from philosopher Massimo Pigliucci.

Scientific knowledge is better imagined as a web than as a tower.

A tower can collapse if its foundation fails.

A web can lose individual strands and remain intact while damaged connections are repaired and strengthened.

That may be the best image with which to end this series.

Individual papers can fail.

Famous results can disappear.

Methods can be corrected.

Entire subfields can revise their practices.

Yet the larger scientific network can become stronger precisely because those weaknesses are exposed.

The real danger is not that science sometimes fails to replicate.

The danger would be a scientific culture in which nobody tried.

When Science News Loses the Error Bars

A scientific paper usually comes with uncertainty attached.

Confidence intervals.

Error bars.

Alternative explanations.

Methodological limitations.

Cautious language.

Then the finding begins travelling.

Paper → press release → headline → social media.

And somewhere along the journey, the uncertainty can fall off the vehicle.

Headlines prefer discoveries to accumulated evidence

The National Academies report notes that journalism is event-driven.

A surprising new study is news.

The fact that 37 earlier studies broadly agree with one another is less likely to become tomorrow's headline.

Scientific news therefore tends to emphasize single studies, contradictions and apparent breakthroughs rather than slow accumulation of evidence.

This naturally amplifies instability.

“Study confirms existing understanding with slightly improved estimate” is scientifically useful.

It is not headline catnip.

Exaggeration can start before journalists arrive

Researchers sometimes blame journalists for overselling science.

The report complicates that story.

One study comparing scientific papers, university press releases and subsequent news coverage found that more than one-third of press releases contained exaggerated advice, causal claims or inappropriate extrapolation.

When the press release exaggerated, subsequent news stories were far more likely to exaggerate too.

So the distortion pipeline may begin surprisingly close to the laboratory.

Uncertainty is part of the result

Imagine a study reports:

Effect estimate = 20%

95% confidence interval = 2% to 38%

A headline may compress that into:

“Scientists discover 20% effect.”

A great deal of information has disappeared.

The uncertainty is not decorative statistical parsley.

It tells us how much confidence to place in the estimate.

The public notices the problem

The report cites survey data in which 73% of Americans said that a major problem with scientific news concerned how reporters covered research, while 43% described the media as too quick to report findings that might not hold up.

Scientists face a similar concern: surveys found substantial numbers of respondents believed researchers sometimes overstate the implications of their own work.

The remedy is not to make science communication dull.

It is to make uncertainty intelligible.

“Promising but preliminary” is not a weakness.

“Consistent with” is not cowardice.

“Evidence suggests” is not an admission that scientists know nothing.

Good science communication preserves the shape of the evidence even after removing the equations.

Keep the story.

But keep the error bars too.

Genetics and Psychology Took Different Roads Toward Reliable Science

Talk about reproducibility often collapses science into one giant category.

But the replication problem faced by a psychologist studying human behaviour is not identical to the problem faced by a geneticist conducting a genome-wide association study.

Different fields have different sources of noise, different instruments and different traditions.

The National Academies uses genetics and psychology to make exactly this point.

Genetics built a large-scale replication machinery

Genome-wide association studies scan enormous numbers of genetic variants for associations with disease or other traits.

That creates obvious statistical dangers.

With huge numbers of tests, chance associations are inevitable unless stringent safeguards are used.

The GWAS community responded by developing norms around:

large sample sizes,

international consortia,

quality control,

phenotype and genotype harmonization,

control of population stratification,

stringent multiple-testing thresholds,

independent replication,

and meta-analysis across cohorts.

This was not merely a statistical reform.

It was sociological.

Reliability increasingly depended on collaboration, infrastructure, data sharing and agreement about analytical standards.

Psychology studies a very different kind of system

Human behaviour is extraordinarily context-sensitive.

Researchers cannot control people with the same precision with which a physicist controls a laboratory instrument.

Psychology has consequently been at the centre of public discussion about a supposed “replication crisis.”

But the report is deliberately cautious.

There is no consensus that the field should be characterized as being in crisis.

Some researchers point to low power, p-hacking and inadequate distinctions between prior and post hoc hypotheses.

Others warn that framing psychology as broken may exaggerate the evidence, discourage risky research and overemphasize false positives at the cost of missing genuine discoveries.

Replication rates cannot simply be compared like cricket scores

The report reviewed replication efforts across fields ranging from genetics and chemistry to psychology and economics and found reported rates ranging from below one in five to above three in four.

But it explicitly warns that these numbers are not directly comparable because studies were selected differently and success was defined differently.

That is a crucial lesson.

The question should not be:

Which discipline has the highest replication percentage?

It should be:

What kind of system is being studied?

How much can its variables be controlled?

How noisy are the measurements?

What counts as successful replication?

And what practices would meaningfully improve reliability in that particular field?

Reproducibility has universal principles.

Its implementation is local.

What If a Journal Accepted Your Experiment Before You Knew the Result?

Scientific publishing usually happens backwards.

Researchers conduct a study.

They discover the result.

Then journals decide whether the study is interesting enough to publish.

That arrangement creates an obvious temptation.

Interesting outcomes become valuable.

Null outcomes become disappointing.

What if we reversed the chronology?

Enter the registered report.

Review the science before seeing the answer

In the model described by the National Academies, researchers prepare the parts of the manuscript that can be written before data collection:

the scientific rationale,

research question,

methods,

and planned analyses.

The journal peer-reviews that proposal.

If it is accepted, the researchers conduct the study and later submit the results.

The publication decision is therefore less dependent on whether the final answer looks exciting.

This attacks publication bias at its root.

The question becomes: Was this worth testing?

Conventional publishing can accidentally encourage:

“Interesting result, therefore interesting paper.”

Registered reports attempt to replace this with:

“Important question + rigorous design, therefore worthwhile study.”

That distinction is subtle but profound.

If the hypothesis is scientifically important, then discovering that it is unsupported is also scientifically useful.

But preregistration is not a halo

The National Academies carefully avoids treating preregistration as a universal cure.

Critics worry that mandatory preregistration could favour easily testable hypotheses and discourage open-ended exploration.

And the report makes an especially important point:

Poor ideas and weak methods can be preregistered too.

Preregistration records a plan.

It does not certify that the plan is intelligent.

Exploration and confirmation need different rules

Many important discoveries arise because scientists notice something unexpected.

Exploratory science needs room to wander.

Confirmatory science needs stronger protection against rewriting the story after seeing the outcome.

The real value of preregistration may therefore lie not in banning exploration, but in distinguishing the two.

There is nothing wrong with saying:

“We discovered this pattern unexpectedly.”

The problem begins when an unexpected pattern is written as though it had been predicted from the beginning.

Registered reports are one attempt to keep those narratives honest.