When researchers evaluate a piece of work, we often compress several different judgments into one vague impression: This is a strong paper. Or: This is not very impressive.
But what exactly does “strong” mean?
A paper can be highly novel and poorly executed. It can be technically excellent but scientifically unimportant. It can contain an important result presented terribly. It can require extraordinary technical skill while contributing almost no new idea. And sometimes a beautifully written paper is mistaken for a deeper contribution than it really contains.
A more useful way to think about research is to separate at least five axes:
Novelty, Skill, Quality, Significance, and Presentation.
These axes are related, but they are not interchangeable. More importantly, they can be aligned in very different ways.
Novelty: How new is the idea?
Novelty asks whether the work introduces something genuinely new.
This might be a new hypothesis, method, dataset, experimental setup, theoretical result, empirical observation, or way of framing an old problem.
But novelty by itself says surprisingly little about the value of the work.
Imagine a paper proposing an entirely new neural architecture inspired by the branching pattern of river deltas. Nobody has tried this before. The idea is unquestionably novel.
Now suppose the architecture performs worse than standard baselines, the motivation is weak, and the authors do not establish why the design should matter.
Its novelty may be very high.
Its significance may be low.
Its quality may also be low if the experiments are incomplete.
Novelty therefore answers:
“Has this been done before?”
It does not answer:
“Was this worth doing?”
Skill: How difficult was the work to carry out?
Skill concerns the technical competence, intellectual effort, engineering ability, mathematical sophistication, experimental craftsmanship, or domain expertise required to produce the work.
This dimension is frequently confused with significance.
Consider a team that reimplements a very complicated multimodal model across twenty datasets, builds a large distributed training infrastructure, carefully reproduces hundreds of experiments, and conducts an enormous ablation study.
The work may require exceptional skill.
But suppose the scientific conclusion is simply:
“The previously reported result also holds on more datasets.”
That may still be useful work, but the skill involved in executing it does not automatically make the scientific contribution profound.
The reverse also occurs.
A researcher may notice an extremely simple relationship between two variables that everyone else overlooked. The experiment takes a few hours. The mathematics fits on half a page.
Yet the observation changes how an entire community thinks about a problem.
The technical skill required to execute the experiment may be modest, while the significance is enormous.
Scientific importance is not proportional to GPU hours, lines of code, theorem length, or experimental pain.
Sometimes the hardest part of research is noticing the right question.
Quality: How well was the research actually done?
Quality concerns the reliability and rigor of the work.
Are the experiments properly controlled?
Are the baselines appropriate?
Are the statistical conclusions justified?
Are the assumptions clear?
Can the result be reproduced?
Are alternative explanations considered?
A paper can be highly novel but low quality.
For example, imagine a fascinating claim:
“A particular visual representation spontaneously encodes causal structure.”
That would be highly novel and potentially significant.
But if the claim is supported by one dataset, one random seed, weak baselines, and a questionable evaluation metric, the research quality remains low.
The opposite is equally possible.
A replication study may contain almost no novelty. Yet it may be executed exceptionally well, with careful controls, large samples, preregistered analyses, multiple datasets, and transparent reporting.
Novelty: low.
Quality: extremely high.
And depending on what it discovers, significance could still be high.
Significance: If this is true, how much does it matter?
Significance is perhaps the most difficult axis to judge.
A contribution can be technically correct, novel, and beautifully executed while still being unimportant.
Suppose a paper improves accuracy from 91.73% to 91.79% on a benchmark.
The method is new. The experiments are rigorous. The implementation is excellent.
But does the improvement change anything?
Perhaps not.
Compare this with a paper showing that a widely used benchmark contains a systematic flaw that invalidates conclusions from hundreds of previous papers.
The method for discovering the problem might be simple.
But the significance is substantial because the result changes what researchers should believe.
Significance therefore concerns consequences.
Does the work change:
what we know?
what we can do?
what we should measure?
what assumptions we should stop making?
what research direction becomes possible?
what previously accepted conclusion must be reconsidered?
This is why significance and novelty often diverge.
A completely new idea may be insignificant.
A seemingly modest observation may be transformative.
Presentation: How well can the reader understand the work?
Presentation concerns writing, figures, organization, notation, argument structure, and the ability to communicate the contribution.
This dimension exerts an enormous psychological influence on reviewers.
A polished paper feels more convincing.
A confusing paper feels weaker.
Yet neither reaction necessarily reflects the underlying science.
Consider two papers containing essentially the same quality of research.
Paper A has a clear introduction, one excellent conceptual figure, carefully motivated experiments, readable tables, and precise claims.
Paper B contains the same underlying ideas but buries the contribution in notation, introduces terminology unnecessarily, uses unreadable plots, and never clearly states what was learned.
Paper A will almost certainly be perceived as stronger.
That is not entirely unreasonable. Communication is part of scientific work.
But presentation should still be distinguished from the underlying scientific contribution.
A poorly presented important discovery remains an important discovery.
A beautifully presented trivial result remains trivial.
The interesting cases occur when the axes disagree
The most revealing papers are often those where the five axes point in different directions.
Consider several combinations.
High novelty, high skill, low significance
A researcher creates an extremely sophisticated optimization algorithm unlike anything previously published. The mathematics is difficult and the implementation is impressive.
After extensive experiments, however, it performs approximately the same as standard optimizers.
This may be novel and technically demanding, but its practical or scientific significance may remain limited.
Low novelty, high quality, high significance
A large replication study carefully reevaluates a celebrated result and finds that the effect disappears under proper controls.
There may be little methodological novelty.
Yet the work could be extremely important because it changes what the field believes.
Low skill, high novelty, high significance
Someone notices that a benchmark label accidentally leaks into an input feature.
Discovering the issue may require only a few lines of code.
But if hundreds of papers rely on that benchmark, the scientific significance is enormous.
The insight matters more than the complexity of the implementation.
High skill, high quality, low novelty
A team conducts an extraordinary engineering effort to reproduce a difficult system at scale.
Everything is carefully done.
But the conceptual contribution is primarily confirmation.
Such work can be valuable infrastructure or validation without being highly novel.
High novelty, low quality, potentially high significance
A paper proposes a radical hypothesis and provides intriguing preliminary evidence.
The experiments are not yet convincing.
This kind of work creates a difficult reviewing problem.
Should reviewers reward the originality and potential impact, or penalize the incomplete evidence?
Different venues and reviewers answer that question differently.
High significance, poor presentation
Occasionally an important idea appears in a paper that is extremely difficult to read.
The terminology may be confusing, the figures weak, and the argument disorganized.
A careful reviewer may nevertheless recognize that something important is hiding underneath.
A rushed reviewer may not.
Excellent presentation, modest everything else
There are also papers that are exceptionally easy to read, beautifully illustrated, and narratively elegant.
The reader finishes the paper with a strong impression.
But after stripping away the presentation, the contribution may amount to a small variation of existing work.
Presentation can amplify perceived contribution, sometimes dramatically.
Why peer review becomes difficult
Peer review is difficult partly because reviewers are rarely asked to score these dimensions independently.
A typical review form might ask about originality, technical quality, clarity, and significance, but the final decision often collapses everything into a single score:
Accept or reject.
This compression creates disagreements.
One reviewer may prioritize novelty.
Another may prioritize rigor.
Another may care primarily about whether the work will influence the field.
Another may strongly penalize unclear writing.
Two reviewers can therefore agree completely about the facts of a paper while disagreeing about its score.
For example:
Reviewer A:
“The method is incremental, but the experiments are extremely thorough and the empirical finding is important.”
Reviewer B:
“The experiments are strong, but there is insufficient methodological novelty for this venue.”
These reviewers may not actually disagree about the paper.
They disagree about the weighting function.
In simplified form, every reviewer implicitly computes something like:
w_N N
+
w_S S
+
w_Q Q
+
w_I I
+
w_P P
]
where (N) is novelty, (S) is skill, (Q) is quality, (I) is significance, and (P) is presentation.
The interesting part is that the weights (w_N,\ldots,w_P) differ between reviewers, venues, disciplines, and even years.
Peer review disagreement is therefore not always evidence that reviewing is random.
Sometimes reviewers are measuring different things.
What should peer review actually do?
A useful review should separate diagnosis from verdict.
Instead of saying:
“This paper is weak.”
the reviewer should ideally explain:
“The empirical methodology is strong, but the conceptual novelty is limited because the central mechanism closely resembles X and Y. The main remaining question is therefore whether the new empirical finding is significant enough to justify publication.”
That is far more informative.
Likewise:
“The idea is highly original and potentially important, but the current evidence does not yet support the central claim because the comparison against Z is missing.”
Now the authors know exactly where the problem lies.
A good review maps the paper onto the relevant axes before collapsing them into an overall recommendation.
What does the length of a review tell us?
Surprisingly little.
Review length is another quantity that researchers often misinterpret.
A three-page review is not necessarily more thoughtful than a six-paragraph review.
A short review can identify the decisive issue immediately:
“The claimed improvement comes entirely from using additional training data that the baselines do not receive. Under matched data, the reported advantage disappears.”
That may be the only issue that matters.
Conversely, a long review may contain twenty minor comments about notation, figure captions, hyperparameters, wording, and formatting while never engaging with the central scientific claim.
Length is therefore not the same as depth.
The relationship between review length and the five axes is more subtle.
A quality problem often produces a long review because methodological concerns require explanation. Missing controls, data leakage, statistical issues, baseline mismatches, and experimental confounds may each require careful discussion.
A novelty problem can sometimes be explained very briefly:
“The proposed loss is mathematically equivalent to the objective introduced in Smith et al.”
If that observation is correct, five sentences may outweigh two pages of other comments.
A significance concern may require substantial argument because significance is contextual. The reviewer may need to explain why the measured improvement does or does not change existing understanding.
A presentation problem often produces many small comments. The resulting review may become long even when the underlying science is relatively sound.
A skill assessment rarely needs much space at all. Reviewers can often recognize that a system was technically difficult to build, but the more important question remains what was learned from building it.
Thus review length can be almost orthogonal to review importance.
One devastating conceptual observation may fit in a paragraph.
Twenty cosmetic suggestions may occupy two pages.
The asymmetry of peer review
There is another reason review length can be misleading.
Positive judgments are often compact.
A reviewer might write:
“The experimental design is rigorous, the central result is convincing, and the work addresses an important unresolved question.”
There may simply be less to diagnose.
Negative judgments often require justification.
If a reviewer recommends rejection, especially for a strong-looking paper, they may feel obligated to document the weaknesses carefully.
This creates a strange asymmetry:
A longer review can sometimes indicate that the reviewer engaged deeply with the paper.
But it can also indicate that the reviewer found many problems.
Or that the reviewer is verbose.
Or that the paper was confusing.
Or merely that the reviewer had more time.
The number of words alone tells us almost nothing.
A better mental model for reading reviews
Authors often read reviews emotionally as a single signal:
Do they like the paper?
A more productive approach is to classify every substantial reviewer comment according to the axis it concerns.
For example:
“Similar to prior work X.”
→ Novelty.
“Only one random seed is reported.”
→ Quality.
“The improvement is less than 0.2% and does not change the practical conclusion.”
→ Significance.
“The architecture requires a complicated distributed implementation.”
→ Skill.
“The main contribution is difficult to identify until Section 4.”
→ Presentation.
Once comments are separated this way, patterns become much clearer.
Three reviewers may appear to disagree dramatically while actually saying:
Reviewer 1: Novelty is moderate, quality is excellent.
Reviewer 2: Novelty is moderate, significance is high.
Reviewer 3: Novelty is insufficient for this venue.
The disagreement may therefore concern only one question:
How much novelty should this venue require?
That is a much more precise disagreement than “reviewers were inconsistent.”
Papers are vectors, not scores
Perhaps the simplest way to express all of this is to stop imagining research quality as a number.
A paper is better represented as a vector:
[
(\text{Novelty},\text{Skill},\text{Quality},\text{Significance},\text{Presentation})
]
One paper might look like:
[
(9, 8, 5, 6, 7)
]
Another:
[
(3, 7, 10, 9, 8)
]
Which paper is “better”?
There is no universal answer.
A theory conference, systems conference, journal, workshop, industry venue, or funding panel may value these vectors differently.
This is also why apparently contradictory publication decisions can sometimes make sense. Different venues are not merely applying different thresholds. They may be projecting the same multidimensional object onto different axes.
The deeper lesson
Scientific evaluation becomes healthier when we stop treating words such as novel, rigorous, important, technically impressive, and well written as synonyms.
They describe different properties.
Great research often scores well on several of them simultaneously, but it does not have to.
Some of the most important scientific results were conceptually simple.
Some technically magnificent projects produced modest scientific insight.
Some highly original ideas failed because the evidence was weak.
Some excellent research struggled because nobody could understand the paper.
And some beautifully presented papers made small ideas feel much larger than they were.
Peer review works best when reviewers make these distinctions explicit.
The purpose of a good review is not merely to produce a verdict.
It is to explain the shape of the work.
And once we see research as having a shape rather than a single score, disagreements about papers become much easier to understand.
No comments:
Post a Comment