The P-Value Was Never Supposed to Be a Truth Machine

Few numbers in science have acquired as much psychological power as 0.05.

A result at p = 0.049 can feel publishable.

A result at p = 0.051 can feel disappointing.

Yet nature does not know that somebody drew a line at 0.05.

The National Academies report devotes considerable attention to the misuse of statistical significance.

A p-value does not tell us the probability that a scientific hypothesis is true.

It is calculated under an assumption about the null hypothesis and describes the compatibility of the observed data with that framework.

Reducing an entire study to significant versus non-significant therefore throws away enormous amounts of information.

What was the effect size?

How precise was the estimate?

How plausible was the hypothesis beforehand?

Were multiple hypotheses tested?

How large was the sample?

What assumptions were built into the model?

What uncertainty accompanies the result?

The report notes that judging replication simply by asking whether both studies cross the same statistical-significance threshold can be unreliable. Researchers should instead examine distributions, estimated effects, variability and uncertainty.

There is another fascinating tension.

A scientific field could achieve excellent replication rates by studying only extremely predictable phenomena.

But science also exists to discover things we do not already know.

The report points out that bold, low-prior-probability hypotheses are naturally less likely to replicate than safe hypotheses. Yet some transformative discoveries necessarily begin as improbable ideas.

The goal, therefore, cannot be to engineer science so that every experiment succeeds.

A perfectly predictable scientific enterprise would be reliable.

It might also be spectacularly boring.

Good science needs both exploration and confirmation.

The challenge is knowing which mode we are in.

########################################################################

Further reading: This post is based on Reproducibility and Replicability in Science, a consensus study report of the National Academies of Sciences, Engineering, and Medicine. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. DOI: 10.17226/25303.