Statistics, data and evidence · Methods and tools
What does the replication crisis reveal about how statistics turns data into knowledge?
A common rule in research has been to call a result 'significant' if its p-value is below 0.05. When researchers tried to repeat well-known studies, many results did not hold up. This question asks what went wrong: the mathematics, the way it was used, or the system that rewarded it.
Claims
- The mathematics of significance testing is sound; the crisis came from misuse, such as testing many hypotheses and reporting only the ones that worked.
- Replication is how science corrects itself, so the crisis shows statistical methods working as intended over time.
Counterclaims
- A fixed threshold such as 0.05 invites people to treat a continuous measure of evidence as a yes-or-no verdict, so the method itself encourages error.
- Journals and careers reward surprising significant results, so the statistics cannot be separated from the incentives of the people using them.
Real-life situations from mathematics
Reproducibility Project: Psychology
In 2015 the Open Science Collaboration reported repeating 100 published psychology studies. Almost all the originals had reported significant results, but only about a third of the repeats did.
The ASA statement on p-values
In 2016 the American Statistical Association published an unusual formal statement warning that a p-value does not measure the probability that a hypothesis is true, and that scientific conclusions should not rest on whether it passes a single threshold.
Check dates and figures in a reliable source before you use them, and cite that source.
Use this in your TOK work
Essay. Very strong for titles about the reliability of the human or natural sciences, or about methods and trust.
Exhibition. A printed statistics table with 'p < 0.05' highlighted makes a pointed object for a prompt about how we decide what counts as evidence.
Link it to the prescribed title or the exhibition prompt you are working on, in your own words. See using maths examples in your essay and choosing exhibition objects.
Themes and study heading
Knowledge and the knower Knowledge and politics Methods and tools
How mathematical knowledge is produced and justified.