With cases, deaths, reproduction numbers, opinion polls and more, we get bombarded with statistics every day. But how can you spot a naughty number, a shabby statistic or dubious data? Lists of questions have been given by Tim Harford, Dave and Tom Chivers, and in The Art of Statistics (which David wrote), with considerable overlap as they grapple with the same essentials. Here is the short list that we use ourselves.
The first question: how trustworthy is the source of the story? Are they honest, competent and reliable or are they trying to manipulate my emotions, say by provoking anxiety or reassurance? Are they giving the whole picture or selecting bits to suit themselves? Like the review that found insects had declined, but it turned out had only searched for studies that showed a decline.
Second: how reliable is the statistic? A count may not be counting what we think it is: Covid-19 cases only represent those who decide to have viral tests, often after showing symptoms, and are becoming increasingly distinct from infections. Survey estimates seek to generalise from a sample to a target population, but have both margins of error and possible systematic biases, while model outputs reflect assumptions and selected scenarios. The uncertainty and the quality of the underlying evidence should be communicated with due humility.
Third: how trustworthy is any claim being made on the basis of the data? Context is all; a “100% increased risk” may make the headlines, but twice not very much is still not very much. Historical statistics can provide a guide to whether something really is a big number. Differences may be attributed to cause rather than just correlation, since observational analyses can have systematic biases or lingering confounders. And sometimes there is basic misunderstanding of good data; ONS found that, among people testing positive, vaccination increased the chance someone would be Omicron rather than Delta, but this led to the claim that vaccination increased the risk of testing positive.
Statistics enlighten our uncertain world, helping people make decisions. We need thoughtful consideration, not reflexive cynicism, when we see claims derived from statistical evidence.
This article was amended on 27 December 2021. An earlier version said that “Covid-19 cases only represent those who decide to have PCR tests”. In fact the reference should have been to viral tests to reflect the inclusion of positive tests from lateral flow devices.