Here on The Value Perspective, we consider ourselves no slouches when it comes to poking fun at poor medical research (Jellybean Trilogy I, say), poor financial research (Jellybean Trilogy II) and poor statistical science (Jellybean Trilogy III) – not to mention the unquestioning way the media can lap up so-called academic studies. Compared with John Bohannon, however, it appears we are mere amateurs.
Even if you have not heard of Bohannon, you may well have seen some of the news stories generated around the globe by his study into how eating chocolate can aid weight loss. They appeared in the likes of the Daily Star in the UK, the Irish Examiner, the Times of India, German tabloid Bild and websites for Cosmopolitan and the Huffington Post as well as on US and Australian television shows.
Unfortunately for dieters everywhere, the study was a spoof. Though real enough, it was based – deliberately – on the same less than rigorous scientific and statistical processes we discussed in The Jellybean Trilogy. A mere 15 subjects were randomly split between a control group, a group on a specific diet and a group on the same diet plus a bar of chocolate and then monitored for three weeks.
All the subjects were tested for a variety of different responses and then Bohannon and his fellow researchers set about crunching the numbers to find some hook – any hook – on which they could hang a study. As we saw in The Jellybean Trilogy, if you test a small enough number of people for a large enough number of things, you are bound to turn up something – and so they did.
In fact, given the insignificant number of people tested, the researchers found quite a few ‘statistically significant’ points, including that the chocolate-eaters had healthier cholesterol levels, increased wellbeing and accelerated weight loss. Remember – to be of statistical significance, scientists must, broadly speaking, be able to show any new research has lower than 1-in-20 odds of merely being chance. Statistical significance is by no means the same thing as ‘being a fact’ or ‘having a significant impact’.
Once they had found their hook, Bohannon and the others set about releasing the news to the world. The first step here was to have their research reviewed in a number of academic journals. To explain the speed with which they managed to see this done, it may be helpful to explain the alternative business models on which scientific journals tend to operate these days.
Under the more established model, the readers pay for the journal and scientists have their research published without charge, while a newer innovation has seen the journal distributed free to readers but scientists paying up to have their research included. Glossing swiftly over whether that reminds you of any aspect of the world of financial services, can you guess which route Bohannon and co took?
Within a few weeks of their credit card being charged, the research began appearing in scientific journals, always intact and never peer-reviewed, and so it was time for the next stage – attracting the attention of the world’s media. This required a suitably worded press release and Bohannon – a journalist as well as a PhD (molecular biology of bacteria, not humans) – knew just what to do.
“The key is to exploit journalists’ incredible laziness,” he writes in I fooled millions into thinking chocolate helps weight loss. Here's how “If you lay out the information just right, you can shape the story that emerges in the media almost like you were writing those stories yourself. In fact, that’s literally what you’re doing, since many reporters just copied and pasted our text.”
Ouch. We have already mentioned some of the places the story appeared but nowhere did any journalist spot the holes in the research process, rarely was a word changed from the press release and, even in the handful of instances when fact-checkers became involved, only a few cursory questions were asked rather than any attempt made to understand the study.
The only ray of light in this otherwise rather unedifying tale is that, while the world’s press may have been caught hook, line and sinker, plenty of participants in associated online discussion forums were not. They were not slow to ask the questions the journalists ought to have done, with the general air of scepticism being summed up by the line: “Every day is April Fools’ Day when it comes to nutrition.”
Read enough health-related news articles and eventually you could find that anything – salt, red meat, red wine, reading health-related news articles – will either kill you or help you live for ever. And, as likely as not, these stories will have been inspired by research that – like Bohannon’s – was not based on real science and rigorous statistics.
There are a number of lessons we might take here about academic research that, as we saw in The Jellybean Trilogy, apply equally to the financial and medical varieties. They include – if a study does not list how many people took part in it, ignore it; and, if a study makes a bold claim that is statistically significant but does not say how big the effect size is, ignore it.
And just because research uses scientific jargon and has appeared in scientific journals does not mean you should not take it with a huge pinch of salt – or a large steak and a big glass of Malbec