Every week, there’s a new study out proclaiming how some beloved consumable — beer, honey, wine, coffee, chocolate — is either Good For You or Bad For You. A new survey from the Pew Research Center indicates that most Americans take that to be a sign that science is constantly advancing, and thus studies come into conflict. But here’s a sobering thought: Maybe it’s the studies themselves that create the conflict.
This is a point raised by statistician Andrew Gelman, who highlights some savvy analysis from the New York Times’ Aaron Carroll. The problems with nutrition studies, Carroll contends, start in their methodology. A representative case came in the form of a paper in the Journal of Nutrition from last year that found that honey, sugar, and high-fructose corn syrup were all basically equivalent, healthwise. But here’s the thing: That study followed just 55 people over two weeks on each sweetener. Yes, Carroll reasoned, glucose and insulin levels were similar across the three, but how much can you know about nutritional truths from a fortnight-length sample?
Unfortunately, the sticky situation presented by the honey study is more rule than exception.
When it comes to nutrition, Carroll says, just about everything “we ‘know’ is based on small, flawed studies” that deliver limited conclusions that then get the hard sell by researchers (who need to publish to advance their careers) and bloggers (who lean on emotion-baiting stories to get shares that get the traffic that give them job security).
Even a 2011 review of studies on artificial sweeteners suffered from poorly designed research: Of 53 randomized control trials (the most rigorous way to find cause and effect), just 13 lasted for over seven days and had ten or more participants. Even worse, Carroll notes, participants knew which sweetener they were getting, further skewing results. “Think about that,” he warns. “This is the sum total of evidence available to us. These are the trials that allow articles, books, television programs and magazines to declare that ‘honey is healthy’ or that ‘high-fructose corn syrup is harmful.’”
Compared to pharmaceutical research, nutrition has the deck stacked against it even further. There’s human behavior, in that pills are easy to take or not take, but it’s notoriously difficult to get people to change their eating habits, even for a study. Then there’s money: Pharma is worth over a $1 trillion globally and the sales of their product depend on getting approval, which in the U.S. requires lots of trials, so the studies (and their conclusions) are more robust. Not entirely dissimilar from the structures that pushed along fake news on Facebook, nutrition news’ serial contradiction is what happens when flimsy research is paired with click-o-centric media. This is 2016: Who cares about truth, anyway?