As Science of Us frequently notes, there’s a lot of bad science — and a lot of bad science reporting — out there. Oftentimes the two reinforce one another: Scientists will overstate or misinterpret something they’ve found (intentionally or, more often, as a result of unconscious bias), their university’s PR team will write an overhyped press release, and then journalists at headline-hungry media outlets compound the error by publicizing the “result” without actually understanding it or reading the underlying research itself.
In a funny, playful response to this, the economist Jonathan Falk and the statistician Andrew Gelman presented a paper (PDF) at the International Conference on Machine Learning in New York City on Friday — one “reporting” a connection between the rise of Donald Trump and professional bridge players’ strategies.
Let’s take it to the abstract:
How are people unconsciously influenced by the rise of Donald Trump? We test the theory that Trump’s rise has irrationally changed the behavior of one group of people: elite bridge players, whom we assume are otherwise completely typical. We examine the hands played in one of the premier North American bridge events, the Vanderbilt Knockout Tournament, in 1999 and 2015. We find that players had significantly higher probabilities of making No Trump contracts in the 2015 period compared with the earlier periods. We conclude that in the latter period, defending players are subtly deranged by the prospect of Trump and play their hands worse. By contrast, a 2015 European tournament shows no significant difference with the earlier 1999 tournament. This strengthens our conclusion.
In the abstract and throughout his paper, Falk and Gelman are taking aim at a variety of headline-ready research that tends to fall short in two ways. First, it fishes for interesting science that’s of only borderline statistical significance. The co-authors allude to this with a not particularly subtle reference to P values, which measure the likelihood that an event occurred by chance rather than because of a meaningful statistical relationship. P values below .05 are generally considered statistically significant. “When we look at the percentage of No Trump contracts made, the Vanderbilt 2015 tournament is significantly higher than the Vanderbilt 1999 tournament,” they write. “The t-test yields a p-value of 0.0492, easily passing the traditional significance level of 5 percent.” “Easily,” indeed.
Having established a “significant” statistical relationship, this type of research then spins an elaborate, mostly unfalsifiable story out of that finding.
Falk and Gelman again:
It would be natural to attribute the findings of this paper to a direct response to the candidacy of Donald Trump; however we must consider other pathways as well, as it is well understood from social psychology that seemingly trivial inputs such as football games, and subliminal images, and shark attacks can be more important then actual policy positions when affecting political attitudes. An example is the well-established finding that visual contrast polarizes moral judgment (Zarkadi and Schnall, 2013), which was later reinterpreted in terms of latent associations of chessboard patterns with Russia, and thus foreign policy (Gelman, 2016).
In the present example, the relevant indirect pathway comes through the well-established principle of embodied cognition (e.g., Bargh, Chen, and Burrows, 1996). In the political context the word Bridge triggers immediate association with the recent “Bridgegate” scandal in New Jersey, a sequence of events closely tied to Chris Christie, the New Jersey governor, former presidential candidate, possible future Secretary of Transportation, and the first major Republican leader to endorse Donald Trump (e.g., Diamond et al., 2016). [footnotes deleted]
Makes perfect sense! And it’s just too exciting a finding to be ignored. The authors note that they “expect that after these results appear in NPR, TED, Gladwell, and the prestigious Proceedings of the National Academy of Sciences, there will be pushback from the inevitable replication bullies, those uncreative types who seem to exist only to criticize.”
So the next time you’re reading coverage of a scientific study that feel a bit … much, maybe pull up Gelman’s fake paper to check for similarities.
