There is an important distinction between saying that something could be true and saying that it is true. This distinction is so important that it is usually marked in a language’s grammar, in a feature known as grammatical mood. Consequently, it is nearly impossible to speak about any anything without taking a stance on its existence. To speak in the realis mood is to make an assertion about the state of the world. When I say, “pigs fly,” (realis mood) I assert that the conditions that must hold to make this statement true actually do hold. To speak in the irrealis mood is to talk about how things could or might be; such as when I say, “pigs could fly.” These latter irrealis statements have far less impact, because they only commit me to the possibility that something is true. It’s the realis statements that grab your attention.
There is a magical social process that takes carefully hedged irrealis assertions from the scientific realm and transmutes them into unqualified realis assertions in the journalistic realm, a transmuting of scientific lead into journalistic gold. I refer to this process as the Innuendo Machine, because for the reader at the end of the chain, the realis assertion is only justified by innuendo: The effect is real because it was published in a top journal by scientists at top institutions; therefore the evidence must have met conventional standards of scientific rigor. The media hype surrounding a recent paper gives us revealing glimpses into how this machine operates.
This paper, which suggests that ideology can be communicated to potential sexual partners through body odor, appeared in the American Journal of Political Science, widely recognized as one of the top journals in the field.
Note the odd irrealis mood in the title, committing only to the possibility that people screen ideological compatibility through body odor. But should top journals be in the business on reporting on things that could be possible? It could be possible that people suss out ideologically compatible mates based on the lengths of their eyelashes, or telepathically. Isn’t it the job of academic journals to report on things that might be possible in the world as we know it, and to reduce the uncertainty that stands between “might” and “is”?
In the abstract, the authors were careful to point out that they only “explored the possibility” for such a mechanism, and that they “observed that individuals are more attracted to their ideological concomitants [sic]”. Nowhere does one encounter any realis claim that people are attracted to mates of similar ideologies through body odor.
What is all this hedging about? As some bloggers have already noted (here and here), the statistical evidence for the effect is very weak. In the article itself, the authors do not report any p-values (except in the supplementary materials), only test statistics. And the main statistics for their findings from three analyses of the same data are t-values of 1.69, 1.48, and 1.45. The authors note that these t-values are in the right direction and are “consistent with their theoretical expectations.” These t-values correspond to two-tailed p-values of approximately .09, .14, and .15, or, to be extra generous, one-tailed values of .045, .070, and .075. So, the evidence for any relationship is extremely weak; but apparently, not too weak for the American Journal of Political Science.
So, how was this all reported in the media? Here’s the Washington Post:
Pure realis mood. All hedging and qualification expunged.
Welcome to the Innuendo Machine
Let’s retrace some of the steps by which the Innuendo Machine operates.
An article’s Discussion section is a useful incubator for realis assertion. In McDermott et al., this is where we see realis in its embryonic form (p. 5):
First, individuals find the smell of those who are more ideologically similar to themselves more attractive than those endorsing opposing ideologies…
Note that by saying individuals find instead of individuals found, the authors have stealthily set the stage for false generalization, by implying that their effect holds beyond the individuals in the study to a potentially larger population. The Discussion section is also a useful place to get your audience comfortable with the reality of the effect, by avoiding discussion of alternative explanations and possible weaknesses. It is far more effective to engage your reader with language that presupposes the effect’s existence. One strategy is to plant a few media friendly cherry-picked anecdotes (p. 6):
In one particularly illustrative case, a participant asked the experimenter if she could take one of the vials home with her because she thought it was “the best perfume I ever smelled”; the vial was from a male who shared an ideology similar to the evaluator. She was preceded by another respondent with an ideology opposite to the person who provided the exact same sample; this participant reported that that vial had “gone rancid” and suggested it needed to be replaced. In this way, different participants experienced the exact same stimulus in radically different ways only moments apart.
It is also useful to discuss how the presumed mechanism it might operate in a complex world, which not only further presupposes the existence of the phenomenon, but also gives you the opportunity to cite prominent researchers in your field.
As editors and reviewers of such a paper, you might want to set aside any nitpicky qualms about extraordinary claims requiring extraordinary evidence, because by engaging in such skepticism you are probably discouraging creativity and squelching innovation. No one likes a boring spoilsport! Evaluate the paper on the basis of how potentially mind-blowing the idea would be (assuming it’s true), the reputation of the authors, and how quickly you would run down the hall to tell your colleagues about the finding (assuming it’s true).
Is the paper accepted yet? It is?
Next stage: Press release!
The press release is where the rubber really hits the realis. It is the key cog in the Innuendo Machine. For publications in top journals, the press release is usually taken care of by the publisher (in this case, Wiley), who are typically more than willing to sex-up the claims to a sufficient level of virality. For instance, they might say that your finding suggests that
…one of the reasons why so many spouses share similar political views is because they were initially and subconsciously attracted to each other’s body odor.
Ah, “because”, beautiful because! Ah, the allure of the subconscious mind! Ah, the thousand domestic mini-dramas that will ensue among spouses with incompatible politics or incompatible body odors!
In contrast to the usual narrative in which a scientist’s cautious claims are exaggerated by the media, here we see it come about through the collusion of:
- the authors (overgeneralizing claims in the article and in the press release)
- the editor (giving the paper the journal’s seal of approval despite shaky evidence)
- the publisher (sexing up the findings for public consumption)
But the Media Puts the Icing on the Cake
As noted above, none of the effects in the article reached statistical significance, at least as conventionally defined. Gelman even praises the editor for not relying on conventional significance levels (though in fairness, he does think that the article is bunk). But apparently, the media hasn’t gotten the memo about The New Statistics. Here’s the Washington Post:
I suppose that is correct, if by “significant” you mean p<.1, one-tailed. Innuendo in action.