The Innuendo Machine: Turning scientific lead into journalistic gold

There is an important distinction between saying that something could be true and saying that it is true. This distinction is so important that it is usually marked in a language’s grammar, in a feature known as grammatical mood. Consequently, it is nearly impossible to speak about any anything without taking a stance on its existence. To speak in the realis mood is to make an assertion about the state of the world. When I say, “pigs fly,” (realis mood) I assert that the conditions that must hold to make this statement true actually do hold.   To speak in the irrealis mood is to talk about how things could or might be; such as when I say, “pigs could fly.” These latter irrealis statements have far less impact, because they only commit me to the possibility that something is true.  It’s the realis statements that grab your attention.

There is a magical social process that takes carefully hedged irrealis assertions from the scientific realm and transmutes them into unqualified realis assertions in the journalistic realm, a transmuting of scientific lead into journalistic gold.  I refer to this process as the Innuendo Machine, because for the reader at the end of the chain, the realis assertion is only justified by innuendo: The effect is real because it was published in a top journal by scientists at top institutions; therefore the evidence must have met conventional standards of scientific rigor.  The media hype surrounding a recent paper gives us revealing glimpses into how this machine operates.

abstract

This paper, which suggests that ideology can be communicated to potential sexual partners through body odor, appeared in the American Journal of Political Science, widely recognized as one of the top journals in the field.

Note the odd irrealis mood in the title, committing only to the possibility that people screen ideological compatibility through body odor.  But should top journals be in the business on reporting on things that could be possible?  It could be possible that people suss out ideologically compatible mates based on the lengths of their eyelashes, or telepathically.  Isn’t it the job of academic journals to report on things that might be possible in the world as we know it, and to reduce the uncertainty that stands between “might” and “is”?

In the abstract, the authors were careful to point out that they only “explored the possibility” for such a mechanism, and that they “observed that individuals are more attracted to their ideological concomitants [sic]”. Nowhere does one encounter any realis claim that people are attracted to mates of similar ideologies through body odor.

What is all this hedging about?  As some bloggers have already noted (here and here), the statistical evidence for the effect is very weak.  In the article itself, the authors do not report any p-values (except in the supplementary materials), only test statistics. And the main statistics for their findings from three analyses of the same data are t-values of 1.69, 1.48, and 1.45. The authors note that these t-values are in the right direction and are “consistent with their theoretical expectations.” These t-values correspond to two-tailed p-values of approximately .09, .14, and .15, or, to be extra generous, one-tailed values of .045, .070, and .075. So, the evidence for any relationship is extremely weak; but apparently, not too weak for the American Journal of Political Science.

So, how was this all reported in the media?  Here’s the Washington Post:

wapo

Pure realis mood. All hedging and qualification expunged.

Welcome to the Innuendo Machine

Let’s retrace some of the steps by which the Innuendo Machine operates.

An article’s Discussion section is a useful incubator for realis assertion.  In McDermott et al., this is where we see realis in its embryonic form (p. 5):

First, individuals find the smell of those who are more ideologically similar to themselves more attractive than those endorsing opposing ideologies…

Note that by saying individuals find instead of individuals found, the authors have stealthily set the stage for false generalization, by implying that their effect holds beyond the individuals in the study to a potentially larger population.  The Discussion section is also a useful place to get your audience comfortable with the reality of the effect, by avoiding discussion of alternative explanations and possible weaknesses.  It is far more effective to engage your reader with language that presupposes the effect’s existence.  One strategy is to plant a few media friendly cherry-picked anecdotes (p. 6):

In one particularly illustrative case, a participant asked the experimenter if she could take one of the vials home with her because she thought it was “the best perfume I ever smelled”; the vial was from a male who shared an ideology similar to the evaluator. She was preceded by another respondent with an ideology opposite to the person who provided the exact same sample; this participant reported that that vial had “gone rancid” and suggested it needed to be replaced. In this way, different participants experienced the exact same stimulus in radically different ways only moments apart.

It is also useful to discuss how the presumed mechanism it might operate in a complex world, which not only further presupposes the existence of the phenomenon, but also gives you the opportunity to cite prominent researchers in your field.

As editors and reviewers of such a paper, you might want to set aside any nitpicky qualms about extraordinary claims requiring extraordinary evidence, because by engaging in such skepticism you are probably discouraging creativity and squelching innovation.  No one likes a boring spoilsport!  Evaluate the paper on the basis of how potentially mind-blowing the idea would be (assuming it’s true), the reputation of the authors, and how quickly you would run down the hall to tell your colleagues about the finding (assuming it’s true).

Is the paper accepted yet?  It is?

Next stage: Press release!

The press release is where the rubber really hits the realis.  It is the key cog in the Innuendo Machine.  For publications in top journals, the press release is usually taken care of by the publisher (in this case, Wiley), who are typically more than willing to sex-up the claims to a sufficient level of virality.  For instance, they might say that your finding suggests that

…one of the reasons why so many spouses share similar political views is because they were initially and subconsciously attracted to each other’s body odor.

Ah, “because”, beautiful because!  Ah, the allure of the subconscious mind!  Ah, the thousand domestic mini-dramas that will ensue among spouses with incompatible politics or incompatible body odors!

In contrast to the usual narrative in which a scientist’s cautious claims are exaggerated by the media, here we see it come about through the collusion of:

  • the authors (overgeneralizing claims in the article and in the press release)
  • the editor (giving the paper the journal’s seal of approval despite shaky evidence)
  • the publisher (sexing up the findings for public consumption)

But the Media Puts the Icing on the Cake

As noted above, none of the effects in the article reached statistical significance, at least as conventionally defined.  Gelman even praises the editor for not relying on conventional significance levels (though in fairness, he does think that the article is bunk).  But apparently, the media hasn’t gotten the memo about The New Statistics.  Here’s the Washington Post:

smallbutsignificant

I suppose that is correct, if by “significant” you mean p<.1, one-tailed.  Innuendo in action.

Advertisements

4 thoughts on “The Innuendo Machine: Turning scientific lead into journalistic gold

  1. The subtle slip from “individuals found” to “individuals find” is of course absolutely necessary to the sexiness (no pun intended) of the findings. Generally this kind of thing reaches a peak in the last paragraph of the Discussion section.

    Then, someone does a null replication (of course, in this case, that is going to have to be a very, very null replication indeed, as the bar for insignificance has been set pretty high), and the original authors run around saying “Oh, well, perhaps the effect doesn’t obtain in or among “. They will not, however, issue a correction to the gee-whiz conclusions of their first article. Cake eaten? Check. Cake sitting right here in front of me? Check.

    In this specific case, there is also the huge problem of the implication that humans evolved in a way that just happens to favour the detection of socially-constructed polar opposites in the abstract domain of politics in one country with (basically) a binary system. If tomorrow a third party were to emerge that regularly took 25% of the vote in U.S. elections, what would that imply for these findings? The whole thing is utterly absurd on its face.

    • Getting published in psychology or other social science fields is basically irreversible and permanent. As you well know, a paper that has been accepted but that is later shown to be full of errors will never be retracted; it is even unlikely that it will even be corrected. All of this makes a mockery of the notion of a “scientific body” of accepted findings. What is to be done about this?

      I am sympathetic to the view that we should basically publish anything that seems like a sound effort regardless of whether it is significant or even newsworthy. The act of publishing and archiving research should be separated from the scientific evaluation of that research. I like Tal Yarkoni’s ideas about a Stack Overflow style of reputation system for researchers and papers. Under this view, papers like the one above would get voted down by people with high reputation, and perhaps this would make media outlets more reluctant to report on the research (perhaps media outlets themselves could get voted down by scientists for covering bad research). And the reputation of researchers who publish bad papers would suffer, possibly motivating them to retract papers that are lowering their scores.

      But right now, we seem to be in an unhappy transition stage where some editors are reluctant to hold to traditional standards of significance (who wants to seem like a stickler for p-values?) but without any change whatsoever in editorial policies favoring newsworthiness. This makes it even easier for weak but flashy papers to get published in top journals, while the media and lay public remain blissfully unaware that standards are on a downward slide.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s