Truth in Advertising

As I indicated in my previous post, it is not easy to estimate beforehand to which situations your conclusions generalize. But it is important to at least  make an effort. Often, conclusions are wildly oversold, creating a paradoxical situation when the results are not replicated. Usually, a hidden moderator is invoked and all of a sudden the scope of conclusions previously advertised as far-reaching is drastically narrowed. An example of this arrived in my mailbox the other day.

A while ago, our registered replication report of Hart & Albarracin (2011) came out. I already blogged about it. The first author, William Hart, has now written a response to our report. It will appear in the next issue of Perspectives on Psychological Science; I have an advanced copy of the response.

Hart doesn’t raise substantive concerns about our report but he does suggest that maybe we didn’t replicate the original findings because the original study was run in Florida; he doesn't specify where, but I'm assuming Gainesville. Most of the studies in our replication were conducted with what he views as more liberal samples.

A number of issues are relevant here. For example, how strong were the original findings in the first place? Another question is how predictive the conservativeness of a county is of the conservativeness of the student body at a university situated in that county. Large universities attract students from all over the country and the way I understand it--political scientists might want to correct me here--students typically vote in their home state. 

I'm going to ignore these questions here because this post is about truth in advertising. Did the original study warn us that the conclusions would only hold for conservative samples? The answer is simple. No, it didn’t. All we get is this.

This doesn't even tell us the university that these 48 students were from, let alone how conservative they are. At least we now have Hart's response, which has narrowed our location down to the Sunshine State. Clearly, at the time the authors didn't think the geographical location of the student sample (let alone its conservativeness) was worth mentioning.

But the discrepancy between truth and advertising is even larger. Here are the article's conclusions.

Rather than alerting the reader that the effect is limited to conservative student samples, this statement suggests that the findings might generalize from the lab to the courtroom!

Am I being fair here? After all, isn't it normal scientific progress when later research finds the limitations of earlier findings? Yep, that's true. But I'm not so sure this is the case here. For one, in his response Hart doesn't provide any evidence that the finding replicates in a conservative sample--he merely offers the suggestion. 

And there also is this question. Does it make sense to generalize from findings with p-values of .01, .03, .02, obtained with a small (N=48 in a between-subjects design) sample, and a single vignette to courtroom behavior?

Rather than using the discussion section for overgeneralizations, it makes more sense to use it for specifying the situations under which the conclusions can be expected to hold. Not only does this provide more truth in advertising but it's also an important theoretical exercise. It's not easy, though.

I plan to pursue the topic of calibrating our conclusions in future posts. Now please excuse me while I try to assemble a Billy, Hemnes, Klippan, Poäng, or Bestå.