Replication Done Right

When I started this blog, I was a little worried that I might soon run out of topics. So far, however, the topics have been presenting themselves. And now readers have even started to suggest topics for blog posts!

I recently received an email message from Etienne LeBel who said he’d enjoyed my Lazy Susan and Bruce Springsteen post (Lazy Susan, the gift that keeps on giving) and suggested I write a post about a recent positive replication experience of his. Specifically, he said: We really want to get the message out that these replication efforts need not be adversarial and antagonistic, and that it should be considered as a normal part of ensuring our science is self-correcting.

It thought this was a great idea—who wouldn’t want to be the bearer of good news? So here goes.

The study Lebel and his co-author Lorne Campbell (a great name for a sheriff in a Western) set out to replicate study 1 of a paper by Matthew Vess published in Psychological Science. Vess compared individuals who were more or less “anxiously attached.” (Being a non-specialist in this area, my first association with the phrase anxiously attached was that of a severed limb having been put back in place by a nervous surgeon, but I think I have a global idea of what it means now.) Vess asked both groups of subjects about their food preferences. The more anxiously attached subjects reported heightened preferences for warm foods compared to the more securely attached subjects. But this occurred only when attachment concerns were activated (i.e., reflecting on a romantic breakup) and not in a control condition.

LeBel and Campbell (L&C) say that they are sympathetic to the theoretical integration of the study but that they wanted to assess the reproducibility of that finding, given that it was only reported in one study with 56 subjects.

How did they go about it?

(1) They made sure they had sufficient power to detect the effect. They didn’t take half-measures and quadrupled the original sample size so that they had a power of .95 to detect an effect. I recently said on Twitter that an underpowered (and failed) replication attempt is more like libel than like research, which seemed to resonate with several people interested in replication. Clearly, L&C are not guilty of libel.

(2) They contacted the original author, Matthew Vess, for details about the procedure and materials.

(3) They preregistered the studies prior to data collection.

(4) As in any good replication study, they faithfully copied the procedure and nature of the sample of the original study.

(5) They went the extra mile by asking the original author to critique their first attempt. Vess noticed some small discrepancies between L&C’s first attempt and his original experiment. These discrepancies were resolved before the second attempt.

(6) They used the exact same analytical procedures as were used in the original study.

(7) They report their findings in a concise and respectful manner.

So what did L&C find? Well, they did not replicate the original finding in either sample. We all can check this because they made all project materials, raw data, and syntax files available online.

L&C argue that their findings are difficult to reconcile with the original ones. There were no major procedural discrepancies between their replication attempts and the original study and they had sufficient power to detect an effect. And because the replication attempts were preregistered, selective reporting was not an issue.

L&C conclude: Our findings, however, do not provide empirical support for the notion that activating the attachment system of more anxious individuals increases sensitivity to temperature cues, although it is possible that this theoretical idea reflects a reproducible phenomenon under a different set of operationalizations. We therefore advise researchers to proceed with caution when exploring links between anxious attachment and temperature experiences in potentially relationship threatening contexts.

This is a nicely worded conclusion. It contains no criticism of the original study, merely suggesting that researchers in this area should tread lightly, given that the empirical foundation may not be as strong as formerly believed. As such, this study is a prime example of what I talked about in my very first post: replications should be about checking the structural integrity of the empirical foundations of the field rather than about pointing fingers.

The steps that L&C followed might serve as a blueprint for other replication studies. Clearly, researchers need to be true to the original study in terms of design, procedure, exclusion criteria, and data analysis. Clearly, they need sufficient power to detect an effect of the size reported in the original paper. Because of publication bias, this means running considerably more subjects than in the original study. Pre-registration seems the way to go, as it prevents a reverse file-drawer problem in which people will only report non-replications, as these might seem more informative than replications.

It is also important to consult the original author.  Obviously, original authors may not always be as helpful as Vess evidently was in this case but they should at least be consulted regarding the materials, design, procedure, and analyses of the intended replication attempt.

L&C’s paper is currently in press in Psychological Science, the journal that also published the original study by Vess.

The original findings may not have been replicated, but I’d still call this a successful replication attempt.