You Say You Want a Revolution

The popular press as well as the psychological literature itself is abuzz with reports about the field trying to clean up its act, especially by demanding replications (though not everyone agrees). Many reports do not fail to mention that this sudden sanitation urge was prompted by the Stapel fraude case. This is ironic, because it is logically impossible to replicate Stapel. His work was never “plicated” in the first place!

Nevertheless, it is clear that the field is in a state of revolution. There are large-scale replication efforts under way, most notably the Open Science Framework. But replication starts at home. So here are some things we can do as authors, reviewers, and editors.

Authors

Incorporate direct replications into your workflow (I hesitate to use this buzzword but it seemed quite popular among those interested in replication during a recent Google hangout). 

I have reported direct replications in one published paper and I have recently submitted two other papers in which each of the experiments has a direct replication. I won’t lie. It is quite cumbersome to have to do a direct replication for every experiment you’ve run and so far it has been really easy for me because I have been using Mechanical Turk (and even so, I have repeatedly cursed myself for wanting to do replications).

An attractive alternative to direct replication—already quite common in cognitive psychology (but maybe not so much other areas of psychology)—is to run an experiment that is close to the original but changes one aspect. In an earlier post, I called this the Bruce Springsteen or Eagles model.

A direct replication is especially called for if the original finding meets one or more of the following criteria: it (1) is highly novel and surprising, (2) has strong practical or theoretical import, (3) runs contrary to established findings or theories, (4) is underpowered. If readers can think of additional criteria, I’d like to hear them.

Reviewers

State explicitly in your review if a direct replication is required. If the study meets one of the criteria listed above, it is a candidate for direct replication.  However, do consider the feasibility of a direct replication. Obviously, a direct replication is going to be difficult with special populations, longitudinal studies, and time-sensitive experiments (e.g., voters’ reactions to Obama’s re-election). So be reasonable. Don’t ask for an impossible replication attempt just because you don’t like the study or its author.

Editors

If the reviewers identify aspects of the research that overlap with the criteria listed above and call for additional data without mentioning a direct replication, call for a direct replication. If reviewers ask for a direct replication but this is unreasonable, say so.

I recently asked for a direct replication of a study that met some of the criteria I listed above (the reviewers had not asked for a direct replication but wanted to see additional data). One of the co-authors happened to be Bobbie Spellman (I only realized this after I’d composed my action letter), herself a staunch advocate of replication. She calls my action letter her first “post-revolution action letter. ” It is post-revolution because it asks for a direct replication [postscript: and for reasons Bobbie mentions in her comment to this post].

It is actually the second post-revolution action letter I’ve written. I probably should have written more in the past and I plan to write more in the future. I hope other editors will start writing post-revolution action letters as well.

These are obviously the first baby steps toward a full-scale integration of replication into our scientific practice. But as Oasis (themselves a failed replication of the Beatles) stated: I start a revolution from my bed…