I’m working on a paper
with Alex Etz, Rich Lucas, and Brent Donnellan. We had to cut 2,000 words and the text below is one of the darlings we killed. I’m reviving it as a blog
post here because even though it made sense to cut the segment from the manuscript (I
cut it myself, the others didn’t make me), the notion of concurrent replication
is an important one.
The current replication debate has, for various reasons, construed
replication as a retrospective process. A research group decides to replicate a
finding that is already in the published literature. Some of the most
high-profile replication studies, for example, have focused on findings
published decades earlier, for example the registered replication projects on
verbal overshadowing (Alogna et al, 2014) and facial feedback (Wagenmakers et
al., in press). This retrospective approach, however timely and important,
might be partially responsible for the controversial reputation that
replication currently enjoys.
A form of replication that has received not much attention yet is
what I will call concurrent replication.
The basic idea is this. A research group formulates a hypothesis that they want
to test. At the same time, they desire to have some reassurance about the
reliability of the finding they expect to obtain. They decide to team up with
another research group. They provide this group with a protocol for the
experiment, the program and stimuli to run the experiment, and the code for the
statistical analysis of the data. The experiment is preregistered. Both groups
then each run the experiment and analyze the data independently. The results of
both studies are included in the article, along with a meta-analysis of the
results. This is the simplest variant. A concurrent replication effort can
involve more groups of researchers.
A direct exchange of experiments (a straight “study swap”) is the
simplest model of concurrent replication. It is possible to accomplish such
study swaps on a larger scale where participating labs offer and request
subject hours. This will likely result in a network of labs each potentially
simultaneously engaged in forming and testing novel hypotheses as well as
concurrently replicating hypotheses formed by other labs. The Open Science
Framework features a site that has recently been developed to facilitate
concurrent replication, Study Swap, see also this article. At the time of
this writing, there are four projects listed on Study Swap. We hope this number
will increase soon.
Aside from this, there already are several large-scale concurrent
replication efforts. An example is the Pipeline
Project, a systematic effort to conduct prepublication replications,
independently performed by separate labs. The first instalment was recently
published (Schweisberg et al. 2016) and a second project is underway.
Concurrent replication has several advantages. First, researchers
have a better sense of the reliability of their findings prior to
publication. After all, the results have
been independently replicated before submission of the article. Likewise,
journal editors and reviewers will have more confidence in the findings
reported in the manuscript they are asked to evaluate. Journals have the luxury
of publishing findings that have already been independently replicated. As a
result, the reproducibility of the findings in the literature will start to
increase. The Schweisberg et al. (2016) study demonstrates that concurrent
replication is not only possible but also useful.
Concurrent replication forces researchers to be explicit about the
procedure by which they expect to obtain the effect. If they do indeed obtain
the finding both in the original study and in an independent replication, they
have what amounts to a scientific finding according to the criteria established
by Popper: They can describe a procedure by which the finding can reliably be
produced. It will be easy and natural to include the protocol into the method
section of the article. A positive side-effect of this will be a marked
improvement in the quality of method sections in the literature. As a result,
researchers who want to build on these findings have two advantages that
researchers currently do not enjoy. First, they can build on a firmer
foundation. After all, the reported finding has already been independently
replicated. Second, a replication recipe doesn’t have to be laboriously
reconstructed. It is readily available in the article.
Of course, concurrent replication is not without challenges. For
instance, how should authorship be determined given such an arrangement? A
flexible approach is best here. At one extreme the original group’s hypothesis
might be very close to the replicating group’s own interest. In this case it
would therefore be logical to make members of both groups co-authors; each
group may have something to add to the paper both in terms of data and analysis
and in terms of theory. At the other extreme, the second group has no direct
interest in the hypothesis but may be willing to run a replication, perhaps in
exchange for a replication of one of their own experiments. In this case it
might be sufficient to acknowledge the other group’s involvement without
offering co-authorship.
Thus far, the discussion here has only involved a scenario in which
the hypothesis is supported in both the initiating as in the replicating lab.
However, other scenarios are also possible. The second scenario is one in which
the hypothesis is supported in one of the labs but not in the other. If the
meta-analysis shows heterogeneity among the findings, researchers might
hypothesize about a potential difference between the experiments, preregister
that hypothesis and test it, again with a direct replication. If the
meta-analysis does not show heterogeneity, it might be decided that it is
sufficient to report the meta-analytic effect. If neither lab shows the effect,
the research groups might report the results without engaging in follow-up
studies. Alternatively, they might decide the experimental procedure was
suboptimal, revise it, preregister the new experiment and run it, along with one
or more concurrent replications.
To summarize, concurrent replication forms an underrepresented but
potentially extremely valuable form of replication. Several concurrent
large-scale replication efforts are currently underway and a platform that also
facilitates conducting smaller-scale projects is available for use. The fact
that concurrent replications are often viewed positively by the field is
further evidence of the importance of replication for scientific endeavors.
References
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S.,
Birch, S., Birt, A. R., ... Zwaan, R. A. (2014). Registered replication report:
Schooler & Engstler-Schooler (1990). Perspectives
on Psychological Science, 9, 556–578.
Schweinsberg, M. et al. (2016). The pipeline project:
pre-publication independent replications of a single laboratory's research
pipeline. journal of experimental social psychology, 66, 55–67.
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q.
F., Acosta, A., Adams, R. B., Jr., . . . Zwaan, R. A. (2016). Registered
Replication Report: Strack, Martin, & Stepper (1988). Perspectives
on Psychological Science, 11, 917–928.
Reacties
Een reactie posten