Friday, May 18, 2018

A Career Niche for Replicators?

My former colleague Roy Baumeister famously said that replication is a "career niche for bad experimenters.”* I like to use this quote in my talk. Roy is wrong, of course. As anyone who has tried to conduct a replication study knows, it requires a great deal of skill to perform replications. This leads to the question Is there a career niche for replicators?

I was asked this question yesterday when I gave a talk on Making Replication Mainstream at the marvellous Donders Institute for Cognition, Brain, and Behaviour in Nijmegen. I get asked this question regularly. My standard answer is that it is not a good career choice. Implicit in this answer is the idea that in order to become a tenured faculty member, one has to make a unique contribution to the literature. Promotion-and-tenure writers are always asked to comment on the uniqueness of a candidate’s work. Someone who only conducts replication studies would run the risk of not meeting the current requirements to become and remain faculty members.

During lunch, a group of us got to talking some more about this issue, to which I hadn't given sufficient thought, as it soon turned out.

It was pointed out that there is a sizeable group of researchers who would like to remain in science, have excellent methodological skills but don’t necessarily have the ambition/creativity/chutzpah/temerity to pursue a career as faculty member.

These researchers, was the thinking at our lunch table, are perfectly suited to conduct replication research. The field would benefit greatly from their work. If we truly want to make replication mainstream, there ought to be a career niche for them.

If faculty member is not a viable option, then what would be a good career niche for replicators? It was suggested at our table that replicators should become staff members, much like lab managers. They would not be evaluated on the originality or uniqueness of their publications. In fact, maybe they would not even be on the publications, just as lab managers often are not on publications. Faculty members select studies for replication and replicators conduct them and by doing so make a value contribution to our science.

I think this is a fair summary of our discussion. I have no strong opinions on this career niche for replicators yet but I wonder what ya'lls thoughts on this are.

----
* The link is to a paywalled article but I'm sure you can scihub your way to it.

8 comments:

  1. I think this is an interesting idea. If we imagine that such a person might also want to teach some classses (maybe especially classes in research methods?) and the replication studies provide opportunities for undergraduate students to learn valuable research skills in a hands-on way, then the faculty member option might be viable. I agree that replications probably wouldn't meet the criteria for tenure, but perhaps this could be a non-tenure track option, or a strategy for teaching-oriented institutions that don't expect their faculty to do a lot of original research.

    I bring this up because there is limited funding and institutional support for replication research. In the framework I described, the research space, participant pool, IRB, etc. could come from the institution, the research staff could be undergrad students, and the funding could be a combination of funds from the department/institution (as a form of teaching) and research funds from (tenure-track) faculty who want to see a particular effect replicated because it is relevant to their own research.

    ReplyDelete
    Replies
    1. Yes, the person could teach classes or labs. In my department, lab managers are sometimes involved in lab classes. I worry about non tenure-track positions, as this wouldn't really give these researchers much of a career prospect.

      Your points about funding are well-taken and your suggestions in this regard are interesting.

      Delete
  2. With all due respect, I think this is a (very) bad idea. It institutionalizes the implicit, popular, and wrong assumption that a) being a good researcher primarily means having original and sexy ideas (a notion that has brought trouble upon us), and b) doing replication is somehow second rate and/or less important. As for a) it is obviously very important that a researcher doesn't only have original and sexy ideas but also that they are able to provide solid and reliable evidence for that or any other idea. With regards to b) performing a good, reliable, valid, and preferably collaborative replication requires *more* communicative, experimental, and statistical skills than finding evidence that (in the eyes of reviewers) justifies publishing a new idea.

    ReplyDelete
    Replies

    1. I noticed you snuck "sexy" in there. Promotion and tenure letter writers are not asked to judge whether the research is sexy--in my experience at least. They are asked to consider whether the research is unique (often meaning separate from that of the graduate advisor). I also think you're introducing a false dichotomy. Original is not the opposite of solid and reliable. The best use of replication will be in the context of original research. Collaborative efforts such as PsyAccellerator will be crucial in this endeavor.

      Delete
    2. Apologies for using the word "sexy", I didn't want to offend anyone, but I did want to express that I believe that "different from grad advisor" is in practice not seen as sufficient for research to be "original". And I am most certainly not introducing a false dichotomy. I *clearly* wrote we need both (note the "not only ... also" construct). I think we'd agree that an "original researcher" must be able to do good replications, but I also believe that a good replicator must be able to be an original researcher.

      Delete
    3. "Different from grad advisor" is indeed not sufficient for being considered "original" but it is necessary. Yep, you're right. You didn't introduce a false dichotomy. We do indeed agree that an original researcher must be able to conduct good replications. As I noted in the post, there is a large group of researchers who a comfortable performing replication studies but have difficulty coming up with their own ideas (this is not just my observation, it was raised by an ECR at the colloquium). Currently, there are no real career prospects for these individuals. A staff position might be an option.

      Delete
    4. I agree, replicators need a great deal of skill. However, it is not sufficient to be a skillfull methodologist, you also need to know the field in which you are operating. As "direct" replications are never identical with the original studies, minor deviations may exert serious psychological consequences. Of course, I am thinking of the camera in the pen-study replications. Had the replicators been knowledgeable about the impact of objective self-awareness, the would have hidden the camera to avoid the original effect to be suppressed. The same is true for replications in other areas. Unless replicators have a sufficient expertise in the area of interest, nonreplications have to be taken with great restraint. Thus, I am skeptical that replicators could make up a special branch of researchers.

      Delete
    5. I agree that there needs to be knowledge about the field one is operating in. In the proposal discussed in the post, this would be the bailiwick of the faculty member who supervising the replicator.

      Delete