The Open Review Survey: First Peek

A few days ago, I conducted a survey among psychologists about open reviews. More than 250 colleagues volunteered their time, for which generosity I'd to thank them.

The survey was prompted by my being senior editor at Collabra, a journal that has the option of open reviews (other journals that I know have this feature are PeerJ and BMC Psychology). A fellow editor and I were wondering about people's views of open review. I conducted an initial quick survey on Twitter and followed this up with the more extensive one I'm describing here.

Participants were recruited via social media (Twitter, Facebook, LinkedIn) and the survey was open between Friday, April 8 and Tuesday April 12.

The survey consisted of open and closed questions. The former were included because I didn’t want to steer the participants too much; I was interested in their views. However, analyzing the open responses takes quite a bit of time, so here is an initial peek at the quantitative data only. I’m planning to write a paper that includes all of the data. (If editors are interested, please contact me.)

Although I specifically asked psychologists to participate, there were some respondents from other fields as well but their number is not large and some of the fields are likely close to psychology (e.g., communication, behavioral economics, psycholinguistics), so I’ve decided to include all respondents here.

My field of inquiry is

Answer	Response	%
psychology	250	90%
other social science	18	6%
natural sciences	9	3%
humanities	2	1%
Total	279	100%

The sample contains sizeable numbers from the ranks of graduate student to full professor; there are additional responses from a few undergraduate students and people with an “other” position; I’m assuming the latter are research associates. 

My current position is

Answer	Response	%
graduate student	66	24%
postdoc	54	19%
assistant professor	52	19%
associate professor	49	18%
full professor	40	14%
other	13	5%
Total	279	100%

I was interested in people’s views on open reviews both from the perspective of a reviewer and from the perspective of an author. Let’s first look at the reviewer perspective.

As a reviewer, I'd like to have my review published along with the accepted paper

Answer	Response	%
Agree, and I would like to sign my review.	108	40%
Agree, but only if I can remain anonymous.	87	32%
Disagree.	77	28%
Total	272	100%

Perhaps surprisingly, a large majority of respondents (72%) prefer to have their review published along with the article. For a third of the respondents, open reviews are desirable only if they can remain anonymous.

One concern that editors might have is that having an open-review option would decrease the likelihood that people will accept requests to review. However, as the results show, the prospect of open reviews would not make a difference for the majority of respondents. For about a quarter of the respondents, the likelihood would increase and for a fifth it would decrease. 

If there was an opportunity for open review, the likelihood that I'd accept an invitation to review would ….. if there were open reviews.

Answer	Response	%
increase	60	24%
decrease	51	20%
neither increase nor decrease	144	56%
Total	255	100%

How do open reviews look from the perspective of an author? Slightly more than half the respondents would like to have open reviews, while a third has no preference. Only 12% of the respondents would prefer not to have open reviews.

As an author,  I'd like to have open reviews

Answer	Response	%
Agree	137	55%
Disagree	29	12%
No preference	85	34%
Total	251	100%

Do response patterns vary as a function of having tenure? The perception exists that having (signed) open reviews, or maybe even unsigned ones, might make one vulnerable to retaliations. This would especially be worrisome for pre-tenure researchers.

To address this question, I collapsed across undergraduates, other, graduate students, postdocs, and assistant professors. I realize that there are considerable differences among these groups along many dimensions, but what they have in common is that they are pre-tenure. Why didn’t you directly ask about tenure?, you might wonder. I considered doing this but decided against it because it would make the issue of tenure more salient and might create demand effects.

Here are the, perhaps surprising, results.

Among the untenured respondents, a large majority (78%) favors open review. Of this latter group, half is even prepared to sign the reviews. Puzzling, to me at least, is the relatively large percentage of tenured respondents who are against having their review published (41%), almost twice the percentage found among untenured respondents! I'll look at the open responses in a later post to see what's going on here. Of the tenured respondents who favor open review, there appears to be less concern about being identified than among the untenured respondents. (Exercise: run your own Chi-square test here.)

Untenured respondents seem somewhat more likely to accept invitations when reviews are open than tenured respondents although openness wouldn't matter either way for the majority in each group.

How does open review look from the perspective of an author? Here, the pattern for the two groups is quite similar. 

So there you have it: our initial peek at the data. Of course, there are limitations to this survey. For one, it is not clear that this sample is representative for psychology at large. Perhaps it contains a larger percentage of proponents of openness than one would find in the population. Furthermore, there are other questions I could have asked. However, I wanted to keep the survey under five minutes to maximize participation.

With this in mind, it is possible to draw some conclusions. There clearly is support for open reviews, especially from prospective reviewers. However, the prospect of open reviews does not have much impact on researchers' willingness to accept an invitation to review. The support for open reviews is greater among untenured than among tenured researchers. However, for those in favor of open reviews, tenured faculty are less concerned about singing their reviews. Prospective authors are more in favor of open reviews than against and this does not differ between tenured and untenured researchers.

In the next post post I delve into the written responses.