Credit, Workload, Accountability, & Fear: Opinions About Open Review

Update May 11, 2016. In a talk, given at Psychonomics in Granada Spain on Saturday, May 7, I discuss the contents of this and the previous post in a symposium on open science, organized by Richard Morey. My talk starts at 43:50. The other talks are definitely worth a watch.

Last week, I reported some quantitative analyses of my open-review survey. In this post I am going to focus on the respondents’ written sentiments regarding open reviews from the perspective of a reviewer.

Many respondents provided written motivations for whether they disagreed or agreed with the statement “as a reviewer, I'd like to have my review published along with the accepted paper.” They could also indicate whether they agreed only if their review would remain anonymous. A large majority (72%) indicated that they would like to see their review published. Forty-five percent of these respondents (87 out of 195) indicated that they only wanted to have their review published if they could remain anonymous.

I then divided the respondents into two groups: tenured and untenured. The distribution of responses over the three answer categories (disagree, agree provided anonymous, agree) differed for the two groups. Perhaps most surprisingly, a much larger percentage of tenured respondents (41%) than untenured respondents (22%) were against having their reviews published. Also interesting, though perhaps less surprising, was that a much larger percentage of untenured (39%) than tenured (18%) respondents wanted to have their reviews published only if their reviews were anonymous. About equal percentages were in favor of having signed reviews published.

What are the motivations behind these numbers? To examine this, I grouped the written responses into several (often related) categories. Sometimes respondents provided multiple reasons. For this post, I decided to go with the first reason provided.* Let’s first look at the motivations provided for not wanting reviews to be published. Not every respondent provided written responses. The two bottom rows show the total number of written responses as well as the total number of responses.

	Untenured	Tenured
"different audience"	8	10
"too much work"	7	2
"don't see the relevance"	5	6
"I fear retribution"	3	4
“other”	5	5
Total written responses	28	26
Total responses	41	36

For both groups the most prevalent response for not wanting to see reviews published was the view that reviews are intended for the authors/editor and not for a broader audience. As one respondent stated: 

“If I want to publish a commentary, I will. But reviews are constructive feedback designed to help improve research articles: Why would the review be interesting once the original paper has been revised to address whatever concerns emerged in the review?”

This respondent distinguishes between reviews and commentaries and thinks each has a different role (and audience) in scientific discourse. A related concern, especially among untenured respondents, was that getting reviews in a publishable format is a lot of work, especially if you’re not a confident writer. Here is how one respondent put it:

“Writing reviews already takes too much time. If I know that it will be published, I will care too much about making sure it's free of typo's, grammatical errors, and bad writing style. This will make the reviewing burden even larger.”

Also related is the concern that publishing reviews would be uninformative, given that some reviews point out only minor flaws and that flaws are fixed in the final manuscript anyway. Interestingly, some respondents expressed fear of retribution, apparently even if there is the option of anonymous review.

Let’s turn now to those who are in favor of open reviews but want them to be anonymous. As expected, fear of retribution figures very prominently here.

	Untenured	Tenured
"I fear retribution"	33	9
"I feel insecure"	5	0
“other”	13	3
Total written responses	51	12
Total responses	71	16

In fact, fear of retribution is by far the most common response in both groups. But there is an interesting difference between the groups. Whereas the untenured respondents are concerned about retribution against themselves, 6 of the tenured respondents express concerns for others, junior faculty and other potentially vulnerable groups. It is also interesting to ask whom respondents are fearing retribution from. Most fear the reviewed authors as the source of reprisal, but a few see the social media as the chief danger. Being on record as having endorsed publication of a controversial paper may make you the target of criticism:

"There could be mistakes that I am happy editors could spot and other reviewers could counterbalance, but I am not sure I would survive harsh open social media bashing for very long."

Some respondents provide more general observations about humanity to explain why they are hesitant to sign open reviews. This respondent who put it very succinctly:

“People, including professors, are dicks.”

I’m sure I speak for all of us when I respond to this sentiment with a heartfelt “amen!”

Finally, there are those who are in favor of signing their reviews and having them published. What are their motivations?

	Untenured	Tenured
"transparency"	23	6
"getting credit"	9	8
"accountability"	6	3
"quality"	6	9
"fairness”	5	2
“other”	11	5
Total written responses	60	33
Total responses	72	36

A plurality of untenured respondents mentioned transparency as the main reason for open, signed reviews. Related to this are the notions of accountability and fairness. It is only fair that if the authors are known that the reviewers should also be known appears to be the reasoning here. As one respondent put it:

“I feel like I have myself received a number of (anonymous) reviews that seemed done in a rush and sometimes unfair. It was in particular the way they were written that made for unpleasant reading. I think that having reviews published will improve the way they are written, because no one would want a badly written review out there.”

Another reviewer noted:

“...it holds me, and other reviewers accountable (no mean-spirited bashing, or self-serving "cite ref-to-my-work-here" tactics without looking like a jerk) “

Separate from this is the notion of credit. Quiet a few respondents mention they like the idea of receiving credit for their work as reviewers:

“Reviews are a lot of work, and I am proud of the work that I do in them. I'd like to get some credit for that work.”

Several respondents were of the opinion that open review enhances the quality of the reviews:

“I think that publishing your review and name provides a social incentive to do a good job with the review.”

So what can we conclude from this? I think quite a bit. But rather than drawing conclusions myself at this point, I’m interested in hearing your views.

Note:

*This obviously is just a first pass through the data. I’ll need someone to provide an independent coding and I need to analyze more than just the first response people gave.