Is there a crisis of arbitration in philosophy? There has been quite a bit of discussion about this over the past two months. What was missing in much of this discussion, however, was data. So I asked for it.
I heard of about 40 philosophy journals, but only half of them were able to provide the kind of information I was looking for in order to understand how difficult it is for journal editors to find enough qualified and willing referees: the percentage of those they invite to referee an article who accept the invitation and write the review.
Twenty or so reviews isn’t as big a pool of useful data as I hoped to dive into, but it’s not nothing. What did they say?
The journals that provided useful data were a diverse group in terms of prestige, breadth of topic (general/specialized), and popularity of the topic in the profession. They also varied in the time periods from which they drew their answers, but most of the information came from the past year (with only a few going back further in time than that). I told the editors that I would not reveal journal-specific information, so in what follows, no particular journal is named.
Across the entire sample, I found that on average, about 40% of invitations that journals send out to potential referees are accepted. About half of the reviews in this group were within ±7 points of this mean. Of the other half, a few were down between 20% and 29% in referee acceptance rates, and a few were up between 55 and 60%.
Were there any trends that journals had what rates of acceptance of invitations to review by referees? Given the relatively small sample size, it is difficult to draw firm conclusions. I will say that one thing that is true of all journals with rather low referee acceptance rates is that they are not the most well-known versions of the type of journal that they are. . (And by type, I mean general or specialist, and if specialist, the specific field.) That said, generalist journals in this subgroup are definitely known to most philosophers, and specialist journals in this subgroup are definitely known to philosophers working in these areas of specialization. So the obscurity of the diary does no job of explanation here. What is? Perhaps it is more competition (i.e. more journals) in this space relative to the number of potential reviewers.
One reason to think this explanation has merit is that it is consistent with the following data, which at first seemed odd to me. As I do not name specific journals, I will use nicknames for three of them: the Very good newspaper in an unpopular neighborhoodthe Possibly the best newspaper in a popular neighborhoodand the Great journal of generalist philosophy. Of these three, the Very good newspaper in an unpopular neighborhood has the highest referee invitation acceptance rate (in the top 50): 4 points above the Possibly the best newspaper in a popular neighborhood and 18 points above the Great journal of generalist philosophy. The first journal is one of the few such journals, and faces little competition for reviewers, while the other two, although having a better reputation in the profession generally, are each one many of their respective types, and thus each faces a lot of competition for reviews.
That said, the supply of potential examiners is lower for the Very good newspaper in an unpopular neighborhood than for the other two, due to differences in the popularity of the domains they cover, which may somewhat work against the competition explanation. There’s also the possibility that there’s a bit of an “outsider” mentality among those working in the unpopular field that motivates them to be more willing to accept referee requests; or perhaps, knowing that they work in a relatively small area, they feel a relatively less diffuse sense of responsibility to help institutions that build credibility in that area, such as the journal.
A different possibility is that the Very good newspaper in an unpopular neighborhood is seen by its community of scholars as “their journal” – one that matters to them and is somewhat definitive for the area it covers, and therefore members of that community are more motivated to arbitrate it. Such a possibility was suggested by a journal editor (not associated with any of the three journals above) in reflections they shared with me about their experiences at two different journals: a leading generalist journal and a leading specialist journal: refused [to referee] for [the specialist journal]the same people did maybe a third of the time to [the generalist journal]. This could…be it [the specialists in this area] seen [the specialist journal] as “their” newspaper in a way that they did not consider a [top] general review.
The explanations of the “competition” and “their review”, as well as some observations about the types of reviews that had an acceptance rate of referee invitations of 30% or less, suggest (but hardly prove, given the small amount of data) something like the following generalization: if the journal you edit is not among the very first places authors might send their manuscripts, you will have a harder time finding referees.
This explanation is consistent with the fact that there are multiple reasons why a journal may be among the very first places an author sends their manuscript: the journal may be particularly prestigious or of high quality, or be focused on distinctive content , or be the professional center of gravity. in a particular subdomain (because of its content, or perhaps the editor’s influential role in the subdomain, outside of their work for the journal). Correlatively, it suggests possible ways for journals to improve their rate of acceptance of referee invitations: excellence, distinction and influence.
Another possible explanation for the disparate pricing comes from the aforementioned editor: “the personal touch made all the difference.” In the trade journal, the editor personally emailed potential arbitrators, and although they used a standard letter template for such messages, he believes they were still more likely to garner positive responses than messages automatically generated by editorial management programs: “I” I am inclined to blame editors who insist on seeking referees through editorial management software as a major (if not the only) reason why editors have so much more hard to find referees.
Some additional information:
- the journal that reported the highest rate of accepting referee invitations is also a journal that has a reputation for having a high rate of office rejections
- very few referees agree to referee an article and then never submit a report (except in one of the responding journals, where the editor said their referee referral rate was between 10% and 20%)
- almost no article in the 20 journals that sent information was rejected simply because of difficulties in obtaining referees and reports (again, except in the same journal referenced in the previous point, where this happens “occasionally “)
Finally, it should be noted that very few journals maintain this type of data. For almost all of the journals that provided data for this article, the data had to be produced first. In many cases, publishers have found it impossible or too time-consuming to do so, or have found the information inaccessible due to publishers’ practices or editorial management software. As with questions about submission topics, potential author demographics, and related issues, it seems like it would be useful for our understanding of our profession (what happens in it, its challenges, its history, its progress, etc.) journals with a history of maintaining and occasionally publishing data relevant to their editorial operations.