How to solve the crisis of referees in professional philosophy (guest post)

In the following guest post, Eric Schliesser (Amsterdam) offers a two-step solution to the crisis of referees in philosophy.

[Theo Van Doesburg – Contra Construction]

It has come to the point that the most appropriate referees for an article are almost never available. It takes us so long to find reviewers, sometimes a month, six weeks or more (this situation is also exacerbated by the fact that many people do not respond to requests for references at all). This further extends the duration of the total review process. All sorts of little fixes no longer work, eg shortening the time people have to review, asking for alternatives (this is always much appreciated, but unfortunately the alternate referees are just as unavailable).

Of course, my editorial experience might be atypical – I’m just an academic.

But I strongly suspect that the peer review system is ultimately broken beyond reasonable repair. We have seen the situation slowly escalate and the pandemic has finally broken the system. People are exhausted and overworked, job applicants increasingly desperate.

De Cruz’s article generated a huge discussion, not only among Cocooners, but also here at We every day. The situation she describes was a long time coming (and for once I can claim I partially predicted it – remember here and here). There are a number of major interrelated reasons for the current status quo (extreme shortage of referees):

(I) a extreme scarcity of decent jobs leading to tenure compared to the number of doctorates minted. It’s a major source of anxiety across the profession
(ii) the unreasonably low acceptance rate of philosophy journals such that perfectly competent articles must be arbitrated several times by a fixed number of arbitrators
(iii) the perceived role that prestige and quality of publications play in hiring
and maybe,
(iv) the relative ease of free-riding in the current system. The real free-riders are universities (who outsource quality control without rewarding it internally – my own university never even asks to list arbitration anywhere) rather than individuals.
In what follows, I take it for granted that low acceptance rates are functional, i.e. maintained in order to produce prestige (remember this post). And the combination of (i-iii) generated a real arms race in the publication. Besides,
(v) the long delays in arbitration, made it prudent to have multiple items being reviewed at the same timethus aggravating the underlying problem.

In what follows, I take it for granted that the standard of regular publication has been good for professional philosophy. Reasonable people may disagree on this. But in my opinion, it allowed for finer discussions and the emergence of specializations which improved the quality. Yes, this risks generating hyper-specialized echo chambers with smaller common denomination salami publications, but these, in turn, generate arbitrage opportunities for people who read a lot or practice what I is called synthetic philosophy.

I also take for granted that (i) is embedded in the political economy of modern universities. I am an accomplice; if you find a way to create a revolution, good for you. But what we can do, as a profession, is sever the ties between prestige, journal publishing and hiring decisions and thereby reduce the need for referees. And we can do this if we destroy the preeminence role of journals in the manufacture of prestige, and restore them to the role of disseminators of knowledge and places of discussion.

This can be achieved in two steps: first, all major philosophy journals should commit to publishing, say, 25% of the papers received. This is considerably higher than the status quo (Philosophical Quarterly publishes only 4% of the more than 850 articles received). This will have six obvious effects: it will reduce the load on the referees; it will put an end to the excessive role of chance in where your article is published; it will reduce (but not eliminate) the steep prestige hierarchy of our current journal system; it will reduce excessive delays in the printing of work caused by the difficulty of finding references; it will transform most of the big newspapers into stationery; this will increase citation rates within the profession.

There is concern that once journals can no longer play the role of generators of extreme prestige, they will then reinforce the roles of departments that grant doctorates. I agree and that would be a shame. This is therefore the second stage: another institution must take its place. Fortunately, such a practice already exists in neighboring disciplines and is present in a nascent form in ours: the price of the article (and of the book). What we need is a multiplication of philosophy prizes: congress prizes (which are pre-publications); trade association awards (large and small companies); prize for the best article awarded by the journals themselves; and we need more versions of Philosopher’s Annual. In fact, as Don Ainslie reminded me, there should be more prizes for different reasons too: philosophy has far too few, which impacts success in interdisciplinary competitions (e.g., societies national scholarly events, scholarship competitions, etc.)+.

At this point, there is concern that these prizes are easily shaped by politics and that our time is spent awarding the prizes rather than arbitrating. On the first, I don’t see how politics can be taken out of the equation entirely (as it stands, the review system has an important “trust us” component reminder). I don’t deny that true masked overhaul, when it occurs, is true equalizing potential. And notice that on my proposal, we don’t eliminate that. People from non-prestigious intellectual backgrounds can still use journal publishing to get noticed. (But the strength of the initial signal may well be reduced.) And one can also use the masked review in the price committees. Over time, awards that perform well in their judgment will become more prestigious and thus complement the reviews in prestige generation and quality control.

In a way, my proposal is a variant and complements the post-peer crowd review advocated by Marcus Arvan, Liam Kofi Bright and Remco Heesen which they have championed in the press and at We every day* and the ideas promoted by Katzav and Vaessen in an article The Philosopher’s Footprint. (Also remember my message here during the Hypatia controversy, but I had limited the idea of ​​work that might be controversial in various ways and in order to internalize political considerations transparently and fruitfully in the review process.) The use of awards is a mechanism to address the coordination and collective action problem inherent in such proposals, and to allay fears that non-experts may overwhelm the wisdom of our own crowds.

On that last concern, about more time spent judging work for prizes, it’s undeniably true. But I suspect it’s more fun to review articles that have already been published and have therefore undergone non-trivial editorial and quality control. Additionally, although writing a prize report is a non-trivial exercise, it is only required for the winning item(s). So overall I would expect this to reduce the burden on those in the profession who are the backbone of our current adjudication system. **

+This comment was added after the message was originally posted on Digressions and impressions.

* I also suspect that my proposal resolves an objection against those who think that Condorcet’s jury theorem is inappropriate here, since awarding a prize is standard-setting, not truth-finding. But that will be for another time.

**Thanks to Helen de Cruz and Neil Levy for the discussion.