Today I’m coming out of the closet: I declare myself a bad reviewer.
To me, peer-reviewing a journal or conference paper implies two main tasks:
- Deciding whether the paper should be accepted for publication
- Providing constructive feedback to the authors so that they can improve their paper
While I´m convinced I’m doing a good job at 1) (or at least I make sure I invest enough time in deciding my recommendation to be sure about it) I suck at 2)
Sure, by deciding 1) you come up with some reasons that could be regarded as part of the constructive feedback but nevertheless, the feedback should go beyond merely pointing out some criticisms (lack of evaluation/implementation / …, unreadable, contribution not clear,…). In an ideal world you should treat each paper you review as this paper was a paper proposal written by somebody from your own team (I’m assuming here that you at least pay attention to the paper coming out of your own group 🙂 ). Since my world is far from ideal, this is not something I do. My reviews are short and to the point and the only feedback I give is the few thoughts that came to my mind when reading the paper.
By looking at the reviews of my own papers I know I´m not alone. There are many researchers out there in the same situation. Some conferences are now starting to give “Best reviewer” awards to encourage us to improve our reviews but we need much more than that to invest our time in writing more useful reviews.
Obviously, one alternative would be to review fewer papers but this is just shifting the problem (the number of potential reviewers is limited, at the very least this opens a new discussion: who should be asked to review papers? only people with a PhD? people at least close to finishing the PhD? anybody?). Other people propose to make the reviewing time proportional to the quality of the paper (shitty papers wouldn´t deserve quality reviews) but that’s a risky proposition (we could also argue the opposite, assuming the paper is not shitty because the authors didn´t put too much effort on it but really because they don´t know better). Maybe reviewing time should also be related to the seniority of the authors (i.e. if one of the authors is a full professor then s/he should be able to understand the weaknesses of the paper even with a short review).
As usual, I don´t have a good solution (that´s why the blog is called “research rants” and not “research solutions”) but I´ll be happy to listen to your proposals.