Today I’m coming out of the closet: I declare myself a bad reviewer.
To me, peer-reviewing a journal or conference paper implies two main tasks:
- Deciding whether the paper should be accepted for publication
- Providing constructive feedback to the authors so that they can improve their paper
While I´m convinced I’m doing a good job at 1) (or at least I make sure I invest enough time in deciding my recommendation to be sure about it) I suck at 2)
Sure, by deciding 1) you come up with some reasons that could be regarded as part of the constructive feedback but nevertheless, the feedback should go beyond merely pointing out some criticisms (lack of evaluation/implementation / …, unreadable, contribution not clear,…). In an ideal world you should treat each paper you review as this paper was a paper proposal written by somebody from your own team (I’m assuming here that you at least pay attention to the paper coming out of your own group 🙂 ). Since my world is far from ideal, this is not something I do. My reviews are short and to the point and the only feedback I give is the few thoughts that came to my mind when reading the paper.
By looking at the reviews of my own papers I know I´m not alone. There are many researchers out there in the same situation. Some conferences are now starting to give “Best reviewer” awards to encourage us to improve our reviews but we need much more than that to invest our time in writing more useful reviews.
Obviously, one alternative would be to review fewer papers but this is just shifting the problem (the number of potential reviewers is limited, at the very least this opens a new discussion: who should be asked to review papers? only people with a PhD? people at least close to finishing the PhD? anybody?). Other people propose to make the reviewing time proportional to the quality of the paper (shitty papers wouldn´t deserve quality reviews) but that’s a risky proposition (we could also argue the opposite, assuming the paper is not shitty because the authors didn´t put too much effort on it but really because they don´t know better). Maybe reviewing time should also be related to the seniority of the authors (i.e. if one of the authors is a full professor then s/he should be able to understand the weaknesses of the paper even with a short review).
As usual, I don´t have a good solution (that´s why the blog is called “research rants” and not “research solutions”) but I´ll be happy to listen to your proposals.
I think I’m a pretty good reviewer – but I struggle more with your first point (making a recommendation) than providing detailed feedback. Part of my difficulty is that venues (journals, conferences) are often confused as to what they want: do they want exciting material that will spark discussion, or do they want mature work? Some places (like ICSE) claim to want both but really don’t – there are unwritten rules that you have to tease out. This makes reviewing hard: you don’t know what your mandate is.
I find the “average/mediocre” papers the hardest to review. These are the ones with a not-particularly-novel idea, presented reasonably well – you know, papers that are professionally done, but don’t really tell us much that’s new. You can put an infinite amount of effort into reviewing such papers, but it’s not clear what the ROI is to both me and the community. If the paper has a really cool idea, then I tend to go beyond the call of duty.
The problem really is whole-system: if I had to review 25 rather than 125 papers a year, I’d do a better job. So people need to submit fewer papers (I’m doing that!). Hiring committees need to expect to see fewer, but higher quality papers on the CVs of our researchers who are applying for jobs. This doesn’t mean that we should push our students to follow the homogeneous model I see in a lot of North American CVs (“You must have an ICSE paper, a POPL paper, a PLDI paper, and an ESEC/FSE paper or you will not be hired.”) which just serves to make everyone indistinguishable — that’s another rant. But committees should go back to assessing quality of research rather than filtering based on quantity.
Well said. That’s what I though very often while doing reviews. I always tried to point the author to the right direction, give hints and to show the issues. I wrote exhaustive reviews for bad ones as they should learn and short reviews for excellent ones as they seem to know who to write good papers. However, once I left my position at university, I wasn’t asked anymore to perform reviews. On the other hand, I would have done a review from time to time even after that time. Then again, if you are not willing to perform a whole bunch of reviews, you are not asked at all. To answer your question, what would be necessary for me to do a review: a) I have to be asked (politely), b) I have to be asked to do one maybe two reviews, not six of them, c) there should be a at least some kind of acknowledgment on the conference site/book that you’ve spent time and effort in doing a review.