As organizer/PC member/author in many conferences and workshops, I’ve seen too many times the hypercriticality of reviewers of papers/projects and I couldn’t agree more on this great column on the topic
I could reject all papers assigned to me. I can always point out to a poor (or lack of) implementation, empirical validation, real case study, formal proof… you name it. I could easily create a “reject template” to automatically apply to all my reviews (maybe I should 🙂 I would save a lot of time!)
But this is not the point of being a reviewer. To me a review has two main parameters, the paper to review and the venue (journal or conference) where the paper has been submitted. IMHO, many reviews forget about this second variable. It’s not the same to review a paper for a workshop than to review a paper for a top journal and reviewers many times apply the same criteria in both scenarios.
If you’re reviewing a paper for a middle-tier conference, you cannot pretend that the paper presents a mature approach with a complete tool implementation, empirical validation on real industrial use cases, a formal proof, ….
Let’s be honest, if my work had all this I wouldn’t be sending it to your conference. I’d send it to a journal (at least until evaluation agencies don’t change their policy and keep considering conference publications as second level results; whatever they should considered as such it’s a completely different discussion).
As organizer/PC member/author in many conferences and workshops, I’ve also seen hypercritical reviews. I fully agree on the point that the venue is many times neglected, as I agree on the fact (pointed in the ACM column) that in many cases “reject” is the default mode. My two cents are to point out that reviewers can not “guess” the rules for a conference. Therefore, organizers, and in particular Program Chairs should do the work of providing clear guidelines about what is “acceptable” and what should be rejected. Having said this, I plead guilty, cause in the past when I have served as Program Chair I have assumed my fellow colleagues of the PC would know what to do with the reviews. However, I take note and plan to do it from now on.
PS. One additional thing to take into account is the issue of anonimous reviews, but I think that deserves its own thread 😉
And for another post we could also add the topic of the right policies to declare conflicts of interest between reviewers and authors