As organizer/PC member/author in many conferences and workshops, I’ve seen too many times the hypercriticality of reviewers of papers/projects and I couldn’t agree more on this great column on the topic

I could reject all papers assigned to me. I can always point out to a poor (or lack of) implementation, empirical validation, real case study, formal proof… you name it. I could easily create a “reject template” to automatically apply to all my reviews (maybe I should 🙂  I would save a lot of time!)

But this is not the point of being a reviewer. To me a review has two main parameters, the paper to review and the venue (journal or conference) where the paper has been submitted. IMHO, many reviews forget about this second variable. It’s not the same to review a paper for a workshop than to review a paper for a top journal and reviewers many times apply the same criteria in both scenarios.

If you’re reviewing a paper for a middle-tier conference, you cannot pretend that the paper presents a mature approach with a complete tool implementation, empirical validation on real industrial use cases, a formal proof, ….

Let’s be honest, if my work had all this I wouldn’t be sending it to your conference. I’d send it to a journal (at least until evaluation agencies don’t change their policy and keep considering conference publications as second level results; whatever they should considered as such it’s a completely different discussion).