I took the time to count how many new reviews for the ECMFA’14 conference were uploaded each day to the easychair account for the conference.
The results, displayed below, are exactly what I was expecting (it’s also my own behaviour 🙂 ). Even if the reviewers had one full month to complete the reviews, 80% of them came in during the last week (day 30 in the graphic was the deadline, as part of the last week I count late reviews arrived days 31 and 32).
Even if this data is taken from a single conference (hey, this a rant blog post, not a scientific paper!) I’m sure you have the same feeling: no matter how much time you give to the reviewers, most of them they will always do the reviews last minute. If so, then, why do we need to give so much time to review (conference) papers? We could have a quicker turnaround (which should be one of the main benefits of sending a paper to a conference) if we just drastically cut the reviewing period and give just two weeks.
Based on our collective behaviour (there are always so many “urgents” things to do that we don´t plan, we just react so until we start getting the warning about upcoming deadlines we don´t put that on top of our to-do list) I don’t think the quality of the reviews would be worse than what we have now and authors would get their notifications earlier.
What do you think?
I wonder how many reviewers read the papers first, contemplate, then write and submit their reviews? I generally do that, but usually fairly quickly (i.e., read the papers, spend a day or two musing, then sit down to write the reviews). A shorter window may affect some reviewers who like to operate in this manner.
I considered that but I´m skeptical. I also print the papers and manually mark them before entering the reviews online but I do it almost the same day (my memory is quite short term!).
I don´t think there are many people that review the papers and wait two weeks or more to upload the reviews.
Sure, the easychair stats doesn´t give us this info so if a conference would like to try a really short review period may want to ask first the PC members for their input.
I would consider a few threats to validit:
– Maybe people write reviews but wait to submit that (hoping to have more time to read them again… a time that will never come, of course).
– People gave papers to sub-reviewers and wait for them to gave back the paper; sub-reviewers could wait some time to pretend having worked accurately on the review…
– Reviewer prefer to reaf first other reviews to then just agree with them (and avoiding possible discussions).
Number 1 could be an option as discussed in the previous comments. Number 2, well, senior researchers are experienced enough to know that subreviewers will also do it last minute (agreed, maybe not so last last minute). Number 3 is not possible (in any reasonable conference you don’t see other reviews for the same paper you’re reviewing until you’ve submitted yours)
It is easier to agreed on being PC member if you have a larger window. I have the kind of behavior you point out, but what I try to do for example is to use travel time (long flight) or week-ends (I know, I know, not need to tell me this is bad, I have a wife!). Here’s what happen: I made the reviews (most of them at least), in terms of hand-written notes on the papers themselves, but… I don’t upload the review itself! I wait for the deadline to pop-up on my computer on in my mailbox. Reason is mainly because I have other more important stuff to work on and doing something that can be done later instead of something that cannot wait does not seems OK. So:
– yes the window can be reduced
– no it can’t be one week 😉
Nice feedback anyway Jordi.
I´m sure we can find a middle point between one week and two months (as some conferences do)!
I have always been led to believe that a long review period just ensures that all the reviewers could indeed find that one week when they have the time to read the papers, think about them and write down their thoughts. No conference paper really needs several weeks to review, it’s not a journal 100 page monster, the trick is to find the time to dedicate to it, which can get tricky for active senior researchers with lots of administrative, teaching and travelling activities and responsibilities.
It turns out that “that one week” is always the last one 🙂
As other people also point out above, the last week is the one when the reviews actually get uploaded, it does not mean they are being written during the last week. Writing a small review from red pen notes on a paper is a matter of half an hour for me, and usually I reserve that pleasure for the last hours of the last day ;）
Unless we ask directly the PC members we can’t know whether many people do as you say. I insist I don’t believe this is the general case.