In the opening session of any conference, the PC Chairs give a brief presentation of the conference. This typically includes informing about the number of abstracts and full papers submitted, the number of papers accepted, the corresponding acceptance rate and some kind of map / graphic displaying the same information by countries.
Usually, that’s it. For the opening session of ECMFA’14 we wanted to give some more (hopefully interesting) data. In the end, due to time constraints, what we gave as additional data was:
- Percentage of accepted papers where none of the authors was (or had been in the last four years) a PC member of the conference
- Acceptance rates of papers were at least one author was (or had been) a PC member
- Number of papers where none of the authors had participated in the community (as a PC member or author) before (again “before” means in the last 4 years)
With the first two we wanted to show that you didn’t need to be a PC member to get your paper in the conference (for ECMFA 42% of papers were from non-PC members) and that having a PC member as co-author did not increase dramatically the probability of getting your paper accepted (acceptance rate for PC co-authored papers was only 10% higher than acceptance rates for papers with no PC member). The third was a way to see how endogamic was the conference (turned out to be quite a lot since only one paper had a complete set of “fresh” authors).
Can I ask you if you would like all conferences to include these three stats in their presentation? And, regardless of your answer, what other data/statistic would you like to know about a confenrence?
Attendance (by authors, by PC members, neither, first-time, repeat, number of organisations/countries represented)
Good point. We didn’t think of it because as PC Chairs we didn´t have this data but I agree it´s a very important view of the conference
+ whether the acceptance ratio was higher for people who have never published anything at this venue but have attended in the past (as opposed to complete “outsiders”)
I would like to read stats about citations of the papers published in previous editions of the conference. It could help understanding which level of “echo” the conference has. The max, mean and median number of citions could be interesting to have.
(on behalf of Björn Thrandur Björnsson) I’d like to see gender statistics, e.g. what is the gender ratio of the conference participants and how does this match up with the gender ratio of plenary, SOTA, oral and poster presenters as decided by the conference organizers.
Would’ve liked to see progress statistics (how many times on average has an accepted paper been rejected before; same per rejected paper), but it’s not going to happen since it requires too much openness for sharing data among conferences and demands too much honesty from the authors who often change the title and shamefully hide the fact they ever produced any work that has been rejected 😉
Agreed (specially in the part about this not going to happen anytime soon 🙂 )
I am a little surprised why it is ‘shameful’ to resubmit (with or without changed title) an earlier rejected paper. The whole idea is to improve the work, and the PC comments help exactly to rectify mistakes or imrpove the presentation (mostly), probably in experiments, etc. So, I fail to understand what is achieved by tracking “progress statistics”.
Not sure why you seem to suggest that Vadim was saying it was shameful to resubmit. I think he means whether knowing the community (the vocabulary, how they typically structure papers,…) helps to write a papers that are a better fit for that community and thus have higher chances of getting accepted.
For a conference, I would like to see the average number of citations of self-authored papers in the same and other conferences separately, as well as to other (non self-authored) papers of the same conference. It can help indicate an aspect of the quality of the conference.
This may be harder to achieve, but I would like to know what was the process put in place to ensure fairness in the review process.
My biggest concern about peer reviewing is that two-three people are deciding on the fate of a research paper. Answers such as “we had a program board meeting” are only partially satisfactory, as it is still an elitary choice (realistically, in the PB/SPC, there may be a few more people with expertise on the paper topic, which does not significantly increase the significance of the judgement).
Interestingly, we proud ourselves and say that the paper, by being accepted at conference X, was accepted by the research community.
I have seen too many good papers that were rejected, and too many not-so-good papers that were accepted, depending on their luck. This, of course, includes many of my papers, in both categories.
Well, I agree with you diagnosis of the problem but not sure what kind of process could be put in place to convince submitters of the fairness of the reviewing process. Maybe make public the internal discussions of the PC members (still hiding their names, though)? suggestions?
Long-term answer: reviews and reviewers should be public information. Just the same way an art/movie/game reviewer does provide his/her own opinion on an artifact by revealing the identity. That is IMO the only way to ensure fairness. If you are confident your opinion is fair, you won’t be worried in revealing who you are. If you (i) don’t write carefully the paper, (ii) are not an expert, (iii) are biased, you will have a problem in revealing your identity. This requires a paradigmatic shift, but that’s the way to go 🙂
Short-term answer: making reviews and internal discussions public would be a good starting point, perhaps publishing them on a social network and allowing everyone to comment/like/dislike the opinions of the reviewers.
Whether reviews should be anonymous or not is a very controversial discussion that goes beyond this post. Myself, I don´t have a clear opinion on that. There are also some important drawbacks (e.g. people not daring to criticize too much papers from the top guys in the community because they don´t want to make them angry). It´s complicated 🙂
See my short-term answer though: you can keep the identity private, but make reviews and internal discussions public. Perhaps this can all be published on a social network where everyone would be allowed to comment/like/dislike the opinions of the anonymous reviewers
This would not reveal any identity.
On your point, I do not see any controversy: just the fear of change and moving on from the status quo 🙂