Empirical validation is becoming a MUST if you want a publish a research paper on software engineering topics. In principle, this is a good thing, right? I mean, for sure, in a field like ours validating the usefulness/applicability/.. of our new methods/techniques/… can only be good.
The problem is that we are bringing this to the extreme and we know since Aristotle that a virtue is a point between a deficiency and an excess of a trait. Do NOT request validation for workshop papers (yes, I’ve seen this). In fact, do NOT request validation for many conferences. At this point we should still be in the proposal / maturity ideas phase so rejecting papers based on the lack of validation just kills creativity (if you don’t know how / have the resources to validate the idea and know your paper has no chance without it you may decide just to skip the topic) or, even worse, results in garbage validation.
Validation is only good for science if reviewers not only check whether some validation was done but also evaluate whether the validation was adequate (and this requires some knowledge on empirical validation that many don’t have). Accepting as validation experiments with students or with such a small group of professionals that more than validation what we have is an anecdotical evidence is even worse than no validation at all.
When I started doing research, I experienced a similar phenomenon. Then, the “fever” (in my research niche) was the need to show some kind of prototype tool for the ideas of the paper. If you didn’t have an implementation section you had very little chances to get your paper accepted. Do you think this “pressure” contributed to making sure researchers started developing good tools that could benefit the community?. In case you wonder, the answer is a clear no. This only made all of us to mock up some shitty tools to capture a screenshot to put in the paper.
Validation yes but well done and when needed!
(and from a different perspective and better reasoning, I recommend you to read Experiments as Research Validation – Have We Gone too Far? by Jeffrey D. Ullman )
Can you elaborate on what validation or empirical validation means in your field? I’ve found that the definition of verification and validation varies widely between fields. For example, in my area, validation means comparison to experimental results. So, I’m curious as to what it means in your field regarding a workshop paper vs. a journal paper.
Indeed, validation may mean different things depending on what you want to validate. I was mainly referring to the validation of the usefulness of the proposed techniques. In my field many papers claim the new technique they propose improves productivity/quality/…ty of software development so a good validation should imply some kind of controlled experiment where two groups of professional developers (only one using the new technique) do the same task and then results are compared to see if the improvement really exists.
To me, doing this with groups of students (good luck trying to convince professionals to participate in research experiments) is a strong threat to the validity of the experiment
Jordi, you may know I am a big supporter of the claims you are making here. Empirical papers are truly important, if the validation is properly conducted. My opinion is that there are very few researchers who are going to provide an opinion about the proposed technique itself, perhaps (I may be wrong) because this requires the effort of providing arguments for/against the opinion. Much easier to bypass the “problem” and say: this is similar to X/Y, no tool, no validation, you missed (my) related work, and the alike.