Empirical validation is becoming a MUST if you want a publish a research paper on software engineering topics. In principle, this is a good thing, right? I mean, for sure, in a field like ours validating the usefulness/applicability/.. of our new methods/techniques/… can only be good.

The problem is that we are bringing this to the extreme and we know since Aristotle that a virtue is a point between a deficiency and an excess of a trait. Do NOT request validation for workshop papers (yes, I’ve seen this). In fact, do NOT request validation for many conferences. At this point we should still be in the proposal / maturity ideas phase so rejecting papers based on the lack of validation just kills creativity (if you don’t know how / have the resources to validate the idea and know your paper has no chance without it you may decide just to skip the topic) or, even worse, results in garbage validation.

Validation is only good for science if reviewers not only check whether some validation was done but also evaluate whether the validation was adequate (and this requires some knowledge on empirical validation that many don’t have). Accepting as validation experiments with students or with such a small group of professionals that more than validation what we have is an anecdotical evidence is even worse than no validation at all.

When I started doing research, I experienced a similar phenomenon. Then, the “fever” (in my research niche) was the need to show some kind of prototype tool for the ideas of the paper. If you didn’t have an implementation section you had very little chances to get your paper accepted. Do you think this “pressure” contributed to making sure researchers started developing good tools that could benefit the community?. In case you wonder, the answer is a clear no. This only made all of us to mock up some shitty tools to capture a screenshot to put in the paper.

Validation yes but well done and when needed!

(and from a different perspective and better reasoning, I recommend you to read Experiments as Research Validation – Have We Gone too Far? by Jeffrey D. Ullman )