Analysing Agreement Among Different Evaluators in God Class and Feature Envy Detection

Abstract

The automatic detection of Design Smells has evolved in parallel to the evolution of automatic refactoring tools. There was a huge rise in research activity regarding Design Smell detection from 2010 to the present. However, it should be noted that the adoption of Design Smell detection in real software development practice is not comparable to the adoption of automatic refactoring tools. On the basis of the assumption that it is the objectiveness of a refactoring operation as opposed to the subjectivity in definition and identification of Design Smells that makes the difference, in this paper, the lack of agreement between different evaluators when detecting Design Smells is empirically studied. To do so, a series of experiments and studies were designed and conducted to analyse the concordance in Design Smell detection of different persons and tools, including a comparison between them. This work focuses on two well known Design Smells: God Class and Feature Envy. Concordance analysis is based on the Kappa statistic for inter-rater agreement (particularly Kappa-Fleiss). The results obtained show that there is no agreement in detection in general, and, in those cases where a certain agreement appears, it is considered to be a fair or poor degree of agreement, according to a Kappa-Fleiss interpretation scale. This seems to confirm that there is a subjective component which makes the raters evaluate the presence of Design Smells differently. The study also raises the question of a lack of training and experience regarding Design Smells.