Viviane Furtado
Marcos Kalinowski
ABSTRACT
The importance of experimentation for Software Engineering research has been notably established in the last years. The software engineering community has discussed how to proper report and evaluate experiments using different approaches, such as quality criteria, scales, and checklists. Nevertheless, there are no guidelines to support researchers and practitioners active in straightforward software engineering research areas, as in Software Product Lines (SPL), at conducting experiments. We hypothesize that experimentation guidelines may aid such a specific area by providing advice and actual excerpts reflecting good practices of SPL experimentation, thus experimentally evolving this area. Therefore, the goal of this paper is to provide guidelines for properly reporting and promoting SPL experiments. We defined such guidelines based on well-known software engineering experiment reports, quality evaluation checklists, and data extracted from 211 SPL experiments identified in a systematic mapping study. We evaluated the guidelines with a qualitative study with SPL and experimentation experts applying open and axial coding procedures. The evaluation enabled us to improve the guidelines. The resulting guidelines contain specific advice to researchers active in SPL and provide examples taken from published SPL experiments. The experts’ positive points indicate that the proposed guidelines can aid SPL researchers and practitioners. Sharing the resulting guidelines could support conducting SPL experiments and allow further area evolution based on prospective experiment replications and reproductions from well-designed and reported experiments.
- Mohsen Asadi, Samaneh Soltani, Dragan Gašević, and Marek Hatala. 2016. The effects of visualization and interaction techniques on feature model configuration. Empirical Software Engineering 21 (2016), 1–38.Google Scholar
Digital Library
- Victor R. Basili and H. Dieter Rombach. 1988. The TAME project: Towards improvement-oriented software environments. IEEE Trans. Soft. Eng. 14, 6 (1988), 758–773.Google ScholarDigital Library
- Juliet Corbin and Anselm Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3 ed.). Sage Publications.Google Scholar
- Oscar Dieste and Natalia Juristo. 2013. Challenges of Evaluating the Quality of Software Engineering Experiments. Springer Berlin Heidelberg, 159–177.Google Scholar
- Tore Dybå, Vigdis By Kampenes, and Dag I.K. Sjøberg. 2006. A systematic review of statistical power in software engineering experiments. Information and Software Technology 48, 8 (2006), 745–755.Google ScholarCross Ref
- Hamzeh Eyal-Salman, Abdelhak-Djamel Seriai, and Christophe Dony. 2014. Feature location in a collection of product variants: Combining information retrieval and hierarchical clustering. In SEKE. 426–430.Google Scholar
- Eduardo Figueiredo, Alessandro Garcia, Marcelo Maia, Gabriel Ferreira, Camila Nunes, and Jon Whittle. 2011. On the impact of crosscutting concern projection on code measurement. In AOSD. ACM, 81–92.Google Scholar
- Viviane R. Furtado, Henrique Vignando, Victor França, and Edson OliveiraJr. 2019. Comparing Approaches for Quality Evaluation of Software Engineering Experiments: An Empirical Study on Software Product Line Experiments. Journal of Computer Science 15, 10 (2019), 1396–1429.Google ScholarCross Ref
- Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting Experiments in Software Engineering. Springer London, 201–228.Google Scholar
- Ian Jolliffe. 2011. Principal component analysis. Springer, 1094–1096.Google Scholar
- Derek M. Jones. 2020. Evidence-based Software Engineering(1 ed.). Knowledge Software, Ltd. 454 pages.Google Scholar
- Natalia Juristo and Omar S. Gómez. 2012. Replication of Software Engineering Experiments. Springer Berlin Heidelberg, Berlin, Heidelberg, 60–88.Google Scholar
- Vigdis Kampenes. 2007. Quality of design, analysis and reporting of software engineering experiments: A systematic review. Ph. D. Dissertation. Department of Informatics, Faculty of Mathematics and Natural Sciences, University of Oslo.Google Scholar
- Barbara Kitchenham, Hiyam Al-Khilidar, Muhammed Ali Babar, Mike Berry, Karl Cox, Jacky Keung, Felicia Kurniawati, Mark Staples, He Zhang, and Liming Zhu. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Engineering 13, 1 (2008), 97–121.Google ScholarDigital Library
- B. Kitchenham and S. Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007-001. Keele University and Durham University Joint Report.Google Scholar
- Barbara Kitchenham, Dag I. K. Sjøberg, O. Pearl Brereton, David Budgen, Tore Dybå, Martin Höst, Dietmar Pfahl, and Per Runeson. 2010. Can We Evaluate the Quality of Software Engineering Experiments?. In ESEM. 1–8.Google Scholar
- Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2016. Evidence-Based Software engineering and systematic reviews. Vol. 4. CRC Press.Google Scholar
- Barbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on software engineering 28, 8 (2002), 721–734.Google ScholarDigital Library
- Xiaoli Lian LiZhang. 2014. An Evolutionary Methodology for Optimized Feature Selection in Software Product Lines. In SEKE. 63–66.Google Scholar
- Roberto E. Lopez-Herrejon, Lukas Linsbauer, José A. Galindo, José A. Parejo, David Benavides, Sergio Segura, and Alexander Egyed. 2015. An assessment of search-based techniques for reverse engineering feature models. Journal of Systems and Software 103 (2015), 353–369.Google ScholarDigital Library
- Ivan do Carmo Machado, Paulo Anselmo da Mota Silveira Neto, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. RiPLE-TE: A Process for Testing Software Product Lines. In SEKE. 711–716.Google Scholar
- Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (2015), 1–18.Google ScholarDigital Library
- Iris Reinhartz-Berger, Kathrin Figl, and Øystein Haugen. 2014. Comprehending feature models expressed in CVL. In MODELS. Springer, 501–517.Google Scholar
- Rolando P. Reyes, Oscar Dieste, Efraín R. Fonseca, and Natalia Juristo. 2018. Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review. In ICSE. 1195––1206.Google Scholar
- Márcio Ribeiro, Paulo Borba, and Christian Kästner. 2014. Feature Maintenance with Emergent Interfaces. In ICSE. ACM, 989––1000.Google Scholar
- Ildevana Poltronieri Rodrigues, Ana Paula Terra Bacelo, Milene Selbach Silveira, Márcia de Borba Campos, and Elder Macedo Rodrigues. 2016. Evaluating the Representation of User Interface Elements in Feature Models: an Empirical Study. In SEKE. 628–633.Google Scholar
- Alcemir Rodrigues Santos, Ivan do Carmo Machado, and Eduardo Santana de Almeida. 2016. RiPLE-HC: Javascript Systems Meets SPL Composition. In SPLC. ACM, 154–163.Google Scholar
- W. B. Santos, E. S. de Almeida, and S. R. de L. Meira. 2012. TIRT: A Traceability Information Retrieval Tool for Software Product Lines Projects. In EUROMICRO. 93–100.Google Scholar
- Carolyn B. Seaman. 1999. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Trans. Softw. Eng. 25, 4 (1999), 557–572.Google ScholarDigital Library
- Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018. The role and value of replication in empirical software engineering results. Information and Software Technology 99 (2018), 120 – 132.Google ScholarDigital Library
- Forrest Shull, Manoel G. Mendoncça, Victor Basili, Jeffrey Carver, José C. Maldonado, Sandra Fabbri, Guilherme Horta Travassos, and Maria Cristina Ferreira. 2004. Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering 9, 1 (2004), 111–137.Google ScholarDigital Library
- Paulo Anselmo da Mota Silveira Neto, Ivan Machado, Yguarata Cerqueira Cavalcanti, Eduardo Santana de Almeida, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2010. A regression testing approach for software product lines architectures. In SBCARS. IEEE, 41–50.Google Scholar
- Paulo Anselmo da Mota Silveira Neto, Ivan Machado, Yguarata Cerqueira Cavalcanti, Eduardo Santana de Almeida, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2012. An experimental study to evaluate a SPL architecture regression testing approach. In IRI. IEEE, 608–615.Google Scholar
- D. I. K. Sjoberg, B. Anda, E. Arisholm, T. Dyba, M. Jorgensen, A. Karahasanovic, E. F. Koren, and M. Vokac. 2002. Conducting realistic experiments in software engineering. In ISESE. 17–26.Google Scholar
- Dag I. K. Sjøberg, Tore Dybå, Bente C. D. Anda, and Jo E. Hannay. 2008. Building Theories in Software Engineering. Springer London, London, 312–336.Google Scholar
- M. Solari. 2013. Identifying Experimental Incidents in Software Engineering Replications. In ESEM. 213–222.Google Scholar
- Martín Solari, Sira Vegas, and Natalia Juristo. 2018. Content and structure of laboratory packages for software engineering experiments. Information and Software Technology 97 (2018), 64–79.Google ScholarCross Ref
- Guilherme Horta Travassos and Márcio Barros. 2003. Contributions of in virtuo and in silico experiments for the future of empirical studies in software engineering. In Workshop on empirical software engineering the future of empirical studies in software engineering. 117–130.Google Scholar
- Claes Wohlin. 2016. Is There a Future for Empirical Software Engineering?. In ESEM. ACM, New York, NY, USA.Google Scholar
- Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Bjöorn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer.Google ScholarCross Ref
Recommendations
Evaluating guidelines for empirical software engineering studies
ISESE '06: Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineeringBackground. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for ...
Describing what experimental software engineering experts do when they design their experiments: a qualitative study
ESEM '17: Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementBackground: Although there has been a significant amount of research focused on designing and conducting controlled experiments, few studies report how experienced experimental software engineering researchers actually design and conduct their studies. ...
Guidelines for conducting and reporting case study research in software engineering
Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting ...
Comments