Sallam Abualhaija
Davide Fucci
Skip Abstract Section
Abstract
[Context] Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks’ inherent hairiness, and, in turn, the heterogeneous reporting structure. [Solution] To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. [Results] In this paper: (i) we report on hands-on experiences of replication, (ii) we review the state-of-the-art and extract replication-relevant information, (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction, and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. [Contribution] This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication, but can also be used in other contexts, e.g., for educational purposes.
- Sallam Abualhaija, Fatma Başak Aydemir, Fabiano Dalpiaz, Davide Dell’Anna, Alessio Ferrari, Xavier Franch, and Davide Fucci. 2024. Online Annex: Replication in Requirements Engineering: the NLP for RE Case. https://doi.org/10.6084/m9.figshare.21824481Google Scholar
Cross Ref
- Sharif Ahmed, Arif Ahmed, and Nasir U. Eisty. 2022. Automatic Transformation of Natural to Unified Modeling Language: A Systematic Review. In 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA). 112–119. https://doi.org/10.1109/SERA54885.2022.9806783Google ScholarCross Ref
- Carlos E. Anchundia and Efraín R. Fonseca C. 2020. Resources for Reproducibility of Experiments in Empirical Software Engineering: Topics Derived From a Secondary Study. IEEE Access 8(2020), 8992–9004. https://doi.org/10.1109/ACCESS.2020.2964587Google ScholarCross Ref
- Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (2016).Google Scholar
- D. Berry, E. Kamsties, and M. Krieger. 2003. From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity, A Handbook. http://se.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdfGoogle Scholar
- Rosanna L Breen. 2006. A practical guide to focus-group research. Journal of geography in higher education 30, 3 (2006), 463–475.Google ScholarCross Ref
- Jeffrey C Carver. 2010. Towards reporting guidelines for experimental replications: A proposal. In 1st international workshop on replication in empirical software engineering, Vol. 1. 1–4.Google Scholar
- Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 18th Conference on Empirical Methods in Natural Language Processing (EMNLP’14).Google ScholarCross Ref
- Jane Cleland-Huang, Raffaella Settimi, Xuchang Zou, and Peter Solc. 2006. The detection and classification of non-functional requirements with application to early aspects. In 14th IEEE International Requirements Engineering Conference. 39–48.Google ScholarDigital Library
- Jane Cleland-Huang, Raffaella Settimi, Xuchang Zou, and Peter Solc. 2007. Automated classification of non-functional requirements. Requirements engineering 12, 2 (2007), 103–120.Google Scholar
- Margarita Cruz, Beatriz Bernárdez, Amador Durán, Jose A Galindo, and Antonio Ruiz-Cortés. 2019. Replication of studies in empirical software engineering: A systematic mapping study, from 2013 to 2018. IEEE Access 8(2019), 26773–26791.Google ScholarCross Ref
- Fabio QB Da Silva, Marcos Suassuna, A César C França, Alicia M Grubb, Tatiana B Gouveia, Cleviton VF Monteiro, and Igor Ebrahim dos Santos. 2014. Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering 19, 3 (2014), 501–557.Google Scholar
- Fabiano Dalpiaz, Davide Dell’Anna, Fatma Basak Aydemir, and Sercan Çevikol. 2019. Requirements Classification with Interpretable Machine Learning and Dependency Parsing. In Proceedings of the 27th IEEE International Requirements Engineering Conference, RE 2019. 142–152. https://doi.org/10.1109/RE.2019.00025Google ScholarCross Ref
- Fabiano Dalpiaz, Davide Dell’Anna, Fatma Başak Aydemir, and Sercan Çevikol. 2019. explainable-re/RE-2019-Materials v0. https://doi.org/10.5281/zenodo.3309669Google ScholarCross Ref
- Fabiano Dalpiaz, Alessio Ferrari, Xavier Franch, and Cristina Palomares. 2018. Natural language processing for requirements engineering: The best is yet to come. IEEE software 35, 5 (2018), 115–119.Google Scholar
- Fabiano Dalpiaz, Ivor van der Schalk, and Garm Lucassen. 2018. Pinpointing Ambiguity and Incompleteness in Requirements Engineering via Information Visualization and NLP. In Proceedings of the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’18).Google ScholarCross Ref
- Fred D. Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 13, 3 (1989), 319–340. http://www.jstor.org/stable/249008Google ScholarDigital Library
- F. De Bruijn and H. Dekkers. 2010. Ambiguity in Natural Language Software Requirements: A Case Study. In Proceedings of the International Workshop on Requirements Engineering: Foundation of Software Quality (REFSQ) (Essen, DE). 233–247.Google Scholar
- Cleyton VC De Magalhães, Fabio QB da Silva, Ronnie ES Santos, and Marcos Suassuna. 2015. Investigations about replication of empirical studies in software engineering: A systematic mapping study. Information and Software Technology 64 (2015), 76–101.Google ScholarDigital Library
- Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Angelo Susi. 2023. Mining and searching app reviews for requirements engineering: Evaluation and replication studies. Information Systems 114(2023), 102181. https://doi.org/10.1016/j.is.2023.102181Google ScholarDigital Library
- Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2021. taphsir v6. https://doi.org/10.5281/zenodo.6009790Google ScholarCross Ref
- Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2022. Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering.Google ScholarDigital Library
- Saad Ezzini, Sallam Abualhaija, Chetan Arora, Mehrdad Sabetzadeh, and Lionel C Briand. 2021. Using domain-specific corpora for improved handling of ambiguity in requirements. In 2021 IEEE/ACM 43rd International Conference on Software Engineering.Google ScholarDigital Library
- D Méndez Fernández, Stefan Wagner, Marcos Kalinowski, Michael Felderer, Priscilla Mafra, Antonio Vetrò, Tayana Conte, M-T Christiansson, Des Greer, Casper Lassenius, et al. 2017. Naming the pain in requirements engineering. Empirical software engineering 22, 5 (2017), 2298–2338.Google Scholar
- Alessio Ferrari, Felice Dell’Orletta, Andrea Esuli, Vincenzo Gervasi, and Stefania Gnesi. 2017. Natural Language Requirements Processing: A 4D Vision.IEEE Softw. 34, 6 (2017), 28–35.Google Scholar
- Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A Dataset of Public Requirements Documents. In 2017 IEEE 25th International Requirements Engineering Conference. 502–505. https://doi.org/10.1109/RE.2017.29Google ScholarCross Ref
- Xavier Franch, Cristina Palomares, Carme Quer, Panagiota Chatzipetrou, and Tony Gorschek. 2023. The state-of-practice in requirements specification: an extended interview study at 12 companies. Requirements Engineering(2023). https://doi.org/10.1007/s00766-023-00399-7Google ScholarDigital Library
- Vincenzo Gervasi, Alessio Ferrari, Didar Zowghi, and Paola Spoletini. 2019. Ambiguity in Requirements Engineering: Towards a Unifying Framework. In From Software Engineering to Formal Methods and Tools, and Back. Springer.Google Scholar
- Martin Glinz. 2007. On non-functional requirements. In 15th IEEE international requirements engineering conference (RE 2007). IEEE, 21–26.Google ScholarCross Ref
- Jesús M González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering 17 (2012), 75–89.Google ScholarDigital Library
- Jesus M Gonzalez-Barahona and Gregorio Robles. 2023. Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Information and Software Technology 164 (2023), 107318.Google ScholarDigital Library
- Ben Hermann, Stefan Winter, and Janet Siegmund. 2020. Community expectations for research artifacts and evaluation processes. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 469–480.Google ScholarDigital Library
- David W Johnson and Frank P Johnson. 1991. Joining together: Group theory and group skills. Prentice-Hall, Inc.Google Scholar
- Natalia Juristo and Sira Vegas. 2011. The role of non-exact replications in software engineering experiments. Empirical Software Engineering 16, 3 (2011), 295–324.Google ScholarDigital Library
- Erik Kamsties and Barbara Peach. 2000. Taming ambiguity in natural language requirements. In Proceedings of the 13th International Conference on Software and Systems Engineering and Applications (ICSSEA’00).Google Scholar
- Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, 2676–2686. https://doi.org/10.18653/v1/P18-1249Google ScholarCross Ref
- Nadzeya Kiyavitskaya, Nicola Zeni, Luisa Mich, and Daniel Berry. 2008. Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. Requirements Engineering 13, 3 (2008).Google Scholar
- Zijad Kurtanović and Walid Maalej. 2017. Automatically classifying functional and non-functional requirements using supervised machine learning. In 2017 IEEE 25th International Requirements Engineering Conference (RE). Ieee, 490–495.Google ScholarCross Ref
- Zijad Kurtanović and Walid Maalej. 2018. On user rationale in software engineering. Requirements Engineering 23, 3 (2018), 357–379.Google ScholarDigital Library
- J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics 33, 2 (1977).Google Scholar
- Feng-Lin Li, Jennifer Horkoff, John Mylopoulos, Renata SS Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014. Non-functional requirements as qualities, with a spice of ontology. In 2014 IEEE 22nd International Requirements Engineering Conference (RE). IEEE, 293–302.Google ScholarCross Ref
- Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2020. Open science in software engineering. Contemporary empirical methods in software engineering (2020), 477–501.Google Scholar
- Daniel Méndez Fernández, Martin Monperrus, Robert Feldt, and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24 (2019), 1057–1060.Google ScholarDigital Library
- Lloyd Montgomery, Davide Fucci, Abir Bouraffa, Lisa Scholz, and Walid Maalej. 2022. Empirical research on requirements quality: a systematic mapping study. Requirements Engineering(2022), 1–27.Google Scholar
- Erik Jan Philippo, Werner Heijstek, Bas Kruiswijk, Michel R. V. Chaudron, and Daniel M. Berry. 2013. Requirement Ambiguity Not as Important as Expected — Results of an Empirical Evaluation. In Proceedings of the International Workshop on Requirements Engineering: Foundation of Software Quality (REFSQ) (Essen, DE). 65–79.Google ScholarDigital Library
- Shekoufeh Rahimi, Kevin Charles Lano, and Chenghua Lin. 2022. Requirement Formalisation Using Natural Language Processing and Machine Learning: A Systematic Review. In International conference on Model-Based Software and Systems Engineering. SCITEPRESS Digital Library, 1–8.Google Scholar
- Cristina Ribeiro and Daniel Berry. 2020. The Prevalence and Severity of Persistent Ambiguity in Software Requirements Specifications: Is a Special Effort Needed to Find Them?Science of Computer Programming 195 (2020), 102472.Google Scholar
- Faiz Ali Shah, Kairit Sirts, and Dietmar Pfahl. 2019. Is the SAFE Approach Too Simple for App Feature Extraction? A Replication Study. In Requirements Engineering: Foundation for Software Quality, Eric Knauss and Michael Goedicke (Eds.). Springer International Publishing, Cham, 21–36.Google Scholar
- Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018. The role and value of replication in empirical software engineering results. Information and Software Technology 99 (2018), 120–132.Google ScholarDigital Library
- Forrest J Shull, Jeffrey C Carver, Sira Vegas, and Natalia Juristo. 2008. The role of replications in empirical software engineering. Empirical software engineering 13, 2 (2008), 211–218.Google Scholar
- Roel J Wieringa. 2014. Design science methodology for information systems and software engineering. Springer.Google ScholarDigital Library
- Jonas Winkler and Andreas Vogelsang. 2016. Automatic Classification of Requirements Based on Convolutional Neural Networks. In 24th IEEE International Requirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016. IEEE Computer Society, 39–45. https://doi.org/10.1109/REW.2016.021Google ScholarCross Ref
- Stefan Winter, Christopher S Timperley, Ben Hermann, Jurgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. 2022. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 145–156.Google ScholarDigital Library
- Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.Google ScholarCross Ref
- Hui Yang, Anne de Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. 2011. Analysing anaphoric ambiguity in natural language requirements. Requirements Engineering 16, 3 (may 2011), 163. https://doi.org/10.1007/s00766-011-0119-yGoogle ScholarDigital Library
- Hui Yang, Anne de Roeck, Alistair Willis, and Bashar Nuseibeh. 2010. A Methodology for Automatic Identification of Nocuous Ambiguity. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010). Coling 2010 Organizing Committee, Beijing, China, 1218–1226. https://aclanthology.org/C10-1137Google Scholar
- Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro. 2021. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. ACM Computing Surveys (CSUR) 54, 3 (2021), 1–41.Google Scholar
- Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223(2023).Google Scholar
Index Terms
- Replication in Requirements Engineering: the NLP for RE Case
Recommendations
Natural Language Processing for Requirements Engineering: A Systematic Mapping Study
Natural Language Processing for Requirements Engineering (NLP4RE) is an area of research and development that seeks to apply natural language processing (NLP) techniques, tools, and resources to the requirements engineering (RE) process, to support ...
Slr on Identification & Classification of Non-Functional Requirements Attributes, and Its Representation in Functional Requirements
CSAI '18: Proceedings of the 2018 2nd International Conference on Computer Science and Artificial IntelligenceSoftware requirements are crucial. Tight relationship between functional and non-functional will determine the quality of requirements, in which ultimately determines the quality of the software being built. Non-Functional Requirements (NFR) often get ...
Branch replication scheme: A new model for data replication in large scale data grids
Data replication is a practical and effective method to achieve efficient and fault-tolerant data access in grids. Traditionally, data replication schemes maintain an entire replica in each site where a file is replicated, providing a read-only model. ...
Comments