A mapping study on design-time quality attributes and metrics
Introduction
Software quality is an ambiguous term, in the sense that: (a) from the viewpoint of the user, quality is about how software meets its purpose, (b) from the developers’ point of view, quality is about the conformance of software to its specifications, (c) from the product view, quality deals with the structural characteristics of the software, and (d) from a monetary viewpoint, quality is about the amount of money that a client is willing to pay to obtain it (Kitchenham and Pfleeger, 1996). Additionally, quality assurance cannot be performed in the same way across different software projects. Instead, assuring the levels of quality for a specific project requires answering the following questions, as outlined in Fig. 1:
- •
What quality attributes should be monitored? One of the first activities in software development is the selection of quality attributes (QAs) that are the most important for the specific project (usually termed as forces or architecture key-drivers) (Bass et al., 2003). Quality attributes are project-specific since different software applications have different priorities, concerns and constraints. Nevertheless, we anticipate that projects belonging to the same application domain are presenting a similar prioritization for their key-drivers (Eckhardt et al., 2017). For example, critical-embedded systems put special emphasis on run-time quality attributes (e.g., performance, energy efficiency, etc.), whereas applications with intense interaction with the users (e.g., enterprise applications), focus on design-time ones (e.g., maintainability, extendibility, etc.). However, monitoring quality attributes cannot be performed in the same way in all phases of software development, in the sense that different phases focus on different quality aspects of the software. For example, during the requirements phase the engineers are expected to be less focused to code-level quality aspects (e.g., cohesion, coupling, etc.), whereas during the testing phase the engineers are more probably concerned about the correctness and completeness of the implementation. Therefore, quality attributes should not only be prioritized by application domain, but by development phase, as well.
- •
How can these quality attributes be monitored? After selecting the quality attributes of interest for every type of product development phase, the next step is the development of a measurement plan to monitor the levels of the specific quality attributes, given the constraints of the specific phase (e.g., available artifacts) (ISO/IEC 25023, 2003). However, there are no widely accepted sets of metrics for assessing a quality attribute across all development phases, since: (a) there is no set of metrics that is appropriate for all phases, and (b) quality attributes are not always associated with metrics. The usefulness of metrics that are accurately mapped to quality attributes has been extensively discussed by Harrison et al. (1998). However, only lately there have been efforts to develop a quality model where quality attributes are associated with measurable elements. For example, ISO/IEC 25023 (2003) provides measures for the characteristics in the product quality model. An additional information that is needed for the selection of specific metrics is their validity as assessors of the targeted quality attribute, and the availability of tools that can automate their calculation.
The goal of this study is to provide the necessary guidance to researchers and practitioners for answering the aforementioned questions. In this paper, we summarize the state-of-research on design-time quality attributes and metrics by conducting a mapping study. Therefore, the goal of this paper is to conduct a fair overview of “good quality” studies1 on design-time quality attributes and related quality metrics. In particular, we identify and analyse research on quality in software engineering, without focusing on any programming paradigm / language, any application domain (e.g., telecommunication, embedded systems), or any software engineering phase (e.g., requirements engineering, software architecture, etc.). Thus, the outcome of this study provides the following contributions:
- [c1]
Highlight the most important design-time quality attributes in different application domains and development phases. This overview contributes a comprehensive list of design-time quality attributes that have been identified in different application domains, and which are of paramount importance in each development phase. Based on this, researchers can spot: (a) the most important design time quality attributes for each domain and development phase, and propose domain- or phase-specific approaches that tackle them, and (b) the aspects of quality in a specific application domains or development phases that have not been studied in detail and therefore might require more attention. Practitioners can use this comprehensive list of design time quality attributes as a checklist to find potential quality attributes for their particular project in every phase of development, based on the application domain in which the project belongs. Based on the outcome of this contribution, practitioners will be able to perform the process for quality attribute selection.
- [c2]
A mapping of design-time quality attributes and metrics. Software metrics are used to quantify quality attributes. Thus, our study compiles a catalogue of metrics related to design-time quality attributes. In particular, we study five perspectives of this relation:
- (a)
we identify if a quality attribute is quantified through a formula that is based on aggregating other metrics, or is assessed through a set of metrics that cannot be aggregated (i.e., the quality attribute would be measured by individual metrics),
- (b)
we map quality attributes to the metrics that can be used for their quantification,
- (c)
we present the validation on the relationship between metrics and quality attributes and the provided level of evidence,
- (c)
we discuss the development phase in which different metrics can be calculated, and
- (d)
we provide a list of tools that can be used for automatically calculating the metric scores for a specific system.
- (a)
By exploiting these five perspectives, practitioners can be guided in their metric selection and application processes. More specifically, after a practitioner picks a quality attribute for each development phase (based on c1), he/she: (i) inspects how the quality attribute can be quantified (through a formula or a set of metrics), (ii) after checking the available metrics for its quantification in the current phase, and considering their validity levels, he/she can select the set of metrics that will be used, and (iii) based on the selected metrics, he/she will decide which tools can be reused or developed from scratch. Similarly, researchers can check which quality attributes are well-supported by metrics and which quality attributes might require novel metrics. Additionally, based on metric validity assessment, researchers can identify quality attributes whose quantification requires further evaluation.
In this mapping study, we are interested only in “good quality” studies, in order to provide researchers and practitioners with an analysis of thoroughly conducted, validated and reliable research efforts (for further justification on the selection of “good quality” studies, see Section 3.2). Therefore, we aimed at identifying studies published only in particular “good quality” venues (more details on venue selection are presented in Section 3.2). Additionally, we focus on studies that introduce or evaluate quality attributes and metrics, excluding papers that use metrics for other software engineering purposes. We exclude these studies, since we expect that the employed metrics have already been identified in the studies in which they have been introduced (if published in one of the examined publication venues). Using snowballing to identify metric definitions that were published at venues outside the searching scope of the study was not applied since searching specific publication venues already resulted in a large number of primary studies.2 Nevertheless, in such papers metrics are only the means for conducting the study rather than the goal/focus of the study. Therefore, they have been excluded from our secondary study. Including them as primary studies, would bias the obtained dataset by double-counting metrics that are used for different reasons. For example, a study that uses Coupling Between Objects (CBO) and Lack of Cohesion of Methods (LCOM) (Chidamber and Kemerer, 1994) to measure the effect of applying a certain refactoring is not evaluating the usefulness of the metrics, but the usefulness of the refactoring. Furthermore, since there is a lot of literature on software quality (Jabangwe et al., 2004), we decided to narrow the scope of this study to one type of quality attributes, namely design-time quality attributes (Abran and Moore, 2004). The Software Engineering Body of Knowledge (SWEBOK) defines design-time quality attributes as any aspect of software quality that is not discernible at run-time, (e.g., modifiability, reusability) (Abran and Moore, 2004).
The rest of the paper is organized as follows: In Section 2, we discuss other secondary studies that are related to quality attributes or metrics. Next, in Section 3, we present the systematic mapping protocol, whereas in Sections 4 and 5, we present and discuss the results of this mapping study, respectively. Finally, in Section 6, we present threats to validity, and in Section 7 we conclude the paper.
Section snippets
Related work
In this section, we present secondary studies (namely systematic literature reviews and mapping studies) that are related to quality attributes and metrics. Whenever possible, we compare the goals of related work to our study and discuss points of differentiation. We do not discuss secondary studies that focus on run-time quality attributes, e.g., fault prediction (Catal and Diri, 2009, Radjenović et al., 2013), reliability (Febrero et al., 2014), etc., since our work is exploring design-time
Study design
This section presents the protocol of the systematic mapping study. A protocol constitutes a plan that describes research questions and how the secondary study has been conducted. Our protocol is presented according to the guidelines of Petersen et al. (2008).
Results
After searching and filtering as described in Sections 3.2 and 3.3, we obtain a dataset of 154 primary studies. In Table 3, for each considered publication venue, we present the number of papers that have been returned as candidate primary studies (step 1), the number of papers qualified after primary study selection based on title and abstract (step 2), and the final number of primary studies (step 3)—see Fig. 3.
The main reason for excluding papers in the manual filtering phase (based on the
Discussion
In this section, we discuss the main outcomes of this mapping study by interpreting our findings, by comparing them to related work (when applicable), by synthesizing the results of each sub-research question to answer the main RQs, and by providing implications to researchers and practitioners.
Threats to validity
In this section, we present the threats to validity that concern our mapping study.
First, we discuss threats to validity that can lead to errors in the primary study identification process (e.g., the selection of digital libraries and search string construction, and study selection bias). Retrieving primary studies from selected publication venues and not by searching in digital libraries, might lead to missing studies that are published in other venues. However, as stated before, the aim of
Conclusions
This study aimed at providing a detailed panorama of the state-of-the-art on design time quality attributes and metrics to assess them. To achieve this goal, we performed a systematic mapping study and investigated studies that are published in top software engineering venues. The results of the study suggest that maintainability is the most commonly studied quality attribute, regardless of the application domain or the development phase. High-level quality attributes are most commonly assessed
Elvira Maria Arvanitou is a PhD Student at the University of Groningen, the Netherlands, in the group of Software Engineering and Architecture. She holds an MSc degree in Information Systems from the Aristotle University of Thessaloniki, Greece (2013), and a BSc degree in Information Technology from the Technological Institute of Thessaloniki, Greece (2011). Her research interests include software quality assurance and metrics, software maintainability and stability.
References (49)
- et al.
A mapping study to investigate component-based software system metrics
J. Syst. Softw.
(2013) - et al.
Requirements engineering for software product lines: a systematic literature review
Inf. Softw. Technol. Elsevier
(2010) - et al.
Empirical studies of quality models in object oriented systems
Adv. Comput.
(2002) - et al.
An An analysis of research topics in software engineering – 2006
J. Syst. Softw.
(2008) - et al.
A systematic review of software fault prediction studies
(2009) - et al.
A systematic mapping study on the combination of static and dynamic quality assurance techniques
Inf. Softw. Technol.
(2012) - et al.
A systematic mapping study of software reliability modeling
Inf. Softw. Technol.
(2014) - et al.
Systematic literature reviews in software engineering: a systematic literature review
Inf. Softw. Technol.
(2009) - et al.
Object-oriented metrics that predict maintainability
J. Syst. Softw.
(1993) - et al.
Quality models for web services: a systematic mapping
Inf. Softw. Technol.
(2014)
Software fault prediction metrics: a systematic literature review
An application of Bayesian network for predicting object-oriented software maintainability
Inf. Softw. Technol.
Identifying relevant studies in software engineering
Inf. Softw. Technol.
An assessment of systems and software engineering scholars and institutions (2003–2007 and 2004–2008)
J. Syst. Softw.
Guide to the Software Engineering Body of Knowledge
A precise method-method interaction-based cohesion metric for object-oriented classes
Trans. Softw. Eng. Methodol. Article 8 ACM
Software Architecture In Practice
A hierarchical model for object-oriented design quality assessment
Trans. Softw. Eng. IEEE Comput. Soc.
Goal question metric paradigm
Encyclopedia of Software Engineering
Cohesion and reuse in an object-oriented system
A unified framework for coupling measurement in object-oriented systems
Trans. Softw. Eng. IEEE Comput. Soc.
A metrics suite for object oriented design
Trans. Softw. Eng. IEEE Comput. Soc.
Are “non-functional” requirements really non-functional? An investigation of non-functional requirements in practice
Variability in software systems—a systematic literature review
Trans. Softw. Eng. IEEE Comput. Soc.
Cited by (36)
A tertiary study on links between source code metrics and external quality attributes
2024, Information and Software TechnologyEmpirical investigation in embedded systems: Quality attributes in general, maintainability in particular
2023, Journal of Systems and SoftwareOn the relationship between source-code metrics and cognitive load: A systematic tertiary review
2023, Journal of Systems and SoftwareQuality measurement in agile and rapid software development: A systematic mapping
2022, Journal of Systems and SoftwareCitation Excerpt :Kupiainen et al. (2015) report on using metrics in industrial lean and agile software development teams analyzing the reasons for and the effects of using them. Arvanitou et al. (2017) report on design-time quality attributes and related metrics. Biesialska et al. (2020) analyze how big data analytics is used in the agile development lifecycle.
A mapping study on documentation in Continuous Software Development
2022, Information and Software TechnologyCitation Excerpt :[1], [2], [3], [4], [5], [6], [7], [10], [15], [18], [19], [20], [22], [26], [27], [29], [30], [33], [34], [36], [37], [40], [42], [44], [45], [47], [50], [51], [52], [53], [54], [60], [61], [62], [63], [64], [65], [66], [69], [70], [71], [76], [80], [86], [91], [92], [93], [94], [97], [100], [102], [105], [106], [108], [117], [119], [122], [123], [124], [129], [130], [131], [133], [134], [135], [136], [140], [142], [143], [144], [146], [147], [148], [151], [154], [157], [158], [162], [163], [168], [170], [171], [174], [176], [180], [183], [185], [186], [189], [190], [191], [201], [202], [203], [204], [208], [209], [210], [211], [215], [216], [219], [222], [223], [229], [230], [231], [233], [234], [236], [238], [239], [241], [244], [247], [249], [252], [253], [256], [257], [259], [262], [264], [266], [270], [278], [281], [285], [286], [287], [288], [289], [293], [294], [295], [297], [298], [299], [300], [301], [302], [303], [307], [308], [310], [311], [314], [315], [321], [326], [327], [329], [331], [333], [335], [336], [338], [342], [343], [345], [347], [348], [351], [353], [354], [357], [358], [360], [362], [363], [365], [366], [367], [368], [369], [370], [374], [378], [380], [383], [386], [391], [392], [393], [397], [399], [401], [403], [412], [426], [428], [429], [430], [431], [432], [433], [434], [435], [437], [438], [440], [441], [S11], [S14], [S16], [S25], [S31], [S32], [S35], [S74], [S77], [S79], [S82], [S83], [S84], [S87], [S88], [S103], [S110], [S111], [S113], [S118], [S120], [S121], [S127], [S128], [S137], [S138], [S145], [S153], [S155], [S160], [S161], [S165], [S172], [S173], [S179], [S192], [S195], [S196], [S207], [S212], [S213], [S227], [S240], [S242], [S243], [S248], [S258], [S263], [S267], [S271], [S272], [S273], [S275], [S276], [S282], [S291], [S296], [S304], [S318], [S323], [S324], [S337], [S339], [S341], [S344], [S350], [S352], [S356], [S371], [S381], [S382], [S385], [S390], [S395], [S396], [S398], [S405], [S408], [S409], [S415], [S416], [S417], [S418], [S420], [S421], [S425], [S439]
Change impact analysis: A systematic mapping study
2021, Journal of Systems and SoftwareCitation Excerpt :To validate the final version of the search string, we have performed a piloting before applying it to all venues. In particular, we have checked that all primary studies of a broader secondary study (i.e., (Arvanitou et al., 2017b)) published in three venues (namely TSE, IST, and JSS) have been retrieved from applying our search string. Although no publication indexing sites (e.g., Scopus, Google Scholar, etc.) have been used, we have performed this step, since some conferences publish their papers in more than one digital library, e.g., ESEM conference is hosted in ACM and IEEE DLs.
Elvira Maria Arvanitou is a PhD Student at the University of Groningen, the Netherlands, in the group of Software Engineering and Architecture. She holds an MSc degree in Information Systems from the Aristotle University of Thessaloniki, Greece (2013), and a BSc degree in Information Technology from the Technological Institute of Thessaloniki, Greece (2011). Her research interests include software quality assurance and metrics, software maintainability and stability.
Dr. Apostolos Ampatzoglou is a Guest Researcher in the Johann Bernoulli Institute for Mathematics and Computer Science of the University of Groningen (Netherlands), where he carries out research in the area of software engineering. He holds a BSc on Information Systems (2003), an MSc on Computer Systems (2005) and a PhD in Software Engineering by the Aristotle University of Thessaloniki (2012). His current research interests are focused on reverse engineering, software maintainability, software quality management, open source software engineering and software design. He has published more than 50 articles in international journals and conferences. He is / was involved in over 10 R&D ICT projects, with funding from national and international organizations.
Dr. Alexander Chatzigeorgiou is Professor of Software Engineering in the Department of Applied Informatics at the University of Macedonia, Thessaloniki, Greece. He received the Diploma in Electrical Engineering and the PhD degree in Computer Science from the Aristotle University of Thessaloniki, Greece, in 1996 and 2000, respectively. From 1997 to 1999 he was with Intracom S.A., Greece, as a telecommunications software designer. Since 2007, he is also a member of the teaching staff at the Hellenic Open University. His research interests include object-oriented design, software maintenance, and software evolution analysis. He has published more than 100 articles in international journals and conferences. He is a member of the Technical Chamber of Greece.
Dr. Matthias Galster is a Senior Lecturer in the Department of Computer Science and Software Engineering at the University of Canterbury, New Zealand. His current work aims at improving the way we develop high quality software, with a focus on software requirements engineering, software architecture, development processes and practices, and empirical software engineering.
Dr. Paris Avgeriou is Professor of Software Engineering in the Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen, the Netherlands where he has led the Software Engineering research group since September 2006. Before joining Groningen, he was a post-doctoral Fellow of the European Research Consortium for Informatics and Mathematics (ERCIM). He has participated in a number of national and European research projects directly related to the European industry of Software-intensive systems. He has co-organized several international conferences and workshops (mainly at the International Conference on Software Engineering - ICSE). He sits on the editorial board of Springer Transactions on Pattern Languages of Programming (TPLOP). He has edited special issues in IEEE Software, Elsevier Journal of Systems and Software and Springer TPLOP. He has published more than 130 peer-reviewed articles in international journals, conference proceedings and books. His research interests lie in the area of software architecture, with strong emphasis on architecture modeling, knowledge, evolution, patterns and link to requirements.