Metrics for analyzing variability and its implementation in software product lines: A systematic literature review

Abstract

Context: Software Product Line (SPL) development requires at least concepts for variability implementation and variability modeling for deriving products from a product line. These variability implementation concepts are not required for the development of single systems and, thus, are not considered in traditional software engineering. Metrics are well established in traditional software engineering, but existing metrics are typically not applicable to SPLs as they do not address variability management. Over time, various specialized product line metrics have been described in literature, but no systematic description of these metrics and their characteristics is currently available.

Objective: This paper describes and analyzes variability-aware metrics, designed for the needs of software product lines. More precisely we restrict the scope of our study explicitly to metrics designed for variability models, code artifacts, and metrics taking both kinds of artifacts into account. Further, we categorize the purpose for which these metrics were developed. We also analyze to what extent these metrics were evaluated to provide a basis for researchers for selecting adequate metrics.

Method: We conducted a systematic literature review to identify variability-aware implementation metrics. We discovered 42 relevant papers reporting metrics intended to measure aspects of variability models or code artifacts.

Results: We identified 57 variability model metrics, 34 annotation-based code metrics, 46 code metrics specific to composition-based implementation techniques, and 10 metrics integrating information from variability model and code artifacts. For only 31 metrics, an evaluation was performed assessing their suitability to draw any qualitative conclusions.

Conclusions: We observed several problematic issues regarding the definition and the use of the metrics. Researchers and practitioners benefit from the catalog of variability-aware metrics, which is the first of its kind. Also, the research community benefits from the identified observations in order to avoid those problems when defining new metrics.

Introduction

Software Product Line Engineering (SPLE) has been established to minimize costs and efforts, while maximizing the quality of products in a family of software products [1]. This requires additional implementation concepts to manage variability and to facilitate the derivation of individual products based on a common platform. These concepts are the variability model and the variation points, i.e., elements in code, which describe the point of impact of a variation in an artifact. Both concepts are not required for the development of single systems and, thus, are not considered in traditional software engineering. At the same time, this is the minimum information required to realize a product line and is thus often considered jointly.

In software engineering, software metrics are an established approach to characterize properties of software, for instance, [2] and [3] are two well-cited publications from academia and industry dealing with the application of metrics in software engineering. However, while they are well-analyzed and surveyed for systems without variability, there is currently no broad characterization of variability management concepts of SPLs.

Here, we present a systematic literature review to identify and characterize variability-aware metrics designed for the needs of SPLs. Our study aims at identifying existing metrics as a basis to draw qualitative conclusions on implementation properties of product lines. We include variability model metrics, because they are linked to all levels of product line realization, including implementation. Thus, we focus only on variability model metrics and code metrics that take variation points into account to characterize product line implementation. We explicitly exclude metrics for requirements, architecture, and testing to focus our research. We pursue the following research questions:

• RQ1 Which metrics have been defined for variability models and implementation artifacts of SPLs?

  • RQ1.1 Which metrics have been defined for variability models?

  • RQ1.2 Which metrics have been defined for code artifacts, which consider variability?

  • RQ1.3 Which metrics have been defined, which combine information of the variability model and variability of code artifacts?

Besides the mere description of variability metrics, we also want to understand to what extent the metrics have been used to derive qualitative characterizations of software product lines.

• RQ2 Which correlations between these measures and quality characteristics of product lines have been studied?

  • RQ2.1 Which correlations have been studied?

  • RQ2.2 Which methods and examples have been used to determine the validity of the correlations?

  • RQ2.3 Which correlations could be successfully demonstrated?

In summary, the main contributions of this study are¹:

• We present a systematic literature review performed in early 2017 to identify the state of the art of implementation-related metrics for software product lines. This literature review comprises peer-reviewed articles of the last 10 years (2007–2016).

• We present a catalog of variability-aware metrics for software product lines. This comprises metrics for variability models, code artifacts, and metrics measuring both artifact types. This catalog provides an overview of metrics, which are available in this domain, but we also identify open issues for the research community.

• We analyze to what extent these metrics have been evaluated to show their ability to draw conclusions about qualitative aspects of the analyzed artifacts. Only 53 metrics ( ≈  36%) have been evaluated in 14 out of 42 identified papers.

This paper is structured as follows: In the following section, we present related work. Section 3 provides an overview about required background. Section 4 presents our search strategy to identify relevant work. In Sections 5 – 8, we answer RQ1 by presenting the identified metrics. These metrics are structured according to the measured artifact types: variability models, code artifacts (separated into annotative approaches for handling negative variability and metrics specific to compositional approaches, for handling positive variability), and metrics measuring both artifact types. Afterwards, we analyze the quality of the identified papers in Section 9. First, we answer RQ2 by analyzing to what extent evaluations have been applied to prove their ability to serve as a basis to draw conclusions about qualitative aspects of the analyzed artifacts in Section 9.1. Subsequently, we discuss some key findings regarding the definition of the metrics in Section 9.2. We report the limitations of our study in Section 10 and our conclusions in Section 11.