MS-QuAAF: A generic evaluation framework for monitoring software architecture quality

Abstract

Context

In a highly competitive software market, architecture quality is one of the key differentiators between software systems. Many quantitative and qualitative evaluation frameworks were proposed to measure architecture. However, qualitative evaluation lacks statistical significance, whereas quantitative methods are designed for evaluating specific quality attributes, such as modifiability and performance. Besides, the assessment covers usually a single development stage, either at the design stage or at the implementation stage.

Objective

A lack of generic frameworks that can support the assessment of a broad set of attributes and ensure continuous evaluation by covering the main development stages is addressed. Accordingly, this paper presents MS-QuAAF, a quantitative assessment framework destined for evaluating software architecture through a set of generic metrics.

Method

The quantitative evaluation consists of checking architecture facets mapped to quality attributes against the early specified meta-models. This process starts by analyzing rules infringements and calculating architecture defects after accomplishing the design stage. Second, the assigned responsibilities supposed to promote stakeholders’ quality attributes are assessed quantitatively at the end of the implementation stage. Third, the final evaluation report is generated.

Results

We made specifically three main contributions. First, the proposed metrics within the framework are generic, which means that the framework has the ability to assess any inputted quality. Second, the framework proposes a set of evaluation services capable of assessing the architecture at two main development stages, which are design and implementation. Third, we proposed a quantitative assessment tree within the framework called the Responsibilities Satisfaction Tree (RST) that uses NFR responsibilities nodes to evaluate the implemented architectures.

Conclusion

The conducted experiment showed that the framework is capable of evaluating quality attributes based on architecture specification using the proposed metrics. Furthermore, these metrics contributed to enhancing architecture quality during the development stages by notifying architects of the discovered anomalies.

Introduction

Software architecture has emerged as a mature sub-discipline of software engineering after several years of the software crisis in the mid-1970s [1]. It can be defined as a set of significant decisions about the organization of a software system. These decisions include the selection of structural elements and their interfaces, the collaboration among those elements (the behavior), the composition of these structural elements into large subsystems, and the architectural styles that guide this organization [1], [2], [3]. The architecture is generally designed based on architectural decisions made in the early stages of the development process to achieve functional and non-functional requirements (NFRs) [4]. However, the ability of a software system to fulfill its assigned functional responsibilities does not imply that the predefined quality attributes are met. For instance, a system can deliver efficiently correct results but does not satisfy the security requirement by allowing malicious users to access easily its confidential data. This implies that architectural decisions made are violated or not suitable for producing software of high quality. It must be a continuous quality evaluation and architecture monitoring throughout the whole development process and at each maintenance activity. It should be an innovative and powerful approach to do that, yet simple and easy to implement, which is our primary focus in this paper.

Nowadays, the software marketplace is highly competitive, which means that delivering high-quality software is not only an obligatory but also a differentiator factor for companies. Critical sectors that depend heavily on software expect to deploy error-free systems that are supposed to fulfill their quality requirement flawlessly. Therefore, the slight difference between the quality of a software product and its competitors is considered decisive in this highly demanding market. Furthermore, poor software quality is responsible for severe business and safety disasters [5]. Companies must boost up software quality perception and integrate it profoundly into the development process. They need tools and frameworks to support continuous architecture assessment even if they cost them the changing of their cultural, technical, and organizational reasoning. To do that, there are two basic classes of evaluation at the architecture level, a qualitative evaluation, and a quantitative evaluation [6,7]. The former uses questioning techniques to generate qualitative questions to be asked of an architecture. These techniques include questionnaires, checklists, and scenarios. The latter suggests quantitative measurement to evaluate an architecture using, for example, metrics, simulations, and experimentations.

Indeed qualitative evaluation is thought to be applicable to assess any given architecture quality [3,7]; however, it lacks statistical significance. Questioning techniques like questionnaires and checklists are mostly based on the evaluators’ perspective and subjectivity, which may decrease the evaluation accuracy and objectivity. Additionally, the results of scenario-based analysis depend on the selection of the scenarios and their relevance for identifying architecture's weaknesses. In this context, there is no fixed number of scenarios to guarantee that the evaluation analysis is meaningful [6]. On the other hand, the essence of measuring techniques is to deliver assessors with quantitative results and views that can reflect the state of the architecture quality more accurately. However, this type of evaluation can suffer from the following issues:

• Quantitative evaluation frameworks are addressed to answer specific questions, and thus evaluating specific qualities, such as performance [8] and modifiability [9]. This is due to the fact that some quality attributes are hard to quantify.

• Defining metrics for some attributes, such as security, and usability have proven difficult to develop [10].

• Many metrics are used to evaluate the architecture at one specific development stage, more specifically, at design-time or after the product is complete or nearly complete [11]. Therefore, a lack of evaluation and risk-management frameworks that can cover these main stages throughout the development process (design-time, implementation, and deployment) can be addressed.

In this paper, we attempt to overcome the main shortcomings of quantitative and qualitative techniques by proposing a quantitative evaluation framework (belongs to the metrics-based category) called MS-QuAAF (Multi-Service - Quantitative Architecture Assessment Framework). The framework defines a suite of generic evaluation metrics to help evaluators in:

• Assessing any quality attribute inputted into the framework.

• Assessing architecture throughout two main stages of the development process (design and implementation).

• Estimating architecture defectiveness and detecting potential deviations.

• Evaluating the achievement of the NFR responsibilities assigned to promote quality attributes.

• Making decisions about the architecture (e.g., validating or disapproving the design).

MS-QuAAF is a multi-service assessment framework derived from ISO/IEC/IEEE 42030:2019 [12], the generic architecture evaluation framework that specifies objectives (quality attributes), approaches, factors, and evaluation results as the key elements of any instantiated framework. MS-QuAAF performs architecture evaluation through two main modules. The first module proposes the concept of facet projection to extract from architecture meta-models only information of interest to the evaluation task. The second module proposes seven metrics applied to the target architectures through three assessment services to evaluate these architectures at the design and implementation stages. These metrics are called generic because they can measure the satisfaction of any quality attribute inputted into the framework. The latter does not provide specific metrics for maintainability, or performance, or any other quality attribute. In contrast, the proposed metrics are common to measure all targeted quality attributes on the condition that the architectural decisions taken to promote these attributes are specified at early development stages. In the design stage, metrics are used to judge the correctness of the designed architecture against the established specification. In the implementation stage, metrics are used to judge the fulfillment of the responsibilities assigned to promote quality attributes. The evaluation results at each stage allow architects to accept or refuse the deviation from architecture specification. Architectural debts [13] can be used to expedite the development process; however, deviations must be adjusted to clear up or reduce these debts in the subsequent architecture releases.

The conducted experiment showed that the proposed framework is able to evaluate the conformance and the properties of architectures against the specified meta-models using the proposed metrics. First, the framework can evaluate the designed architectures by determining the fulfilled and violated NFR design rules, the satisfaction rate, and the defectiveness rate of those architectures. Second, the framework can evaluate the implemented (realized) architectures by assessing the satisfaction of the assigned NFR responsibilities using assessment trees. Third, based on the previous results, the framework generates the final assessment report using dedicated evaluation matrices.

The remainder of this paper is organized as follows. Section 2 discusses related work. Section 3 presents briefly the framework's evaluation methodology. Section 4 introduces the concept of architecture facets. The evaluation services and metrics are presented in sections 5 and 6. The experimental evaluation and its results are discussed in section 7. Section 8 provides a conclusion.