MS-QuAAF: A generic evaluation framework for monitoring software architecture quality
Introduction
Software architecture has emerged as a mature sub-discipline of software engineering after several years of the software crisis in the mid-1970s [1]. It can be defined as a set of significant decisions about the organization of a software system. These decisions include the selection of structural elements and their interfaces, the collaboration among those elements (the behavior), the composition of these structural elements into large subsystems, and the architectural styles that guide this organization [1], [2], [3]. The architecture is generally designed based on architectural decisions made in the early stages of the development process to achieve functional and non-functional requirements (NFRs) [4]. However, the ability of a software system to fulfill its assigned functional responsibilities does not imply that the predefined quality attributes are met. For instance, a system can deliver efficiently correct results but does not satisfy the security requirement by allowing malicious users to access easily its confidential data. This implies that architectural decisions made are violated or not suitable for producing software of high quality. It must be a continuous quality evaluation and architecture monitoring throughout the whole development process and at each maintenance activity. It should be an innovative and powerful approach to do that, yet simple and easy to implement, which is our primary focus in this paper.
Nowadays, the software marketplace is highly competitive, which means that delivering high-quality software is not only an obligatory but also a differentiator factor for companies. Critical sectors that depend heavily on software expect to deploy error-free systems that are supposed to fulfill their quality requirement flawlessly. Therefore, the slight difference between the quality of a software product and its competitors is considered decisive in this highly demanding market. Furthermore, poor software quality is responsible for severe business and safety disasters [5]. Companies must boost up software quality perception and integrate it profoundly into the development process. They need tools and frameworks to support continuous architecture assessment even if they cost them the changing of their cultural, technical, and organizational reasoning. To do that, there are two basic classes of evaluation at the architecture level, a qualitative evaluation, and a quantitative evaluation [6,7]. The former uses questioning techniques to generate qualitative questions to be asked of an architecture. These techniques include questionnaires, checklists, and scenarios. The latter suggests quantitative measurement to evaluate an architecture using, for example, metrics, simulations, and experimentations.
Indeed qualitative evaluation is thought to be applicable to assess any given architecture quality [3,7]; however, it lacks statistical significance. Questioning techniques like questionnaires and checklists are mostly based on the evaluators’ perspective and subjectivity, which may decrease the evaluation accuracy and objectivity. Additionally, the results of scenario-based analysis depend on the selection of the scenarios and their relevance for identifying architecture's weaknesses. In this context, there is no fixed number of scenarios to guarantee that the evaluation analysis is meaningful [6]. On the other hand, the essence of measuring techniques is to deliver assessors with quantitative results and views that can reflect the state of the architecture quality more accurately. However, this type of evaluation can suffer from the following issues:
- •
Quantitative evaluation frameworks are addressed to answer specific questions, and thus evaluating specific qualities, such as performance [8] and modifiability [9]. This is due to the fact that some quality attributes are hard to quantify.
- •
Defining metrics for some attributes, such as security, and usability have proven difficult to develop [10].
- •
Many metrics are used to evaluate the architecture at one specific development stage, more specifically, at design-time or after the product is complete or nearly complete [11]. Therefore, a lack of evaluation and risk-management frameworks that can cover these main stages throughout the development process (design-time, implementation, and deployment) can be addressed.
In this paper, we attempt to overcome the main shortcomings of quantitative and qualitative techniques by proposing a quantitative evaluation framework (belongs to the metrics-based category) called MS-QuAAF (Multi-Service - Quantitative Architecture Assessment Framework). The framework defines a suite of generic evaluation metrics to help evaluators in:
- •
Assessing any quality attribute inputted into the framework.
- •
Assessing architecture throughout two main stages of the development process (design and implementation).
- •
Estimating architecture defectiveness and detecting potential deviations.
- •
Evaluating the achievement of the NFR responsibilities assigned to promote quality attributes.
- •
Making decisions about the architecture (e.g., validating or disapproving the design).
MS-QuAAF is a multi-service assessment framework derived from ISO/IEC/IEEE 42030:2019 [12], the generic architecture evaluation framework that specifies objectives (quality attributes), approaches, factors, and evaluation results as the key elements of any instantiated framework. MS-QuAAF performs architecture evaluation through two main modules. The first module proposes the concept of facet projection to extract from architecture meta-models only information of interest to the evaluation task. The second module proposes seven metrics applied to the target architectures through three assessment services to evaluate these architectures at the design and implementation stages. These metrics are called generic because they can measure the satisfaction of any quality attribute inputted into the framework. The latter does not provide specific metrics for maintainability, or performance, or any other quality attribute. In contrast, the proposed metrics are common to measure all targeted quality attributes on the condition that the architectural decisions taken to promote these attributes are specified at early development stages. In the design stage, metrics are used to judge the correctness of the designed architecture against the established specification. In the implementation stage, metrics are used to judge the fulfillment of the responsibilities assigned to promote quality attributes. The evaluation results at each stage allow architects to accept or refuse the deviation from architecture specification. Architectural debts [13] can be used to expedite the development process; however, deviations must be adjusted to clear up or reduce these debts in the subsequent architecture releases.
The conducted experiment showed that the proposed framework is able to evaluate the conformance and the properties of architectures against the specified meta-models using the proposed metrics. First, the framework can evaluate the designed architectures by determining the fulfilled and violated NFR design rules, the satisfaction rate, and the defectiveness rate of those architectures. Second, the framework can evaluate the implemented (realized) architectures by assessing the satisfaction of the assigned NFR responsibilities using assessment trees. Third, based on the previous results, the framework generates the final assessment report using dedicated evaluation matrices.
The remainder of this paper is organized as follows. Section 2 discusses related work. Section 3 presents briefly the framework's evaluation methodology. Section 4 introduces the concept of architecture facets. The evaluation services and metrics are presented in sections 5 and 6. The experimental evaluation and its results are discussed in section 7. Section 8 provides a conclusion.
Section snippets
Related work
There are three central research areas related to our work, which are evaluation frameworks and architecture analysis methods, goal decomposition frameworks and NFR assurance, and model slicing (model projection).
- a)
Architecture analysis and evaluation
Quality evaluation frameworks have been addressed in the field of software quality assessment and monitoring. SAAM [14,15] and ATAM [16] are two known classical methods for architecture analysis and evaluation. SAAM is a scenario-based analysis
The MS-QuAAF's evaluation methodology in a nutshell
Architecture evaluation can be defined by the judgments about architectures according to the specified evaluation objectives [12]. MS-QuAAF judges software architecture by analyzing architecture specifications with respect to stakeholders’ non-functional requirements (NFR) goals. An architecture specification encapsulates all architectural decisions taken to satisfy these goals, such as architectural styles, tactics, and rules (constraints). These architectural decisions constitute meta-models
Architecture facets
Software architectures are so complex, which means that describing and treating them in a one-dimensional fashion is impossible [40]. The tremendous complexity can make the architecture evaluation even harder and trickier. Therefore, we should expand our view to handle the architecture from multiple sides and angles according to stakeholders’ quality attributes. We call these sides the architecture facets (AF). Accordingly, we believe that depicting software architecture as facets can enhance
The evaluation effort
The generic AE framework defines the evaluation effort by the activity that determines the actual value of architecture. The evaluation effort can be performed at many stages during the development entity [12]. Accordingly, MS-QuAAF allows performing this effort through a set of services at two main development stages, which are the conceptual design and the implementation stages. In this paper, the AE effort consists of accepting any quality attribute as input, extracting the corresponding
Evaluation services and metrics
In this section, we will present the second module of the framework, which is the quality evaluator module (Figure 1). The latter consists of three evaluation services, each of which provides a set of evaluation metrics. The succession of these services constitutes the architecture evaluation process that starts after the facet projection process.
Experimental evaluation
In this section, we have conducted an experimental evaluation to assess the proposed framework and to answer the following research questions:
- •
RQ1: Does the proposed framework capable of calculating the defectiveness of the designed architecture?
This question aims to evaluate the effectiveness of the metrics QuARF, QuARD, OVF, and OVD.
- •
RQ2: Does the proposed framework capable of estimating the satisfaction of NFR responsibilities of the implemented architecture?
This question aims to evaluate the
Conclusion
In this paper, we proposed MS-QuAAF, a multi-service evaluation framework dedicated to assessing software architectures quantitatively. In this context, we made three mains contributions. First, the evaluation effort is independent of quality attributes and the relevant quality models. The independency is promoted by the proposed generic metrics that direct the ability of the framework to evaluating any inputted quality attribute. However, this is feasible as long as the architectural decisions
CRediT authorship contribution statement
Salim Kadri: Writing – review & editing, Methodology, Writing – original draft. Sofiane Aouag: Supervision, Writing – review & editing, Validation. Djalal Hedjazi: Writing – review & editing, Formal analysis.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
REFERENCES (59)
- et al.
A software quality framework for large-scale mission-critical systems engineering
Information and Software Technology
(2018) Performance evaluation of component-based software systems: A survey
Performance Evaluation
(2010)Technical debt and agile software development practices and processes: An industry practitioner survey
Information and Software Technology
(2018)- et al.
The essential components of software architecture design and analysis
Journal of Systems and Software
(2006) - et al.
Architecture-level modifiability analysis (ALMA)
Journal of Systems and Software
(2004) - et al.
Assessing the use of slicing-based visualizing techniques on the understanding of large metamodels
Information and Software Technology
(2015) - et al.
Controlling software architecture erosion: A survey
Journal of Systems and Software
(2012) - et al.
Specification and automatic checking of architecture constraints on object oriented programs
Information and Software Technology
(2018) The analytic hierarchy process—what it is and how it is used
Mathematical Modelling
(1987)- et al.
Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools
Computer Science Review
(2015)
The Decision View's Role in Software Architecture Practice
IEEE Software
Software Architecture : Perspectives on an Emerging Discipline
Software Architecture in Practice
Decision-Making Techniques for Software Architecture Design: A Comparative Survey
ACM Computing Surveys
A survey on software architecture analysis methods
IEEE Transactions on Software Engineering
Designing software architectures to achieve quality attribute requirements
IEE Proc., Softw
A hierarchical model for object-oriented design quality assessment
IEEE Transactions on Software Engineering
Scenario-Based Analysis of Software Architecture
Software, IEEE
Cited by (2)
EsArCost: Estimating repair costs of software architecture erosion using slice technology
2024, Journal of Systems and SoftwareQuality Metrics in Software Architecture
2023, Proceedings - IEEE 20th International Conference on Software Architecture, ICSA 2023