Quantitatively measuring a large-scale agile transformation

https://doi.org/10.1016/j.jss.2016.03.029Get rights and content

Highlights

  • We provide a quantitative metrics model to evaluate the impact of an agile and lean transformation.

  • We propose eight rigorously described metrics within that model.

  • We establish and apply the model in a large international telecommunication organization with 350 employees in two sites.

  • Our findings show significant improvement in six of the eight metrics whereas one metric showed deteriorated results.

Abstract

Context: Agile software development continues to grow in popularity and is being adopted by more and more organizations. However, there is a need for empirical evidence on the impact, benefits and drawbacks of an agile transformation in an organization since the cost for such a transformation in terms of money, disrupted working routines and quality of development can become considerable. Currently, such evidence exists in the form of success stores and case studies, mostly of qualitative nature.

Objective: Provide a metrics model to quantitatively measure the impact of an agile transformation in a software development organization.

Method: The metrics model was elicited with the use of the Goal Question Metric approach.

Results: A quantitative metrics model containing eight rigorously described metrics is presented and followed by its application to evaluate an agile and lean transformation in a large international telecommunication organization with 350 employees in two sites.

Conclusions: The metrics model was sensitive to the changes that occurred in the organization and revealed significant improvements in six of the eight metrics and a deterioration in one of the metrics.

Introduction

The IT world of today is highly competitive and value oriented. Companies active in this field strive to be flexible and adaptive to change due to constant business and technological changes in requirements and in the environment. As a consequence, agile and lean software development methods are gaining popularity among companies of various sizes and domains (Rodríguez, Markkula, Oivo, Turula, 2012, Dingsøyr, Moe, 2013). Agility is in itself a desired characteristic with over a decade of successful adoption both as a new development process or a changes in existing processes.

Transforming the way of working in a large organization takes time, effort and resources and there is a need to constantly evaluate the need and benefits of such initiatives. Software metrics can be used to provide objective insights and evaluate the effect of software process changes (Kitchenham, 1996).

One challenge when trying to apply process metrics to evaluate a transformation is that metrics can be process specific. That is, a metric is related to a task, role or artifact that is defined in one way of working or process but not in the other. We try to overcome this and other challenges related to application of software metrics in the study of large agile transformations and formulate the research question to be answered in this study as:

RQ: How can the changes of an agile transformation be measured by quantitative objective metrics?

To answer this question, we present a metrics model to measure the impact of agile transformations and its application in the context of a large-scale telecommunication company. In particular, we use metrics and measurements that are feasible to analyze both plan driven development and agile and lean development. The goal is to use metrics and measurements for the purpose of comparing the state of the organization before and after an agile and lean transformation. However, the metrics can be applied with other purposes, for instance, to provide transparency, feedback and aid for the self-organization of teams.

We use as the starting point the metrics model presented in our previous work, where we proposed a set of metrics for evaluation of improvement of the development process. In this paper we refine the earlier established metrics model and the metrics themselves. We base our enhancement on iterative collaboration with practitioners, e.g. via workshops and meetings. Thus, we develop the model based on empirical data. We formulate the goal of measurement in a neutral manner, so that the change itself is assessed—not the improvement. We depict the extended metrics in detail by using structured and formalized description.

Our contribution in this paper is as follows. We provide a structure for the description of metrics and thoroughly define each metric accordingly. Moreover, we discuss the validity and appropriateness of the proposed metrics for their intended use. Furthermore, we apply the metrics in a study of a large organization from the telecommunication domain. We perform validation on two levels: (i) when presenting the analysis and investigating the weaknesses and strengths of the proposed metrics, (ii) when discussing the validity of the study. Finally, we examine our results on a generic level, i.e., the feasibility of the proposed metrics for the purpose of the study.

Our results are presented in such a way that organizations of similar size and organizational context could use as a reference to monitor an agile and lean transformation. Experience and continuous learning are central tenets in both the agile and lean communities. We hope that our results contribute to this idea by extending the body of knowledge in the agile and lean domain as well as enabling further empirical investigations by providing a re-usable metrics model.

This paper is structured as follows. Section 2 introduces some relevant terminology and related work, followed by a description of the research question and the investigation strategy in Section 3. In Section 4 the metrics model is introduced at a high level, whereas each metric is described in detail in Section 5. Following the metrics description, the main results of an empirical study are presented in Section 6. Section 7 contains a discussion on validity concerns regarding the metrics. Our conclusions are presented in the last section.

Section snippets

Background and related work

There are plenty of existing metric models available, just to mention the ones of Boehm (1978), McCall et al. (1977) (known as General Electric Model) and Dromey (1995), or standards that include quality models, e.g. SQuaRE (ISO/IEC, 2010), ISO 9126 (ISO, 2001) and ISO 9004:2000 (ISO, 2000). However, tailoring measurements for specific context might require the “define-your-own-model” approach (Gilb, 1988), especially in order to encapsulate both product and process metrics. We use the

Research question, method and context

In this section we describe our research by first explaining the research question, followed by the description of the context of the study and the data collection process.

Goal, questions and metrics for the transformation metrics model

In this section we describe the application of the GQM (Basili et al., 1994) to create our metrics model. We derive the goal and questions from the collaboration with industry partners, progress with literature survey and finally identify metrics for our model.

Description of metrics

We concentrate on software process quality metrics and measurements which are relevant for both agile and plan driven settings. In the remainder of this section we describe in detail, and provide the reasoning behind, the choice of software metrics and measurements for our model. For each metric we follow a descriptive structure which is a merge of the framework for evaluating metrics presented by Kaner and Bond (2004) with property-based software engineering measurement given by Briand et al.

Application of the metrics model

We have applied the proposed metrics model to study the agile transformation at the case organization. In this section, we describe the process of data collection and analyze the gathered data.

Validation of metrics

In Section 3.1 we listed five criteria used for selecting the metrics of the proposed model. In this section we explain why we consider these criteria to be fulfilled by the proposed metrics. We also consider the validity of the metrics model according to the criteria given by Meneely et al. (2013). Finally, we study the validity of the application of the metrics model in the case organization and point out the limitations of our study.

Conclusions

Measurements of agile development process and especially the organizational changes leading to establishing this process in large-scale organizations have been neglected in research (Dybåand Dingsøyr, 2008). In this article, we present a metrics model to quantitatively compare a software development organization before and after an agile and lean transformation. Our goal with this work was to provide a quantitative approach as an alternative to qualitative studies such as interviews and surveys.

Acknowledgments

This work was supported by the DIGILE project Cloud Software Finland. The work of first author was partially funded by the Academy of Finland project ADVICeS (no. 266373).

Marta Olszewska (née Pląska) is a postdoctoral researcher at the Distributed Systems Design Laboratory at Åbo Akademi University, in Turku, Finland. Currently she is involved in the Academy of Finland funded project ADVICeS on Adaptive Integrated Formal Design of Safety-Critical Systems. Her research interests focus on establishing and validating metrics and measurements in perspective of how the development processes and practices impact the software quality. She obtained her Ph.D. degree in

References (50)

  • DybåT. et al.

    Empirical studies of agile software development: a systematic review

    Inf. Softw. Technol.

    (2008)
  • ACM Digital Library....
  • Agile Alliance. Agile Manifesto....
  • Agile Journal....
  • AndersonD.J.

    Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results

    (2004)
  • BasiliV.R. et al.

    The goal question metric approach

    Encyclopedia of Software Engineering

    (1994)
  • BasiliV.R. et al.

    The tame project: towards improvement-oriented software environments

    IEEE Trans. Softw. Eng.

    (1988)
  • BoehmB.W.

    Characteristics of Software Quality

    (1978)
  • BriandL.C. et al.

    Property-based software engineering measurement

    IEEE Trans. Softw. Eng

    (1996)
  • Cloud Software Programme, 2013. Cloud Software...
  • CockburnA.

    What engineering has in common with manufacturing and why it matters

    J. Defence Softw. Eng

    (2006)
  • CohnM.

    Succeeding with Agile: Software Development Using Scrum

    (2009)
  • ConcasG. et al.

    An empirical study of software metrics for assessing the phases of an agile project

    Int. J. Softw. Eng. Knowl. Eng.

    (2012)
  • DeemerP. et al.

    The Scrum Primer—A Lightweight Guide to the Theory and Practice of Scrum

    Technical Report

    (2012)
  • DestefanisG. et al.

    Software metrics in agile software: an empirical study

    Agile Processes in Software Engineering and Extreme Programming

    (2014)
  • DingsøyrT. et al.

    What is large in large-scale? A taxonomy of scale for agile software development

  • DingsøyrT. et al.

    Research challenges in large-scale agile software development

    SIGSOFT Softw. Eng. Notes

    (2013)
  • DromeyR.G.

    A model for software product quality

    IEEE Trans. Softw. Eng.

    (1995)
  • DubinskyY. et al.

    Agile metrics at the Israeli Air Force

    Proceedings of the Agile Development Conference (ADC’05)

    (2005)
  • EbertC. et al.

    Lean software development

    IEEE Softw.

    (2012)
  • GilbT.

    Principles of Software Engineering Management

    (1988)
  • HartmannD. et al.

    Appropriate agile measurement: using metrics and diagnostics to deliver business value

    AGILE 2006 Conference (Agile’06)

    (2006)
  • HazzanO. et al.

    Agile software engineering

    Undergraduate Topics in Computer Science

    (2008)
  • HeikkiläV. et al.

    Continuous release planning in a large-scale scrum development organization at Ericsson

    Proceedings of the 2013 International Conference, XP2013 on Agile Processes in Software Engineering and Extreme Programming

    (2013)
  • IEEE. IEEE Xplore Digital Library....
  • Cited by (34)

    View all citing articles on Scopus

    Marta Olszewska (née Pląska) is a postdoctoral researcher at the Distributed Systems Design Laboratory at Åbo Akademi University, in Turku, Finland. Currently she is involved in the Academy of Finland funded project ADVICeS on Adaptive Integrated Formal Design of Safety-Critical Systems. Her research interests focus on establishing and validating metrics and measurements in perspective of how the development processes and practices impact the software quality. She obtained her Ph.D. degree in 2011 with the thesis “On the Impact of Rigorous Approaches on the Quality of Development ”. So far he published 18 articles, of which 13 were refereed.

    Jeanette Heidenberg D.Sc.(Tech.) is an Innovation and Business Architect at Ericsson Finland. She did her Ph.D. in Software Engineering at Åbo Akademi University in 2011 with a title of “Towards Increased Productivity and Quality in Software Development Using Agile, Lean and Collaborative Approaches”. She has over a decade of software industry experience (telecommunications) in the areas of software development and software process improvement deploying agile methods.

    Max Weijola holds an M.Sc. in Computer Engineering with a major in Software Engineering from Åbo Akademi University. He currently works as Quality Assurance Manager at Lumi Technologies, a company focusing in audience engagement technology. His research has been mainly focused on agile software development processes and metrics. With a background in both research and industry his goal is to find ways to improve software development processes in a quantifiable way from a combined research and practitioner point of view. He has co-published three articles in various proceedings.

    Kirsi Mikkonen is currently responsible of external collaboration and funding within Ericsson R&D Finland. In her role as Change Manager and organizational Coach, she has driven the sustainable self-learning organization culture in both R&D and administrative organizations. She has strong experience in People management, Lean Leadership, Brain based coaching, Group facilitation, Agile and Lean methodology. She has Master of Science in Electrical Engineering from Helsinki University of Technology and Norwegian University of Science and Technology, in 1995.

    Ivan Porres PhD (Eng.) is professor in Software Engineering at Åbo Akademi University, in Turku, Finland and the leader of the Software Engineering Laboratory at TUCS, the Turku Centre for Computer Science. He is the principal investigator at Åbo Akademi for the Cloud Software Finland (2009–2013) and N4S (2014–2016) projects at DIGILE, the Finnish Strategic Centre for Science, Technology and Innovation in the ICT. He has received the Ten-Year Most Influential Paper Award at the ACM/IEEE Conference on Model Driven Engineering Languages and Systems in two occasions. He is the author of more than 100 scientific articles in Software Engineering.

    View full text