Keywords

1 Introduction

Agile approaches have reached a level of acceptance that has led many organisations to promote them to ever wider contexts than those initially envisioned of small projects and teams [1, 2]. Large-scale agile development is one such context, but agile is also being promoted outside the context of software development. Organisations are adopting agile principles outside of IT, hoping to cope with rapidly changing environments, and increase their capabilities for delivery and customer satisfaction; making organisations more agile is not always driven by the need to cope with agile software development at scale. Although there is no single agreed definition of business, organisational or enterprise agility [3] it is seen as a set of desirable qualities that demand a transformation affecting the whole organisation. Such transformations are hard as they require a multi-disciplinary approach, and need to balance maintaining business-as-usual with significant and disruptive change. Approaches to achieve enterprise agility through business transformation [3] can be grouped into three categories: scaled-framework-driven (operational agility), business-driven (strategic orientation) and sustainable agility (cultural orientation). Scaled-framework-driven approaches include frameworks that have been used in software development environments to support large scale projects [1], e.g. DSDM, AgilePM, SAFe, LeSS; they address operational aspects to help with improving flow, value creation activities and delivery cycles. Business-driven approaches take a strategic view of agility considering how the business model can become more agile [4]. Sustainable agility [5] approaches take the view that the organisation culture is key in supporting the long-term objectives of a transformation. Approaches in this category view culture as the main focus of the transformation, with people’s behaviours and values being central to its success and sustainability.

This paper explores the transformation of a local district council, in the UK. It was the first council to follow a behaviour-led approach focusing on cultural orientation, making it a unique case study for enterprise agility. Interviews were conducted with internal stakeholders and meetings were observed over a five month period. The council wanted to have an external view on how they were performing and how far they had travelled in their journey to be a more agile organisation; they also believed that understanding and changing organisational culture was an essential part of their transformation. The two research questions addressed were: RQ1: What successes and challenges are identified towards the end of a behaviour-led transformation to become an agile organisation? and RQ2: What improvements suggested in the literature are applicable in this context?

This work contributes to the growing area of enterprise agility when agile principles are applied in non-software development areas and organisations.

This paper is structured as follows: Sect. 2 introduces related work; Sect. 3 describes the method followed; Sect. 4 gives the context for the case study; the findings about the transformation are presented in Sect. 5, followed by discussion and conclusions in Sects. 6 and 7.

2 Related Work: Transformation Towards Enterprise Agility

Within the software agile community, there is a growing body of research into large-scale agile transformations and impact on the wider organisation [1, 2, 6, 7]. While the focus of this work is on transformations triggered by scaling agile software development, many of the challenges identified are not specific to software development; for example, change resistance, lack of investment, coordination challenges or hierarchical management and organisational boundaries [1].

Success factors in these transformations, are also mostly not software development specific as shown in the following categories [1]: management support, commitment to change, leadership, choosing and customising the agile approach, piloting, training and coaching, engaging people, communication and transparency, mindset and alignment, team autonomy and requirements management. While some of these categories are software-specific (e.g. choosing and customising the agile approach, piloting, training and coaching and requirements management) the others are not. Challenges [2] have also been identified that are software-specific (method, technology and ability-related) and non-software specific such as organisation, culture and motivation-related. Among the 11 categories of challenges identified by Uludag et al. [7] we also find two non-software specific categories: Culture & Mindset and Communication & Coordination. The former being about change, management buy-in and trust, and the latter about inter and intra-team communication in agile development teams and communication gaps with stakeholders.

Apart from scaling agile software development, enterprise, or business agility [8, 9] has become a desirable outcome for many organisations trying to survive in a continuously changing and competitive environment. It is the ability to adapt to change and continuously improve [10] that makes an enterprise agile. In a transformation process to achieve agility, organisations strive to develop capabilities to become adaptable and to develop a culture that will sustain the transformation in the long term. Teece [11] defends the need for dynamic capabilities to adapt to, and change in order to respond to a volatile environment. Dynamic capabilities are: sensing, i.e. identifying, developing and assessing opportunities and threats in relation to users’ needs, using all available data to identify coherent patterns and imaginatively creating hypotheses about the future; seizing, i.e. mobilising resources to address needs and opportunities for which internal structures are needed to support flexibility and slack; and, transforming/shifting, i.e. continued renewal, for which organisations need to be very good at learning how to do new things [12].

It is not easy to establish a causal relationship between culture change and the development of these capabilities; however, it is recognised that an agile mindset needs to be promoted to sustain success over time [5], and that the organisational culture needs to be transformed to support the engagement of every person contributing to the work of the organisation [5, 13]. Carvalho et al. [5] propose an integration between organisational agility, organisational excellence, and organisational culture leading to sustainable organisational excellence and promoting adaptability. They highlight that the failure of many excellence programmes in organisations is due to neglect of how to sustain them in the long term. This continuous push for sustainability requires that:

“(1) senior leadership must be united in driving excellence, (2) the organisation, in a holistic perspective, must be committed and engaged, (3) the organisation strategy must be clear, defined and communicated, (4) the organisation must have process improvement ongoing activities together with self-assessment and (5) the use of information and data analysis must be a daily practice of the organisation.” [10 cited in 5].

The role of senior leadership to achieve strategic agility is also addressed by Doz and Kosonen [4]; they propose an agenda constructed with a set of actions in three areas: strategic sensitivity, leadership unity, and resource fluidity. Increased sensitivity to internal and external environments, achieving true engagement and commitment of all, and making the required ingredients available will help foster a successful transformation.

There is a gap in the literature between research coming from a software development background and that coming from a business context. More cross-disciplinary learning is required between these domains. The work presented in this paper contributes to address this gap.

3 Method

We conducted a qualitative single-case case study [15, 16] to follow part of the journey for a local council that was undergoing a comprehensive transformation programme. We identified their successes and challenges, answering RQ1, and provided feedback to the council for continuous improvements, simultaneously addressing RQ2.

Data collection consisted of semi-structured interviews, meeting observations and studying official documents. Ethical permission was received from the University to conduct the study, and all participants consented to take part after reading an information leaflet. Data collection was conducted between January and May 2018. During this five-month period the research team observed and took notes of regular weekly meetings of the assistant directors, and carried out 19 interviews with employees in senior management roles. Of the people interviewed most had been employed at the Council throughout the transformation with only two participants having been recruited as a result of it. Each interview lasted around half an hour and was conducted by at least one of the first two authors plus the acknowledged researcher. All interviewees were asked about: their views of the transformation journey so far, the successes and challenges of the transformation and what they considered the next steps.

An inductive thematic analysis was undertaken to identify the main themes for the successes, challenges, and steps ahead [17]. The thematic analysis was carried out independently by two researchers, using the interview data and meeting notes, with the final analysis resulting from a comparison between both lists of themes. This final list was then discussed by the wider team. Literature on organisational culture and agility (such as that in Sect. 2) was used to help identify and structure potential areas of improvement highlighted through the empirical work. We also identified recommendations from this literature for the organisation to consider in their own context and decide whether and how to apply them.

For a more in-depth analysis focusing on the organisation’s culture, we used the Agile Business Consortium’s (ABC) Culture Development Matrix [18] (Fig. 1). The full matrix has seven elements, but we used six in our analysis because the Innovation & Learning element (omitted in Fig. 1) was not covered through the interviews.

Fig. 1.
figure 1

Agile culture development matrix (adapted from https://www.agilebusiness.org/agile-culture)

Full size image

Organisations can be assessed at 5 different levels (surviving, stabilising, secure, thriving and transformational) for each of the elements. Figure 1 shows the elements across the top row of the table, with the levels listed in the first column of the table. By mapping an organisation’s behaviours against the relative development level in each element, a snapshot of readiness for transformation emerges, which can indicate a starting point for improvement.

4 Case Study

The council covers an area just outside greater London; it serves around 180 k residents, is the second largest district council (in the UK) and a major area for growth. The services provided by this council are: household recycling and waste collection, local planning and building regulations, housing advice, licensing (e.g. alcohol and entertainment, animal related, gambling, market stalls, sex establishments, taxis, etc.), environmental problems, benefits, council tax collections, community safety, public car parks and parks and community centres.

For the last decade this council has undertaken a top-down internal transformation, inspired by Simon Sinek’s Start with Why [19]. Senior management had sensed the external environment and realised the need to achieve financial stability, given the threats to government grants for local authorities, while at the same time to continue to deliver improved services to their customers. It was a long transformation process that proceeded in stages and on different strands: commercially minded, community focused, customer and innovation, and financially fit.

The aim of this transformation was to achieve ‘world-class support for those who need it’ being ‘the best place to work in the area with the best people’. It began in 2008 and had a number of milestones; trade unions were involved throughout. In 2008 the change programme was introduced by senior management to set managers on the road to cultural change; in 2010 this was one of the first councils to adopt a Cloud IT strategy; from 2011 onwards the total removal of the government grant by 2020 was foreseen and the need to change became a priority; in 2012 a new business model was deployed to explore opportunities in the market place; an ideas hub for the change process was created in 2013; and in 2014 the vision for moving into an income generating entrepreneurial culture took shape. In 2015 a new website was developed around residents’ desires and needs with the digitalisation of services. In 2016, a new organisational structure was created.

Central to the transformation plan was a desire for all staff employed by the council to exhibit commercially-minded behaviours, and this underpinned the more practical milestones mentioned above. Most existing staff (320, excluding the CEO and 2 directors) went through a behavioural assessment exercise in the process of applying for jobs at the council – either in their original roles, or in new ones. The aim was for all staff employed by the council to adhere to the specified behaviours, rather than to change the behaviour of existing staff. Staff could apply for any job at any level, and some ended up being promoted several levels. As a result, around 70 people left the organisation (some through early retirement) and 100 new people were recruited. This behaviour-led programme resulted in a commercially-minded restructuring of the whole council based on the five behaviours shown in Figs. 2 and 3; big saving targets were also put in place. As a public service entity, the council cannot make a profit, so any surplus from commercial ventures must feed back into better service delivery.

Fig. 2.
figure 2

The Council’s Behaviour Framework (© Aylesbury Vale District Council 2017)

Full size image
Fig. 3.
figure 3

Commercial Behaviours (© Aylesbury Vale District Council 2017)

Full size image

This transformation focused on ‘commercial’ behaviours, but these behaviours map directly to the organisational culture factors that correlate with agile method usage described in Strode et al. [20]. These factors include, for example: ‘the organization is results oriented’, and ‘the organization enables empowerment of people’. Only Customer Focus & Insight does not appear in Strode et al.’s list but it equates to ‘customer collaboration over contract negotiation’ in the Agile Manifesto [21]. So although employees at the council rarely spoke of an agile transformation, their goal was an agile organisation.

To sustain these behaviours, as well as the actions described above, the council implemented a new business model with a more commercial approach, reviewed every service, introduced charges for some non-essential services, and introduced new chargeable services.

5 A Transformation Towards Business Agility

5.1 Findings from the Thematic Analysis

Thematic analysis of the interviews, with meetings and documents as context, was used to answer RQ1: What successes and challenges are identified towards the end of a behaviour-led transformation to become an agile organisation? We found evidence for many positive elements of an agile culture as in Table 1; namely, that the organisation is results oriented, the management style is supportive and collaborative, the organisation values feedback and learning, social interaction in the organisation is trustful, collaborative, and competent, the organisation enables empowerment of people, and the leadership in the organisation is entrepreneurial, innovative, and risk taking [20].

Table 1. Successes
Full size table

Analysis of the interviews highlighted challenges that were identified at the time of the interviews, the five-month period just after the main transformation (see Table 2).

Table 2. Challenges
Full size table

5.2 Findings from the Agile Culture Development Matrix Assessment

The council wanted to achieve a deeper understanding of their culture after its most significant period of transformation, and to highlight areas that needed attention. We assessed our findings (Tables 1 and 2) against the Agile Culture Development Matrix (see Fig. 1). Based on this, the council scored as follows:

  • Purpose and Results (Thriving to Transformational)

  • Agile Leadership (Secure to Thriving)

  • Well-being & Fulfilment (Thriving to Transformational)

  • Collaboration & Autonomy (Secure to Thriving)

  • Trust & Transparency (Secure to Thriving)

  • Adaptability to Change (Secure to Thriving)

We identified, in particular, two areas for improvement towards a more agile organisation, Collaboration & Autonomy and Adaptability to Change, which are discussed below. For the former, the data underpinning two challenge themes of ‘Silos’ and ‘Internal processes and procedures’ indicated a lack of collaboration but also confusion around autonomy. For the latter, the theme of ‘Leadership vulnerability and resilience to change’ indicated an uncertainty about any changes to the leadership team. Looking back at the behaviour-led approach undertaken by the transformation (Fig. 3), the first area relates to Building effective relationships and the second to Innovating and adapting to change behaviours.

Collaboration and Autonomy

A transformational organisation is characterised by “a network of collaborative teams” and “authority is distributed with an appropriate level of autonomy” [22]. Our data indicates that the council does not meet either of these at this time. Although we found evidence of cohesive teams

the team is pretty cohesive and we’ve all had to support each other… If somebody is struggling a little bit and not wanting to admit it, the rest of the team actually notice and go and give support; got to know some things about staff you didn’t necessarily know about them before learning about other colleagues; and learning all of that sort of stuff together is quite good

it is unclear whether there is a network of collaborative teams and a clear understanding of responsibilities and priorities. Networked teams need to operate in the context of everyone working together, but also to an agreed way of working. One of the challenges raised through the interviews is the misalignment between autonomy for decision making (empowered teams, get on and do it attitude) and the lack of processes and procedures, with people creating them on the fly impacting the organisation’s reputation.

There is a recognition of the need to maintain the momentum and revitalise, while also consolidating processes and procedures

[..] there is very much an attitude of get on and do it which I think is a double-edged sword [..] things are happening but it does mean that some of the processes and procedures aren’t being followed or if they aren’t existing processes and procedures people are creating them in the fly [..] sometimes we do things without having a solid robust procedure behind it

This also suggests that they didn’t have an appreciation of what it is to be self-organising, i.e. people went off and made decisions without reference back to (or independent from) the core (a characteristic of the ‘secure’ assessment)

[..] there is a risk that we started to see things that are happening and [..] we didn’t even know we were doing that

Adaptability to Change

A transformational organisation is characterised by having a strong core, i.e. a team of people that provides the stability to support the change [22]. There is definitely an ability to change as the council has gone through a big transformation and has come out of it successfully. However, it is too early to judge whether there is a strong core that can provide stability and flexibility to adapt and change, and internal challenges were identified (e.g. vulnerability of core team, leadership still forming as a team, …).

the organisation is still very reliant, I think, on the top team being very clear what it is trying to achieve.

We found examples of innovative approaches but we also found some concerns that ‘the need to deliver today’s results is an inhibitor to bold action’ [22].

6 Discussion

In this section we discuss our findings in the context of the research questions, and highlight observations about the ‘balancing act’ we perceive.

6.1 RQ1: What Successes and Challenges Are Identified Towards the End of a Behaviour-Led Transformation to Become an Agile Organisation?

Table 1 provides a list of successes identified by our interviewees. Many of the factors for a successful transformation, highlighted by Dikert et al. [1], are reflected in those themes, namely:

  • management support – strong team supporting each other (theme in Table 1);

  • commitment to change – change was initiated by a very committed leadership in the council;

  • leadership – a supportive leadership (theme in Table 1);

  • training and coaching – all staff were well supported in going through the transformation and the behavioural assessments;

  • engaging people – collective ownership (theme in Table 1)

  • communication and transparency – good communication (theme in Table 1).

While Dikert et al.’s success factors focus on what needs to be in place in order for the transformation to be successful, our data was collected once the bulk of the transformation activity had taken place. But when interviewees were asked about successes of the transformation undergone, most talked not only about where they had got to, but also about the process itself; even the interviewees who had been recruited at the end of the significant transformation period were well aware of the process and referred back to it. Success factors for an agile transformation are also relevant to long-term sustained agility [5]. Carvalho et al. talk about agility enablers rather than success factors; enablers characterise agility in an organisation and some of our themes also appear as enablers, such as their organisational commitment and employee empowerment.

Some of the successes we encountered do not appear in Dikert et al’s categories, in particular, the following themes (discarding the ones specific to the context of the case study): a clear and inspiring purpose focusing on results to stakeholders; a feeling of achievement; fluid, constantly changing, iterative; and, restructuring, consolidating, learning. There are naturally differences when looking at agility from the perspective of software development and from the perspective of the whole organisation. We suggest, however, that the two perspectives are complementary and that the agile software community can benefit from understanding the wider perspective of the organisation. Lenberg et al. [23] stress the importance of organisational values in software companies, as successful transformations depend on organisation-wide aligned values.

Having undergone such a radical transformation, the local council is at a point where it can be considered successful as an agile organisation. The survival strategy adopted by the council was to undergo a behaviour-led transformation to become a commercially-minded organisation; however, the behaviours chosen are those of an organisational culture related to agile use [20]. An agile organisation is characterised by its capability of sensing and responding [4, 11], which was the aim of the council.

We also found challenges in two areas when mapping to the Agile Culture Development Matrix (Fig. 1), Collaboration & Autonomy and Adaptability to Change. The challenges encountered are not about the behaviours chosen, but rather about their implementation. The first falls within the Building effective relationships behaviour and the second under Innovating and adapting to change behaviour. Both of these behaviours were well accepted by interviewees but for both there were disconnects between the behaviour and practice; in the former, around the theme of Internal processes and procedures, and in the latter around the theme Leadership vulnerability and resilience to change (see Table 2). Only the first of these two themes resonates with a challenge in large-scale transformations [1]: Autonomous team model challenging. Although this challenge in Dikert et al. is about software teams it also emerged in the council: the lack of balance between the autonomy of teams and the broader goals of the organisation.

We concentrated on these two challenge areas as they were the most relevant to the council to assess and improve where they were, towards an agile culture.

6.2 RQ2: What Improvements Suggested in the Literature Are Applicable in This Context?

What can be done to implement intended behaviours better and to sustain what has been achieved? To address the first of the challenge areas, Collaboration & Autonomy, we drew on established frameworks to make suggestions for the council. Doz and Kosonen [4] developed a framework for strategic agility with 3 areas: strategic sensitivity, leadership unity, and resource fluidity. Of particular relevance to developing collaborative communities are actions suggested under leadership unity: dialoguing, surfacing and sharing assumptions and understanding contexts; and, aligning, rallying around a common interest. However, a balance needs to be struck between empowering collaborative communities and setting macro level goals while distributing authority. The fine tuning of this balance between autonomy and accountability requires the communities to have a clear strategy and clarity of purpose [24], and boundary conditions and expectations [25]; these help with establishing accountability within defined limits giving employees the freedom to decide how to achieve objectives.

Another balancing act has to be achieved between how much is left to autonomous teams and how much is documented in processes and procedures. Agile developers recognise that documentation is important for some projects, but are selective. Lessons can be learned from their practice to help achieve an optimal balance [26]; for example, checking whether and why documentation is needed, and for whom.

Addressing the second challenge area, Adaptability to change, and in particular the need for a strong core that provides stability, requires a succession plan, and relevant capabilities to be developed and supported [11, 12].

Sustaining agility is not mentioned as a challenge in large-scale agile transformations triggered by software development (e.g. [1]), but the need for sustainability is recognised as a challenge in organisational transformations (e.g. [5]). Further research is needed to understand why sustainability is not apparently an issue when software triggers the transformation.

6.3 The Balancing Act

The behaviour-led transformation undertaken by the council was intended to achieve a change of culture by only employing people who exhibit certain behaviours. It seems that this approach had a positive impact for the survival and financial sustainability of the council. But the challenges in their implementation require a balancing act between opposing forces:

  • disruptive transformation activities while carrying on business as usual;

  • empowering collaborative communities (resources) while setting macro level goals and distributing authority (responsibility) – a three-way balancing act;

  • autonomy for decision making while defining and documenting processes and procedures that need to be followed, e.g. for regulatory reasons; and,

  • adopting the desired behaviours while demonstrating the required skills.

We suggest that to achieve a successful transformation to an agile enterprise compromises have to be struck between these contradictory forces, with fine tuning actions to achieve the correct balance. The example of Spotify [25] suggests that to achieve the right balance between autonomy and accountability requires: a strategy and clarity of purpose, transparent boundary conditions, and expectations. This framework gives employees freedom to choose how they achieve objectives within exisiting constraints. But, is adopting and fine tuning the right behaviours sufficient to sustain a change in culture that can be sustained? Robinson and Sharp [27] discussed the relationship between behaviours (in their case XP practices) and culture and the difficulties in relating the practices adopted with the underlying culture. We suggest that achieving a change in culture through behaviours is not necessarily a guarantee for that change to be sustained. A continuous process of revisiting behaviours, learning lessons, and adjustment is under way in the local council and that is supportive of sustainability. But more research is needed to understand how agility and transformations can be sustained in the long term [28].

7 Limitations

There are limitations in the work presented here. The constraints of how the case study was conducted only allowed for a partial view of the local council with no access to staff below middle management. From our analysis, we also did not have enough data to consider all elements of the Agile Culture Development Matrix; to assess all the areas of the matrix would have required an organisation-wide consultation. Also, although we carried out the work after the main period of the transformation, the council has not stopped and changes have been happening since and will continue to happen.

The threats to validity [29] were addressed as follows: for internal validity, data was collected by three researchers who also carried out the analysis and discussed the data with the wider author set; for construct validity, the constructs emerged from the participants and were not imposed; for reliability, it is quite likely that the same results would emerge if conducted again with the same questions. As for external validity, the case study in this paper is a snapshot of a continuous journey; it is difficult to generalise what we found to other contexts.

8 Conclusions

The literature on large-scale agile transformations has been mainly focusing on software development transformations; concerns about the wider organisation are acknowledged but the assumption is often that these transformations are triggered by the digitisation of organisations. The case study in this paper presents a different angle: that of a local council that realises the need for transformation as the only way to survive and be financially sustainable. This was achieved successfully through ‘commercially’ oriented behaviours. The challenges encountered were about achieving the right balance in the implementation of these behaviours between: disruption and business as usual, empowerment and goal setting, autonomy and processes and procedures, and behaviours and skills. In this case study, behaviour change has led to evidence of an agile culture but a change in culture through behaviours alone is not necessarily a guarantee for that change to be sustained [14]. More effort is needed to achieve an appropriate balance, and work to maintain the behaviours and hence to sustain the change. These balancing acts were encountered in a transformation towards business agility, but they also need to be addressed by the agile software community. The focus on sustaining agility and on an organisation-wide perspective is important to both enterprise agility and to large scale software development agile transformations.