Gaming as a research method in humanitarian logistics

2.1 The need for relevant and contextualized research in humanitarian logistics

Although many authors have stressed the need for empirical research (Starr and Van Wassenhove, 2014; Van Wassenhove et al., 2012), the most prominent research method in humanitarian logistics is still analytical modelling and simulation (Kunz and Reiner, 2012; Leiras et al., 2014). At the same time, there is increasing concern about the relevance of humanitarian logistics research for practice (Laguna et al., 2015; Leiras et al., 2014).

Kunz et al. (2017) identify eight barriers to relevant research, falling broadly into barriers for research and barriers for communication and data sharing (including trust and competition). We will use the first set of barriers to develop requirements for gaming as a research method, and to identify problem areas for which gaming is a particularly promising method:

1. Problem definition driven by standard problems: much research is dedicated to improving operations research models and simulations (see above). But humanitarian logistics problems are often ill-structured and messy, requiring the use of multi-disciplinary cross-functional approaches (Näslund, 2002). At the same time, Jahre et al. (2009) stress the need for theory building within the field of humanitarian logistics. Particularly to explore and understand the impact of new, emerging and innovative practices and technologies, such as cash transfers, digital identity, block-chain based smart contracts or new monitoring and tracking systems, research methods are required that support theory building and problem formulation across different research areas and disciplines.

2. Lack of contextualization: as humanitarian responders are working in a complex systems of different organizations, mandates, norms and supported by a range of technologies (Van de Walle and Comes, 2014), it is important to consider interventions in the context. To overcome the lack of context, which is crucial for socio-technical systems, case studies for theory building, field work or participatory action research have been advocated in logistics research (Jahre et al., 2009). Being bound to past or on-going cases, however, implies that logistics research is not oriented towards the future and the fundamental changes that it may bring (Näslund, 2002). Maybe most prominent in this context is the discussion on trends such as climate adaptation or urbanization (Kovács and Spens, 2011), and the interplay between response and development via social protection systems or local procurement programmes.

3. Difficult data collection: access to regions that are affected by disasters may put humanitarians, beneficiaries and researchers at risk and adds a burden to already stressed system. The lack of research on conflicts has already been mentioned before. But even if access to selected sites or responders can be achieved, there are often only few data points are interviewees (Chan and Comes, 2014). Here, methods are needed that foster the reproducibility and generalizability of results in a safe environment and allow for comparing different variables and settings.

4. Lack of validation with practice: model validation is most commonly done by comparing computational results with benchmark cases (e.g. performance for a past case) without being reflective on why specific decisions were made and if the underlying modelling assumptions on the constraints or objectives are correct. Here, methods are needed to foster the participation of humanitarians in a way that fosters learning in both academia and practice.

2.2 Simulation gaming

Simulation gaming is an increasingly accepted research method to study complex systems (Kurapati et al., 2018). Simulation games can be described as experimental, rule-based, interactive and social organization, constituted by the players themselves, who learn by taking actions and by experiencing their effects through feedback mechanisms within the game (Mayer, 2009; Lukosch et al., 2018). They can be distinguished based on the purpose they serve (Shubik, 1983; Ståhl et al., 1983), leading to following taxonomy:

1. Training games – with a wide use and acceptance in industry, military and education, and the aim to train specific skills without too much conceptual detail.

2. Teaching games – broader in scope than training games, and try to address wider concepts and more abstract ideas.

3. Experimentation games – aim at testing theories or hypotheses, and can also be used to test the effects of certain variables on a given situation.

4. Research games – are designed and/or used to obtain data or empirical material, dealing with a more or less realistic situation or scenario, addressing experienced players.

5. Operational games – to aid decision making, planning and policy implementation in specific situations.

In simulation games, players enact a certain role in a simulated environment (Duke and Geurts, 2004). As pointed out by Klabbers (2018), when games are used for research, they should address players in their capacity of being reflexive actors; while playing being engaged with introspection, allowing for reflection-in, and reflection-on-action. In the language of Sociologist Anthony Giddens, games represent a duality of structure as players of games constitute the emerging social organization, while at the same time the game is the very medium of its constitution (Giddens, 1993; Klabbers, 2018). Thus, while playing a game, a player continuously sets new (social) rules, and observes the consequences of his or her actions and decisions. In games, transfer of knowledge occurs through the player’s background that influences game play, and through the process of acquiring knowledge and skills from the game play that can be useful in a specific real context (Copier, 2007). As Klabbers (2018) discusses, the player is not only a subject, but also on agent or actor, who demonstrates purposeful behaviour, based on a certain set of skills and knowledge. Such behaviour can be observed and analysed in a qualitative way.

As such, simulation gaming has a long tradition and proven value in education, policy making and research (Duke and Geurts, 2004; Klabbers, 2006). While games are clearly distinguished from the real world (Klabbers, 2006), their boundary is open to transfers between game and reality, which makes them applicable tools to understand and design real complex systems.

Nowadays, games come in many forms and are based on different technologies. From role-playing games that are based on (almost) no materials, to physical board games, to high-end games that use advanced digital technology. The choice of the applicable game type is based on factors such as fidelity or level of realism that is needed for the purpose of the game, the size of the target group and the quality and quantity of data to collect. Physical board games, for example, have the advantage to enable direct communication and collaboration amongst a relatively small group of players, and allow for open game play, as often rules can be changed by the players (and/or the facilitator) during game play, if necessary. Disadvantages of this type of games are that usually it is difficult to scale them up to a large group of players, and that data collection is limited to observation and reflection. In comparison, digital games are more expensive in development cost and time. On the other hand, data collection and processing can be automated and can be used for quantitative data collection. The implementation and adjustment of scenarios as context and storyline for the game can be easier than in physical games. In some cases, it is not necessary to design complete games, but to make use of the engaging nature of game elements, in order to, e.g., use scenarios to increase the realism of a given exercise, and to foster more realistic behaviour of test persons.

2.3 Gaming as a research method

While in the past simulation gaming has been largely ignored as a research method (Greenblat, 1975), the increasing need to understand complex phenomena slowly turn simulation games into an accepted research instrument (Klabbers, 2006). Especially the use of simulation games to integrate different perspectives, concepts, theories, data, information, methods and techniques from various disciplines makes this method a valuable tool for meaning making and trans-disciplinary research (Klabbers, 2009). As a research method, simulation games are usually combined with other instruments such as questionnaires, tests and debriefing (Kurapati et al., 2015; Lukosch et al., 2018).

As other methods that help in understanding complex problems, simulation games are based on a model. Bradley et al. (1977) characterize the use of models in providing guidelines to decision-makers for effective decisions as operational research. Gaming can thus be described as an approach of operational research, applicable to complex systems. Bradley et al. (1977) categorize tools for operational research into four types: operational exercise, simulation gaming, simulation and analytical model:

1. In an operational exercise, the (research) experiments are directly implemented in the real environment. As discussed above, this is often not possible in humanitarian logistics research, as the real environment can be very dangerous for both participants and researchers as well as for the technology to be tested. Yet, the advantage of an operational exercise is that it can deliver the highest degree of realism.

2. Simulation games are interactive environments that allow for direct engagement and provide immediate feedback to players and researchers. The level of realism is lower than in operational exercises (Bradley et al., 1977), as they represent a more abstract environment. Based on the technology and game process used, it can be easy or difficult to collect valuable data for research purposes. While a pure simulation aims at representing a system as realistic without human interference (Bailey, 1982), a simulation game is a more simplified, yet realistic representation of a system that can be manipulated by (the decisions of) players. Games are hence characterized by a game layer on top of a simulation.

3. In simulations, an environment is represented by the use of mathematics or objective representation (Feinstein and Cannon, 2002). Simulation models are inductive and can be used to evaluate the performance of a given system (Bradley et al., 1977). A limitation of most forms of simulation is that the human decision-maker is removed from the representation.

4. Analytical models are theoretical constructs of a given system, expressed in mathematical terms, usually based on a simplified framework, to visualize complex processes (Choi et al., 2016). Analytical models are easy und cheap to develop (Bradley et al., 1977). Yet, their level of simplification of a given system is usually the highest, hence their level of realism is very low.

Above, we identified gaps and requirements for research methods in humanitarian logistics. Table I compares aforementioned OR research methods based on these aspects, highlighting that simulation games bridge the gap between operational exercises and simulations or analytical modelling.

As in any other method, the validity of the results from gaming of course depends on defining a rigorous set-up and design. Traditional research designs follow four steps: random selection of subjects; random assignment of subjects to different treatment conditions; experimenter manipulation of treatments and experimenter control over the conduct of the experiment (Bachrach and Bendoly, 2011). The strict control over some of these aspects is not given in simulation games. For example, a random selection of subjects can be difficult, when only a limited number of experts is available for a gaming session. Yet, the rigour of the method can be guaranteed, when the process of experimental research is followed. This process can be formulated as conceptualizing the research question; operationalization and design; methodology and collecting data; validity testing and interpretability; and effect and relationship testing (Bachrach and Bendoly, 2011). All steps can and need to be followed when a game is used as instrument.

One of the main advantages of simulation games is that they can provide rich output and data. For example, reactions and responses of players in scenario-based role-playing exercises can be recorded, quantified and analysed (Rungtusanatham et al., 2011). As in any other experiment, researchers have to check whether the abstraction of a simulation game influences the results, and how to apply the results to the outside world (Deck and Smith, 2013). Compared to simulations and analytical models, however, the level of complexity and realism of games usually higher.

In sum, games can be seen as operational and interactive representations of a real system, allowing participants and researchers to literally “play” with game elements and observe the consequences. We highlighted above, that there are many choices in game design that influence the validity, type and quantity of data that can be collected through simulation-game-based research and the importance of following design guidelines to develop games for given contexts and research aims. In the following section, we show which design methodology we apply for the use of gaming as research method in humanitarian logistics.

2.4 Designing games for research

To design games for research of complex socio-technical systems, it is crucial to address three aspects: the actor layer, the technological layer and the relationships among actors and between actors and technology (Klabbers, 2018). Especially defining the relationships can lead to interesting game mechanisms, as forms and rules of communication, information sharing, or power relationships can be displayed. Games can be used to research these relationships and can motivate players to change them to improve the real system (Lukosch et al., 2018). In a rigorous way, games can be seen as an artefact in a design science cycle, or to facilitate “research through game design”, as Figure 1 illustrates.

The proposed research through game design process starts with identifying a research gap, and formulating related research questions, as shown in the lower left of the figure. This is mainly a deductive activity. From these theoretical considerations, game requirements are formulated based on theoretical insights and practical considerations. Such rather abstract requirements are then further developed in simple, often paper-based game prototypes. In test sessions with game design and content experts, these prototypes are tested and evaluated. After consolidating a final design, the experimental set-up is developed.

The data collected in such research cycle can be very different in nature. Typically, game data are collected either directly from the game (scores, decision, actions) or from (structured) observations (validated) material can be used, such as debriefing material, pre- and post-test surveys and other instruments that are related to the research questions. Various approaches towards data analysis can be used, such as content analysis, coding of verbal feedback or Bayesian analysis.

The cycle illustrates that gaming can be seen as part of a whole research design, where a theoretical gap analysis leads to research questions that inform both the design of a game prototype, and the development of hypotheses or a research framework.

3.1 Case study: information technology for humanitarian logistics in conflicts

To illustrate how gaming can be applied as research method in humanitarian aid logistics, and how this field can benefit from gaming research, we focus on information technology developments. Humanitarian organizations are confronted with mounting tensions as they seek to maintain access to populations in need and provide aid to the most vulnerable populations in crisis regions such as the Middle East or Africa. Technological innovation is seen as one of the major game changers to enable humanitarians to “Stay and Deliver” (Egeland et al., 2011). For example, satellite imagery, real-time analysis of camera footage from cars or UAVs and more generally speaking remote sensing and monitoring are seen as promising approaches to help humanitarian organizations to rapidly evaluate access conditions and risks (Comes and Van de Walle, 2016). While in 2015, cash and markets-based interventions only amounted for about 6 per cent of all aid, relief provision is beginning to shift towards virtual distributions through digital payment systems, or “mobile money” (Logistics Cluster, 2015). Orders and shipment logistics can be complemented by social media feeds and analyses of customer buying patterns (Majewski et al., 2010), and biometric identification technologies are increasingly used for refugee management (Jacobsen, 2015). Real-time tracking and monitoring is expected to improve efficiency and effectiveness of logistics by enabling decision-makers to adapt to the ever-changing context of the operations. As such, technological innovation entails the need to adapt decision processes, coordination and management structures (Comes et al., 2018).

Introducing new technologies or processes in an on-going conflict poses a significant risk, because any deviation from procedures, malfunction, unforeseen use or even possession of technology can cause harm. At the same time, it is important to take into account the specifics of the context, such as the sensitivity of data and information, or the legal and technological constraints, security and logistics guidelines and policies (Van de Walle and Comes, 2015). As such, the challenge of evaluating the impact of monitoring and tracking technologies on humanitarian logistics in conflict is used as example for many other innovations that are currently being explored as new humanitarian technologies.

Van Wassenhove (2006) argues that a humanitarian supply chain needs to be designed to align material and informational flows. To evaluate the impact of innovation on supply chain performance and the use in terms of (improved) decision making and coordination, we therefore analyse a humanitarian supply chain across three layers: in terms of physical movement of goods and people; informational flows; and decision making and planning.

3.2 Game design approach

To explore which technologies could be beneficial for humanitarian operations; and to identify what the impact of the developed solutions is on all the three levels, we developed and used a two games in the context of a European research project that develops technologies and policies for staff safety and logistics support in conflicts. First, we conducted a requirements analysis for the development of a tracking, monitoring and logistics support system. We supported this step with the development and implementation of a board game. Second, we used game elements in a large-scale exercise to evaluate the usability and usefulness of the developed system.

Here, we use the game research cycle (Figure 1) to discuss how the respective games were designed and used. All documentation of the concrete game designs, execution and evaluation are available publicly and openly in the project deliverables on www.itrack-project.eu/page/en/documentation/public-deliverables.php

For the board game, we first conducted a systems analysis of the processes of humanitarian logistics, building upon theoretical knowledge in humanitarian logistics, information management and ICT. Second, we reviewed the proposed solutions from technical partners in the research project. We chose to use the game to explore decisions and preferences of aid workers towards different technical components of the proposed system. This set-up led to a number of game requirements, such as that the game should be placed in the context of a crisis situation, that it should foster the discussion between different humanitarian aid organizations, and that it should be playable within half a day. Based on this identification of requirements, a first paper-based prototype was developed and test-played with game design experts. Based on their feedback, and after some design iterations, the final design of “Plaitra” was developed. As experimental set-up, it was decided to conduct a qualitative study, in which the game served in first instance for making the players aware of the different technical solutions, and to observe their choices in the game. Second, a debriefing structure and questions were developed, following the guidelines by Kriz (2010).

For the exercise, the research aimed at testing a prototype of the technologies and its impact on workflows, communication, decision making, coordination and performance. Here, we used game elements, especially scenario building and role-playing, to support a realistic system test. Again, theoretical articles as well as working reports were analysed to identify current processes and policies in humanitarian logistics. Those were translated into scenarios and role descriptions that related to different system components. A large number of additional research instruments was selected, such as usability test instruments, demographical data collection, questionnaires to evaluate the experienced usefulness of the system, as well as debriefing structures. A combination of qualitative and quantitative data analysis methods was applied to interpret the data collected. The results of this test informed the development of the final, integrated version of the socio-technical system.

4.1 Design cycle: needs analysis, prototype development, tests

The physical board game “Plaitra” is the result of an iterative design process, including academics, game designers and experts from the field, following the research design cycle as shown in Figure 2. First, interviews were conducted with humanitarian aid workers to explore what kind of technology is used in conflicts, and to carry out a needs analysis. Via a literature review, we identified requirements for innovations to be of use for humanitarian operations in this context. Based on both empirical work and literature review, research gaps were identified. Second, a game prototype was developed and tested with students of a technical university as well as experts with a background in humanitarian aid (for details, see Schwarz et al., 2017). The main requirement for the game design was that the game session should allow researchers to observe the decisions of the players for a technology. Furthermore, the impact of that choice on logistics operations should immediately be visible both for players as well as for researchers.

In the game, players take over the role of humanitarian organizations, who have to efficiently allocate limited resources while dealing with multiple risks, uncertainties and impediments to communication. Additionally, no single organization can cover all humanitarian assistance needed; instead, organizations have to collaborate. Consequently, humanitarian actors have to share information and coordinate their operations in order to ensure a coherent response.

The game material consists of two boards that represent a crisis region, as illustrated in Figure 3. Four teams of two players play the game, while two researchers/facilitators support them. Players have to carry out assessments to indicate the humanitarian needs in particular places. Players can invest in technologies to gather and share information.

We identified three main categories of information that were translated into the game: humanitarian needs, actions of other humanitarians and security situation. In order to let players experience how technology impacts information flows and thereby logistics processes and coordination, the following goals for the game have been identified:

1. The main purpose of the board game is supporting requirements analysis. After game play, a structured debriefing is used to discuss technologies and policies with the players.

2. The technological components that are available for the players in the game represent the components proposed by the technical partners of the research project, and beyond. They include advanced technologies that are not deployed in the field of humanitarian logistics yet. The extensive list of available technologies in the game is designed to make the player aware of the possibilities of technology for the humanitarian field.

3. Information management workflows and processes have been modelled for implementation in the game, which allows us to explore the processes and policies of their use, too.

4. The role of coordination is addressed in the game. Communication is not allowed in the first rounds of the game. Players need to invest in technologies to be able to share information and coordinate with each other.

5. Humanitarian responders operate with constrained resources. This has been modelled in the game by disruptive events happening during the game play, and the limited availability of technology at the beginning, but also during the game.

4.2 Empirical cycle: experimental set-up, game session and data collection

We decided for qualitative data analysis of the game session, as we expected only a limited number of test persons being able to participate in a game session, and our research was primarily of exploratory nature. We first created an observer protocol and an in-game observer role to make the role less intrusive. During game play, a journalist notes down observations of player actions and decisions along a pre-structured protocol. Furthermore, journalists can provide some information to the players. In addition, the game session includes a debriefing phase, following the structure of debriefing as proposed by Kriz (2010).

We invited 16 humanitarian practitioners with experience in the field to take part in a full-day game workshop. The 16 players were randomly distributed to two parallel sessions that took place at a University in Finland. In each session, a facilitator as well as a journalist joined. The game play started with a briefing, introduction to gaming, purpose and main rules of the game. After that, the players played 16 rounds with the game (see for an impression Plate 1).

The observers stayed during the whole game play and wrote down their observations. After game play, players first filled in a questionnaire about their experience of the game play. Then, all 16 players gathered for a plenary debriefing that was facilitated by a researcher who was not involved in the game play. After the plenary debriefing, the players discussed in smaller groups which technology would be useful and usable in humanitarian aid logistics. The results of these discussions were collected, too.

4.3 Results

Via the questionnaires, we collected experiences with the game, and how useful the players valued the game as instrument itself. The outcome of this research step was foremost positive. This feedback showed that the game design was perceived as engaging, purposeful, and – despite the high level of abstraction – realistic. Some comments from the questionnaires and debrief allowed us to improve the fidelity and realism of the set-up and workflows.

Through a combination of performance measuring of the play itself, observation of the discussion during the games, the de-brief sessions conducted after the game play, a number of issues were raised that are relevant for the further discussion of technology innovation:

• the general scepticism against technology innovation particularly at operational level;

• the need to address secondary impact of using technology in the operating environment;

• the need to support decision making under security threats during convoys; and

• the need for decentralized communication structures among operational staff; and the need to align technology and workflows and processes.

These results show that the research through game design approach allowed for broader insights than the choices of players alone. Especially, the combination of research instruments, namely game play, observation, questionnaire and debriefing, provided rich material that allowed for above mentioned conclusions. To specifically address the last point in the list, the simulation exercise was designed, again using the game design research cycle.

5.1 Design cycle: exercise requirements, tests and overall design

The first step in the design of the simulation exercise was to define the requirements for the exercise. The exercise itself was meant to close the first development cycle of the integrated version of the tracking, monitoring and logistics support system and to inform the second development phase with additional requirements. Two sets of requirements for the exercise were identified: first, an evaluation framework was developed that addressed the individual technological components, as well as the integrated platform. An inventory of all technological components and related hard and software needs was made, and we chose to evaluate their usability with help of existing and accepted instruments.

Second, the purpose was to create a realistic stress test for the system beyond the limitations of a clean, experimental test situation. To this end, scenarios, understood as game elements to increase the fidelity or realism of the exercise, were developed, to enable participants to test the system in a realistic quasi-experimental environment. Different locations were chosen and decorated into realistic mission control room, warehouse and checkpoint (Plate 2). In addition to the experiment materials such as laptops and mobile phones, other facilities used to increase the fidelity included a convoy of three vehicles, radios and a UAV.

The exercise was monitored at both the system and the process level. The system was monitored at the back-end and fine-tuned immediately when necessary. Operations at all three exercise locations were filmed through webcams and all communications made in the system were logged for post exercise analysis.

5.2 Empirical cycle: realization of the exercise and data collection

The simulation exercise was carried out in April 2018 with participants from the project consortium, humanitarian organizations and EU civil protection. The usability tests focussed on testing each component. The participants were asked to fill in various surveys to evaluate the user experience (Laugwitz et al., 2008), the components’ usefulness and ease of use (Davis, 1989), as well as the ethics and privacy issues addressed by the system. The usability test was completed with a short debriefing for direct comments and feedback to the tech partners.

In order to effectively test the integrated system, the simulation exercise adopted scenarios of basic logistics workflows that were combined with a collection of special injects to represent various field situations. The special injects covered issues from the perspectives of logistics procedures, humanitarian information management and security risk management. All exercise data, including communication and system operations, were logged automatically on a secure local server for analysis. A debriefing session was held at the end of each day to discuss issues encountered from the exercise and collect feedback on both the system and user behaviour. At the end of the week, a final reflection meeting was organized to summarize the progress made and the lessons learnt throughout the simulation exercise.

5.3 Results

All participants viewed the simulation exercise valuable and as an insightful way of testing technology. The scenario-based testing approach allowed participants to experiment new technology in a realistic and safe environment. The exercise week also brought the technology developers and end users together and shortened the feedback loop between the test lab and the fields. In addition, playing logistics themed scenarios and injects also provided a platform to reflect specific policies and protocols in carrying out missions. In the exercise, especially the realistic documents used for information management as well as the realistic communication protocols applied added to a realistic feeling. They contributed to a realistic, while safe test environment in the context of humanitarian aid logistics.

7.1 Implications for practice

As discussed prototypically for Plaitra, games have a great potential to connect research and innovation for practice if they are useful for training purposes. We think that this is a unique way to use the power of engagement and the “fun” factor of the games that players generally agreed on for both research and practice.

However, these games have to go beyond the traditional exercises that focus on situations that can be anticipated or that have occurred before to prepare responders and logisticians for the response. In a more and more uncertain and complex world, the exact conditions that field logisticians will be confronted with are harder and harder to predict, and particularly the impact of new technologies on the future operations is hard to foresee. In such uncertain situations, different types of preparation are necessary, that relates to general skills such as flexibility, adaptability, creativeness, communication and decision making. Results of such empirical research can help to support actors in both ways.

Design recommendations for practice:

1. Be aware of the dynamic and uncertain nature of the field and make these characteristics a vital element of the game to be used.

2. Games are only one method to gather a certain type of data related to the human factor. Make sure to include data from other sources to develop a complete picture of a problem.

3. Games are dynamic and the players create their own reality. Let them explore and observe, use the observations to increase the feedback of the game.

7.2 Outlook and implications for research

We have argued that gaming as a research method has potential to contribute to theory building and exploration of new emerging technologies; in situations of limited access and high risk; or for rare and extreme situations. Gaming as a method particularly allows us to explore the interplay between an intervention (e.g. a policy, process or technology) and work practices, decision making and team performance.

Design recommendations for research:

1. Ensure that design and empirical cycle in the research process are well related.

2. Start with formulating a problem, based on theory (and practice), and define game requirements including data to collect.

3. Go beyond designing a game as isolated tool. In some circumstance, it can be useful to use game elements instead of a complete game. Carefully consider and choose additional research instruments including a reflection moment.

Gaming as a research method can open the path to new insights and models, but because of the natural restrictions in numbers of players and experts, duration of play, facilities and equipment, gaming is an addition to other research instruments such as field studies or optimization and simulation. Here, approaches need to be explored that enable a translation of the data and insights collected through the games into valid theories and models, and that embed these insights back into a new cycle of game design and testing.