Position PaperTools and methods in participatory modeling: Selecting the right tool for the job
Introduction
Numerous tools and methods facilitate stakeholder engagement in participatory modeling (PM), which Stave (2010) defined broadly as “… an approach for including a broad group of stakeholders in the process of formal decision analysis.” In the PM process, participants co-formulate a problem and use modeling to describe the problem, to identify, develop and test solutions, and to inform the decision-making and actions of the group. Therefore, we define PM specifically as a purposeful learning process for action that engages the implicit and explicit knowledge of stakeholders to create formalized and shared representations of reality. Since PM is heavily focused on collaborative learning, the tools and methods used during PM projects are expected to promote system understanding and awareness for all stakeholders. By stakeholders we mean all who have a ‘stake’ in the project. This includes modelers and researchers themselves, who are often considered external to the project but still have interests in it, come with their own biases, and cannot be assumed totally objective and neutral (Voinov et al., 2014). The level of engagement differs across stakeholders and varies from one stage of the project to another (Arnstein, 1969; Hurlbert and Gupta, 2015; Reed et al., 2009).
Argyris and Schön (2002) showed that there are two levels of learning, referred to as “single loop” and “double loop” learning. In single loop learning, individuals and groups act within a single reference frame, where specific hypotheses, values, norms, beliefs and objectives are assumed to describe the world. Learning in these systems consists of observing the results of actions and, potentially, modifying future actions based on what is observed. In double loop learning, actors question and learn about the reference frame itself, and may change their fundamental hypotheses, values, norms, and beliefs based on what they learn about the system, as well as what they learn about the outcomes of specific actions (Zellner and Campbell, 2015).
The transition between single and double loop learning can result from the interaction between individual and organizational learning. Argyris and Schön (2002) found complex retrospection and feedback mechanisms between individual and organizational learning. The individual mental models that are used to construct shared mental models of an organization coalesce, thereby modifying the perception of the organization and transforming organizational values and paradigms. In turn, this modifies the environment of the individuals and affects their own mental models (Daré et al., 2014). As a result, the act of model co-creation is, in itself, an act of knowledge construction at both the single and double loop learning levels. In some cases, PM processes deliberately avoid formal model co-creation to first allow the identification and challenging of stakeholders’ causal beliefs and expectations and, consequently, a reconstruction of knowledge (Habermas, 1990; Smajgl and Ward, 2013).
The goal of this paper is to provide an overview of some of the methods and tools for PM, identify some of their strengths and weaknesses, and provide some guidance for practitioners as they select methods for their PM projects. For the purposes of this paper, we define a tool as a modeling technique used to carry out a particular function to achieve a specific goal. Tools are defined, documented, do not change significantly through use, and are clearly external to their users and often not created by them. In contrast, a method is a way of doing something, in particular, a way of using tools. According to Mingers (2000), a method is “a structured set of processes and activities that includes tools, techniques, and models, that can be used in dealing with a problem or problem situation.” A particular method can be supported by one or several tools. For example, in this context agent-based modeling (ABM) is a method; Netlogo, Mason, or RePast are some of the possible open-source tools used to perform ABM. Multiple tools often exist to support a single method, and some tools also serve several methods. For example, Netlogo, AnyLogic™, or Numerus™ are tools that can be used within both ABM and System Dynamics (SD) methods.
While the choice of methods used can heavily impact both processes and decisions, there is little scholarly discussion about how tools and methods are chosen during PM. Certainly, decisions about methods are more influential for the whole process than the choice of a particular tool, and therefore should come before choice of tools. For instance, there are not many implications in deciding to use Stella® rather than Vensim® or Simile; all are well-established tools that support the SD method. But the decision to implement a more quantitative method rather than a qualitative or conceptual one can potentially significantly change the outcome of a PM process. For example, a companion ABM based on role-playing games (see Barreteau et al., 2001) can increase stakeholder involvement in the PM process and may generate much different results than computational ABM using only computer simulations and modelers’ assumptions.
Previously, Voinov et al. (2016) reviewed several participatory tools and methods that have been used to enhance stakeholder participation for different components of the PM process. They concluded that, while many different methods are used for various stages of the process, in practice, there is rarely much justification given for the use of a particular method. It is difficult to find examples of participatory projects that used different combinations of modeling methods when dealing with the same problem. In most cases, once the method (or combination of methods) is chosen, it becomes the only one reported. We recently reviewed 180 papers related to participatory modeling as part of a SESYNC project on “Synergizing public participation and participatory modeling methods for action oriented outcomes” (https://www.sesync.org/project/enhancing-socio-environmental-research-education/participatory-modeling). We found no papers that reported using one method and then a switch to another method. This may be due to a general reluctance to report failures rather than only success stories, but it complicates the comparison of different methods. Another reason most studies report only one method might be that switching from one method to another is costly in terms of time and resources. A similar, though much more limited effort in healthcare research, which focused on comparison of three dynamic simulation methods, SD, ABM and Discrete Event simulation (Marshall et al., 2015) also reports very few failures of particular methods that led to switching to other methods.
A careful and conscious selection of methods is important for the modelling process and its outcomes. Ideally, the selection would be accompanied by effective evaluations to monitor the impact of individual methods used during in PM (Hassenforder et al., 2015; Smajgl and Ward, 2015). However, in many case studies, the choice of methods and tools seems largely driven by the experiences of participating researchers (Prell et al., 2007). This is a manifestation of the hammer and nail’ syndrome: once someone learns to use a hammer, everything starts to look like a nail. A researcher with expertise in system dynamics is very likely to apply system dynamics for the next modeling project, even if other methods could be equally or even more appropriate to address the full set of driving questions. Retraining is time-consuming and resources are always scarce. Engaging colleagues with experience in alternative approaches could help expand the scope of methods considered, but this is not always feasible. There are practical and social reasons why this experience-driven approach to method and tool selection is not optimal, especially in the field of PM. First, the value of PM in developing models that effectively and efficiently meet participants' requirements will be improved by using methods that best fit the project purpose and context. The modeling skill set available should be considered only to identify gaps in the skills required to address the problem in question. PM seeks to be transparent to the users and it is critical to make sure that PM practitioners are not treating all problems as nails just because they are good at using a hammer. Stakeholders, defined broadly as above, are expected to engage in all steps of the PM process, which includes method selection as well as the modeling steps. While the participation of various groups of stakeholders will certainly be different, at each stage all stakeholders should understand why the chosen methods and tools are appropriate. This requires some flexibility in the PM process, whereby stakeholders move collectively from the problem to an appropriate method, and onto tools and associated skills found within the project team. A sharper focus on method and tool selection is needed. This requires understanding stakeholders’ preferences and constraints, including their experience with particular methods, the availability of training for specific methodologies, the ability to use and maintain a particular tool for the long term given the costs to do so, and/or the ability to combine a new tool with existing tools or methods (Smajgl, 2015).
Second, social factors may also affect method and tool selection. The choice of methods is more than a technical decision; it can also involve ethical or other social judgments. It may make it easier or more difficult for specific groups to participate effectively, and to adequately represent specific technical aspects of the problem. In implementing a PM process, decisions must be made about who is involved and what is included (Midgley, 1995). Tradeoffs between narrow technical accuracy and more inclusive participation in the modeling processes themselves may add more legitimacy to the process (Nabavi et al., 2017), or help to “level the playing field” in the case of asymmetries in the power (i.e. influence or control) or knowledge of different stakeholders (e.g. Barnaud and Van Paassen, 2013; Campo et al., 2010). When the choice of modeling methods and tools becomes largely a personal decision of certain more knowledgeable stakeholders, it represents an ethical posture based on their own preferences and experiences and may not reflect the larger PM group. Methods (and tools) ought to be chosen in service to ethical or social needs. In contrast, method-driven PM practice can result in methods that are ‘epistemically violent’ to vulnerable participants; they forcibly replace one structure of beliefs with another. Individuals must be invited to join the process, but it is rarely possible to invite every individual who might be interested in the questions being addressed. Time and resource constraints, as well as the need to have effective and useful interactions among the participants means that some individuals are necessarily excluded. Further, because modeling often requires some element of rules or strategy guiding the approach prior to the decision-making process, certain participants may have greater power.
The choice of methods and tools can significantly empower some participants at the expense of others. Often these others may already be traditionally disenfranchised. If the method chosen is one with which the project leaders have a lot of experience, it might give them substantial advantage in understanding and controlling the process, relative to other participants for whom the method is novel. The confidence and knowledge they have, make them more likely to guide the participatory process while subordinating the novices. But using a method that the practitioner is not familiar with just to maintain equality of power would be also unrealistic and unproductive. Because inequality in power can manifest itself in many ways (Kraus, 2014), it is important for a truly participatory process to have all individuals informed not only about the decisions being made, but also about the decision-making process itself. Ultimately, the research team can be even assembled after stakeholders co-designed the project and select the most effective methods based on the policy indicators and the scale they perceived as most relevant (Smajgl, 2010; Smajgl et al., 2009).
On the positive side, methods can also empower and integrate many perspectives. Any of the methods and tools described in this paper may promote both individual and social learning through the use of the model as a “boundary object,” a representation with a shared meaning that can facilitate exchanges of ideas and worldviews between participants (Schmitt Olabisi et al., 2014; Johnson et al., 2012; Zellner, 2008). A boundary object implies a distance from reality and situations that are sometimes tense and painful. This distancing can allow for discussions on subjects that are conflictual or taboo. By fitting into a social issue, the model, co-designed with the stakeholders, becomes an “object of mediation” (D’Aquino et al. 2002, 2003), promoting conflict resolution and collective decision-making.
When selecting methods for participatory modeling, modelers and facilitators should consider how the methods or tools will provide evidence of learning. For example, a ‘before and after’ systems diagram may reveal shifts in mental models that occur as the result of a PM exercise. Discourse analysis may demonstrate changes in the ways groups conceptualize problems and problem-solve as the result of interaction with the model (Radinsky et al., 2017). Consideration of learning is therefore an integral part of method selection and process design in PM.
The selection of methods is both a critical and a difficult task that ideally requires (1) knowledge of available methods and tools, and (2) careful examination of selection criteria and trade-offs. This paper addresses both of these issues. Section 2 describes a broad array of available methods and tools available to scientists, modelers and stakeholders, and Section 3 systematically examines PM practice and the issue of method and tool selection.
Section snippets
Overview of PM methods
There are numerous methods used in PM projects. In Fig. 1, we propose a typology of methods (and some possible combinations thereof). It is sometimes difficult to distill the particular methods and tools used within the context of broader methodologies proposed for PM. These methodologies tend to cover the whole process and assume a particular type or set of tools embedded within. For example, the Soft Systems Methodology (SSM) (Checkland and Holwell, 1998), and the Companion Modeling (ComMod)
Selecting appropriate methods
As summarized above, there are a large number of methods and tools that can and have been used in PM processes. Yet it is difficult to identify the best strategy for deciding on what methods and tools and/or combinations thereof are most appropriate for a particular PM project. What makes these decisions especially difficult is that, as previously mentioned, there are hardly any reported cases where more than one method has been tried for the same problem within the same project. Combining
Conclusions
There is much improvement yet to be made in how modeling methods are selected for PM projects. There are many methods already available, and choices are not simple. In too many cases, the selection process seems to be largely driven by the past experience of participants, rather than by the particular needs of the project. While logic tells us that this is probably not the best strategy, we do not have much, if any, evidence that this is a bad thing. To a large extent, this is because there are
Acknowledgements
This work was supported by the National Socio-Environmental Synthesis Center (SESYNC) under funding received from the National Science Foundation DBI-1052875. Any use of trade, product, or firm names in this publication is for descriptive purposes only and does not imply endorsement by the U.S. Government. The authors thank Toni Lyn Morelli, Bruce Taggart and two other anonymous reviewers for helpful comments.
References (172)
Agent Based Modelling and Simulation tools: a review of the state-of-art software
Computer Sci. Rev.
(2017)- et al.
A review of scenario planning
Futures
(2013) Modeling human decisions in coupled human and natural systems: review of agent-based models
Ecol. Model.
(2012)- et al.
Collaborative modelling or participatory modelling? A framework for water resources management
Environ. Model. Software
(2017) - et al.
Modeling urban dynamics through GIS-based cellular automata
Comput. Environ. Urban Syst.
(1999) - et al.
An overview of the model integration process: from pre-integration assessment to testing
Environ. Model. Software
(2017) - et al.
Multi-agent simulations and ecosystem management: a review
Ecol. Model.
(2004) - et al.
Modelling with stakeholders within a development project
Environ. Model. Software
(2010) - et al.
Participatory modelling to support decision making in water management under uncertainty: two comparative case studies in the Guadiana river basin, Spain
J. Environ. Manag.
(2013) - et al.
Bayesian Networks and participatory modelling in water resource management
Environ. Model. Software
(2007)
Good practice in Bayesian network modelling
Environ. Model. Software
Cognitive mapping expert views for policy analysis in the public sector
Eur. J. Oper. Res.
Spatial agent-based models for socio-ecological systems: challenges and prospects
Environ. Model. Software
Combining fuzzy cognitive maps with agent-based modeling: frameworks and pitfalls of a powerful hybrid modeling approach to understanding human-environment interactions
Environ. Model. Software
Dynamic adaptive policy pathways: a method for crafting robust decisions for a deeply uncertain world
Global Environ. Change
Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways
Environ. Model. Software
A comparison of multiple criteria analysis techniques for water resource management
Eur. J. Oper. Res.
Towards understanding participatory processes: framework, application and results
J. Environ. Manag.
The analytic hierarchy process in an uncertain environment: a simulation approach
Eur. J. Oper. Res.
Mixing OR methods in practice: past, present and future directions
Eur. J. Oper. Res.
Multi-criteria decision analysis in environmental sciences: ten years of applications and trends
Sci. Total Environ.
The split ladder of participation: a diagnostic, strategic, and evaluation tool to assess when participation is necessary
Environ. Sci. Pol.
Fuzzy Cognitive Maps for future studies -- a methodological assessment of concepts and methods
Futures
Integrated environmental modeling: a vision and roadmap for the future
Environ. Model. Software
What it took to catalyse uptake of dynamic adaptive pathways planning to address climate change uncertainty
Environ. Sci. Pol.
Spatial system dynamics: new approach for simulation of water resources systems
J. Comput. Civ. Eng.
Using participatory system dynamics in environmental and sustainability dialogues
Apprentissage Organisationnel: Théorie, Méthode, Pratique
A ladder of citizenship participation
J. Am. Inst. Plan.
Equity, power games, and legitimacy: dilemmas of participatory natural resource management
Ecol. Soc.
Role-playing games for opening the black box of multi-agent systems: method and lessons of its application to Senegal River Valley irrigated systems
J. Artif. Soc. Soc. Simulat.
Our companion modelling approach
J. Artif. Soc. Soc. Simulat.
Collaborative Modelling. Engaging Stakeholders in Solving Complex Problems of Water Management
Collaborative modelling for informed decision making and inclusive water development
Water Resour. Manag.
Rich pictures: sustainable development and stakeholders–the benefits of content analysis
Sustain. Dev.
Rich pictures: a means to explore the ‘sustainable mind’?
Sustain. Dev.
A theory of spatial system archetypes
Syst. Dynam. Rev.
Cormas: an agent-based simulation platform for coupling human decisions with computerized dynamics
Theorizing benefits and constraints in collaborative environmental governance: a transdisciplinary social-ecological network approach for empirical investigations
Ecol. Soc.
Agent-based modeling: methods and techniques for simulating human systems
Proc. Natl. Acad. Sci. Unit. States Am.
Managing complex issues through evolutionary learning laboratories
Syst. Res. Behav. Sci.
Multi-agent systems and role games: collective learning processes for ecosystem management
Participatory ethics: politics, practices, institutions
ACME: Int. J. Critic. Geograph.
Flint Water Crisis - Michigan 2016
Participatory mapping and geographic information systems: whose map? Who is empowered and who disempowered? Who gains and who loses?
Electron. J. Inf. Syst. Dev. Ctries.
PRA, PLA and pluralism: practice and theory
Information, Systems and Information Systems: Making Sense of the Field
Soft Systems Methodology in Action
Concept mapping, mind mapping and argument mapping: what are the differences and do they matter?
High Educ.
Cited by (269)
Sustainable integration of desalinated seawater into regional water supply networks using a participatory modelling framework
2024, Environmental Science and PolicyAgent-based models of groundwater systems: A review of an emerging approach to simulate the interactions between groundwater and society
2024, Environmental Modelling and SoftwareAdvancing participatory energy systems modelling
2024, Energy Strategy ReviewsNavigating community engagement in participatory modeling of food systems
2024, Environmental Science and PolicyField experiences and lessons learned from applying participatory system dynamics modelling to sustainable water and agri-food systems
2024, Journal of Cleaner Production