Elsevier

Environmental Modelling & Software

Volume 109, November 2018, Pages 232-255
Environmental Modelling & Software

Position Paper
Tools and methods in participatory modeling: Selecting the right tool for the job

https://doi.org/10.1016/j.envsoft.2018.08.028Get rights and content

Highlights

  • Participatory Modeling (PM) is a purposeful learning process for action.

  • We review some of the methods for PM, and identify their strengths and weaknesses.

  • We provide guidance for practitioners as they select methods for their PM projects.

  • A web portal for the Community of Practice on PM can assist in method selection.

Abstract

Various tools and methods are used in participatory modelling, at different stages of the process and for different purposes. The diversity of tools and methods can create challenges for stakeholders and modelers when selecting the ones most appropriate for their projects. We offer a systematic overview, assessment, and categorization of methods to assist modelers and stakeholders with their choices and decisions. Most available literature provides little justification or information on the reasons for the use of particular methods or tools in a given study. In most of the cases, it seems that the prior experience and skills of the modelers had a dominant effect on the selection of the methods used. While we have not found any real evidence of this approach being wrong, we do think that putting more thought into the method selection process and choosing the most appropriate method for the project can produce better results. Based on expert opinion and a survey of modelers engaged in participatory processes, we offer practical guidelines to improve decisions about method selection at different stages of the participatory modeling process.

Introduction

Numerous tools and methods facilitate stakeholder engagement in participatory modeling (PM), which Stave (2010) defined broadly as “… an approach for including a broad group of stakeholders in the process of formal decision analysis.” In the PM process, participants co-formulate a problem and use modeling to describe the problem, to identify, develop and test solutions, and to inform the decision-making and actions of the group. Therefore, we define PM specifically as a purposeful learning process for action that engages the implicit and explicit knowledge of stakeholders to create formalized and shared representations of reality. Since PM is heavily focused on collaborative learning, the tools and methods used during PM projects are expected to promote system understanding and awareness for all stakeholders. By stakeholders we mean all who have a ‘stake’ in the project. This includes modelers and researchers themselves, who are often considered external to the project but still have interests in it, come with their own biases, and cannot be assumed totally objective and neutral (Voinov et al., 2014). The level of engagement differs across stakeholders and varies from one stage of the project to another (Arnstein, 1969; Hurlbert and Gupta, 2015; Reed et al., 2009).

Argyris and Schön (2002) showed that there are two levels of learning, referred to as “single loop” and “double loop” learning. In single loop learning, individuals and groups act within a single reference frame, where specific hypotheses, values, norms, beliefs and objectives are assumed to describe the world. Learning in these systems consists of observing the results of actions and, potentially, modifying future actions based on what is observed. In double loop learning, actors question and learn about the reference frame itself, and may change their fundamental hypotheses, values, norms, and beliefs based on what they learn about the system, as well as what they learn about the outcomes of specific actions (Zellner and Campbell, 2015).

The transition between single and double loop learning can result from the interaction between individual and organizational learning. Argyris and Schön (2002) found complex retrospection and feedback mechanisms between individual and organizational learning. The individual mental models that are used to construct shared mental models of an organization coalesce, thereby modifying the perception of the organization and transforming organizational values and paradigms. In turn, this modifies the environment of the individuals and affects their own mental models (Daré et al., 2014). As a result, the act of model co-creation is, in itself, an act of knowledge construction at both the single and double loop learning levels. In some cases, PM processes deliberately avoid formal model co-creation to first allow the identification and challenging of stakeholders’ causal beliefs and expectations and, consequently, a reconstruction of knowledge (Habermas, 1990; Smajgl and Ward, 2013).

The goal of this paper is to provide an overview of some of the methods and tools for PM, identify some of their strengths and weaknesses, and provide some guidance for practitioners as they select methods for their PM projects. For the purposes of this paper, we define a tool as a modeling technique used to carry out a particular function to achieve a specific goal. Tools are defined, documented, do not change significantly through use, and are clearly external to their users and often not created by them. In contrast, a method is a way of doing something, in particular, a way of using tools. According to Mingers (2000), a method is “a structured set of processes and activities that includes tools, techniques, and models, that can be used in dealing with a problem or problem situation.” A particular method can be supported by one or several tools. For example, in this context agent-based modeling (ABM) is a method; Netlogo, Mason, or RePast are some of the possible open-source tools used to perform ABM. Multiple tools often exist to support a single method, and some tools also serve several methods. For example, Netlogo, AnyLogic™, or Numerus™ are tools that can be used within both ABM and System Dynamics (SD) methods.

While the choice of methods used can heavily impact both processes and decisions, there is little scholarly discussion about how tools and methods are chosen during PM. Certainly, decisions about methods are more influential for the whole process than the choice of a particular tool, and therefore should come before choice of tools. For instance, there are not many implications in deciding to use Stella® rather than Vensim® or Simile; all are well-established tools that support the SD method. But the decision to implement a more quantitative method rather than a qualitative or conceptual one can potentially significantly change the outcome of a PM process. For example, a companion ABM based on role-playing games (see Barreteau et al., 2001) can increase stakeholder involvement in the PM process and may generate much different results than computational ABM using only computer simulations and modelers’ assumptions.

Previously, Voinov et al. (2016) reviewed several participatory tools and methods that have been used to enhance stakeholder participation for different components of the PM process. They concluded that, while many different methods are used for various stages of the process, in practice, there is rarely much justification given for the use of a particular method. It is difficult to find examples of participatory projects that used different combinations of modeling methods when dealing with the same problem. In most cases, once the method (or combination of methods) is chosen, it becomes the only one reported. We recently reviewed 180 papers related to participatory modeling as part of a SESYNC project on “Synergizing public participation and participatory modeling methods for action oriented outcomes” (https://www.sesync.org/project/enhancing-socio-environmental-research-education/participatory-modeling). We found no papers that reported using one method and then a switch to another method. This may be due to a general reluctance to report failures rather than only success stories, but it complicates the comparison of different methods. Another reason most studies report only one method might be that switching from one method to another is costly in terms of time and resources. A similar, though much more limited effort in healthcare research, which focused on comparison of three dynamic simulation methods, SD, ABM and Discrete Event simulation (Marshall et al., 2015) also reports very few failures of particular methods that led to switching to other methods.

A careful and conscious selection of methods is important for the modelling process and its outcomes. Ideally, the selection would be accompanied by effective evaluations to monitor the impact of individual methods used during in PM (Hassenforder et al., 2015; Smajgl and Ward, 2015). However, in many case studies, the choice of methods and tools seems largely driven by the experiences of participating researchers (Prell et al., 2007). This is a manifestation of the hammer and nail’ syndrome: once someone learns to use a hammer, everything starts to look like a nail. A researcher with expertise in system dynamics is very likely to apply system dynamics for the next modeling project, even if other methods could be equally or even more appropriate to address the full set of driving questions. Retraining is time-consuming and resources are always scarce. Engaging colleagues with experience in alternative approaches could help expand the scope of methods considered, but this is not always feasible. There are practical and social reasons why this experience-driven approach to method and tool selection is not optimal, especially in the field of PM. First, the value of PM in developing models that effectively and efficiently meet participants' requirements will be improved by using methods that best fit the project purpose and context. The modeling skill set available should be considered only to identify gaps in the skills required to address the problem in question. PM seeks to be transparent to the users and it is critical to make sure that PM practitioners are not treating all problems as nails just because they are good at using a hammer. Stakeholders, defined broadly as above, are expected to engage in all steps of the PM process, which includes method selection as well as the modeling steps. While the participation of various groups of stakeholders will certainly be different, at each stage all stakeholders should understand why the chosen methods and tools are appropriate. This requires some flexibility in the PM process, whereby stakeholders move collectively from the problem to an appropriate method, and onto tools and associated skills found within the project team. A sharper focus on method and tool selection is needed. This requires understanding stakeholders’ preferences and constraints, including their experience with particular methods, the availability of training for specific methodologies, the ability to use and maintain a particular tool for the long term given the costs to do so, and/or the ability to combine a new tool with existing tools or methods (Smajgl, 2015).

Second, social factors may also affect method and tool selection. The choice of methods is more than a technical decision; it can also involve ethical or other social judgments. It may make it easier or more difficult for specific groups to participate effectively, and to adequately represent specific technical aspects of the problem. In implementing a PM process, decisions must be made about who is involved and what is included (Midgley, 1995). Tradeoffs between narrow technical accuracy and more inclusive participation in the modeling processes themselves may add more legitimacy to the process (Nabavi et al., 2017), or help to “level the playing field” in the case of asymmetries in the power (i.e. influence or control) or knowledge of different stakeholders (e.g. Barnaud and Van Paassen, 2013; Campo et al., 2010). When the choice of modeling methods and tools becomes largely a personal decision of certain more knowledgeable stakeholders, it represents an ethical posture based on their own preferences and experiences and may not reflect the larger PM group. Methods (and tools) ought to be chosen in service to ethical or social needs. In contrast, method-driven PM practice can result in methods that are ‘epistemically violent’ to vulnerable participants; they forcibly replace one structure of beliefs with another. Individuals must be invited to join the process, but it is rarely possible to invite every individual who might be interested in the questions being addressed. Time and resource constraints, as well as the need to have effective and useful interactions among the participants means that some individuals are necessarily excluded. Further, because modeling often requires some element of rules or strategy guiding the approach prior to the decision-making process, certain participants may have greater power.

The choice of methods and tools can significantly empower some participants at the expense of others. Often these others may already be traditionally disenfranchised. If the method chosen is one with which the project leaders have a lot of experience, it might give them substantial advantage in understanding and controlling the process, relative to other participants for whom the method is novel. The confidence and knowledge they have, make them more likely to guide the participatory process while subordinating the novices. But using a method that the practitioner is not familiar with just to maintain equality of power would be also unrealistic and unproductive. Because inequality in power can manifest itself in many ways (Kraus, 2014), it is important for a truly participatory process to have all individuals informed not only about the decisions being made, but also about the decision-making process itself. Ultimately, the research team can be even assembled after stakeholders co-designed the project and select the most effective methods based on the policy indicators and the scale they perceived as most relevant (Smajgl, 2010; Smajgl et al., 2009).

On the positive side, methods can also empower and integrate many perspectives. Any of the methods and tools described in this paper may promote both individual and social learning through the use of the model as a “boundary object,” a representation with a shared meaning that can facilitate exchanges of ideas and worldviews between participants (Schmitt Olabisi et al., 2014; Johnson et al., 2012; Zellner, 2008). A boundary object implies a distance from reality and situations that are sometimes tense and painful. This distancing can allow for discussions on subjects that are conflictual or taboo. By fitting into a social issue, the model, co-designed with the stakeholders, becomes an “object of mediation” (D’Aquino et al. 2002, 2003), promoting conflict resolution and collective decision-making.

When selecting methods for participatory modeling, modelers and facilitators should consider how the methods or tools will provide evidence of learning. For example, a ‘before and after’ systems diagram may reveal shifts in mental models that occur as the result of a PM exercise. Discourse analysis may demonstrate changes in the ways groups conceptualize problems and problem-solve as the result of interaction with the model (Radinsky et al., 2017). Consideration of learning is therefore an integral part of method selection and process design in PM.

The selection of methods is both a critical and a difficult task that ideally requires (1) knowledge of available methods and tools, and (2) careful examination of selection criteria and trade-offs. This paper addresses both of these issues. Section 2 describes a broad array of available methods and tools available to scientists, modelers and stakeholders, and Section 3 systematically examines PM practice and the issue of method and tool selection.

Section snippets

Overview of PM methods

There are numerous methods used in PM projects. In Fig. 1, we propose a typology of methods (and some possible combinations thereof). It is sometimes difficult to distill the particular methods and tools used within the context of broader methodologies proposed for PM. These methodologies tend to cover the whole process and assume a particular type or set of tools embedded within. For example, the Soft Systems Methodology (SSM) (Checkland and Holwell, 1998), and the Companion Modeling (ComMod)

Selecting appropriate methods

As summarized above, there are a large number of methods and tools that can and have been used in PM processes. Yet it is difficult to identify the best strategy for deciding on what methods and tools and/or combinations thereof are most appropriate for a particular PM project. What makes these decisions especially difficult is that, as previously mentioned, there are hardly any reported cases where more than one method has been tried for the same problem within the same project. Combining

Conclusions

There is much improvement yet to be made in how modeling methods are selected for PM projects. There are many methods already available, and choices are not simple. In too many cases, the selection process seems to be largely driven by the past experience of participants, rather than by the particular needs of the project. While logic tells us that this is probably not the best strategy, we do not have much, if any, evidence that this is a bad thing. To a large extent, this is because there are

Acknowledgements

This work was supported by the National Socio-Environmental Synthesis Center (SESYNC) under funding received from the National Science Foundation DBI-1052875. Any use of trade, product, or firm names in this publication is for descriptive purposes only and does not imply endorsement by the U.S. Government. The authors thank Toni Lyn Morelli, Bruce Taggart and two other anonymous reviewers for helpful comments.

References (172)

  • S.H. Chen et al.

    Good practice in Bayesian network modelling

    Environ. Model. Software

    (2012)
  • C. Eden et al.

    Cognitive mapping expert views for policy analysis in the public sector

    Eur. J. Oper. Res.

    (2004)
  • T. Filatova et al.

    Spatial agent-based models for socio-ecological systems: challenges and prospects

    Environ. Model. Software

    (2013)
  • P.J. Giabbanelli et al.

    Combining fuzzy cognitive maps with agent-based modeling: frameworks and pitfalls of a powerful hybrid modeling approach to understanding human-environment interactions

    Environ. Model. Software

    (2017)
  • M. Haasnoot et al.

    Dynamic adaptive policy pathways: a method for crafting robust decisions for a deeply uncertain world

    Global Environ. Change

    (2013)
  • M. Haasnoot et al.

    Fit for purpose? Building and evaluating a fast, integrated model for exploring water policy pathways

    Environ. Model. Software

    (2014)
  • S. Hajkowicz et al.

    A comparison of multiple criteria analysis techniques for water resource management

    Eur. J. Oper. Res.

    (2008)
  • E. Hassenforder et al.

    Towards understanding participatory processes: framework, application and results

    J. Environ. Manag.

    (2015)
  • D. Hauser et al.

    The analytic hierarchy process in an uncertain environment: a simulation approach

    Eur. J. Oper. Res.

    (1996)
  • S. Howick et al.

    Mixing OR methods in practice: past, present and future directions

    Eur. J. Oper. Res.

    (2011)
  • I.B. Huang et al.

    Multi-criteria decision analysis in environmental sciences: ten years of applications and trends

    Sci. Total Environ.

    (2011)
  • M. Hurlbert et al.

    The split ladder of participation: a diagnostic, strategic, and evaluation tool to assess when participation is necessary

    Environ. Sci. Pol.

    (2015)
  • A.J. Jetter et al.

    Fuzzy Cognitive Maps for future studies -- a methodological assessment of concepts and methods

    Futures

    (2014)
  • G.F. Laniak

    Integrated environmental modeling: a vision and roadmap for the future

    Environ. Model. Software

    (2013)
  • J. Lawrence et al.

    What it took to catalyse uptake of dynamic adaptive pathways planning to address climate change uncertainty

    Environ. Sci. Pol.

    (2017)
  • S. Ahmad et al.

    Spatial system dynamics: new approach for simulation of water resources systems

    J. Comput. Civ. Eng.

    (2004)
  • P. Antunes et al.

    Using participatory system dynamics in environmental and sustainability dialogues

  • C. Argyris et al.

    Apprentissage Organisationnel: Théorie, Méthode, Pratique

    (2002)
  • S.R. Arnstein

    A ladder of citizenship participation

    J. Am. Inst. Plan.

    (1969)
  • C. Barnaud et al.

    Equity, power games, and legitimacy: dilemmas of participatory natural resource management

    Ecol. Soc.

    (2013)
  • O. Barreteau et al.

    Role-playing games for opening the black box of multi-agent systems: method and lessons of its application to Senegal River Valley irrigated systems

    J. Artif. Soc. Soc. Simulat.

    (2001)
  • O. Barreteau et al.

    Our companion modelling approach

    J. Artif. Soc. Soc. Simulat.

    (2003)
  • L. Basco-Carrera et al.

    Collaborative Modelling. Engaging Stakeholders in Solving Complex Problems of Water Management

    (2017)
  • L. Basco-Carrera et al.

    Collaborative modelling for informed decision making and inclusive water development

    Water Resour. Manag.

    (2017)
  • S. Bell et al.

    Rich pictures: sustainable development and stakeholders–the benefits of content analysis

    Sustain. Dev.

    (2015)
  • S. Bell et al.

    Rich pictures: a means to explore the ‘sustainable mind’?

    Sustain. Dev.

    (2013)
  • T.K. BenDor et al.

    A theory of spatial system archetypes

    Syst. Dynam. Rev.

    (2012)
  • P. Bommel et al.

    Cormas: an agent-based simulation platform for coupling human decisions with computerized dynamics

  • Ö. Bodin et al.

    Theorizing benefits and constraints in collaborative environmental governance: a transdisciplinary social-ecological network approach for empirical investigations

    Ecol. Soc.

    (2016)
  • E. Bonabeau

    Agent-based modeling: methods and techniques for simulating human systems

    Proc. Natl. Acad. Sci. Unit. States Am.

    (2002)
  • O.J. Bosch et al.

    Managing complex issues through evolutionary learning laboratories

    Syst. Res. Behav. Sci.

    (2013)
  • F. Bousquet et al.

    Multi-agent systems and role games: collective learning processes for ecosystem management

  • C. Cahill et al.

    Participatory ethics: politics, practices, institutions

    ACME: Int. J. Critic. Geograph.

    (2007)
  • Center for Disease Control and Prevention

    Flint Water Crisis - Michigan 2016

    (2016)
  • R. Chambers

    Participatory mapping and geographic information systems: whose map? Who is empowered and who disempowered? Who gains and who loses?

    Electron. J. Inf. Syst. Dev. Ctries.

    (2006)
  • R. Chambers

    PRA, PLA and pluralism: practice and theory

  • P. Checkland et al.

    Information, Systems and Information Systems: Making Sense of the Field

    (1998)
  • P. Checkland

    Soft Systems Methodology in Action

    (1999)
  • M. Davies

    Concept mapping, mind mapping and argument mapping: what are the differences and do they matter?

    High Educ.

    (2011)
  • Cited by (269)

    View all citing articles on Scopus
    View full text