Ecological risk assessments to guide decision-making: Methodology matters

Highlights

• Risk assessment (RA) methodology may substantially affect the outcome of the RA.

• Risk scores that represent the content of the risk categories perform better.

• The number and linkages between the DPS risk components determine RA sensitivity.

• Risk of combined components reflects that of the worst-case single component.

• Choice of appropriate methodology depends on the context in which the RA is applied.

Abstract

Ecological risk assessment is often applied to guide the decision-making process that underpins ecosystem-based management and prioritisation of risk factors for management. Several studies have recently used ecological risk assessment approaches to identify risk factors of greatest concern, but rarely are the underlying methodological decisions discussed in terms of the effect that those decisions have on the outcome of the assessment and ultimately, how that affects prioritisation of risk factors for management.

This study therefore evaluates the effect of methodological decisions involving (1) the choice and definition of risk factors, and (2) the calculation of risk scores, providing, where possible, recommendations on what should be the most appropriate methodologies.

The definition of risk factors is often determined by the policy context and could result in the comparison of one broadly defined risk meta-factor (e.g. Food Production) with corresponding specific risk factors defined more narrowly (i.e. Oil and Gas production or Offshore Wind). Depending on the method to calculate risk this may result in a systematic bias prioritising any risk meta-factor. For the calculation of individual impact chain risk scores we compared weighted scores with ordinal scores, where the former allows more flexibility to represent the qualitative categories that determine risk and provided results better supported by scientific evidence. A consideration of different risk assessment applications in EBM showed there is no one-size-fits-all solution to this as these methodological decisions need to be considered in concert and the preferred methodology may depend on the context in which the risk assessment is applied. The outcome of the risk assessment should always be accompanied by an explicit consideration of these methodological issues and description of the resulting methodological choices.

Introduction

Large areas of marine ecosystems are currently impacted by human activities and many resources are exploited at an unsustainable rate, (e.g. Glover and Smith, 2003, Halpern et al., 2008). As such, sustainable use has become the central paradigm of many recent environmental policies as well as renewed efforts to identify, manage and limit the impact of human activities (e.g. Halpern and Fujita, 2013, Piet et al., 2015, Knights et al., 2013), but the number of threats and constraints on resources can restrict management to a limited number of options and not necessarily those posing the greatest threat to natural systems (Gibbs and Browman, 2015). Decision-support tools are continually being developed (e.g. Jeffrey, 1983, Jeffrey, 1992, Piet et al., 2015, Resnik, 1987, Samhouri and Levin, 2012) to support effective decision-making in light of those constraints. Ecological (or environmental) risk assessment (ERA) is an approach that provides a flexible, problem-solving solution capable of linking the relationship between human activities and the environment, thereby supporting the decision-making needs of environmental managers (Hope, 2006).

Risk assessment per se covers a broad array of approaches for a wide set of applications (see reviews by Holdgate, 1979, Evans, 2004, Fryer et al., 2006); here we focus on the approaches most suited to ecological risk assessments (e.g. Astles et al., 2006, Campbell and Gallagher, 2007, Fletcher, 2005). In general terms, ERA describes the likelihood and consequences of an event and can be used to evaluate the degree to which human activities interfere with the achievement of management objectives (Samhouri and Levin, 2012). In this context, risk can be assessed using quantitative (e.g. Francis and Shotton, 1997, Samhouri and Levin, 2012) or qualitative approaches (e.g. Fletcher, 2005, Breen et al., 2012, Fletcher et al., 2010). Traditionally in ERA the likelihood-consequence approach was used for estimating the risk of a rare or unpredictable event (Williams et al., 2011), but when an assessment of the risks associated with on-going (current) pressure is needed, an exposure-effect analysis is more suitable (Smith et al., 2007). Such approaches have been used to consider the potential for ecosystem-based management (EBM) at sub-regional (Samhouri and Levin, 2012) or regional scales (Piet et al., 2015) by allowing decision makers to explore how different management options could reduce threat to their ecosystem policy objectives across a wide range of risk factors. Environmental risk assessment concepts have also been used to provide a clear structure for cumulative effects assessment for which, according to (Judd et al., 2015), no consistency or standardisation in approaches exists. Risk assessment is therefore playing an increasingly important role in integrating science, policy and management (CENR, 1999).

Any assessment of risk caused by human activities on an ecosystem will be dependent on (1) a correct description of the functioning ecosystem and how this is impacted by those activities, together with (2) an appropriate methodology to translate the impact into risk. Significant progress has been made toward linking human activities to ecosystem impact with the definition and evaluation of the array of sector-pressure-state combinations or “impact chains”, although the resulting network of interactions can be complex (Knights et al., 2013, Tamis et al., 2016; see illustration of impact chains in Fig. 1). Applying a productivity-susceptibility analysis (e.g. Hobday et al., 2011, Samhouri and Levin, 2012, Stobutzki et al., 2001) or an exposure-effect evaluation on an interaction network can enable risk to the ecosystem from a single or combination of (anthropogenic) impacts, to be determined (e.g. Halpern et al., 2007, Knights et al., 2015, Milton, 2001, Stobutzki et al., 2001). If risk represents the cumulative effects of different human activities impacting on multiple ecosystem components through different pressures, then individual impact chains need to be combined into an overall measure of risk such that those risk factors, e.g. sector(s), pressure(s) and ecosystem component(s), introducing the greatest level of risk can be identified (Tamis et al., 2016).

As described above, ERA provides a powerful approach for comparison of the effects of different anthropogenic drivers acting on ecosystems (Gibbs and Browman, 2015), but there are many methodological issues to consider in the design of an ERA (Tamis et al., 2016) and we suggest that there has been little discussion in the academic literature of how the decisions made on methodological design affect the outcomes and the advice that is based on this. For example, the method of combining assessment criteria and/or impact chains can vary between studies, in some instances, calculated as the sum of the impact chain scores (e.g. Fock, 2011, Halpern et al., 2008, HELCOM, 2010, Korpinen et al., 2012, Samhouri and Levin, 2012, Stelzenmuller et al., 2010), in others, by the average of them (e.g. Knights et al., 2015, Samhouri and Levin, 2012). The values assigned to each assessment category can also vary widely (Tamis et al., 2016). Methodological decisions are clearly made when designing policy-relevant ecosystem assessments, but in the ecological risk assessment approaches outlined to date, there is often a limited description of how these decisions were reached. For example, when choosing the scoring and summation approach, no consideration is given to how this might affect the prioritisation of threats to marine ecosystems and their management. We argue that this exploration of methodological decisions must be openly undertaken and the implications for prioritisation of management explored so that informed decisions can be made about the design of risk assessment to best fit the context in which it is applied.

Here we explore how the methods used to score individual impact chains and to aggregate impact risk over these chains can affect ERA outcomes in terms of the prioritisation of threats. We do this by taking an existing risk assessment approach (Knights et al., 2015), and examine how the outcome of the risk assessment in terms of the rank order of risk factors is altered by (1) changes in the way individual impact chains (within a risk factor) are scored, (2) the method by which multiple chains are then aggregated for an overall risk factor score, and (3) the number of impact chains included, which is often determined by (4) the choice and definition of those risk factors. We also use a case study to discuss the findings in the context of two different cumulative effects assessment applications (following Judd et al., 2015), focusing on how differences in approach methodology can affect: (i) the identification of the most threatening impact chains (see Knights et al., 2015 and analogous to the identification of a ‘hazard’), and (ii) the evaluation of the performance of management measures applied to reduce the risk from specific impact chains (described in full in Piet et al., 2015).