Elsevier

Information Sciences

Volume 375, 1 January 2017, Pages 79-97
Information Sciences

Multimodal optimization: An effective framework for model calibration

https://doi.org/10.1016/j.ins.2016.09.048Get rights and content

Highlights

  • We propose a framework for model calibration based on niching genetic algorithms.

  • It consists of interactive multimodal search, visualization, and parameters analysis.

  • The output of the framework is a set of alternative model calibration solutions.

  • The multimodal framework is applied to two different applications.

  • Experiments showed the framework benefits when interactively validating models.

Abstract

Automated calibration is a crucial stage when validating non-linear dynamic systems. The modeler must control the calibration results and analyze parameter values in an iterative way. In many non-linear models, it is usual to find sets of configuration parameters that may obtain the same model fitting. In these cases, the modeler needs to understand the results’ implications and run a sensitivity analysis to check the model validity. This paper presents a framework based on niching genetic algorithms to provide modeler with a set of alternative calibration solutions which also ease the analysis of their parameters, model’s response, and sensitivity analysis. The framework is called MOMCA, an integral and interactive solution for model validation which facilitates the implication of decision makers. The core component of MOMCA is its niching genetic algorithm, able to reach various optima in multimodal optimization problems by keeping the necessary diversity. The proposed framework is applied to two different case studies. The first case study is a biological growth model and the second one is a managerial model to improve brand equity. Both applications show the benefits of the framework when providing a set of calibrated models and a way to analyze and perform sensitivity analysis based on the set of solutions.

Introduction

Non-linear dynamic models are widely used as they characterize real-world systems and the key relationships between their elements. These models are particularly suitable for systems with a high number of interrelated variables, where all relevant data to build the system is not always available or precise. They also provide a way to carry out simulations, understand the effects of alternative strategies, and assist stakeholders in better planning and management [67]. An example of these models is system dynamics which presents methodologies and tools for developing mathematical models of complex systems for social, biological, and economic problems [34], [51], [62], [68].

A decisive phase when modeling non-linear dynamic systems is model validation [67]. The validation requires testing a set of hypothesis, the significance of their behavioral components (by assuming that the behavior is a consequence of the system structure), and the historical model fitting [49]. Validation is also measured in terms of degrees of confidence or quality and this quality is usually difficult to obtain for most non-linear simulation models in use [25]. The search for better validation procedures and methodologies is still necessary to ensure an appropriate level of confidence in the models’ performance [6].

Automated calibration, mainly based on gradient search methods and genetic algorithms (GAs) [3], is a useful tool for model validation, but its results must be analyzed with caution [49]. The modeler needs to use automated calibration methods judiciously and in iterative and controlled way in order to manually filter the different alternatives. Otherwise, if modelers blindly accept the calibrated parameters without studying them, these values will be forced to match the historical behavior, with the subsequent risk of treating the model as a black box [58].

The presence of a multimodal nature of parameters is an another problem while calibrating the model [46]. The existence of several sub-optimal solutions in a multimodal search space [28] causes difficulties to find a unique solution for the parameters. This is also known as “system identifiability” [4]. Modelers also need to study the parameters and outputs of the model as non-linear simulation models cannot be properly understood without exploring their behaviors under different parameter settings [37]. Input/output exploration, sensitivity analysis, and parameters’ distribution visualization are the most valuable validation techniques to help to understand the model’s behavior [37], [49].

In this contribution, we propose a novel calibration framework based on handling the parameter space multimodality to help and support the modeler in an integral model validation process. Our multimodal calibration framework, called multimodal optimization for model calibration (MOMCA), can obtain a set of different and acceptable calibration solutions for the same model in a single run. The framework generates different parameter configurations which show the same or a similar model behavior. This archive of valid calibration solutions are used by MOMCA to perform automatic parameter analysis and run sensitivity analysis to provide additional indications on the model validity [59].

The use of niching genetic algorithms (NGAs) for the optimization process [28], [54], [61] is a key strength of the presented framework. These methods allow the framework to obtain multiple alternatives (calibration solutions) in a single run and to enhance the exploration of possible combinations of parameters [71]. The majority of the existing NGA-related studies tried to find a single best solution without getting stuck in local optima and the assessment is made in terms of the number of found optima from the known set of solutions [61]. However, our MOMCA approach takes advantage of NGAs not only to improve the search performance of standard GAs but to also offer the modeler a set of equally preferable calibration solutions in terms of fitness (model fitting).

Up to our knowledge, this work represents the first NGA application to model calibration. Additionally, the framework extends current state-of-the-art solutions by incorporating interaction methods that achieve the primary MOMCA objectives: to serve as an integral framework for a whole model validation process [49], [56] and facilitate the implication of decision makers and stakeholders [6], [67]. The proposed MOMCA framework is decomposed in three main stages: 1) an optimization algorithm based on NGAs, 2) an evaluation and interactive filtering process on the set of calibration solutions, and 3) an assisted sensitivity analysis and parameters’ study based on quantitative and visualization tools for the same set of calibration solutions.

After presenting the framework we apply it to two non-linear models for different real-world case studies, disparate both in the application field and the used modeling methodology. In the first case, MOMCA is applied to a biological model based on the dynamic energy budget (DEB) theory [36], [41]. DEB interrelates several physiological processes of individual organisms such as ingestion, assimilation, respiration, growth, and reproduction to simulate non-trivial biological processes. The calibration of a DEB model is complex because the observations of some parameters of the models are not directly measurable [24], [57]. We calibrate a growth prediction model for blue mussels using empirical data from a Norwegian bay area by also analyzing the parameters and DEB model response.

The second case study uses system dynamics for modeling a brand value management problem [1]. The model simulates the evolution of the brand equity of a television show, the Indian television show Who wants to be a millionaire? [45]. In this second case, modelers are assisted through MOMCA to estimate the parameters of the effects between the branding variables of the case and to understand the dependencies between those parameters that better fit the interest level of the television show. Our experimentation also shows the generic nature of the MOMCA framework which allows its application to any non-linear simulation model.

The rest of the paper is structured as follows. Section 2 extends this work motivation as well as the benefits of the framework to tackle existing challenges in model calibration. Then, Section 3 explains the MOMCA framework and its components. The computational setup is established in Section 4 while Sections 5 and 6 describe both case studies with more details and report the results analysis and modeling implications. Finally, we present our concluding remarks in Section 7.

Section snippets

Open problems and motivation for the MOMCA framework

In this section, we review related literature, explore the main existing problems in non-linear models validation, and highlight how our proposal can help to face them. A summary of these open problems and MOMCA benefits is shown in the diagram in Fig. 1.

Description of the MOMCA framework

In this section we describe the proposed framework and its main three stages (3.1 Searching for a set of multiple calibration solutions by niching genetic algorithms, 3.2 Evaluating calibration performance and filtering of the solution set, and 3.3). Fig. 2 shows a diagram which summarizes the MOMCA framework, their inputs and outputs as well as its iterative validation loop.

Experimental setup

In the experimentation of the paper we compare our MOMCA framework based on NGAs with two traditional calibration algorithms. The first one is the Nelder–Mead simplex [47], [50]. The second one is a standard generational GA [3]. The Nelder–Mead simplex is a classical optimizer previously used in different model calibration experiments [27] due to its ability to work under low restrictive conditions (e.g., it does not require the objective function to be differentiable). Also, it was used by

Model description

The growth of bivalve species is widely studied due to their role in aquaculture and other ecosystem services such as water filtration. The need to make growth predictions encourages the creation of modeling tools to carry out prospective studies, and has promoted the development of individual bivalve growth models. Dynamic energy budget (DEB) is a non-linear individual-based modeling and one of the main approaches applied to model bivalve growth [35], [57].

DEB theory [35] describes the

Model description

A concrete case of strategic management is brand value management [1]. Brand value management is a complex, adaptive, and dynamic environment. This environment is usually a system with a high number of variables and contains non-linearities, inertia, delays, and bi-directional network feedback loops. System dynamics is an ideal methodology for complex feedback systems like brand management where all brand related components are treated as resources, which grow or erode over time [45]. As

Concluding remarks

In this work we have presented MOMCA, a novel framework for applying multimodal optimization to the calibration of non-linear models. The framework is based on NGAs and has three different stages: the search of a set of calibration solutions, an interactive filtering process, and an automated analysis of the set of solutions for a model sensitivity and parameters’ understanding. The main goal of the presented MOMCA framework is to assist modelers with an integral set of methods for model

Acknowledgments

This work is supported by Spanish Ministerio de Economía y Competitividad under the NEWSOCO project (ref. TIN2015-67661), including European Regional Development Funds (ERDF), statutory activities of the Faculty of Computer Science and Management of Wroclaw University of Technology and by the European Commission under the 7th Framework Programme, Coordination and Support Action, Grant Agreement Number 316097, Engine project. Authors also wish to thank Dr. Ramón Filgueira for his suggestions,

References (72)

  • D.E. Goldberg, J. Richardson, Genetic algorithms with sharing for multimodal function optimization, Hillsdale, NJ:...
  • C. Hamarat et al.

    An exploratory approach for adaptive policy-making by using multi-objective robust optimization

    Simul. Model. Pract. Theory

    (2014)
  • A.J. Heppenstall et al.

    Genetic algorithm optimisation of an agent-based model for simulating a retail market

    Environ. Plann. B

    (2007)
  • S.A.L.M. Kooijman

    Energy budgets can explain body size relations

    J. Theor. Biol.

    (1986)
  • S.A.L.M. Kooijman

    Dynamic Energy and Mass Budgets in Biological Systems

    (2000)
  • C.S. de Magalhães et al.

    A dynamic niching genetic algorithm strategy for docking highly flexible ligands

    Inf. Sci.

    (2014)
  • J. Van der Meer

    An introduction to dynamic energy budget (DEB) models with special emphasis on parameter estimation

    J. Sea Res.

    (2006)
  • J.H. Miller

    Active nonlinear tests (ANTs) of complex simulation models

    Manage. Sci.

    (1998)
  • A. Mukherjee et al.

    A system dynamic model of management of a television game show

    J. Model. Manage.

    (2006)
  • J. Nelder et al.

    A simplex method for function minimization

    Comput. J.

    (1965)
  • R. O’Neill

    Algorithm AS 47: function minimization using a simplex procedure

    Appl. Stat.

    (1971)
  • E. Pérez et al.

    Finding multiple solutions in job shop scheduling by niching genetic algorithms

    J. Intell. Manuf.

    (2003)
  • M. Saleh et al.

    A comprehensive analytical approach for policy analysis of system dynamics models

    Eur. J. Oper. Res.

    (2010)
  • A. Saltelli et al.

    Global Sensitivity Analysis: The Primer

    (2008)
  • S. Wei et al.

    System dynamics simulation model for assessing socio-economic impacts of different levels of environmental flow allocation in the weihe river basin, china

    Eur. J. Oper. Res.

    (2012)
  • A. Will et al.

    On the use of niching genetic algorithms for variable selection in solar radiation estimation

    Renewable Energy

    (2013)
  • E. Yu et al.

    Ensemble of niching algorithms

    Inf. Sci.

    (2010)
  • G. Yücel et al.

    Automated parameter specification in dynamic feedback models based on behavior pattern features

    Syst. Dyn. Rev.

    (2011)
  • D.A. Aaker

    Measuring brand equity across products and markets

    California Manage. Rev.

    (1996)
  • J.J. Aguilera et al.

    Niching genetic feature selection algorithms applied to the design of fuzzy rule based classification systems

    IEEE International Conference on Fuzzy Systems (FUZZ-IEEE’07), London (UK)

    (2007)
  • T. Back et al.

    Handbook of Evolutionary Computation

    (1997)
  • J. Banga et al.

    Parameter estimation and optimal experimental design

    Essays Biochem.

    (2008)
  • J. Barranquero et al.

    Detecting key variables in system dynamics modelling by using social network metrics

  • E. Bonabeau

    Predicting the unpredictable

    Harv. Bus. Rev.

    (2002)
  • B. Calvez et al.

    Automatic tuning of agent-based models using genetic algorithms

    Multi-Agent-Based Simulation VI

    (2005)
  • T. Chai et al.

    Root mean square error (RMSE) or mean absolute error (MAE)?–arguments against avoiding RMSE in the literature

    Geosci. Model Dev.

    (2014)
  • Cited by (32)

    • Coral reefs optimization algorithms for agent-based model calibration

      2021, Engineering Applications of Artificial Intelligence
    View all citing articles on Scopus
    View full text