Multimodal optimization: An effective framework for model calibration
Introduction
Non-linear dynamic models are widely used as they characterize real-world systems and the key relationships between their elements. These models are particularly suitable for systems with a high number of interrelated variables, where all relevant data to build the system is not always available or precise. They also provide a way to carry out simulations, understand the effects of alternative strategies, and assist stakeholders in better planning and management [67]. An example of these models is system dynamics which presents methodologies and tools for developing mathematical models of complex systems for social, biological, and economic problems [34], [51], [62], [68].
A decisive phase when modeling non-linear dynamic systems is model validation [67]. The validation requires testing a set of hypothesis, the significance of their behavioral components (by assuming that the behavior is a consequence of the system structure), and the historical model fitting [49]. Validation is also measured in terms of degrees of confidence or quality and this quality is usually difficult to obtain for most non-linear simulation models in use [25]. The search for better validation procedures and methodologies is still necessary to ensure an appropriate level of confidence in the models’ performance [6].
Automated calibration, mainly based on gradient search methods and genetic algorithms (GAs) [3], is a useful tool for model validation, but its results must be analyzed with caution [49]. The modeler needs to use automated calibration methods judiciously and in iterative and controlled way in order to manually filter the different alternatives. Otherwise, if modelers blindly accept the calibrated parameters without studying them, these values will be forced to match the historical behavior, with the subsequent risk of treating the model as a black box [58].
The presence of a multimodal nature of parameters is an another problem while calibrating the model [46]. The existence of several sub-optimal solutions in a multimodal search space [28] causes difficulties to find a unique solution for the parameters. This is also known as “system identifiability” [4]. Modelers also need to study the parameters and outputs of the model as non-linear simulation models cannot be properly understood without exploring their behaviors under different parameter settings [37]. Input/output exploration, sensitivity analysis, and parameters’ distribution visualization are the most valuable validation techniques to help to understand the model’s behavior [37], [49].
In this contribution, we propose a novel calibration framework based on handling the parameter space multimodality to help and support the modeler in an integral model validation process. Our multimodal calibration framework, called multimodal optimization for model calibration (MOMCA), can obtain a set of different and acceptable calibration solutions for the same model in a single run. The framework generates different parameter configurations which show the same or a similar model behavior. This archive of valid calibration solutions are used by MOMCA to perform automatic parameter analysis and run sensitivity analysis to provide additional indications on the model validity [59].
The use of niching genetic algorithms (NGAs) for the optimization process [28], [54], [61] is a key strength of the presented framework. These methods allow the framework to obtain multiple alternatives (calibration solutions) in a single run and to enhance the exploration of possible combinations of parameters [71]. The majority of the existing NGA-related studies tried to find a single best solution without getting stuck in local optima and the assessment is made in terms of the number of found optima from the known set of solutions [61]. However, our MOMCA approach takes advantage of NGAs not only to improve the search performance of standard GAs but to also offer the modeler a set of equally preferable calibration solutions in terms of fitness (model fitting).
Up to our knowledge, this work represents the first NGA application to model calibration. Additionally, the framework extends current state-of-the-art solutions by incorporating interaction methods that achieve the primary MOMCA objectives: to serve as an integral framework for a whole model validation process [49], [56] and facilitate the implication of decision makers and stakeholders [6], [67]. The proposed MOMCA framework is decomposed in three main stages: 1) an optimization algorithm based on NGAs, 2) an evaluation and interactive filtering process on the set of calibration solutions, and 3) an assisted sensitivity analysis and parameters’ study based on quantitative and visualization tools for the same set of calibration solutions.
After presenting the framework we apply it to two non-linear models for different real-world case studies, disparate both in the application field and the used modeling methodology. In the first case, MOMCA is applied to a biological model based on the dynamic energy budget (DEB) theory [36], [41]. DEB interrelates several physiological processes of individual organisms such as ingestion, assimilation, respiration, growth, and reproduction to simulate non-trivial biological processes. The calibration of a DEB model is complex because the observations of some parameters of the models are not directly measurable [24], [57]. We calibrate a growth prediction model for blue mussels using empirical data from a Norwegian bay area by also analyzing the parameters and DEB model response.
The second case study uses system dynamics for modeling a brand value management problem [1]. The model simulates the evolution of the brand equity of a television show, the Indian television show Who wants to be a millionaire? [45]. In this second case, modelers are assisted through MOMCA to estimate the parameters of the effects between the branding variables of the case and to understand the dependencies between those parameters that better fit the interest level of the television show. Our experimentation also shows the generic nature of the MOMCA framework which allows its application to any non-linear simulation model.
The rest of the paper is structured as follows. Section 2 extends this work motivation as well as the benefits of the framework to tackle existing challenges in model calibration. Then, Section 3 explains the MOMCA framework and its components. The computational setup is established in Section 4 while Sections 5 and 6 describe both case studies with more details and report the results analysis and modeling implications. Finally, we present our concluding remarks in Section 7.
Section snippets
Open problems and motivation for the MOMCA framework
In this section, we review related literature, explore the main existing problems in non-linear models validation, and highlight how our proposal can help to face them. A summary of these open problems and MOMCA benefits is shown in the diagram in Fig. 1.
Description of the MOMCA framework
In this section we describe the proposed framework and its main three stages (3.1 Searching for a set of multiple calibration solutions by niching genetic algorithms, 3.2 Evaluating calibration performance and filtering of the solution set, and 3.3). Fig. 2 shows a diagram which summarizes the MOMCA framework, their inputs and outputs as well as its iterative validation loop.
Experimental setup
In the experimentation of the paper we compare our MOMCA framework based on NGAs with two traditional calibration algorithms. The first one is the Nelder–Mead simplex [47], [50]. The second one is a standard generational GA [3]. The Nelder–Mead simplex is a classical optimizer previously used in different model calibration experiments [27] due to its ability to work under low restrictive conditions (e.g., it does not require the objective function to be differentiable). Also, it was used by
Model description
The growth of bivalve species is widely studied due to their role in aquaculture and other ecosystem services such as water filtration. The need to make growth predictions encourages the creation of modeling tools to carry out prospective studies, and has promoted the development of individual bivalve growth models. Dynamic energy budget (DEB) is a non-linear individual-based modeling and one of the main approaches applied to model bivalve growth [35], [57].
DEB theory [35] describes the
Model description
A concrete case of strategic management is brand value management [1]. Brand value management is a complex, adaptive, and dynamic environment. This environment is usually a system with a high number of variables and contains non-linearities, inertia, delays, and bi-directional network feedback loops. System dynamics is an ideal methodology for complex feedback systems like brand management where all brand related components are treated as resources, which grow or erode over time [45]. As
Concluding remarks
In this work we have presented MOMCA, a novel framework for applying multimodal optimization to the calibration of non-linear models. The framework is based on NGAs and has three different stages: the search of a set of calibration solutions, an interactive filtering process, and an automated analysis of the set of solutions for a model sensitivity and parameters’ understanding. The main goal of the presented MOMCA framework is to assist modelers with an integral set of methods for model
Acknowledgments
This work is supported by Spanish Ministerio de Economía y Competitividad under the NEWSOCO project (ref. TIN2015-67661), including European Regional Development Funds (ERDF), statutory activities of the Faculty of Computer Science and Management of Wroclaw University of Technology and by the European Commission under the 7th Framework Programme, Coordination and Support Action, Grant Agreement Number 316097, Engine project. Authors also wish to thank Dr. Ramón Filgueira for his suggestions,
References (72)
- et al.
Characterising performance of environmental models
Environ. Model. Softw.
(2013) - et al.
Robust optimization - a comprehensive survey
Comput. Methods Appl. Mech. Eng.
(2007) - et al.
A multiobjective model and evolutionary algorithms for robust time and space assembly line balancing under uncertain demand
Omega
(2016) - et al.
Multiobjective, constructive heuristics for the 1/3 variant of the time and space assembly line balancing problem: ACO and random greedy search
Inf. Sci.
(2010) - et al.
Identimod: Modeling and managing brand value using soft computing
Decis. Support Syst.
(2016) - et al.
An integrated approach to automated innovization for discovering useful design principles: case studies from engineering
Appl. Soft Comput.
(2014) - K. Deb, A. Srinivasan, Innovization: innovating design principles through optimization, ACM, 2006. Proceedings of the...
- et al.
Real–coded genetic algorithms and interval-schemata
Found. Genet. Algorithms
(1993) - et al.
A comparison of scope for growth (SFG) and dynamic energy budget (DEB) models applied to the blue mussel (mytilus edulis)
J. Sea Res.
(2011) System dynamics: the next fifty years
Syst. Dyn. Rev.
(2007)
An exploratory approach for adaptive policy-making by using multi-objective robust optimization
Simul. Model. Pract. Theory
Genetic algorithm optimisation of an agent-based model for simulating a retail market
Environ. Plann. B
Energy budgets can explain body size relations
J. Theor. Biol.
Dynamic Energy and Mass Budgets in Biological Systems
A dynamic niching genetic algorithm strategy for docking highly flexible ligands
Inf. Sci.
An introduction to dynamic energy budget (DEB) models with special emphasis on parameter estimation
J. Sea Res.
Active nonlinear tests (ANTs) of complex simulation models
Manage. Sci.
A system dynamic model of management of a television game show
J. Model. Manage.
A simplex method for function minimization
Comput. J.
Algorithm AS 47: function minimization using a simplex procedure
Appl. Stat.
Finding multiple solutions in job shop scheduling by niching genetic algorithms
J. Intell. Manuf.
A comprehensive analytical approach for policy analysis of system dynamics models
Eur. J. Oper. Res.
Global Sensitivity Analysis: The Primer
System dynamics simulation model for assessing socio-economic impacts of different levels of environmental flow allocation in the weihe river basin, china
Eur. J. Oper. Res.
On the use of niching genetic algorithms for variable selection in solar radiation estimation
Renewable Energy
Ensemble of niching algorithms
Inf. Sci.
Automated parameter specification in dynamic feedback models based on behavior pattern features
Syst. Dyn. Rev.
Measuring brand equity across products and markets
California Manage. Rev.
Niching genetic feature selection algorithms applied to the design of fuzzy rule based classification systems
IEEE International Conference on Fuzzy Systems (FUZZ-IEEE’07), London (UK)
Handbook of Evolutionary Computation
Parameter estimation and optimal experimental design
Essays Biochem.
Detecting key variables in system dynamics modelling by using social network metrics
Predicting the unpredictable
Harv. Bus. Rev.
Automatic tuning of agent-based models using genetic algorithms
Multi-Agent-Based Simulation VI
Root mean square error (RMSE) or mean absolute error (MAE)?–arguments against avoiding RMSE in the literature
Geosci. Model Dev.
Cited by (32)
A framework for the comparison of errors in agent-based models using machine learning
2023, Journal of Computational ScienceA multimodal evolutionary algorithm with multi-niche cooperation
2023, Expert Systems with ApplicationsA framework of opinion dynamics using fuzzy linguistic 2-tuples
2021, Knowledge-Based SystemsSimulating the influence of terror management strategies on the voter ideological distance using agent-based modeling
2021, Telematics and InformaticsCoral reefs optimization algorithms for agent-based model calibration
2021, Engineering Applications of Artificial Intelligence