Elsevier

Computers & Chemical Engineering

Volume 108, 4 January 2018, Pages 250-267
Computers & Chemical Engineering

Review
Advances in surrogate based modeling, feasibility analysis, and optimization: A review

https://doi.org/10.1016/j.compchemeng.2017.09.017Get rights and content

Highlights

  • State of the art review of surrogate modeling techniques is provided.

  • Best choice of the surrogate model depends on the type of problem at hand.

  • Recent advances in sampling strategies for surrogate model building are discussed.

Abstract

The idea of using a simpler surrogate to represent a complex phenomenon has gained increasing popularity over past three decades. Due to their ability to exploit the black-box nature of the problem and the attractive computational simplicity, surrogates have been studied by researchers in multiple scientific and engineering disciplines. Successful use of surrogates shall result in significant savings in terms of computational time and resources. However, with a wide variety of approaches available in the literature, the correct choice of surrogate is a difficult task. An important aspect of this choice is based on the type of problem at hand. This paper reviews recent advances in the area of surrogate models for problems in modeling, feasibility analysis, and optimization. Two of the frequently used surrogates, radial basis functions, and Kriging are tested on a variety of test problems. Finally, guidelines for the choice of appropriate surrogate model are discussed.

Introduction

The problem discussed in the paper is assessing the performance of surrogates on the deterministic function f : Rd  R; where the input vector is X=(x1,x2,,xd), d is the number of dimensions of XL  X  XU the problem, and there is a single output y. The input vector X has known lower and upper bounds

Additionally, some constraints fj  0, j  J where J is the set of all constraints, may be present. It is assumed that evaluation of the function as well as constraints is computationally expensive and the symbolic form of the function and that of one or more constraints is unknown. From this assumption, it follows that the analytical form of the derivatives is also unavailable. Surrogate modeling addresses this problem by obtaining a function fˆ(X) that approximates the function f.

This problem occurs frequently in multiple engineering and scientific disciplines where complex computer simulations or physical experiments are used. In these cases, obtaining more data means additional experiments and thus it results in significant material or economic cost as well as highly non-trivial computational expense. As a result, it is difficult to obtain an analytical form of the objective function or that of the derivatives. Deriving this information from surrogate fˆ(x) is relatively easier because its analytical form is known and it is cheaper to evaluate. Several applications of surrogates to address this type of problems can be found in the literature. For example, (Anthony et al., 1997), (Balabanov and Haftka, 1998), use polynomial, linear response surfaces in aircraft design. Artificial neural networks (ANN) is used for process modeling (Meert and Rijckaert, 1998), process control (Bloch and Denoeux, 2003), (Mujtaba et al., 2006), and for optimization (Fernandes, 2006), (Henao and Maravelias, 2011). Kriging is used for process flowsheet simulations (Palmer and Realff, 2002), design simulations (Yang et al., 2005), (Prebeg et al., 2014), pharmaceutical process simulations (Jia et al., 2009), and feasibility analysis (Rogers and Ierapetritou, 2015). Radial basis functions (RBF) is used for feasibility analysis (Wang and Ierapetritou, 2016) and parameter estimation (Müller et al., 2015). It can be observed from the fˆ(x) applications listed above that there are multiple approaches proposed in literature to obtain a surrogate.

Several prior reviews discuss these approaches and related developments in the field of surrogate models. Surrogate models and their potential use in simulations is discussed by (Barton, 1992). They discuss polynomial response surface, spline interpolation, radial basis functions, regression models, and Kriging surrogates. With the focus on modeling and prediction for engineering design, (Simpson et al., 1997) review stationary sampling designs, polynomial response surface methods, Kriging and robust methods. (Jin et al., 2001) studied performance of polynomial regression, multivariate adaptive regression splines, radial basis functions, and Kriging surrogates under multiple criteria such as efficiency, robustness and model simplicity. Motivated from applications in aerospace systems, (Queipo et al., 2005) discuss surrogate based optimization and sensitivity analysis, sampling strategies and surrogate model validation. (Barton and Meckesheimer, 2006) discuss surrogates for guiding optimization of simulations. In this context of guiding search towards optimum, they classify surrogates as local surrogates that are updated within an iterative framework and global surrogates that are fitted only once and the search proceeds using the same surrogate thereafter. For the purposes of design optimization, (Wang and Shan, 2007) provide an overview of surrogate models. Their focus is mainly on solving optimization problems such as global optimization, multi-objective optimization, and probabilistic design optimization. Motivated from computationally intensive aerospace designs, (Forrester and Keane, 2009) discuss details of surrogate modeling methodology focusing on sampling, surrogate model building, and validation. They discuss surrogates such as polynomial interpolation, RBF, Kriging and support vector regression and their advantages and disadvantages for achieving better prediction accuracy. (Razavi et al., 2012) investigate the potential of surrogate modeling techniques with a focus on the use of surrogates in water resources applications. They provide an excellent review on use of surrogates in water resources. (Nippgen et al., 2016) review surrogate modeling strategies from a broader point of view by classifying those as data-driven, projection-based and multi-fidelity surrogate modeling strategies. They focus on potential of using surrogates for applications in groundwater modeling. (Haftka et al., 2016) discuss in detail, several strategies for global optimization using surrogates, criteria for local and global searches from the point of view of parallelization. It is important to note that with respect to applications, the problems requiring surrogates can be classified in to three classes. The first class of problems is the most fundamental use of surrogates i.e. prediction and modeling. The second class of problems is commonly known as derivative-free optimization (DFO) where the objective function to be optimized is expensive and thus derivative information is unavailable. The third class of problems is feasibility analysis where the objective is also to satisfy design constraints. Prior reviews discuss applications of surrogate models for only one or two of these three classes. This review emphasizes that there is a significant difference between using surrogates for each of these three classes of problems and provides a comprehensive understanding of surrogate models for all three classes of problems mentioned. There has been a growing interest in model selection methodologies for regression models where the aim is to choose the best model from a given set of models. This problem has many practical uses in cases where the surrogate model does not generalize well on the test set, a phenomenon commonly known as overfitting or when there are too many input variables that might contain redundant information. In such cases, it is important to select most relevant variables in order to build simple yet effective surrogate models. Even though model selection is popular in the field of statistics for over 50 years, prior reviews in the context of surrogate modeling do not address this issue. As avoiding overfitting an important aspect to consider while building surrogate models, an extensive review of model selection strategies is provided.

The rest of this paper is organized as follows. Section 2 discusses different types of surrogates and their underlying mathematical formulations. Sections 3 and 4 describe surrogates in relation to the DFO problems and feasibility analysis, respectively. Section 5 is devoted towards sampling which is an important component of building the right surrogate model. Section 6 describes approaches for validating surrogates. Section 7 describes some of the existing software implementations. In section 8, a detailed comparison of the performance of RBF and Kriging surrogates on a set of 47 test problems is provided whereas Section 9 provides a summary of the manuscript.

Section snippets

Surrogate models

In this section, frequently used approaches for obtaining the surrogate fˆ(x) are discussed with a focus on the recent advances. The models that are designed to yield unbiased predictions at the sampled data are referred to as interpolation models, whereas models that are built by minimizing the error between given data and model prediction under a certain criterion are referred to as regression models. In this section, regression models such as linear regression, support vector regression are

Derivative-free optimization and surrogates

The optimization problems for which function derivative information is not symbolically or numerically available are classified as DFO problems. There are two sub-categories in algorithms addressing DFO problems, one is local search (referred to as local DFO) algorithms and the other is global search algorithms (referred to as global DFO). Local search algorithms are effective in refining the solution or reaching a local optimum from an initial guess. Global search algorithms, on the other

Feasibility analysis and surrogates

The ability of a process to satisfy all relevant constraints is referred to as feasibility. Feasibility analysis relates to identifying conditions under which the process is feasible. The problem of feasibility analysis arises because of several constraints on operation such as product demand, environmental conditions, the safety of operation, and material properties to name a few. The same problem arises in designing a product or a new material where the design is restricted by factors such as

Sampling

The process of generating data points to be able to build surrogates is referred to as sampling. The performance of surrogate models depends strongly on the quality as well as the number of samples. However, as generating data demands evaluation of the true function, sampling contributes towards significant computational cost. To maintain the quality of surrogates without incurring excessive sampling cost, studying sampling strategies is of immense importance.

Sampling strategies are broadly

Surrogate model validation

Assessing the reliability of surrogate model is one of the major concerns because having an inaccurate surrogate model can lead to waste of resources and have a bad effect on optimization, prediction or feasibility analysis. Surrogate model validation is the process of assessing the reliability of the surrogate model. In addition to assessing accuracy, validation techniques can be used to select a surrogate model from a set of candidate models and to tune hyper-parameters (such as correlation

Software implementations

Several software implementations exist for DFO. List of these software can be found in (Rios and Sahinidis, 2013). Relatively less number of software implementations exist for surrogate model building for prediction and feasibility analysis. Few of the software implementations are listed followed by a recent DFO framework:

1) Automated learning of algebraic models for optimization (ALAMO) (Cozad et al., 2014)

ALAMO is a regression and classification methodology that builds simple and accurate

Computational results and discussion

The purpose of computational experiments in this section is to illustrate the effect of following choices on the performance of surrogates in terms of their predictive ability.

  • 1.

    For Kriging models, the choice of regression and correlation terms

  • 2.

    Initial sample size

  • 3.

    Initial sample design scheme

Summary

Surrogate models have attracted interest from multiple engineering and scientific disciplines and have found a wide range of application domains. However, the choice of surrogate model for the problem at hand is not straight-forward because of trade-offs associated with each surrogate. This choice is better understood if the problem at hand is classified as a prediction, optimization or a feasibility analysis problem based on the use of surrogate model. The difference of approaches for each of

Acknowledgement

Financial support from NSF under grants 1159244 and 1434548 is gratefully acknowledged.

References (137)

  • A.I.J. Forrester et al.

    Recent advances in surrogate-based optimization

    Progr. Aerospace Sci.

    (2009)
  • S.S. Garud et al.

    Smart sampling algorithm for surrogate model development

    Comput. Chem. Eng.

    (2017)
  • I.E. Grossmann et al.

    Evolution of concepts and models for quantifying resiliency and flexibility of chemical processes

    Comput. Chem. Eng.

    (2014)
  • K. Meert et al.

    Intelligent modelling in the chemical process industry with neural networks: a case study

    Comput. Chem. Eng.

    (1998)
  • I.M. Mujtaba et al.

    Neural network based modelling and control in batch reactor

    Chem. Eng. Res. Des.

    (2006)
  • K. Palmer et al.

    Metamodeling approach to optimization of steady-state flowsheet simulations

    Chem. Eng. Res. Des.

    (2002)
  • P. Prebeg et al.

    Application of a surrogate modeling to the ship structural design

    Ocean Eng.

    (2014)
  • N.V. Queipo et al.

    Surrogate-based analysis and optimization

    Prog. Aerosp. Sci.

    (2005)
  • H. Akaike

    A new look at the statistical model identification

    IEEE Trans. Autom. Control

    (1974)
  • A.G. Anthony et al.

    Multidisciplinary Optimization of a Supersonic Transport Using Design of Experiments Theory and Response Surface Modeling

    (1997)
  • V. Balabanov et al.

    Multifidelity response surface model for HSCT wing bending material weight

    Proceedings of 7th …

    (1998)
  • S. Banerjee et al.

    Gaussian predictive process models for large spatial data sets

    J. R. Stat. Soc. Ser. B: Stat. Methodol.

    (2008)
  • R.R. Barton

    Metamodels for simulation input-output relations

  • D. Bertsimas et al.

    OR Forum—an algorithmic approach to linear regression

    Oper. Res.

    (2016)
  • B. Bischl et al.

    Resampling methods for meta-model validation with recommendations for evolutionary computation

    Evol. Comput.

    (2012)
  • B. Bischl et al.

    mlr: Machine learning in r

    J. Mach. Learn. Res.

    (2016)
  • M. Björkman et al.

    Global optimization of costly nonconvex functions using radial basis functions

    Optim. Eng.

    (2000)
  • M. Bouhlel et al.

    Improving Kriging Surrogates of High-dimensional Design Models by Partial Least Squares Dimension Reduction To Cite This Version: Improving Kriging Surrogates of High-dimensional Design Models by Partial Least Squares Dimension Reduction

    (2015)
  • F. Boukouvala et al.

    ARGONAUT: AlgoRithms for Global Optimization of coNstrAined grey-box compUTational problems

    Optim. Lett.

    (2016)
  • F. Boukouvala et al.

    Derivative-free optimization for expensive constrained problems using a novel expected improvement objective function

    AIChE J.

    (2014)
  • A. Boukouvalas et al.

    Learning heteroscedastic gaussian processes for complex datasets

    Group

    (2009)
  • L. Breiman

    Better subset regression using the nonnegative garrote

    Technometrics

    (1995)
  • Cadima, J., Cerdeira, J. O., Silva, P. D., & Minhoto, M., 2012. The subselect R...
  • E. Candes et al.

    The Dantzig selector: statistical estimation when p is much larger than n

    Ann. Stat.

    (2007)
  • H. Chen et al.

    Analysis methods for computer experiments: how to assess and what counts?

    Stat. Sci.

    (2013)
  • Z. Chen

    Extended bayesian information criteria for model selection with large model spaces model selection extended Bayesian information criteria for with large model spaces

    Biometrika

    (2014)
  • T. Chu et al.

    Penalized maximum likelihood estimation and variable selection in geostatistics

    Ann. Stat.

    (2011)
  • S.M. Clarke et al.

    Analysis of support vector regression for approximation of complex engineering analyses

    J. Mech. Des.

    (2005)
  • a. R. Conn et al.

    Recent progress in unconstrained nonlinear optimization without derivatives

    Math. Program.

    (1997)
  • A. Cozad et al.

    Learning surrogate models for simulation-based optimization

    AIChE J.

    (2014)
  • N. Cressie et al.

    Fixed rank kriging for very large spatial data sets

    J. R. Stat. Soc. Ser. B (Stat. Methodol.)

    (2008)
  • E. Dolan et al.

    Benchmarcking optimization software with performance profiles

    Math. Programm.

    (2002)
  • B. Efron et al.

    Least angle regression

    The Ann. Stat.

    (2004)
  • J. Fan et al.

    Variable selection via nonconcave penalized

    J. Am. Stat. Assoc.

    (2001)
  • J. Fan et al.

    Sure independence screening for ultrahigh dimensional feature space

    J. R. Stat. Soc. Ser. B: Stat. Methodol.

    (2008)
  • F.A.N. Fernandes

    Optimization of fischer-tropsch synthesis using neural networks

    Chem. Eng. Technol.

    (2006)
  • C.A. Floudas et al.

    Global optimization in design under uncertainty: feasibility test and flexibility index problems

    Ind. Eng. Chem. Res.

    (2001)
  • D.P. Foster et al.

    The risk inflation criterion for multiple regression

    The Ann. Stat.

    (1994)
  • G.M. Furnival et al.

    Regressions by leaps and bounds

    Technometrics

    (1974)
  • R. Furrer et al.

    Covariance tapering for interpolation of large spatial datasets

    J. Comput. Graph. Stat.

    (2006)
  • Cited by (0)

    View full text