Abstract
Although researchers have successfully incorporated metamodels in evolutionary algorithms to solve computational-expensive optimization problems, they have scarcely performed comparisons among different metamodeling techniques. This paper presents an in-depth comparison study over four of the most popular metamodeling techniques: polynomial response surface, Kriging, radial basis function neural network (RBF), and support vector regression. We adopted six well-known scalable test functions and performed experiments to evaluate their suitability to be coupled with an evolutionary algorithm and the appropriateness to surrogate problems by regions (instead of surrogating the entire problem). Notwithstanding that most researchers have undertaken accuracy as the main measure to discern among metamodels, this paper shows that the precision, measured with the ranking preservation indicator, gives a more valuable information for selecting purposes. Additionally, nonetheless each model has its own peculiarities; our results concur that RBF fulfills most of our interests. Furthermore, the readers can also benefit from this study if their problem at hand has certain characteristics such as a low budget of computational time or a low-dimension problem since they can assess specific results of our experimentation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We will use the terms approximation models, surrogate models, and metamodels interchangeably in this paper.
The SVM for a regression problem is known as a support vector regression (SVR).
Statistical method of stratified sampling that can be applied to multiple variables.
This normalization is known as normalized root-mean-square deviation.
References
Bäck T (1996) Evolutionary algorithms in theory and practice. Oxford University Press, Oxford
Barton R (1992) Metamodels for simulation input-output relations. In: Proceedings of the 24th conference on winter simulation (WSC’92). ACM, New York, pp 289–299
Carpenter W, Barthelemy J (1993) A comparison of polynomial approximations and artificial neural nets as response surfaces. Struct Optim 5(3):166–174
Chow C, Yuen S (2011) An evolutionary algorithm that makes decision based on the entire previous search history. IEEE Trans Evol Comput 15(6):741–769
De Jong K (1975) An analysis of the behavior of a class of genetic adaptive systems. Ph.D. thesis, University of Michigan, Ann Arbor
Díaz-Manríquez A, Toscano-Pulido G, Gomez-Flores W (2011) On the selection of surrogate models in evolutionary optimization algorithms. In: IEEE congress on evolutionary computation, pp 2155–2162
Díaz-Manríquez A, Toscano-Pulido G, Coello Coello CA, Landa-Becerra R (2013) A ranking method based on the \(r2\) indicator for many-objective optimization. In: 2013 IEEE congress on evolutionary computation (CEC’13). IEEE Press, Cancún, pp 1523–1530. ISBN 978-1-4799-0454-9
Draper N, Smith H (1981) Applied regression analysis. In: Wiley series in probability and mathematical statistics, 2nd edn. Wiley, New York
Forgy EW (1965) Cluster analysis of multivariate data: efficiency versus interpretability of classifications. Biometrics 21:768–769
Gaspar-Cunha A, Vieira A (2005) A multi-objective evolutionary algorithm using neural networks to approximate fitness evaluations. Int J Comput Syst Signal 6:18–36
Georgopoulou C, Giannakoglou K (2009) Multiobjective metamodel-assisted memetic algorithms. In: Multi-objective memetic algorithms, studies in computational intelligence, vol 171. Springer, Berlin, pp 153–181
Giunta A, Watson L (1998) A comparison of approximation modeling techniques: polynomial versus interpolating models. Tech. rep., NASA Langley Technical Report Server
Hansen N, Ostermeier A (2001) Completely derandomized self-adaptation in evolution strategies. Evol Comput 9(2):159–195
Hardy R (1971) Multiquadric equations of topography and other irregular surfaces. J Geophys Res 76:1905–1915
Isaacs A, Ray T, Smith W (2007) An evolutionary algorithm with spatially distributed surrogates for multiobjective optimization. In: Randall M, Abbass H, Wiles J (eds) Progress in artificial life, vol 4828., Lecture notes in computer scienceSpringer, Berlin, pp 257–268
Jin R, Chen W, Simpson T (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscip Optim 23(1):1–13
Matheron G (1963) Principles of geostatistics. Econ Geol 58(8):1246–1266
McKay M, Beckman R, Conover W (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245
Myers R, Anderson-Cook C (2009) Response surface methodology: process and product optimization using designed experiments, vol 705. Wiley, New York
Nain P, Deb K (2002) A computationally effective multi-objective search and optimization technique using coarse-to-fine grain modeling. In: 2002 PPSN workshop on evolutionary multiobjective optimization comprehensive survey of fitness approximation in evolutionary computation
Pilat M, Neruda R (2013) Aggregate meta-models for evolutionary multiobjective and many-objective optimization. Neurocomputing 116:392–402
Press W, Teukolsky SA, Vetterling W, Flannery B (2007) Numerical recipes 3rd edition: the art of scientific computing, 3rd edn. Cambridge University Press, New York
Rasheed K, Ni X, Vattam S (2002) Comparison of methods for developing dynamic reduced models for design optimization. In: IEEE congress on evolutionary computation, pp 390–395
Rastrigin L (1974) Extremal control systems. In: Theoretical foundations of engineering cybernetics series. Nauka, Moscow
Sacks J, Welch W, Mitchell T, Wynn H (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–423
Schumaker L (2007) Spline functions: basic theory. Cambridge University Press, Cambridge
Schwefel H (1981) Numerical optimization of computer models. Wiley, New York
Shyy W, Papila N, Vaidyanathan R, Tucker K (2001) Global design optimization for aerodynamics and rocket propulsion components. Prog Aerosp Sci 37(1):59–118
Silverman B, Jones M (1989) An important contribution to nonparametric discriminant analysis and density estimation: commentary on fix and hodges. In: International statistical review/revue internationale de statistique, pp 233–238
Simpson T, Mauery T, Korte J, Mistree F (1998) Comparison of response surface and Kriging models for multidiscilinary design optimization
Storn R, Price K (1997) Differential evolution? A simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359
Vapnik V (1998) Statistical learning theory. Wiley-Interscience, New York
Voutchkov I, Keane A (2006) Multiobjective optimization using surrogates. In: International conference on adaptive computing in design and manufacture. The M.C.Escher Company, Holland, pp 167–175
Willmes L, Baeck T, Jin Y, Sendhoff B (2003) Comparing neural networks and Kriging for fitness approximation in evolutionary optimization. In: IEEE congress on evolutionary computation, pp 663–670
Yuen S, Chow C (2009) A genetic algorithm that adaptively mutates and never revisits. IEEE Trans Evol Comput 13(2):454–472
Acknowledgments
G. Toscano gratefully acknowledges support from CONACyT through Project No. 105060. C. A. Coello Coello gratefully acknowledges support from CONACyT Project No. 221551.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Díaz-Manríquez, A., Toscano, G. & Coello Coello, C.A. Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput 21, 5647–5663 (2017). https://doi.org/10.1007/s00500-016-2140-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-016-2140-z