Abstract
In the context of gradient-free multi-modal optimization, numerous algorithms are based on restarting evolution strategies. Such an algorithm classically performs many local searches, for finding all the global optima of the objective function. The strategy used to select the restarting points (i.e., the initial points of the local searches) is a crucial step of the method. In previous works, a strategy based on reinforcement learning has been proposed: the search space is partitioned and a multi-armed bandit algorithm is used to select an interesting region to sample. This strategy significantly improves the main optimization algorithm but is limited to small dimensional problems. In this paper, we propose an algorithm tackling this problem. The main contributions are (1) a tree-based scheme for hierarchically partition the search space and (2) a multi-armed bandit strategy for traversing this tree. Thus, a node in the tree corresponds to a region of the search space, and its children partition this region according to one dimension. The multi-armed bandit strategy is used to traverse the tree by selecting interesting children recursively. We have experimented our algorithm on difficult multi-modal functions, with small and large dimensions. For small dimensions, we observe performances comparable to previous state-of-the-art algorithms. For large dimensions, we observe better results as well as lower memory consumption.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahrari A, Deb K, Preuss M (2017) Multimodal optimization by covariance matrix self-adaptation evolution strategy with repelling subpopulations. Evol Comput 25(3):439–471
Auer P, Cesa-Bianchi N, Fischer P (2002a) Finite-time analysis of the multiarmed bandit problem. Mach Learn 47(2–3):235–256
Auer P, Cesa-Bianchi N, Fischer P (2002b) Finite-time analysis of the multiarmed bandit problem. Mach Learn 47(2–3):235–256
Auger A, Hansen N (2005) A restart CMA evolution strategy with increasing population size. In: 2005 IEEE congress on evolutionary computation, vol 2. pp 1769–1776
Beyer H-G (2001) The theory of evolution strategies. Springer, Berlin
Bubeck S, Munos R, Stoltz G, Szepesvári C (2011) X-armed bandits. J Mach Learn Res 12(May):1655–1695
De Jong KA (1975) An analysis of the behavior of a class of genetic adaptive systems. PhD thesis, University of Michigan
Deng W, Zhao H, Yang X, Xiong J, Sun M, Li B (2017) Study on an improved adaptive pso algorithm for solving multi-objective gate assignment. Appl Soft Comput 59:288–302
Deng W, Xu J, Song Y, Zhao H (2019a) An effective improved co-evolution ant colony optimization algorithm with multi-strategies and its application. Int J Bio-Inspired Comput
Deng W, Xu J, Zhao H (2019b) An improved ant colony optimization algorithm based on hybrid strategies for scheduling problem. IEEE Access 7:20281–20292
Dubois A, Dehos J, Teytaud F (2018) Improving multi-modal optimization restart strategy through multi-armed bandit. In: IEEE ICMLA 2018: 17th IEEE international conference on machine learning and applications
Gelly S, Silver D (2007) Combining online and offline knowledge in UCT. In: International conference of machine learning
Goldberg DE, Richardson J et al (1987) Genetic algorithms with sharing for multimodal function optimization. In: Genetic algorithms and their applications: proceedings of the second international conference on genetic algorithms. pp 41–49
Kadioglu S, Sellmann M, Wagner M (2017) Learning a reactive restart strategy to improve stochastic search. In: International conference on learning and intelligent optimization. pp 109–123
Kocsis L, Szepesvári C (2006) Bandit based Monte-Carlo planning. In: European conference on machine learning. pp 282–293
Lacroix B, Christie LA, Mcall JA (2017) Interpolated continuous optimisation problems with tunable landscape features. In: Proceedings of GECCO ’17 companion
Li X (2016) Multimodal optimization using niching methods. American Cancer Society, New York, pp 1–8
Mahfoud SW (1995) Niching methods for genetic algorithms. Urbana 51:62–94
Mason K, Duggan J, Howley E (2018) Maze navigation using neural networks evolved with novelty search and differential evolution. In: Adaptive and learning agents workshop (at ICML-AAMAS 2018)
Mengshoel OJ, Goldberg DE et al (1999) Probabilistic crowding: deterministic crowding with probabilistic replacement. In: Proceedings of the genetic and evolutionary computation conference, vol 1. Morgan Kauffman, pp 409–416
Munos R (2011) Optimistic optimization of a deterministic function without the knowledge of its smoothness. In: John S-T, Richard SZ, Peter LB, Fernando CNP, Kilian QW (eds) Advances in neural information processing systems, pp 783–791. http://papers.nips.cc/paper/4304-optimistic-optimization-of-a-deterministic-function-without-the-knowledge-of-its-smoothness
Petrowski A (1996) A clearing procedure as a Niching method for genetic algorithms. In: Proceedings of IEEE international conference on evolutionary computation. pp 798–803
Preuss M (2015) Multimodal optimization by means of evolutionary algorithms, 1st edn. Springer, Berlin
Preux P, Munos R, Valko M (2014) Bandits attack function optimization. In: IEEE congress on evolutionary computation
Rapin J, Teytaud O (2018) Nevergrad—a gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad
Rechenberg I (1973) Evolutionsstrategie: optimierung technischer Systeme nach Prinzipien der biologischen evolution. Number 15 in Problemata. Frommann-Holzboog
Schoenauer M, Teytaud F, Teytaud O (2011) A rigorous runtime analysis for quasi-random restarts and decreasing stepsize. In: Artificial evolution. Angers, France
Silver D (2009) Reinforcement learning and simulation-based search. PhD thesis, University of Alberta
Singh G, Deb DR K (2006) Comparison of multi-modal optimization algorithms based on evolutionary algorithms. In: Proceedings of the 8th annual conference on genetic and evolutionary computation. pp 1305–1312
Teytaud F, Teytaud O (2016) Qr mutations improve many evolution strategies: a lot on highly multimodal problems. In: Proceedings of the 2016 GECCO conference. pp. 35–36
Valko M, Carpentier A, Munos R (2013) Stochastic simultaneous optimistic optimization. In: Proceedings of the 30th international conference on machine learning, vol 28
Zhao H, Liu H, Xu J, Deng W (2019) Performance prediction using high-order differential mathematical morphology gradient spectrum entropy and extreme learning machine. IEEE Trans Instrum Meas 69(7):4165–4172
Zhao H, Zheng J, Deng W, Song Y (2020) Semi-supervised broad learning system based on manifold regularization and broad network. IEEE Trans Circuits Syst I Regul Pap 67(3):983–994
Acknowledgements
Experiments presented in this paper were carried out using the CALCULCO computing platform, supported by SCOSI/ULCO (Service COmmun du Système d’Information de l’Université du Littoral Côte d’Opale).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Amaury Dubois has received research grants from Weenat. Julien Dehos and Fabien Teytaud declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dubois, A., Dehos, J. & Teytaud, F. Upper confidence tree for planning restart strategies in multi-modal optimization. Soft Comput 25, 1007–1015 (2021). https://doi.org/10.1007/s00500-020-05196-w
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-020-05196-w