Abstract
Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist. Recently, successful parameter control methods based on Reinforcement Learning (RL) have been suggested for one-off applications, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. However, the reward function used was not investigated in depth, though it is a non-trivial factor with an important impact on the performance of a RL mechanism. In this paper, we address this issue by defining and comparing four alternative reward functions for such generic and RL-based EA parameter controllers. We conducted experiments with different EAs, test problems and controllers and results showed that the simplest reward function performs at least as well as the others, making it an ideal choice for generic out-of-the-box parameter control.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Notice that, for example, as discussed above, the reward definition that Muller et al. [14] found most efficient cannot be generalised.
- 2.
Any other other information (e.g. diversity) would introduce a bias on the controller’s strategy.
- 3.
If, on the contrary, we consider a controller for a repetitive application, we could train the controller off-line using multiple training runs, thus being able to take the final best fitness of each run as the reward.
- 4.
We did not use the more intuitive ratio \(\frac{\varDelta _f}{Ref(n)}\) because preliminary experiments showed the difference \(\varDelta _f - Ref(N)\) to perform better.
- 5.
- 6.
The source code was acquired directly from the authors.
- 7.
http://www3.ntu.edu.sg/home/epnsugan/index_files/CEC11-RWP/CEC11-RWP.htm. The source code of GA MPC is available at the same competition page.
- 8.
The source code for the (IPOP) CMA-ES was acquired from https://www.lri.fr/hansen/cmaes_inmatlab.html. The 10DDr variation was added by us.
- 9.
- 10.
The source code for this experiment is available for download at http://www.few.vu.nl/~gks290/resources/evostar2015.tar.gz.
References
Alba, E., Dorronsoro, B.: Cellular Genetic Algorithms, 1st edn. Springer, Heidelberg (2008)
Chen, F., Gao, Y., Chen, Z., Chen, S.: SCGA: controlling genetic algorithms with Sarsa(0). In: International Conference on Computational Intelligence for Modelling, Control and Automation, 2005 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, vol. 1, pp. 1177–1183 (2005)
Eiben, A.E., Horvath, M., Kowalczyk, W., Schut, M.C.: Reinforcement learning for online control of evolutionary algorithms. In: Brueckner, S.A., Hassas, S., Jelasity, M., Yamins, D. (eds.) ESOA 2006. LNCS (LNAI), vol. 4335, pp. 151–160. Springer, Heidelberg (2007)
Eiben, A.E., Hinterding, R., Michalewicz, Z.: Parameter control in evolutionary algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999)
Eiben, A.E., Smith, J.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003)
Elsayed, S., Sarker, R., Essam, D.: GA with a new multi-parent crossover for solving IEEE-CEC2011 competition problems. In: Proceedings of the 2011 IEEE Congress on Evolutionary Computation, pp. 1034–1040. IEEE Press, New Orleans, USA (2011)
Fialho, Á., Da Costa, L., Schoenauer, M., Sebag, M.: Extreme value based adaptive operator selection. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 175–184. Springer, Heidelberg (2008)
Gong, W., Fialho, A., Cai, Z.: Adaptive strategy selection in differential evolution. In: Pelikan, M., Branke, J. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2010), pp. 409–416. ACM, Portland (2010)
Holtschulte, N.J., Moses, M.: Benchmarking cellular genetic algorithms on the BBOB noiseless testbed. In: Proceeding of the Fifteenth Annual Conference Companion on Genetic and Evolutionary Computation Conference Companion, GECCO 2013 Companion, pp. 1201–1208. ACM (2013)
Karafotias, G., Eiben, A.E., Hoogendoorn, M.: Generic parameter control with reinforcement learning. In: Arnold, D.V. (ed.) GECCO 2014: Proceedings of the 16th annual conference on Genetic and evolutionary computation, pp. 1319–1326. ACM, New York (2014)
Karafotias, G., Hoogendoorn, M., Eiben, A.E.: Parameter control in evolutionary algorithms: trends and challenges. IEEE Transactions on Evolutionary Computation (2014, to appear). doi:10.1109/TEVC.2014.2308294
Karafotias, G., Hoogendoorn, M., Weel, B.: Comparing generic parameter controllers for EAs. In: 2014 IEEE Symposium on Foundations of Computational Intelligence (FOCI), pp. 46–53, December 2014
Liao, T., Stützle, T.:. Bounding the population size of IPOP-CMA-ES on the noiseless BBOB testbed. In: Proceeding of the Fifteenth Annual Conference Companion on Genetic and Evolutionary Computation Conference Companion, pp. 1161–1168. ACM (2013)
Muller, S., Schraudolph, N., Koumoutsakos, P.: Step size adaptation in evolution strategies using reinforcement learning. In: 2002 Congress on Evolutionary Computation (CEC 2002), pp. 151–156. IEEE Press, Piscataway, NJ, Honolulu, USA 12–17 May 2002
Pettinger, J., Everson, R.: Controlling genetic algorithms with reinforcement learning. In: Langdon, W., et al. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2002), page 692, pp. 9–13. Morgan Kaufmann, San Francisco (2002)
Sakurai, Y., Takada, K., Kawabe, T., Tsuruta, S.: A method to control parameters of evolutionary algorithms by using reinforcement learning. In: 2010 Sixth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), pp. 74–79 (2010)
Schwefel, H.-P.: Evolution and Optimum Seeking. Wiley, New York (1995)
Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)
Uther, W.B., Veloso, M.: Tree based discretization for continuous state space reinforcement learning. In: Proceedings of the Fifteenth National/tenth Conference on Artificial Intelligence/innovative Applications of Artificial Intelligence, AAAI 1998/IAAI 1998, pp. 769–774. American Association for Artificial Intelligence (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Karafotias, G., Hoogendoorn, M., Eiben, A.E. (2015). Evaluating Reward Definitions for Parameter Control. In: Mora, A., Squillero, G. (eds) Applications of Evolutionary Computation. EvoApplications 2015. Lecture Notes in Computer Science(), vol 9028. Springer, Cham. https://doi.org/10.1007/978-3-319-16549-3_54
Download citation
DOI: https://doi.org/10.1007/978-3-319-16549-3_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16548-6
Online ISBN: 978-3-319-16549-3
eBook Packages: Computer ScienceComputer Science (R0)