Skip to main content
SpringerLink
Log in

Analyzing bandit-based adaptive operator selection mechanisms

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

Several techniques have been proposed to tackle the Adaptive Operator Selection (AOS) issue in Evolutionary Algorithms. Some recent proposals are based on the Multi-armed Bandit (MAB) paradigm: each operator is viewed as one arm of a MAB problem, and the rewards are mainly based on the fitness improvement brought by the corresponding operator to the individual it is applied to. However, the AOS problem is dynamic, whereas standard MAB algorithms are known to optimally solve the exploitation versus exploration trade-off in static settings. An original dynamic variant of the standard MAB Upper Confidence Bound algorithm is proposed here, using a sliding time window to compute both its exploitation and exploration terms. In order to perform sound comparisons between AOS algorithms, artificial scenarios have been proposed in the literature. They are extended here toward smoother transitions between different reward settings. The resulting original testbed also includes a real evolutionary algorithm that is applied to the well-known Royal Road problem. It is used here to perform a thorough analysis of the behavior of AOS algorithms, to assess their sensitivity with respect to their own hyper-parameters, and to propose a sound comparison of their performances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multi-armed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)

    Article  MATH  Google Scholar 

  2. Barbosa, H.J.C., Sá, A.M.: On adaptive operator probabilities in real coded genetic algorithms. In: XX Intl. Conference of the Chilean Computer Science Society (2000)

  3. Bartz-Beielstein, T., Lasarczyk, C., Preuss, M.: Sequential parameter optimization. In: McKay, B. (ed.) Proc. Congress on Evolutionary Computation, pp. 773–780. IEEE (2005)

  4. Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Langdon, W.B., et al. (eds.) Proc. Genetic and Evolutionary Computation Conference, pp. 11–18. Morgan Kaufmann (2002)

  5. Collet, P., Schoenauer, M.: GUIDE: unifying evolutionary engines through a graphical user interface. In: Liardet, P., et al. (eds.) Proc. Intl. Conference on Artificial Evolution. LNCS, vol. 2936, pp. 203–215. Springer (2003)

  6. Conover, W.J.: Practical Nonparametric Statistics. Wiley (1999)

  7. Da Costa, L., Fialho, A., Schoenauer, M., Sebag, M.: Adaptive operator selection with dynamic multi-armed bandits. In: Keijzer, M., et al. (eds.) Proc. Genetic and Evolutionary Computation Conference, pp. 913–920. ACM (2008)

  8. Davis, L.: Adapting operator probabilities in genetic algorithms. In: Schaffer, J.D. (ed.) Proc. Intl. Conference on Genetic Algorithms, pp. 61–69. Morgan Kaufmann (1989)

  9. DeJong, K.: Evolutionary Computation. A unified Approach. MIT (2006)

  10. DeJong, K.: Parameter setting in EAs: a 30 year perspective. In: Lobo, F., Lima, C., Michalewicz, Z. (eds.): Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, vol. 54, pp. 1–18. Springer (2007)

  11. Eiben, A.E., Hinterding, R., Michalewicz, Z.: Parameter control in Evolutionary Algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999)

    Article  Google Scholar 

  12. Eiben, A.E., Michalewicz, Z., Schoenauer, M., Smith, J.E.: Parameter control in evolutionary algorithms. In: Lobo, F., Lima, C., Michalewicz, Z. (eds.): Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, vol. 54, pp. 19–46. Springer (2007)

  13. Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer (2003)

  14. Fialho, A., Da Costa, L., Schoenauer, M., Sebag, M.: Extreme value based adaptive operator selection. In: Rudolph, G., et al. (eds.) Proc. Intl. Conference on Parallel Solving from Nature. LNCS, vol. 5199, pp. 175–184. Springer (2008)

  15. Fialho, A., Da Costa, L., Schoenauer, M., Sebag, M.: Dynamic multi-armed bandits and extreme value-based rewards for adaptive operator selection in evolutionary algorithms. In: Stützle, T. (ed.) Proc. 3rd Intl. Conference on Learning and Intelligent Optimization. LNCS, vol. 5851, pp. 176–190. Springer (2009)

  16. Fialho, A., Schoenauer, M., Sebag, M.: Analysis of adaptive operator selection techniques on the royal road and long k-path problems. In: Raidl, G., et al. (eds.) Proc. Genetic and Evolutionary Computation Conference, pp. 779–786. ACM (2009)

  17. Fogel, D.B.: Phenotypes, genotypes and operators in evolutionary computation. In: Proc. Intl. Conference on Evolutionary Computation. IEEE (1995)

  18. Gagliolo, M., Schmidhuber, J.: Algorithm Selection as a Bandit Problem with Unbounded Losses. Tech. Rep. IDSIA-07-08, IDSIA (2008)

  19. Goldberg, D.: Probability matching, the magnitude of reinforcement, and classifier system bidding. Mach. Learn. 5(4), 407–426 (1990)

    Google Scholar 

  20. Gould, S., Eldredge, N.: Punctuated equilibria: the tempo and mode of evolution reconsidered. Paleobiology 3(2), 115–151 (1977)

    Google Scholar 

  21. Hartland, C., Baskiotis, N., Gelly, S., Teytaud, O., Sebag, M.: Change point detection and meta-bandits for online learning in dynamic environments. In: Proc. Conférence Francophone sur l’Apprentissage Automatique (2007)

  22. Hartland, C., Gelly, S., Baskiotis, N., Teytaud, O., Sebag, M.: Multi-armed bandit, dynamic environments and meta-bandits. In: Online Trading of Exploration and Exploitation Workshop, NIPS (2006)

  23. Hinkley, D.: Inference about the change point from cumulative sum-tests. Biometrika 58(3), 509–523 (1970)

    Article  MathSciNet  Google Scholar 

  24. Holland, J.H.: Royal road functions. In: Internet Genetic Algorithms Digest, vol. 7, p. 22. Massachusetts Institute of Technology (1993)

  25. Jones, T.: A description of Holland’s Royal Road. Evol. Comput. 2(4), 409–415 (1994)

    Article  Google Scholar 

  26. Julstrom, B.: What have you done for me lately? Adapting operator probabilities in a steady-state genetic algorithm. In: Eshelman, L.J., et al. (eds.) Proc. Intl. Conference on Genetic Algorithms, pp. 81–87. Morgan Kaufmann (1995)

  27. Kallel, L., Schoenauer, M.: Fitness Distance Correlation for Variable Length Representations. Tech. Rep. 363, CMAP, Ecole Polytechnique (1996)

  28. Lai, T., Robbins, H.: Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6(1), 4–22 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  29. Lobo, F., Goldberg, D.: Decision making in a hybrid genetic algorithm. In: Porto, B. (ed.) Proc. Intl. Conference on Evolutionary Computation, pp. 121–125. IEEE (1997)

  30. Lobo, F., Lima, C., Michalewicz, Z. (eds.): Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, vol. 54. Springer (2007)

  31. Maturana, J., Fialho, A., Saubion, F., Schoenauer, M., Sebag, M.: Extreme compass and dynamic multi-armed bandits for adaptive operator selection. In: Proc. Congress on Evolutionary Computation, pp. 365–372. IEEE (2009)

  32. Maturana, J., Lardeux, F., Saubion, F.: Autonomous operator management for evolutionary algorithms. Journal of Heuristics (2010). doi: 10.1007/s10732-010-9125-3

    Google Scholar 

  33. Maturana, J., Saubion, F.: A compass to guide genetic algorithms. In: Rudolph, G., et al. (eds.) Proc. Intl. Conference on Parallel Solving from Nature. LNCS, vol. 5199, pp. 256–265. Springer (2008)

  34. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs, 3rd edn. Springer, New York (1996)

    MATH  Google Scholar 

  35. Mitchell, M., Forrest, S., Holland, J.H.: The royal road for genetic algorithms: fitness landscapes and GA performance. In: Proc. European Conference on Artificial Life, pp. 245–254 (1992)

  36. Nannen, V., Eiben, A.E.: Relevance estimation and value calibration of evolutionary algorithm parameters. In: Veloso, M. (ed.) Proc. Intl. Joint Conference on Artificial Intelligence, pp. 975–980 (2007)

  37. Quick, R.J., Rayward-Smith, V.J., Smith, G.D.: The royal road functions: description, intent and experimentation. In: Selected Papers from AISB Workshop on Evolutionary Computing. LNCS, vol. 1143, pp. 223–235. Springer (1996)

  38. Spears, W.: Adapting crossover in evolutionary algorithms. In: McDonnell, J.R., et al. (eds.) Proc. Conference on Evolutionary Programming, pp. 367–384. MIT (1995)

  39. Stützle, T. (ed.): Proc. 3rd Intl. Conference on Learning and Intelligent Optimization. LNCS, vol. 5851. Springer (2009)

  40. Thierens, D.: An adaptive pursuit strategy for allocating operator probabilities. In: Beyer, H.G. (eds.) Proc. Genetic and Evolutionary Computation Conference, pp. 1539–1546. ACM (2005)

  41. Thierens, D.: Adaptive strategies for operator allocation. In: Lobo, F., Lima, C., Michalewicz, Z. (eds.): Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, vol. 54, pp. 77–90. Springer (2007)

  42. Tuson, A., Ross, P.: Adapting operator settings in genetic algorithms. Evol. Comput. 6(2), 161–184 (1998)

    Article  Google Scholar 

  43. Whitacre, J., Pham, T., Sarker, R.: Use of statistical outlier detection method in adaptive evolutionary algorithms. In: Keijzer, M. (ed.) Proc. Genetic and Evolutionary Computation Conference, pp. 1345–1352. ACM (2006)

  44. Yu, T., Davis, D., Baydar, C., Roy, R. (eds.): Evolutionary Computation in Practice. Studies in Computational Intelligence, vol. 88. Springer (2008)

  45. Yuan, B., Gallagher, M.: Statistical racing techniques for improved empirical evaluation of evolutionary algorithms. In: Yao, X., et al. (eds.) Proc. Intl. Conference on Parallel Solving from Nature. LNCS, vol. 3242, pp. 172–181. Springer (2004)

Download references

Author information

Authors and Affiliations

  1. Microsoft Research–INRIA Joint Centre, 28 rue Jean Rostand, 91893, Orsay Cedex, France

    Álvaro Fialho, Marc Schoenauer & Michèle Sebag

  2. Project-Team TAO, INRIA Saclay—Île-de-France & LRI (UMR CNRS 8623), Bât. 490, Université Paris-Sud, 91405, Orsay Cedex, France

    Luis Da Costa, Marc Schoenauer & Michèle Sebag

Authors
  1. Álvaro Fialho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis Da Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marc Schoenauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michèle Sebag
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marc Schoenauer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fialho, Á., Da Costa, L., Schoenauer, M. et al. Analyzing bandit-based adaptive operator selection mechanisms. Ann Math Artif Intell 60, 25–64 (2010). https://doi.org/10.1007/s10472-010-9213-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-010-9213-y

Keywords

Mathematics Subject Classifications (2010)

Search

Navigation