Skip to main content
SpringerLink
Log in

Multi-agent Behavior-Based Policy Transfer

  • Conference paper
  • First Online:
Applications of Evolutionary Computation (EvoApplications 2016)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 9598))

Included in the following conference series:

Abstract

A key objective of transfer learning is to improve and speed-up learning on a target task after training on a different, but related, source task. This study presents a neuro-evolution method that transfers evolved policies within multi-agent tasks of varying degrees of complexity. The method incorporates behavioral diversity (novelty) search as a means to boost the task performance of transferred policies (multi-agent behaviors). Results indicate that transferred evolved multi-agent behaviors are significantly improved in more complex tasks when adapted using behavioral diversity. Comparatively, behaviors that do not use behavioral diversity to further adapt transferred behaviors, perform relatively poorly in terms of adaptation times and quality of solutions in target tasks. Also, in support of previous work, both policy transfer methods (with and without behavioral diversity adaptation), out-perform behaviors evolved in target tasks without transfer learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Transfer learning and policy transfer are used interchangeably in this paper.

  2. 2.

    All experiments were run in RoboCup Keep-Away version 6 [6]. Source code and executables can be found at: http://people.cs.uct.ac.za/~gnitschke/EvoStar2016/.

  3. 3.

    NEAT and HyperNEAT average maximum task performance progression graphs can be found at: http://people.cs.uct.ac.za/~gnitschke/EvoStar2016/.

References

  1. Pan, S., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  2. Torrey, L., Shavlik, J.: Transfer learning. In: Olivas, E.S. (ed.) Handbook of Research on Machine Learning Applications, pp. 17–23. IGI Global, Hershey (2009)

    Google Scholar 

  3. Ammar, H., Tuyls, K., Taylor, M., Driessens, K., Weiss, G.: Reinforcement learning transfer via sparse coding. In: Proceedings of the Eleventh International Conference on Autonomous Agents and Multiagent Systems, Valencia, Spain, pp. 4–8. AAAI (2012)

    Google Scholar 

  4. Ramon, J., Driessens, K., Croonenborghs, T.: Transfer learning in reinforcement learning problems through partial policy recycling. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 699–707. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  5. Boutsioukis, G., Partalas, I., Vlahavas, I.: Transfer learning in multi-agent reinforcement learning domains. In: Sanner, S., Hutter, M. (eds.) EWRL 2011. LNCS, vol. 7188, pp. 249–260. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  6. Taylor, M., Stone, P., Liu, Y.: Transfer learning via inter-task mappings for temporal difference learning. J. Mach. Learn. 8(1), 2125–2167 (2010)

    MathSciNet  MATH  Google Scholar 

  7. Stone, P., Kuhlmann, G., Taylor, M.E., Liu, Y.: Keepaway soccer: from machine learning testbed to benchmark. In: Bredenfeld, A., Jacoff, A., Noda, I., Takahashi, Y. (eds.) RoboCup 2005. LNCS (LNAI), vol. 4020, pp. 93–105. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  8. Doncleux, S.: Knowledge extraction from learning traces in continuous domains. In: AAAI 2014 Fall Symposium on Knowledge, Skill, and Behavior Transfer in Autonomous Robots, Arlington, USA, pp. 1–8. AAAI Press (2014)

    Google Scholar 

  9. Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: from architectures to learning. Evol. Intel. 1(1), 47–62 (2008)

    Article  Google Scholar 

  10. Moshaiov, A., Tal, A.: Family bootstrapping: a genetic transfer learning approach for onsetting the evolution for a set of realated robotic tasks. In: Proceedings of the Congress on Evolutionary Computation, pp. 2801–2808. IEEE Press (2014)

    Google Scholar 

  11. Deb, K.: Pareto Based Multi-objectives Optimization Using Evolutionary Algorithms. Wiley, New York (2001)

    MATH  Google Scholar 

  12. Taylor, M., Whiteson, S., Stone, P.: Transfer learning for policy search methods. In: ICML 2006: Proceedings of the Twenty-Third International Conference on Machine Learning Transfer Learning Workshop, Pittsburgh, USA, pp. 1–4. ACM (2006)

    Google Scholar 

  13. Stanley, K., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  14. Verbancsics, P., Stanley, K.: Evolving static representations for task transfer. J. Mach. Learn. Res. 11(1), 1737–1763 (2010)

    MathSciNet  MATH  Google Scholar 

  15. Stanley, K., D’Ambrosio, D., Gauci, J.: A hypercube-based indirect encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  16. Stone, P., Sutton, R., Kuhlmann, G.: Reinforcement learning for robocup-soccer keepaway. Adapt. Behav. 13(3), 165–188 (2006)

    Article  Google Scholar 

  17. Whiteson, S., Stone, P.: Evolutionary function approximation for reinforcement learning. J. Mach. Learn. Res. 7(1), 877–917 (2006)

    MathSciNet  MATH  Google Scholar 

  18. Bahceci, E., Miikkulainen, R.: Transfer of evolved pattern-based heuristics in games. In: Proceedings of the IEEE Symposium on Computational Intelligence and Games, Perth, Australia, pp. 220–227. Morgan Kaufmann (2008)

    Google Scholar 

  19. Lehman, J., Stanley, K.: Abandoning objectives: evolution through the search for novelty alone. Evol. Comput. 19(2), 189–223 (2011)

    Article  Google Scholar 

  20. Mouret, J., Doncieux, S.: Encouraging behavioral diversity in evolutionary robotics: an empirical study. Evol. Comput. 20(1), 91–133 (2012)

    Article  Google Scholar 

  21. Gomes, J., Mariano, P., Christensen, A.: Avoiding convergence in cooperative coevolution with novelty search. In: Proceedings of the International Conference on Autonomous Agents and Multi-agent Systems, pp. 1149–1156. ACM (2014)

    Google Scholar 

  22. Gomes, J., Mariano, P., Christensen, A.: Devising effective novelty search algorithms: a comprehensive empirical study. In: Proceedings of the Genetic Evolutionary Computation Conference, Madrid, Spain, pp. 943–950. ACM (2015)

    Google Scholar 

  23. Degrave, J., Burm, M., Kindermans, P., Dambre, J., Wyffels, F.: Transfer learning of gaits on a quadrupedal robot. Adapt. Behav. 23, 9–19 (2015)

    Article  Google Scholar 

  24. Knudson, M., Tumer, K.: Policy transfer in mobile robots using neuro-evolutionary navigation. In: Proceedings of the Genetic and Evolutionary Computation Conference, Philadelphia, USA, pp. 1411–1412. ACM Press (2012)

    Google Scholar 

  25. Stanley, K.: Compositional pattern producing networks: a novel abstraction of development. Genet. Program Evolvable Mach. 8(2), 131–162 (2007)

    Article  Google Scholar 

  26. D’Ambrosio, D., Stanley, K.: Scalable multiagent learning through indirect encoding of policy geometry. Evol. Intell. J. 6(1), 1–26 (2013)

    Article  Google Scholar 

  27. Gauci, J., Stanley, K.: A case study on the critical role of geometric regularity in machine learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, Menlo Park, USA, pp. 628–633. AAAI Press (2008)

    Google Scholar 

  28. Risi, S., Stanley, K.: Confronting the challenge of learning a flexible neural controller for a diversity of morphologies. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 255–261. ACM (2013)

    Google Scholar 

  29. Gomes, J., Christensen, A.: Generic behavior similarity measures for evolutionary swarm robotics. In: Proceedings of the Genetic and Evolutionary Computation Conference, Amsterdam, The Netherlands, pp. 199–206. ACM Press (2013)

    Google Scholar 

  30. Urbano, P., Georgiou, L.: Improving grammatical evolution in santa fe trail using novelty search. In: Proceedings of the 12th European Conference on Artificial Life, Taormina, Italy, pp. 917–924. MIT Press (2013)

    Google Scholar 

  31. Lehman, J., Stanley, K.: Efficiently evolving programs through the search for novelty. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, Portland, USA, pp. 837–844. ACM (2010)

    Google Scholar 

  32. Velez, R., Clune, J.: Novelty search creates robots with general skills for exploration. In: Proceedings of the Genetic and Evolutionary Computation Conference, Vancouver, Canada, pp. 737–744. ACM (2014)

    Google Scholar 

  33. Cuccu, G., Gomez, F., Glasmachers, T.: Novelty-based restarts for evolution strategies. In: Proceedings of the Congress on Evolutionary Computation, New Orleans, USA, pp. 158–163. IEEE Press (2011)

    Google Scholar 

  34. Gomes, J., Urbano, P., Christensen, A.L.: Progressive minimal criteria novelty search. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds.) IBERAMIA 2012. LNCS, vol. 7637, pp. 281–290. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  35. Lehman, J., Stanley, K.: Revising the evolutionary computation abstraction: minimal criteria novelty search. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 103–110. ACM (2010)

    Google Scholar 

  36. Liapis, A., Yannakakis, G., Togelius, J.: Constrained novelty search: a study on game content generation. Evol. Comput. 23(1), 101–129 (2015)

    Article  Google Scholar 

  37. Ghasemi, A., Zahediasl, S.: Normality tests for statistical analysis: a guide for non-statisticians. Int. J. Endocrinol. Metab. 10(2), 486–489 (2012)

    Article  Google Scholar 

  38. Flannery, B., Teukolsky, S., Vetterling, W.: Numerical Recipes. Cambridge University Press, Cambridge (1986)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Cape Town, Rondebosch, Cape Town, 7700, South Africa

    Sabre Didi & Geoff Nitschke

Authors
  1. Sabre Didi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geoff Nitschke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sabre Didi .

Editor information

Editors and Affiliations

  1. Politecnico di Torino, Turin, Italy

    Giovanni Squillero

  2. Aalborg University, Copenhagen, Denmark

    Paolo Burelli

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Didi, S., Nitschke, G. (2016). Multi-agent Behavior-Based Policy Transfer. In: Squillero, G., Burelli, P. (eds) Applications of Evolutionary Computation. EvoApplications 2016. Lecture Notes in Computer Science(), vol 9598. Springer, Cham. https://doi.org/10.1007/978-3-319-31153-1_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-31153-1_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-31152-4

  • Online ISBN: 978-3-319-31153-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation