Abstract
This paper investigates the learning of both low-level behaviors for humanoid robot controllers and of high-level coordination strategies for teams of robots engaged in simulated soccer. Regarding controllers, current approaches typically hand-tune behaviors or optimize them without realistic constraints, for example allowing parts of the robot to intersect with others. This level of optimization often leads to low-performance behaviors. Regarding strategies, most are hand-tuned with arbitrary parameters (like agents moving to pre-defined positions on the field such that eventually they can score a goal) and the thorough analysis of learned strategies is often disregarded. This paper demonstrates how it is possible to use a distributed framework to learn both low-level behaviors, like sprinting and getting up, and high-level strategies, like a kick-off scenario, outperforming previous approaches in the FCPortugal3D Simulated Soccer team.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdolmaleki, A., Lau, N., Reis, L.P., Neumann, G.: Regularized covariance estimation for weighted maximum likelihood policy search methods. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 154–159. IEEE (2015)
Abdolmaleki, A., Simões, D., Lau, N., Reis, L.P., Neumann, G.: Learning a humanoid kick with controlled distance. In: Robot World Cup, pp. 45–57. Springer, Hedidelberg (2016)
Cruz, L., Reis, L.P., Lau, N., Sousa, A.: Optimization approach for the development of humanoid robots’ behaviors. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds.) Advances in Artificial Intelligence - IBERAMIA 2012, pp. 491–500. Springer, Heidelberg (2012)
Federation, R.: Robocup simulation 3d league rules. https://ssim.robocup.org/wp-content/uploads/2018/12/Rules_RoboCupSim3D2018.pdf. Accessed 21 Apr 2019
Foerster, J.N., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. CoRR abs/1705.08926 (2017)
Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)
Hansen, N.: The CMA evolution strategy: a tutorial. arXiv preprint arXiv:1604.00772 (2016)
Kasaei, S.M., Simões, D., Lau, N., Pereira, A.: A hybrid zmp-cpg based walk engine for biped robots. In: Iberian Robotics Conference, pp. 743–755. Springer, Heidelberg (2017)
Kupcsik, A., Deisenroth, M., Peters, J., Neumann, G.: Data-efficient contextual policy search for robot movement skills. In: Proceedings of the National Conference on Artificial Intelligence (AAAI) (2013)
Lau, N., Reis, L.P., Shafii, N., Ferreira, R., Abdolmaleki, A.: FC Portugal 3D simulation team: team description paper. RoboCup 2013 (2013)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. CoRR abs/1706.02275 (2017)
MacAlpine, P., Stone, P.: Overlapping layered learning. Artif. Intell. 254, 21–43 (2018)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814 (2010)
Peng, P., Yuan, Q., Wen, Y., Yang, Y., Tang, Z., Long, H., Wang, J.: Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. CoRR abs/1703.10069 (2017)
Picado, H., Gestal, M., Lau, N., Reis, L.P., Tomé, A.M.: Automatic generation of biped walk behavior using genetic algorithms. In: Cabestany, J., Sandoval, F., Prieto, A., Corchado, J.M. (eds.) Bio-Inspired Systems: Computational and Ambient Intelligence, pp. 805–812. Springer, Heidelberg (2009)
Rückstieß, T., Felder, M., Schmidhuber, J.: State-dependent exploration for policy gradient methods. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 234–249. Springer, Heidelberg (2008)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Simões, D., Lau, N., Reis, L.P.: Multi-agent neural reinforcement-learning system with communication. In: World Conference on Information Systems and Technologies, pp. 3–12. Springer, Heidelberg (2019)
Stone, P., Veloso, M.: Layered learning and flexible teamwork in RoboCup simulation agents. In: Robot Soccer World Cup, pp. 495–508. Springer, Heidelberg (1999)
Stulp, F., Sigaud, O.: Path integral policy improvement with covariance matrix adaptation. arXiv preprint arXiv:1206.4621 (2012)
Sukhbaatar, S., Szlam, A., Fergus, R.: Learning multiagent communication with backpropagation. CoRR abs/1605.07736 (2016)
Sun, Y., Wierstra, D., Schaul, T., Schmidhuber, J.: Efficient natural evolution strategies. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 539–546. ACM (2009)
Theodorou, E., Buchli, J., Schaal, S.: A generalized path integral control approach to reinforcement learning. J. Mach. Learn. Res. 11(Nov), 3137–3181 (2010)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Acknowledgements
The first author is supported by FCT (Portuguese Foundation for Science and Technology) under grant PD/BD/113963/2015. This research was partially supported by LIACC (UID/CEC/00027/2019) and IEETA (UID/CEC/00127/2019).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Simões, D., Amaro, P., Silva, T., Lau, N., Reis, L.P. (2020). Learning Low-Level Behaviors and High-Level Strategies in Humanoid Soccer. In: Silva, M., Luís Lima, J., Reis, L., Sanfeliu, A., Tardioli, D. (eds) Robot 2019: Fourth Iberian Robotics Conference. ROBOT 2019. Advances in Intelligent Systems and Computing, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-030-36150-1_44
Download citation
DOI: https://doi.org/10.1007/978-3-030-36150-1_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36149-5
Online ISBN: 978-3-030-36150-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)