Abstract
Swarm intelligence has been successfully applied in various domains, e.g., path planning, resource allocation and data mining. Despite its wide use, a theoretical framework in which the behavior of swarm intelligence can be formally understood is still lacking. This article starts by formally deriving the evolutionary dynamics of ant colony optimization, an important swarm intelligence algorithm. We then continue to formally link these to reinforcement learning. Specifically, we show that the attained evolutionary dynamics are equivalent to the dynamics of Q-learning. Both algorithms are equivalent to a dynamical system known as the replicator dynamics in the domain of evolutionary game theory. In conclusion, the process of improvement described by the replicator dynamics appears to be a fundamental principle which drives processes in swarm intelligence, evolution, and learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Börgers, T., Sarin, R.: Learning through reinforcement and replicator dynamics. Journal of Economic Theory 77(1), 1–14 (1997)
Dorigo, M., Colorni, A., Maniezzo, V.: Positive feedback as a search strategy. Technical Report 91-016, Dipartimento di Elettronica, Politecnico di Milano, Milan, Italy (1991)
Dorigo, M., Birattari, M., Sttzle, T.: Ant colony optimization – artificial ants as a computational intelligence technique. IEEE Comput. Intell. Mag. 1, 28–39 (2006)
Dorigo, M., Maniezzo, V., Colorni, A.: The ant system: Optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics-Part B 26(1), 29–41 (1996)
Grosan, C., Abraham, A., Chis, M.: Swarm intelligence in data mining. In: Swarm Intelligence in Data Mining, pp. 1–20 (2006)
Gutjahr, W.J.: A graph-based ant system and its convergence. Future Gener. Comput. Syst. 16, 873–888 (2000)
Gutjahr, W.J.: On the finite-time dynamics of ant colony optimization. Methodology and Computing in Applied Probability 8(1), 105–133 (2006)
Hennes, D., Tuyls, K., Rauterberg, M.: State-coupled replicator dynamics. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2, AAMAS 2009, pp. 789–796. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2009)
Sigmund, K.: Evolutionary Games and Population Dynamics. Cambridge University Press, Cambridge (1998)
Kaisers, M., Tuyls, K.: Frequency adjusted multi-agent Q-learning, pp. 309–316. International Foundation for Autonomous Agents and Multiagent Systems (2010)
Kaisers, M., Tuyls, K.: FAQ-Learning in Matrix Games: Demonstrating Convergence near Nash Equilibria, and Bifurcation of Attractors in the Battle of Sexes. In: Workshop on Interactive Decision Theory and Game Theory (IDTGT 2011). Assoc. for the Advancement of Artif. Intel. (AAAI) (2011)
Kianercy, A., Galstyan, A.: Dynamics of softmax q-learning in two-player two-action games. CoRR, abs/1109.1528 (2011)
Liu, D., Wang, B.: Biological swarm intelligence based opportunistic resource allocation for wireless ad hoc networks. Wireless Personal Communications (2011)
Mahadevan, S., Connell, J., Sammut, C., Sutton, R., Temporal Phd: Automatic programming of behavior-based robots using reinforcement learning (1991)
Merkle, D., Middendorf, M.: Modeling the dynamics of ant colony optimization. Evol. Comput. 10, 235–262 (2002)
Peters, J., Schaal, S.: Reinforcement learning of motor skills with policy gradients (2008)
Smith, J.M.: Evolution and the Theory of Games. Cambridge University Press, Cambridge (1982)
Stone, P., Balch, T., Kraetzschmar, G.K. (eds.): RoboCup 2000. LNCS (LNAI), vol. 2019. Springer, Heidelberg (2001)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)
Taylor, P.D., Jonker, L.B.: Evolutionary stable strategies and game dynamics. Mathematical Biosciences 40(1-2), 145–156 (1978)
Tuyls, K., Hoen, P.J., Vanschoenwinkel, B.: An evolutionary dynamical analysis of multi-agent learning in iterated games. The Journal of Autonomous Agents and Multi-Agent Systems, JAAMAS 12(1), 115–153 (2006)
Tuyls, K., Parsons, S.: What evolutionary game theory tells us about multiagent learning. Artificial Intelligence 171(7), 406–416 (2007)
Vigorito, C.M.: Distributed path planning for mobile robots using a swarm of interacting reinforcement learners. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2007, pp. 120:1–120:8. ACM, New York (2007)
Vrancx, P., Verbeeck, K., Nowé, A.: Networks of Learning Automata and Limiting Games. In: Tuyls, K., Nowe, A., Guessoum, Z., Kudenko, D. (eds.) ALAMAS 2005, ALAMAS 2006, and ALAMAS 2007. LNCS (LNAI), vol. 4865, pp. 224–238. Springer, Heidelberg (2008)
Vrancx, P., Tuyls, K., Westra, R.L.: Switching dynamics of multi-agent learning. In: AAMAS (1), pp. 307–313 (2008)
Watkins, C.J.C.H., Dayan, P.: Technical note q-learning. Machine Learning 8, 279–292 (1992)
Zhou, J., Dai, G., He, D.-Q., Ma, J., Cai, X.-Y.: Swarm intelligence: Ant-based robot path planning. In: Proceedings of the Fifth International Conference on Information Assurance and Security, IAS 2009, Xi An, China, August 18-20, pp. 459–463. IEEE Computer Society (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bou Ammar, H., Tuyls, K., Kaisers, M. (2012). Evolutionary Dynamics of Ant Colony Optimization. In: Timm, I.J., Guttmann, C. (eds) Multiagent System Technologies. MATES 2012. Lecture Notes in Computer Science(), vol 7598. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33690-4_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-33690-4_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33689-8
Online ISBN: 978-3-642-33690-4
eBook Packages: Computer ScienceComputer Science (R0)