Abstract
We describe a framework and equations used to model and predict the behavior of multi-agent systems (MASs) with learning agents. A difference equation is used for calculating the progression of an agent's error in its decision function, thereby telling us how the agent is expected to fare in the MAS. The equation relies on parameters which capture the agent's learning abilities, such as its change rate, learning rate and retention rate, as well as relevant aspects of the MAS such as the impact that agents have on each other. We validate the framework with experimental results using reinforcement learning agents in a market system, as well as with other experimental results gathered from the AI literature. Finally, we use PAC-theory to show how to calculate bounds on the values of the learning parameters.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
W. B. Arthur, S. Durlauf, and D. Lane (ed.), The Economy as an Evolving Complex System II, Series in the Sciences of Complexity, Addison-Wesley: Reading, MA, 1997.
W. B. Arthur, J. H. Holland, B. LeBaron, R. Palmer, and P. Tayler. “Asset pricing under endogenous expectations in an artificial stock market, ” in W. B. Arthur, S. Durlauf, and D. Lane (eds.), The Economy as an Evolving Complex System II, Addison-Wesley: Reading, MA, 1997.
K. Binmore, “Modeling rational players, part 2, ” Economics and Philosophy, vol. 4, pp. 9–55, 1988.
D. Carmel and S. Markovitch, “Exploration and adaptation in multiagent systems: A model-based approach, ” in Proceedings of the International Joint Conference on AI, 1997.
A. Chavez and P. M. Kasbah, “An agent marketplace for buying and selling goods, ” in First International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, pp. 75–90, 1996.
C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems, ” in Proceedings of Workshop on Multiagent Learning, AAAI Press, 1997.
G. A. Cowan, D. Pines, and D. Meltzer (eds.), Complexity: Metaphors, Models and Reality, Addison-Wesley: Reading, MA, 1995.
V. M. Darley and S. Kauffman, “Natural rationality, ” in W. B. Arthur, S. Durlaf, and D. Lane (eds.), The Economy as an Evolving Complex System II, Addison-Wesley: Reading, MA, 1997.
E. H. Durfee, D. L. Kiskis, and W. P. Birmingham, “The agent architecture of the University of Michigan Digital Library, ” IEE Proceedings on Software Engineering, vol. 144, no.1, pp. 61–71, 1997.
O. Etzioni, “Moving up the information food chain: Deploying softbots on the world wide web, ” in Proceedings of the Thirteenth National Conference on Artificial Intelligence and the Eighth Innovative Applications of Artificial Intelligence Conference, Menlo Park: AAAI Press, MIT Press, pp. 1322–1326, 1996.
J. H. Holland, Hidden Order, Addison-Wesley, 1995.
J. Hu and M. P. Wellman, “Self-fulfilling bias in multiagent learning, ” in Proceedings of the Second International Conference on Multi-Agent Systems, pp. 118–125, 1996.
J. Hu and M. P. Wellman, “Multiagent reinforcement learning: Theoretical framework and an algorithm, ” in Proceedings of the Fifteenth International Conference on Machine Learning, 1998.
J. Hu and M. P. Wellman, “Online learning about other agents in a dynamic multiagent system, ” in Proceedings of the Second International Conference on Autonomous Agents, 1998.
A. Hübler and D. Pines, “Prediction and Adaptation in an Evolving Chaotic Environment, ” in G. A. Cowan, D. Pines, and D. Meltzer (eds.), Complexity: Methaphors, Models and Reality, pp. 343–379, 1994.
T. Ishida, Real-Time Search for Learning Autonomous Agents, Kluwer Academic Publishers, 1997.
M. J. Kearns and U. V. Vazirani, An Introduction to Computational Learning Theory, The MIT Press: Cambridge, MA, 1994.
J. O. Kephart, T. Hogg, and B. A. Huberman, “Collective behavior of predictive agents, ” Physica D, vol. 42, no.2, pp. 48–65, 1990.
V. R. Lesser and D. D. Corkill, “Functionally accurate, cooperative distributed systems, ” IEEE Transactions on Systems, Man, and Cybernetics, vol. 11, no.1, pp. 81–96, 1981.
M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning, ” in Proceedings of the Eleventh International Conference on Machine Learning, Morgan Kaufmann, pp. 157–163, 1994.
J. S. Liu and K. P. Sycara, “Multiagent coordination in tightly coupled real-time environments, ” in Victor Lesser (ed.), Proceedings of the First International Conference on Multi-Agent Systems, MIT Press, 1995.
M. J. Matarić, “Issues and approaches in the design of collective autonomous agents, ” Robotics and Autonomous Systems, vol. 16, nos.2–4, pp. 321–331, 1995.
M. J. Matarić, “Learning social behaviors, ” Robotics and Autonomous Agents, vol. 20, pp. 191–204, 1997.
T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.
J. S. Rosenschein and G. Zlotkin, Rules of Encounter, The MIT Press: Cambridge, MA, 1994.
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, Prentice Hall, 1995.
S. Sen (ed.), Working Notes from the AAAI Symposium on Adaptation, Co-evolution and Learning in Multiagent Systems, 1996.
S. Sen and M. Sekaran, “Individual learning of coordination knowledge, ” Journal of Experimental and Theoretical AI, pp. 156–170, 1998.
S. Sen, M. Sekaran, and J. Hale, “Learning to coordinate without sharing information, ” in Proceedings of the Twelfth National Conference on Artificial Intelligence, 1994.
Y. Shoham and M. Tennenholtz, “On the emergence of social conventions: modeling, analysis, and simulations, ” Artificial Intelligence, vol. 94, no.1, pp. 139–166, 1997.
P. Stone and M. Velos, “Multiagent systems: A survery from a machine learning perspective, ” Technical Report CMU-CS–97–193, Carnegie Mellon University, December 1997.
P. Stone and M. Veloso, “Team-partitioned, opaque-transition reinforcement learning, ” in Proceedings of the Third International Conference on Autonomous Agents, Springer-Verlag, 1999.
J. M. Vidal, “Computational agents that learn about agents: Algorithms for their design and a predictive theory of their behavior, ” Ph.D. thesis, University of Michigan, 1998.
J. M. Vidal and E. H. Durfee, “Learning nested models in an information economy, ” Journal of Experimental and Theoretical Artificial Intelligence, vol. 10, no.3, pp. 291–308, 1998.
C. J. Watkins and P. Dayan, “Q-learning, ” Machine Learning, vol. 8, pp. 279–292, 1992.
G. Weiß (ed.), Distributed Artificial Intelligence Meets Machine Learning, Springer, 1997.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Vidal, J.M., Durfee, E.H. Predicting the Expected Behavior of Agents that Learn About Agents: The CLRI Framework. Autonomous Agents and Multi-Agent Systems 6, 77–107 (2003). https://doi.org/10.1023/A:1021765422660
Issue Date:
DOI: https://doi.org/10.1023/A:1021765422660