Abstract
This chapter introduces benchmarking as a special form of evaluation and addresses problems and demands concerning the benchmarking of multiagent systems. It gives an overview of the evaluation concepts used in the German research program SPP 1083 for intelligent agents and realistic commercial application scenarios as well as examples for evaluation and benchmark studies for multiagent systems. The article provides basics for setting-up evaluation studies, regarding special concerns for the evaluation of multiagent systems. Moreover, the exemplary overview may serve as orientation for further evaluation and benchmarking of multiagent systems in realistic and commercial application scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Becker, J.; Schütte, R.: Handels-Informationssysteme. 2. Auflage, Frankfurt am Main, 2004.
Bixby, B.; Reinelt, G.: TSPLIB, Softwarelibrary. Rice University, Houston, Texas, 1990.
Brennan, R. W.; O, W.: A simulation test-bed to evaluate multiagent control of manufacturing systems. In: Joines, J. A.; Barton, R. R.; Kang, K.; Fishwick, P. A. (Eds.): Proceedings of the 2000 Winter Simulation Conference, 2000.
Camp, R. C.: Benchmarking — The Search for Industry Best Practices that Lead to Superior Performance. Quality Press, Milwaukee, 1989.
Camp, R. C.: Business Process Benchmarking. ASQC Press, Milwaukee, 1995.
Cavalieri, S.; Bongaerts, L.; Taisch, M.; Macchi, M.; Wyns, J.: A benchmark framework for manufacturing control. In: Proceedings of the Second International Workshop on Intelligent Manufacturing Systems. Leuven, 1999, pp. 22–24.
Demirkol, E.; Mehta, S.; Uzsoy, R.: Benchmarks for shop scheduling problems. European Journal of Operational Research 109(1998), pp. 137–141.
Dix, A. J.: Human-computer interaction. 2nd ed., Prentice Hall, Paramus, 1998.
Durfee, E. H.: Scaling Up Agent Coordination Strategies. In: Computer 34(2001)7, pp. 39–46.
Gappmair, M.; Häntschel, I.: Die Evaluierung von Workflow-Management-Systemen in Laborstudien. In: Grün, O.; Heinrich, L. J.: Wirtschaftsinformatik — Ergebnisse Empirischer Forschung. Berlin et al., 1997, pp. 63–77.
Hanks, S.; Pollack, M. E.; Cohen, P. R.: Benchmarks, testbeds, controlled experimentation, and the design of agent architectures. In: AI Magazine 14(1993)4, pp. 17–42.
Heinrich, L. J.; Heinzl, A.; Roithmayr, F.: Wirtschaftsinformatik-Lexikon. München, Wien, 2004.
Heinrich, L. J.: Wirtschaftsinformatik — Einführung und Grundlegung. München, Wien, 1993.
Heinrich, L. J.: Informationsmanagement. 7. Auflage, München, Wien, 2002.
Herzwurm, G.; Mellis, W.: Benchmarking der Kundenorientierung in Softwareunternehmen. In: BFuP — Betriebswirtschaftliche Forschung und Praxis 4(1998), pp. 438–450.
Helsinger, A.; Lazarus, R.; Wright, W.; Zinky, J.: Tools and techniques for performance measurement of large distributed multiagent systems. In: Proceedings of the second international joint conference on Autonomous agents and multiagent systems. Melbourne, Australia, 2003, pp. 843–850.
Howe, A. E.; Dahlman, E.: A Critical Assessment of Benchmark Comparison in Planning. In: Journal of Artificial Intelligence Research 17(2002), pp. 1–33.
Korba, L.; Song, R.: The Scalability of a Multi-Agent System in Security Services. In: NRC/ERB-1098, NRC 44952, 2002.
Legner, C.: Benchmarking informationssystem-gestützter Geschäftsprozesse: Methode und Anwendung. Wiesbaden, 1999.
Lee, L. C.; Nwana, H. S.; Ndumu, D. T.; De Wilde, P.: The Stability, Scalability and Performance of Multi-agent systems. In: BT Technology Journal 16(1998)3, pp 94–103.
Marc, M. A.; Greer, J. E.: Evaluation methodologies for intelligent tutoring systems. In: Journal of Artificial Intelligence in Education 4(1993)3.
Rardin, R. L.; Uzsoy, R.: Experimental evaluation of heuristic optimization algorithms: A tutorial. In: Journal of Heuristics 7(2001)3, pp. 261–304.
Rechenberg, P.; Pomberger, G.: Informatik-Handbuch. München, Wien, 1997.
Scriven, M.: The Logic of Evaluation. California, 1980.
Taillard, E.: Benchmarks for basic scheduling problems. In: European Journal of Operations Research 64(1992), pp 278–285.
Teubner, A.; Rentmeister, J.; Klein, S.: IT-Fitness für das 21. Jahrhundert — Konzeption eines Evaluationsinstruments. In: Heinrich, L. J.; Häntschel, I. (Eds.): Evaluation und Evaluationsforschung in der Wirtschaftsinformatik. München, Wien, 2000.
Töpfer, A.; Mann, A.: Benchmarking: Lernen von den Besten. In: Töpfer, A. (Ed.): Benchmarking — Der Weg zur Best Practice. Springer, Belin et al., 1997, pp. 31–75.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer Berlin · Heidelberg
About this chapter
Cite this chapter
Zöller, A., Rothlauf, F., Paulussen, T.O., Heinzl, A. (2006). Benchmarking of Multiagent Systems. In: Kirn, S., Herzog, O., Lockemann, P., Spaniol, O. (eds) Multiagent Engineering. International Handbooks on Information Systems. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-32062-8_26
Download citation
DOI: https://doi.org/10.1007/3-540-32062-8_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-31406-6
Online ISBN: 978-3-540-32062-3
eBook Packages: Computer ScienceComputer Science (R0)