Skip to main content
SpringerLink
Log in

On measuring the accuracy of SLAM algorithms

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

In this paper, we address the problem of creating an objective benchmark for evaluating SLAM approaches. We propose a framework for analyzing the results of a SLAM approach based on a metric for measuring the error of the corrected trajectory. This metric uses only relative relations between poses and does not rely on a global reference frame. This overcomes serious shortcomings of approaches using a global reference frame to compute the error. Our method furthermore allows us to compare SLAM approaches that use different estimation techniques or different sensor modalities since all computations are made based on the corrected trajectory of the robot.

We provide sets of relative relations needed to compute our metric for an extensive set of datasets frequently used in the robotics community. The relations have been obtained by manually matching laser-range observations to avoid the errors caused by matching algorithms. Our benchmark framework allows the user to easily analyze and objectively compare different SLAM approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Amigoni, F., Gasparini, S., &  Gini, M. (2007). Good experimental methodologies for robotic mapping: A proposal. In Proc. of the IEEE int. conf. on robotics & automation (ICRA).

  • Balaguer, B., Carpin, S., & Balakirsky, S. (2007). Towards quantitative comparisons of robot algorithms: Experiences with SLAM in simulation and real world systems. In IROS 2007 workshop.

  • Bar-Shalom, Y., Li, X. R., & Kirubarajan, T. (2001). Estimation with application to tracking and navigation. New York: Wiley.

    Book  Google Scholar 

  • Bonarini, A., Burgard, W., Fontana, G., Matteucci, M., Sorrenti, D. G., & Tardos, J. D. (2006). Rawseeds a project on SLAM benchmarking. In Proceedings of the IROS’06 workshop on benchmarks in robotics research. Available online at http://www.robot.uji.es/EURON/pdfs/LectureNotesIROS06.pdf.

  • Bosse, M., Newman, P. M., Leonard, J. J., & Teller, S. (2003). An ALTAS framework for scalable mapping. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 1899–1906). Taipei, Taiwan.

  • Burgard, W., Stachniss, C., Grisetti, G., Steder, B., Kümmerle, R., Dornhege, C., Ruhnke, M., Kleiner, A., & Tardós, J. D. (2009, to appear). A comparison of slam algorithms based on a graph of relations. In Proc. of the int. conf. on intelligent robots and systems (IROS).

  • Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6), 679–698.

    Article  Google Scholar 

  • Censi, A. (2006). Scan matching in a probabilistic framework. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2291–2296).

  • Darpa (2007). Darpa Urban Challenge. http://www.darpa.mil/grandchallenge/.

  • Dellaert, F. (2005). Square Root SAM. In Proc. of robotics: science and systems (RSS) (pp. 177–184). Cambridge, MA, USA.

  • Dellaert, F., Fox, D., Burgard, W., & Thrun, S. (1998). Monte Carlo localization for mobile robots. In Proc. of the IEEE int. conf. on robotics & automation (ICRA). Leuven, Belgium.

  • Dissanayake, G., Durrant-Whyte, H., & Bailey, T. (2000). A computationally efficient solution to the simultaneous localisation and map building (SLAM) problem. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 1009–1014).

  • Doucet, A., de Freitas, N., & Gordan, N. (Eds.) (2001). Sequential Monte-Carlo methods in practice. Berlin: Springer.

    MATH  Google Scholar 

  • Duckett, T., Marsland, S., & Shapiro, J. (2002). Fast, on-line learning of globally consistent maps. Autonomous Robots, 12(3), 287–300.

    Article  MATH  Google Scholar 

  • EPFL and IROS (2002). Cleaning Robot Contest. http://robotika.cz/competitions/cleaning2002/en.

  • ESA (2008). Lunar robotics challenge. http://www.esa.int/esaCP/SEM4GKRTKMF_index_0.html.

  • Estrada, C., Neira, J., & Tardós, J. D. (2005). Hierarchical SLAM: Real-time accurate mapping of large environments. IEEE Transactions on Robotics, 21(4), 588–596.

    Article  Google Scholar 

  • Eustice, R., Singh, H., & Leonard, J. J. (2005a). Exactly sparse delayed-state filters. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2428–2435).

  • Eustice, R., Walter, M., & Leonard, J. J. (2005b). Sparse extended information filters: Insights into sparsification. In Proc. of the int. conf. on intelligent robots and systems (IROS) (pp. 641–648). Edmonton, Canada.

  • Frese, U. (2006). Treemap: An o(log n) algorithm for indoor simultaneous localization and mapping. Autonomous Robots, 21(2), 103–122.

    Article  Google Scholar 

  • Frese, U. (2008). Dlr spatial cognition data set. http://www.informatik.uni-bremen.de/agebv/en/DlrSpatialCognitionDataSet.

  • Frese, U., Larsson, P., & Duckett, T. (2005). A multilevel relaxation algorithm for simultaneous localisation and mapping. IEEE Transactions on Robotics, 21(2), 1–12.

    Article  Google Scholar 

  • Früh, C., & Zakhor, A. (2004). An automated method for large-scale, ground-based city model acquisition. International Journal of Computer Vision, 60, 5–24.

    Article  Google Scholar 

  • Grisetti, G., Grzonka, S., Stachniss, C., Pfaff, P., & Burgard, W. (2007a). Efficient estimation of accurate maximum likelihood maps in 3D. In Proc. of the int. conf. on intelligent robots and systems (IROS). San Diego, CA, USA.

  • Grisetti, G., Stachniss, C., & Burgard, W. (2007b). Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Transactions on Robotics, 23, 34–46.

    Article  Google Scholar 

  • Grisetti, G., Stachniss, C., Grzonka, S., & Burgard, W. (2007c). A tree parameterization for efficiently computing maximum likelihood maps using gradient descent. In Proc. of robotics: science and systems (RSS).

  • Gutmann, J.-S., & Konolige, K. (1999). Incremental mapping of large cyclic environments. In Proc. of the IEEE int. symposium on computational intelligence in robotics and automation (CIRA).

  • Hähnel, D., Burgard, W., Fox, D., & Thrun, S. (2003). An efficient FastSLAM algorithm for generating maps of large-scale cyclic environments from raw laser range measurements. In Proc. of the int. conf. on intelligent robots and systems (IROS) (pp. 206–211).

  • Hermosillo, J., Pradalier, C., Sekhavat, S., Laugier, C., & Baille, G. (2003). Towards motion autonomy of a bi-steerable car: Experimental issues from map-building to trajectory execution. In Proc. of the IEEE int. conf. on robotics & automation (ICRA).

  • Hoover, A., Jean-Baptiste, G., Jiang, X., Flynn, P. J., Bunke, H., Goldgof, D. B., Bowyer, K. K., Eggert, D. W., Fitzgibbon, A. W., & Fisher, R. B. (1996). An experimental comparison of range image segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(7), 673–689.

    Article  Google Scholar 

  • Howard, A., & Roy, N. (2003). Radish: The robotics data set repository, standard data sets for the robotics community. http://radish.sourceforge.net/.

  • Julier, S., Uhlmann, J., & Durrant-Whyte, H. (1995). A new approach for filtering nonlinear systems. In Proc. of the American control conference (pp. 1628–1632).

  • Kaess, M., Ranganathan, A., & Dellaert, F. (2007). iSAM: Fast incremental smoothing and mapping with efficient data association. In Proc. of the IEEE int. conf. on robotics & automation (ICRA).

  • Kümmerle, R., Steder, B., Dornhege, C., Kleiner, A., Grisetti, G., & Burgard, W. (2009). Large scale graph-based SLAM using aerial images as prior information. In Proc. of robotics: science and systems (RSS).

  • Leonard, J. J., & Durrant-Whyte, H. F. (1991). Mobile robot localization by tracking geometric beacons. IEEE Transactions on Robotics and Automation, 7(4), 376–382.

    Article  Google Scholar 

  • Lu, F., & Milios, E. (1994). Robot pose estimation in unknown environments by matching 2d range scans. In IEEE computer vision and pattern recognition conference (CVPR) (pp. 935–938).

  • Lu, F., & Milios, E. (1997). Globally consistent range scan alignment for environment mapping. Autonomous Robots, 4, 333–349.

    Article  Google Scholar 

  • Montemerlo, M., Thrun, S., Koller, D., & Wegbreit, B. (2003). FastSLAM 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. In Proc. of the int. conf. on artificial intelligence (IJCAI) (pp. 1151–1156).

  • Nüchter, A., Lingemann, K., Hertzberg, J., & Surmann, H. (2005). 6d SLAM with approximate data association. In Proc. of the 12th int. conference on advanced robotics (ICAR) (pp. 242–249).

  • Olson, E. (2008). Robust and efficient robotic mapping. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA.

  • Olson, E., Leonard, J., & Teller, S. (2006). Fast iterative optimization of pose graphs with poor initial estimates. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2262–2269).

  • Pfaff, P., Burgard, W., & Fox, D. (2006). Robust Monte-Carlo localization using adaptive likelihood models. In H.I. Christiensen (Ed.), STAR Springer tracts in advanced robotics : Vol. 22. European robotics symposium 2006 (pp. 181–194). Berlin: Springer.

    Chapter  Google Scholar 

  • Ranganathan, A., Kaess, M., & Dellaert, F. (2007). Loopy sam. In Proc. of the int. conf. on artificial intelligence (IJCAI).

  • RoboCup Federation (2009). RoboCup Competitions. http://www.robocup.org.

  • Scharstein, D., & Szeliski, R. (2002). Middlebury stereo vision page. http://www.middlebury.edu/stereo.

  • Smith, R. C., & Cheeseman, P. (1986). On the representation and estimation of spatial uncertainty. International Journal of Robotics Research, 5(4), 56–68.

    Article  Google Scholar 

  • Smith, R., Self, M., & Cheeseman, P. (1990). Estimating uncertain spatial relationships in robotics. In I. Cox & G. Wilfong (Eds.), Autonomous robot vehicles (pp. 167–193). Berlin: Springer.

    Google Scholar 

  • Stachniss, C., Frese, U., & Grisetti, G. (2007a). OpenSLAM. org—give your algorithm to the community. http://www.openslam.org.

  • Stachniss, C., Grisetti, G., Roy, N., & Burgard, W. (2007b). Evaluation of Gaussian proposal distributions for mapping with Rao-Blackwellized particle filters. In Proc. of the int. conf. on intelligent robots and systems (IROS).

  • Symeo GmbH (2008). http://www.symeo.de.

  • Thrun, S. (2001). An online mapping algorithm for teams of mobile robots. International Journal of Robotics Research, 20(5), 335–363.

    Article  Google Scholar 

  • Thrun, S., Liu, Y., Koller, D., Ng, A. Y., Ghahramani, Z., & Durrant-Whyte, H. (2004). Simultaneous localization and mapping with sparse extended information filters. International Journal of Robotics Research, 23(7/8), 693–716.

    Google Scholar 

  • Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge: MIT Press.

    MATH  Google Scholar 

  • Thrun, S. et al. (2006). Winning the Darpa grand challenge. Journal on Field Robotics.

  • Torralba, A., Murphy, K. P., & Freeman, W. T. (2007). Labelme: the open annotation tool. http://labelme.csail.mit.edu/.

  • Uhlmann, J. (1995). Dynamic map building and localization: New theoretical foundations. PhD thesis, University of Oxford.

  • Wulf, O., Nüchter, A., Hertzberg, J., & Wagner, B. (2008). Benchmarking urban six-degree-of-freedom simultaneous localization and mapping. Journal of Field Robotics, 25(3), 148–163.

    Article  Google Scholar 

  • Yguel, M., Keat, C. T. M., Braillon, C., Laugier, C., & Aycard, O. (2007). Dense mapping for range sensors: Efficient algorithms and sparse representations. In Proc. of robotics: science and systems (RSS).

Download references

Author information

Authors and Affiliations

  1. Dept. of Computer Science, University of Freiburg, Georges Köhler Allee 79, 79110, Freiburg, Germany

    Rainer Kümmerle, Bastian Steder, Michael Ruhnke, Giorgio Grisetti & Cyrill Stachniss

  2. Dept. of Computer Science, University of Freiburg, Georges Köhler Allee 52, 79110, Freiburg, Germany

    Christian Dornhege & Alexander Kleiner

Authors
  1. Rainer Kümmerle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bastian Steder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Dornhege
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Ruhnke
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Giorgio Grisetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cyrill Stachniss
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Alexander Kleiner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rainer Kümmerle.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kümmerle, R., Steder, B., Dornhege, C. et al. On measuring the accuracy of SLAM algorithms. Auton Robot 27, 387–407 (2009). https://doi.org/10.1007/s10514-009-9155-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-009-9155-6

Keywords

Search

Navigation