Skip to main content
SpringerLink
Log in

Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015

  • Published:
Autonomous Robots Aims and scope Submit manuscript

A Correction to this article was published on 21 November 2020

This article has been updated

Abstract

We describe the winning entry to the Amazon Picking Challenge  2015. From the experience of building this system and competing in the Amazon Picking Challenge, we derive several conclusions: (1) We suggest to characterize robotic system building along four key aspects, each of them spanning a spectrum of solutions—modularity versus integration, generality versus assumptions, computation versus embodiment, and planning versus feedback. (2) To understand which region of each spectrum most adequately addresses which robotic problem, we must explore the full spectrum of possible approaches. To achieve this, our community should agree on key aspects that characterize the solution space of robotic systems. (3) For manipulation problems in unstructured environments, certain regions of each spectrum match the problem most adequately, and should be exploited further. This is supported by the fact that our solution deviated from the majority of the other 2015 challenge entries along each of the spectra.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Change history

  • 21 November 2020

    In the original publication of the article, the incorrect author photo was displayed in biography section.

References

  • Aloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988). Active vision. International Journal of Computer Vision, 1(4), 333–356.

    Article  Google Scholar 

  • Atkeson, C. G., Babu, B. P. W., Banerjee, N., Berenson, D., Bove, C. P., Cui, X., DeDonato, M., Du, R., Feng, S., & Franklin, P., et al. (2015). No falls, no resets: Reliable humanoid behavior in the DARPA robotics challenge. In IEEE-RAS 15th international conference on humanoid robots (Humanoids) (pp 623–630). IEEE.

  • Baldwin, C. Y., & Clark, K. B. (2000). Design rules: The power of modularity (Vol. 1). Cambridge: MIT Press.

    Google Scholar 

  • Bertsekas, D. P. (1995). Dynamic programming and optimal control. Belmont: Athena Scientific.

    MATH  Google Scholar 

  • Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., et al. (2017). Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6), 1273–1291.

    Article  Google Scholar 

  • Bradski, G. (2000). The openCV library. Dr Dobb’s Journal: Software Tools for the Professional Programmer, 25(11), 120–123.

    Google Scholar 

  • Brooks, F. P. (1995). The Mythical Man-Month: Essays on software engineering. Boston: Addison-Wesley Longman.

    Google Scholar 

  • Brooks, R. A. (1990). Elephants don’t play chess. Robotics and Autonomous Systems, 6(1), 3–15.

    Article  Google Scholar 

  • Brooks, R. A., Breazeal, C., Irie, R., Kemp, C. C., Marjanovic, M., Scassellati, B., & Williamson, M. M. (1998). Alternative essences of intelligence. In: AAAI/IAAI (pp. 961–968).

  • Brown, E., Rodenberg, N., Amend, J., Mozeika, A., Steltz, E., Zakin, M. R., et al. (2010). Universal robotic gripper based on the jamming of granular material. Proceedings of the National Academy of Sciences, 107(44), 18,809–18,814.

    Article  Google Scholar 

  • Buehler, M., Iagnemma, K., & Singh, S. (2007). The 2005 DARPA grand challenge: The great robot race. Berlin: Springer.

    Book  Google Scholar 

  • Cohen, P. R. (1996). Empirical methods for artificial intelligence. IEEE Intelligent Systems, 11(6), 88.

    Google Scholar 

  • Coleman, D. (2015). MoveIt! strengths, weaknesses, and developer insight, presentation at ROSCon. https://vimeo.com/142621953. Accessed 16 Nov 2017.

  • Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., et al. (2018). Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1), 172–188.

    Article  Google Scholar 

  • Deimel, R., & Brock, O. (2016). A novel type of compliant and underactuated robotic hand for dexterous grasping. The International Journal of Robotics Research, 35(1–3), 161–185.

    Article  Google Scholar 

  • Dollar, A. M., & Howe, R. D. (2010). The highly adaptive SDM hand: Design and performance evaluation. The International Journal of Robotics Research, 29(5), 585–597.

    Article  Google Scholar 

  • Egerstedt, M. (2000). Behavior based robotics using hybrid automata. In Hybrid Systems: computation and control (pp 103–116). Springer.

  • Eppner, C., Deimel, R., Álvarez Ruiz, J., Maertens, M., & Brock, O. (2015). Exploitation of environmental constraints in human and robotic grasping. The International Journal of Robotics Research, 34(7), 1021–1038.

    Article  Google Scholar 

  • Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2016). Lessons from the Amazon Picking Challenge: Four aspects of building robotic systems. In Proceedings of Robotics: Science and Systems.

  • Espiau, B., Chaumette, F., & Rives, P. (1992). A new approach to visual servoing in robotics. IEEE Transactions on Robotics and Automation, 8(3), 313–326.

    Article  Google Scholar 

  • Hastie, T., Tibshirani, R., Friedman, J., & Franklin, J. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85.

    Google Scholar 

  • Hawes, N., Zillich, M., & Jensfelt, P. (2010). Lessons learnt from scenario-based integration. In Cognitive Systems (pp 423–438). Springer.

  • Hernandez, C., Bharatheesha, M., Ko, W., Gaiser, H., Tan, J., van Deurzen, K., de Vries, M., Mil, B. V., van Egmond, J., Burger, R., Morariu, M., Ju, J., Gerrmann, X., Ensing, R., van Frankenhuyzen, J., & Wisse, M. (2016). Team delft’s robot winner of the Amazon Picking Challenge 2016. CoRR arXiv:1610.05514.

  • Hollerbach, J., Khalil, W., & Gautier, M. (2008). Model identification. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 321–344). Berlin: Springer.

    Chapter  Google Scholar 

  • Holmberg, R., & Khatib, O. (2000). Development and control of a holonomic mobile robot for mobile manipulation tasks. The International Journal of Robotics Research, 19(11), 1066–1074.

    Article  MATH  Google Scholar 

  • Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.

    Article  MathSciNet  MATH  Google Scholar 

  • Johnson, M., Shrewsbury, B., Bertrand, S., Wu, T., Duran, D., Floyd, M., et al. (2015). Team IHMC’s lessons learned from the DARPA robotics challenge trials. Journal of Field Robotics, 32(2), 192–208.

    Article  Google Scholar 

  • Jonschkowski, R., & Brock, O. (2015). Learning state representations with robotic priors. Autonomous Robots, 39(3), 407–428.

    Article  Google Scholar 

  • Jonschkowski, R., Eppner, C., Höfer, S., Martín-Martín, R., & Brock, O. (2016). Probabilistic multi-class segmentation for the Amazon Picking Challenge. In IEEE/RSJ international conference on intelligent robots and systems (pp. 822–830).

  • Kamiya, Y. (2016). Team preferred networks. In Talk at workshop on automation for warehouse logistics (CASE2016). https://youtu.be/iBr943C6uxI. Accessed 16 Feb 2016.

  • Katz, D., & Brock, O. (2011). A factorization approach to manipulation in unstructured environments. In Robotics Research (pp. 285–300). Springer.

  • Laumond, J. P. (2006). Kineo CAM: A success story of motion planning algorithms. IEEE Robotics & Automation Magazine, 13(2), 90–93.

    Article  Google Scholar 

  • LaValle, S. M. (2006). Planning algorithms. Cambridge: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Lozano-Perez, T., Mason, M. T., & Taylor, R. H. (1984). Automatic synthesis of fine-motion strategies for robots. The International Journal of Robotics Research, 3(1), 3–24.

    Article  Google Scholar 

  • Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., & Konolige, K. (2010). The office marathon: Robust navigation in an indoor office environment. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 300–307).

  • Marr, D. (1982). Vision: A computational approach. San Francisco: Freeman & Co.

    Google Scholar 

  • Martín-Martín, R., & Brock, O. (2014). Online interactive perception of articulated objects with multi-level recursive estimation based on task-specific priors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 2494–2501).

  • Martín-Martín, R., Höfer, S., & Brock, O. (2016). An integrated approach to visual perception of articulated objects. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5091–5097).

  • Mason, M. T., Rodriguez, A., Srinivasa, S. S., & Vazquez, A. S. (2012). Autonomous manipulation with a general-purpose simple hand. The International Journal of Robotics Research, 31(5), 688–703.

    Article  Google Scholar 

  • Milan, A. (2016). Team NimbRo. In: Talk at Workshop on Automation for Warehouse Logistics (CASE2016). https://youtu.be/K3QQ_ZmImmE. Accessed 16 Feb 2016.

  • Miller, A. T., & Allen, P. K. (2004). GraspIt! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4), 110–122.

    Article  Google Scholar 

  • Morrison, D., Tow, A. W., McTaggart, M., Smith, R., Kelly-Boxall, N., Wade-McCue, S., Erskine, J., Grinover, R., Gurman, A., Hunn, T., Lee, D., Milan, A., Pham, T., Rallos, G., Razjigaev, A., Rowntree, T., Vijay, K., Zhuang, Z., Lehnert, C. F., Reid, I. D., Corke, P., & Leitner, J. (2017). Cartman: The low-cost cartesian manipulator that won the amazon robotics challenge. CoRR arXiv:1709.06283.

  • Nakanishi, J., Cory, R., Mistry, M., Peters, J., & Schaal, S. (2008). Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27(6), 737–757.

    Article  Google Scholar 

  • Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441–450.

    Article  MathSciNet  MATH  Google Scholar 

  • Pfeifer, R., & Gómez, G. (2009). Morphological computation: Connecting body, brain, and environment. In Creating brain-like intelligence (pp. 66–83). Springer.

  • Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A. Y. (2009). ROS: An open-source robot operating system. In ICRA workshop on open source software.

  • Rennie, C., Shome, R., Bekris, K. E., & De Souza, A. F. (2016). A dataset for improved rgbd-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters, 1(2), 1179–1185.

    Article  Google Scholar 

  • Rickert, M., Sieverling, A., & Brock, O. (2014). Balancing exploration and exploitation in sampling-based motion planning. IEEE Transactions on Robotics, 30(6), 1305–1317.

    Article  Google Scholar 

  • Rooks, B. (2006). The harmonious robot. Industrial Robot: An International Journal, 33(2), 125–130.

    Article  Google Scholar 

  • Rusu, R. B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1–4).

  • Scholz, J., Levihn, M., Isbell, C., & Wingate, D. (2014). A physics-based model prior for object-oriented MDPs. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) (pp. 1089–1097).

  • Schwaber, K. (2004). Agile project management with Scrum. Redmond: Microsoft Press.

    MATH  Google Scholar 

  • Schwarz, M., Milan, A., Lenz, C., Munoz, A., Periyasamy, A. S., Schreiber, M., Schüller, S., & Behnke, S. (2017). NimbRo picking: Versatile part handling for warehouse automation. In IEEE International Conference on Robotics and Automation (ICRA).

  • Schwarz, M., Milan, A., Periyasamy, A. S., & Behnke, S. (2018). RGB-D object detection and semantic segmentation for autonomous manipulation in clutter. The International Journal of Robotics Research, 37(4–5), 437–451. https://doi.org/10.1177/0278364917713117.

  • Stulp, F., & Sigaud, O. (2013). Robot skill learning: From reinforcement learning to evolution strategies. Paladyn, Journal of Behavioral Robotics, 4(1), 49–61.

    Article  Google Scholar 

  • Sucan, I. I., & Chitta, S. (2016). MoveIt. http://moveit.ros.org. Accessed 16 Nov 2017.

  • Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge: MIT Press.

    MATH  Google Scholar 

  • Ulrich, K. (1995). The role of product architecture in the manufacturing firm. Research Policy, 24(3), 419–440.

    Article  Google Scholar 

  • van Gigch, J. P. (1991). System design modeling and metamodeling. Berlin: Springer.

    Book  Google Scholar 

  • Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341–1390.

    Article  Google Scholar 

  • Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82.

    Article  Google Scholar 

  • XR4000. (2015). Nomadic technologies inc. XR4000. http://cbcis.ttu.edu/ep/old_netra_site/about/xr4000/XR4000.htm. Accessed 29 Jan 2016.

  • Yu, K. T., Fazeli, N., Chavan-Dafle, N., Taylor, O., Donlon, E., Lankenau, G. D., & Rodriguez, A. (2016). A summary of team MIT’s approach to the Amazon Picking Challenge 2015. ArXiv e-prints arXiv:1604.03639.

  • Yu, P. (2016). Team MIT. In Talk at Workshop on Automation for Warehouse Logistics (CASE2016). https://youtu.be/4OKGev0b9qU. Accessed 16 Feb 2016.

  • Zeng, A., Yu, K. T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J. (2017). Multi-view self-supervised deep learning for 6d pose estimation in the Amazon Picking Challenge. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1386–1383).

Download references

Acknowledgements

We would like to thank Barrett Technology for their support and our team members Raphael Deimel, Roman Kolbert, Gabriel Le Roux, and Wolf Schaarschmidt, who helped creating the winning system.

Author information

Author notes

  1. Clemens Eppner, Sebastian Höfer, Rico Jonschkowski, Roberto Martín-Martín, Arne Sieverling, Vincent Wall contributed equally to this work and are ordered alphabetically.

Authors and Affiliations

  1. Robotics and Biology Laboratory, Fakultät IV, MAR 5-1, Marchstr. 23, 10587, Berlin, Germany

    Clemens Eppner, Sebastian Höfer, Rico Jonschkowski, Roberto Martín-Martín, Arne Sieverling, Vincent Wall & Oliver Brock

Authors
  1. Clemens Eppner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Höfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rico Jonschkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Roberto Martín-Martín
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Arne Sieverling
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vincent Wall
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Oliver Brock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arne Sieverling.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We gratefully acknowledge the Funding provided by the Alexander von Humboldt foundation and the Federal Ministry of Education and Research (BMBF), the European Commission (SOMA Project, H2020-ICT-645599), the German Research Foundation (DFG) (Exploration Challenge, BR 2248/3-1), and the travel grant provided by Amazon Robotics.

S. Höfer: Amazon Research. All work presented in this paper has been conducted during the author’s employment at TU Berlin.

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Robotics Science and Systems”.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eppner, C., Höfer, S., Jonschkowski, R. et al. Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015. Auton Robot 42, 1459–1475 (2018). https://doi.org/10.1007/s10514-018-9761-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-018-9761-2

Keywords

Search

Navigation