Abstract
Ensuring a safe journey with an autonomous vehicle, the surrounding has to be sensed and understood. Especially human intuition about the plans and intentions of traffic participants is hard to model for machines. In literature, there are already several prediction techniques existing for pedestrians, which are based on different features. Some models are very complex, whereas others only rely on the considered person’s motion. The goal of this work is to analyze the importance of different classes of context knowledge for the prediction performance, derive features to remove this lack of information and prove this by an improved prediction algorithm. In order to judge the lack of context knowledge, we analyze the prediction performance and error cases of a long short-term memory (LSTM) Neural Network as State-of-the-Art prediction algorithm, only based on motion data. The Network is trained and evaluated on a benchmark dataset, to make the results comparable to other approaches. Analyzing the most error-prone predictions, the missing context shall be identified, which could improve the prediction results. Since the data was generated by video, we can evaluate the whole scenario and identify the influencing factors. The found influences were classified in categories and their importance for the prediction model estimated. We prove the necessity of additional context knowledge by retraining a neural network with additional context knowledge. In a literature research we compare our found results to existing approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Status at January 27, 2021
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/. Accessed 21 Dec 2020
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social lstm: Human trajectory prediction in crowded spaces. Proceedings of the IEEE conference on computer vision and pattern recognition 961–971,(2016)
Andriluka, M., Roth, S., Schiele, B.: Monocular 3D pose estimation and tracking by detection, In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010), pp. 623–630
Becker, S., Hug, R., Hbner, W., Arens, M.: An evaluation of trajectory prediction approaches and notes on the trajnet benchmark. (2018). arXiv:1805.07663
Bisagno, N., Zhang, B., Conci, N.: Group lstm: Group trajectory prediction in crowded scenarios, Proceedings of the European conference on computer vision (ECCV) (2018)
Chollet, F., et al.: Keras https://keras.io (2015). Accessed 12 Jan 2021
Ferryman, J., Shahrokni, A.: PETS2009: Dataset and challenge, In 2009 Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (2009), pp. 1–6
Garzón, M., Garzón-Ramos, D., Barrientos, A., Cerro, J.: in Proceedings of the 13th International Conference on Informatics in Control, , Automation and Robotics, pp. 29–31. Lisbon, Portugal (2016)
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social gan: Socially acceptable trajectories with generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2255–2264,(2018)
Hamid, O.H., Wendemuth, A., Braun, J.: Temporal context and conditional associative learning. BMC Neurosci. 11(1), 1 (2010)
Hamid, O.H., Braun, J.: Reinforcement learning and attractor neural network models of associative learning, In International Joint Conference on Computational Intelligence (Springer, 2017), pp. 327–349
Hunt, E.: Tay, microsoft’s ai chatbot, gets a crash course in racism from twitter (2016). https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter. Accessed 3 Jan 2021
Kerscher, S., Balbierer, N., Kraust, S., Hartmannsgruber, A., Müller, N. and Ludwig, B.: Intention-based Prediction for Pedestrians and Vehicles in Unstructured Environments. In VEHITS pp. 307–314 (2018)
Lerner, A., Chrysanthou, Y., Lischinski, D.: Lerner, Alon and Chrysanthou, Yiorgos and Lischinski, Dani. Comput. Graph. Forum 26, 655 (2007). https://doi.org/10.1111/j.1467-8659.2007.01089.x
Luber, M., Stork, J.A., Tipaldi, G.D., Arras, K.O.: People tracking with human motion predictions from social forces, In 2010 IEEE International Conference on Robotics and Automation (IEEE, 2010), pp. 464–469
Pellegrini, S., Ess, A., Schindler, K., van Gool, L.: You’ll never walk alone: Modeling social behavior for multi-target tracking, In 2009 IEEE 12th International Conference on Computer Vision (2009), pp. 261–268
Pfeiffer, M., Paolo, G., Sommer, H., Nieto, J., Siegwart, R., Cadena, C.: A data-driven model for interaction-aware pedestrian motion prediction in object cluttered environments, In 2018 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2018), pp. 1–8
Pumperla, M.: Hyperas: a very simple convenience wrapper around hyperopt for fast prototyping with keras models (2017). https://maxpumperla.com/hyperas/. Accessed 29 Oct 2020
Quintero, R., Parra, I., Lorenzo, J., Fernández-Llorca, D., Sotelo, M.: Pedestrian intention recognition by means of a hidden markov model and body language, In 2017 IEEE 20th international conference on intelligent transportation systems (ITSC) (IEEE, 2017), pp. 1–7
Rasouli, A., Kotseruba, I., Tsotsos, J.K.: Understanding pedestrian behavior in complex traffic scenes. IEEE Trans. Intell. Veh. 3(1), 61 (2017)
Rehder, E., Wirth, F., Lauer, M., Stiller, C.: Pedestrian prediction by planning using deep neural networks, In 2018 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2018), pp. 1–5
Reuters. Amazon ditched ai recruiting tool that favored men for technical jobs (2018). https://www.theguardian.com/technology/2018/oct/10/amazon-hiring-ai-gender-bias-recruiting-engine. Accessed 26 Jan 2021
Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016, pp. 549–565. Springer International Publishing, Cham (2016)
Sadeghian, A., Kosaraju, V., Gupta, A., Savarese, S., Alahi, A.: TrajNet: towards a benchmark for human trajectory prediction, arXiv preprint (2018)
Schmidt, S., Faerber, B.: Pedestrians at the kerb-Recognising the action intentions of humans. Transp. Res. Part F 12(4), 300 (2009)
Schöller, C., Aravantinos, V., Lay, F., Knoll, A.: What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robot. Autom. Lett. 5(2), 1696 (2020)
Vemula, A., Muelling, K., Oh, J.: Social attention: Modeling attention in human crowds, In 2018 IEEE international Conference on Robotics and Automation (ICRA) (IEEE, 2018), pp. 1–7
Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, İ, Feng, Y., Moore, E.W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P.: SciPy 1.0 contributors. Nat. Methods 17, 261 (2020). https://doi.org/10.1038/s41592-019-0686-2
Wu, J., Ruenz, J., Althoff, M.: Probabilistic map-based pedestrian motion prediction taking traffic participants into consideration In 2018 IEEE Intelligent Vehicles Symposium (IV) (IEEE, 2018), pp. 1285–1292
Funding
S.K. was supported by the BayWISS Consortium Digitization.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kerscher, S., Müller, N. & Ludwig, B. Assessing the lack of context knowledge for a pedestrian predicting neural network. Int J Intell Robot Appl 6, 467–482 (2022). https://doi.org/10.1007/s41315-021-00208-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41315-021-00208-w