Hua Wei
Skip Abstract Section
Abstract
Traffic signal control is an important and challenging real-world problem that has recently received a large amount of interest from both transportation and computer science communities. In this survey, we focus on investigating the recent advances in using reinforcement learning (RL) techniques to solve the traffic signal control problem. We classify the known approaches based on the RL techniques they use and provide a review of existing models with analysis on their advantages and disadvantages. Moreover, we give an overview of the simulation environments and experimental settings that have been developed to evaluate the traffic signal control methods. Finally, we explore future directions in the area of RLbased traffic signal control methods. We hope this survey could provide insights to researchers dealing with real-world applications in intelligent transportation systems
- M. Abdoos, N. Mozayani, and A. L. Bazzan. Hierarchical control of traffic signals using Q-learning with tile coding. Applied intelligence, 40(2):201--213, 2014. Google Scholar
Digital Library
- I. Arel, C. Liu, T. Urbanik, and A. Kohls. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems, 2010.Google ScholarCross Ref
- K. Arulkumaran, M. P. Deisenroth, et al. A brief survey of deep reinforcement learning. arXiv preprint, 2017.Google Scholar
- M. Aslani, M. S. Mesgari, and M. Wiering. Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events. TRB-C, 2017.Google ScholarCross Ref
- M. Aslani, S. Seipel, M. S. Mesgari, and M. Wiering. Traffic signal optimization through discrete and continuous reinforcement learning with robustness analysis in downtown tehran. Advanced Engineering Informatics, 2018.Google ScholarCross Ref
- J. Ault, J. Hanna, and G. Sharon. Learning an interpretable traffic signal control policy. arXiv preprint, 2019.Google Scholar
- B. Bakker, S. Whiteson, L. Kester, and F. C. Groen. Traffic light control by multiagent reinforcement learning systems. In Interactive Collaborative Information Systems. 2010.Google ScholarCross Ref
- A. L. Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. AAMAS, 2009. Google ScholarDigital Library
- J. A. Calvo and I. Dusparic. Heterogeneous multi-agent deep reinforcement learning for traffic lights control. In AICS, pages 2--13, 2018.Google Scholar
- N. Casas. Deep deterministic policy gradient for urban traffic light control. arXiv preprint, 2017.Google Scholar
- C. Chen, H. Wei, N. Xu, et al. Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control. In AAAI, 2020.Google Scholar
- T. Chu, J.Wang, L. Codec'a, and Z. Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. arXiv preprint, 2019.Google Scholar
- C. Claus and C. Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998. Google ScholarDigital Library
- M. Cos¸kun et al. Deep reinforcement learning for traffic light optimization. In ICDMW. IEEE, 2018.Google ScholarCross Ref
- T. Economist. The cost of traffic jams. https: //www.economist.com/blogs/economist-explains/2014/ 11/economist-explains-1, November 2014.Google Scholar
- S. El-Tantawy and B. Abdulhai. Comprehensive analysis of reinforcement learning methods and parameters for adaptive traffic signal control. Technical report, 2011.Google Scholar
- R. Florin and S. Olariu. A survey of vehicular communications for traffic signal optimization. Vehicular Communications, 2015. Google ScholarDigital Library
- Gao, Y. Shen, J. Liu, M. Ito, and N. Shiratori. Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network. arXiv preprint arXiv:1705.02755, 2017.Google Scholar
- J. Garcia and F. Fern´andez. A comprehensive survey on safe reinforcement learning. JMLR, 2015. Google ScholarDigital Library
- D. Garg, M. Chli, and G. Vogiatzis. Deep reinforcement learning for autonomous traffic light control. In ICITE, 2018.Google ScholarCross Ref
- H. Ge, Y. Song, C. Wu, J. Ren, and G. Tan. Cooperative deep q-learning with q-value transfer for multi-intersection signal control. IEEE Access, 7:40797--40809, 2019.Google ScholarCross Ref
- W. Genders and S. Razavi. Using a deep reinforcement learning agent for traffic signal control. arXiv preprint, 2016.Google Scholar
- Y. Gong, M. Abdel-Aty, Q. Cai, and M. S. Rahman. Decentralized network level adaptive signal control by multi-agent deep reinforcement learning. Transportation Research Interdisciplinary Perspectives, 1:100020, 2019.Google ScholarCross Ref
- A. Haydari and Y. Yilmaz. Deep reinforcement learning for intelligent transportation systems: A survey. arXiv:2005.00935, 2020.Google Scholar
- P. Henderson, R. Islam, P. Bachman, J. Pineau, et al. Deep reinforcement learning that matters. In AAAI, 2018.Google ScholarCross Ref
- L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 1996. Google ScholarDigital Library
- J. R. Kok and N. Vlassis. Using the max-plus algorithm for multiagent decision making in coordination graphs. In Robot Soccer World Cup, pages 1--12. Springer, 2005.Google Scholar
- P. Koonce et al. Traffic signal timing manual. Technical report, United States. Federal Highway Administration, 2008.Google Scholar
- A. Kouvelas, J. Lioris, S. A. Fayazi, and P. Varaiya. Maximum Pressure Controller for Stabilizing Queues in Signalized Arterial Networks. TRB, 2014.Google ScholarCross Ref
- L. Li, Y. Lv, and F.-Y. Wang. Traffic signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 2016.Google Scholar
- L. Li, D. Wen, and D. Yao. A survey of traffic control with vehicular communications. IEEE TITS, 2014. Google ScholarDigital Library
- X. Liang, X. Du, G. Wang, and Z. Han. Deep reinforcement learning for traffic light control in vehicular networks. arXiv preprint arXiv:1803.11115, 2018.Google Scholar
- X. Liang, X. Du, G.Wang, and Z. Han. A deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, 68(2):1243--1253, 2019.Google ScholarCross Ref
- T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint, 2015.Google Scholar
- J. D. Little, M. D. Kelson, and N. H. Gartner. Maxband: A versatile program for setting signals on arteries and triangular networks. 1981.Google Scholar
- X.-Y. Liu, Z. Ding, S. Borst, and A. Walid. Deep reinforcement learning for intelligent transportation systems. arXiv preprint arXiv:1812.00979, 2018.Google Scholar
- Y. Liu, L. Liu, and W.-P. Chen. Intelligent traffic light control using distributed multi-agent q learning. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1--8. IEEE, 2017.Google ScholarCross Ref
- P. Lowrie. Scats--a traffic responsive method of controlling urban traffic. roads and traffic authority, sydney. New South Wales, Australia, 1990.Google Scholar
- P. Mannion, J. Duggan, and E. Howley. An experimental review of reinforcement learning algorithms for adaptive traffic signal control. In Autonomic Road Transport Support Systems. 2016.Google ScholarCross Ref
- F. J. Martinez, C. K. Toh, J.-C. Cano, C. T. Calafate, and P. Manzoni. A survey and comparative study of simulators for vehicular ad hoc networks (VANETs). Wireless Communications and Mobile Computing, 2011. Google ScholarDigital Library
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, et al. Human-level control through deep reinforcement learning. Nature, 2015.Google Scholar
- S. S. Mousavi et al. Traffic light control using deep policygradient and value-function-based reinforcement learning. Intelligent Transport Systems, 2017.Google Scholar
- O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and policy based reinforcement learning. In NeurIPS, 2017. Google ScholarDigital Library
- T. Nishi, K. Otaki, K. Hayakawa, and T. Yoshimura. Traffic signal control based on reinforcement learning with graph convolutional neural nets. In ITSC. IEEE, 2018.Google ScholarCross Ref
- A. Nowe, P. Vrancx, and Y. M. D. Hauwere. Game Theory and Multi-agent Reinforcement Learning. 2012.Google ScholarCross Ref
- V. Pandey and S. D. Boyles. Multiagent reinforcement learning algorithm for distributed dynamic pricing of managed lanes. In ITSC'18. IEEE, 2018.Google ScholarCross Ref
- M. Papageorgiou, C. Diakaki, V. Dinopoulou, A. Kotsialos, and Y. Wang. Review of road traffic control strategies. Proceedings of the IEEE, 2003.Google ScholarCross Ref
- T. T. Pham, T. Brys, M. E. Taylor, T. Brys, et al. Learning coordinated traffic light control. In AAMAS, 2013.Google Scholar
- L. A. Prashanth and S. Bhatnagar. Reinforcement learning with average cost for adaptive control of traffic lights at intersections. ITSC, 2011.Google ScholarCross Ref
- J. Rios-Torres and A. A. Malikopoulos. A survey on the coordination of connected and automated vehicles at intersections and merging at highway on-ramps. IEEE TITS, 2016. Google ScholarDigital Library
- S. G. Rizzo, G. Vantini, and S. Chawla. Time critic policy gradient methods for traffic signal control in complex and congested scenarios. In KDD, 2019. Google ScholarDigital Library
- R. P. Roess, E. S. Prassas, and W. R. McShane. Traffic engineering. Pearson, 2004.Google Scholar
- M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference. Springer, 2018.Google Scholar
- D. Schrank, B. Eisele, T. Lomax, and J. Bak. 2015 urban mobility scorecard. 2015.Google Scholar
- M. Schutera, N. Goby, S. Smolarek, and M. Reischl. Distributed traffic light control at uncoupled intersections with real-world topology by deep reinforcement learning. arXiv preprint arXiv:1811.11233, 2018.Google Scholar
- S. Sukhbaatar, R. Fergus, et al. Learning multiagent communication with backpropagation. In NeurIPS, 2016. Google ScholarDigital Library
- T. Tan, F. Bao, Y. Deng, A. Jin, Q. Dai, and J. Wang. Cooperative deep reinforcement learning for large-scale traffic grid signal control. IEEE transactions on cybernetics, 2019.Google ScholarCross Ref
- E. van der Pol. Coordinated deep reinforcement learners for traffic light control. NeurlPS, 2016.Google Scholar
- P. Veli?ckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. ICLR, 2018.Google Scholar
- Y. Wang, T. Xu, X. Niu, and other. STMARL: A Spatio- Temporal Multi-Agent Reinforcement Learning Approach for Traffic Light Control. arXiv preprint, 2019.Google Scholar
- H. Wei, C. Chen, C. Liu, G. Zheng, and Z. Li. Learning to simulate on sparse trajectory data. 2020.Google Scholar
- H. Wei, C. Chen, G. Zheng, K. Wu, V. Gayah, K. Xu, and Z. Li. PressLight: Learning Max Pressure Control to Coordinate Traffic Signals in Arterial Network. In KDD, 2019. Google ScholarDigital Library
- H. Wei, N. Xu, H. Zhang, G. Zheng, et al. CoLight: Learning Network-level Cooperation for Traffic Signal Control. In CIKM, 2019. Google ScholarDigital Library
- H. Wei, G. Zheng, V. Gayah, and Z. Li. A survey on traffic signal control methods. arXiv:1904.08117, 2019.Google Scholar
- H. Wei, G. Zheng, H. Yao, and Z. Li. IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control. In KDD, 2018. Google ScholarDigital Library
- M. Wiering. Multi-agent reinforcement learning for traffic light control. In ICML, 2000. Google ScholarDigital Library
- C. Wu, A. Kreidieh, K. Parvate, E. Vinitsky, and A. M. Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint, 2017.Google Scholar
- Y. Wu, H. Tan, and B. Ran. Differential variable speed limits control for freeway recurrent bottlenecks via deep reinforcement learning. arXiv preprint arXiv:1810.10952, 2018.Google Scholar
- Y. Xiong, G. Zheng, K. Xu, and Z. Li. Learning traffic signal control from demonstrations. In CIKM, 2019. 70] M. Xu, J. Wu, L. Huang, R. Zhou, T. Wang, and D. Hu. Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning. Journal of Intelligent Transportation Systems, 24(1):1--10, 2020. Google ScholarDigital Library
- K.-L. A. Yau, J. Qadir, H. L. Khoo, et al. A survey on reinforcement learning models and algorithms for traffic signal control. ACM Computing Survey, 2017. Google ScholarDigital Library
- X. Zang, H. Yao, G. Zheng, N. Xu, K. Xu, and Z. Li. Meta- Light: Value-based Meta-reinforcement Learning for Online Universal Traffic Signal Control. In AAAI, 2020.Google Scholar
- H. Zhang, S. Feng, C. Liu, et al. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In The WebConf, 2019. Google ScholarDigital Library
- Z. Zhang, J. Yang, and H. Zha. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint, 2019.Google Scholar
- G. Zheng, C. Liu, H.Wei, C. Chen, and Z. Li. Rebuilding citywide traffic origin destinationfrom road speed data. In ICDE, 2021.Google Scholar
- G. Zheng, H. Liu, and Z. Li. Learning to simulate vehicle trajectory from demonstrations. In ICDE, 2020.Google ScholarCross Ref
- G. Zheng, Y. Xiong, X. Zang, J. Feng, H. Wei, et al. Learning Phase Competition for Traffic Signal Control. In CIKM, 2019. Google ScholarDigital Library
- G. Zheng, X. Zang, N. Xu, H. Wei, Z. Yu, et al. Diagnosing Reinforcement Learning for Traffic Signal Control. arXiv preprint, 2019.Google Scholar
Index Terms
- Recent Advances in Reinforcement Learning for Traffic Signal Control: A Survey of Models and Evaluation
Index terms have been assigned to the content through auto-classification.
Recommendations
Traffic Signal Control Using Reinforcement Learning
CSNT '14: Proceedings of the 2014 Fourth International Conference on Communication Systems and Network TechnologiesProposing an appropriate and dynamic strategy to meet the existing requirements is an important aspect in traffic control system. Continuous changes of states and the necessity to respond quickly are the specific characteristics of the environment in a ...
Parallel Reinforcement Learning for Traffic Signal Control
AbstractDeveloping Adaptive Traffic Signal Control strategies for efficient urban traffic management is a challenging problem, which is not easily solved. Reinforcement Learning (RL) has been shown to be a promising approach when applied to traffic signal ...
Comments