Skip to main content
SpringerLink
Log in

Enhanced Pub/Sub Communications for Massive IoT Traffic with SARSA Reinforcement Learning

  • Conference paper
  • First Online:
Machine Learning for Networking (MLN 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12629))

Included in the following conference series:

Abstract

Sensors are being extensively deployed and are expected to expand at significant rates in the coming years. They typically generate a large volume of data on the internet of things (IoT) application areas like smart cities, intelligent traffic systems, smart grid, and e-health. Cloud, edge and fog computing are potential and competitive strategies for collecting, processing, and distributing IoT data. However, cloud, edge, and fog-based solutions need to tackle the distribution of a high volume of IoT data efficiently through constrained and limited resource network infrastructures. This paper addresses the issue of conveying a massive volume of IoT data through a network with limited communications resources (bandwidth) using a cognitive communications resource allocation based on Reinforcement Learning (RL) with SARSA algorithm. The proposed network infrastructure (PSIoTRL) uses a Publish/Subscribe architecture to access massive and highly distributed IoT data. It is demonstrated that the PSIoTRL bandwidth allocation for buffer flushing based on SARSA enhances the IoT aggregator buffer occupation and network link utilization. The PSIoTRL dynamically adapts the IoT aggregator traffic flushing according to the Pub/Sub topic’s priority and network constraint requirements.

Work supported by CAPES and Salvador University (UNIFACS).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    By initial buffer transmission rate, we mean the configured initial transmission rate to empty buffers without any buffer over-utilization.

References

  1. Alfakih, T., Hassan, M.M., Gumaei, A., Savaglio, C., Fortino, G.: Task offloading and resource allocation for mobile edge computing by deep reinforcement learning based on SARSA. IEEE Access 8, 54074–54084 (2020)

    Article  Google Scholar 

  2. An, K., Gokhale, A., Tambe, S., Kuroda, T.: Wide Area Network-Scale Discovery and Data Dissemination in Data-Centric Publish/Subscribe Systems, ACM Press, pp. 1–2 (2015)

    Google Scholar 

  3. Asghari, A., Sohrabi, M.K., Yaghmaee, F.: Task scheduling, resource provisioning, and load balancing on scientific workflows using parallel SARSA reinforcement learning agents and genetic algorithm. J. Supercomput. 77, 2800–2828 (2021). https://doi.org/10.1007/s11227-020-03364-1

  4. Bibal Benifa, J.V., Dejey, D.: RLPAS: reinforcement learning-based proactive auto-scaler for resource provisioning in cloud environment. Mob. Netw. Appl. 24(4), 1348–1363 (2018). https://doi.org/10.1007/s11036-018-0996-0

    Article  Google Scholar 

  5. Boutaba, R., et al.: A comprehensive survey on machine learning for networking: evolution, applications and research opportunities. J. Internet Serv. Appl. 9(1), 16 (2018)

    Article  Google Scholar 

  6. Ciosek, K., Vuong, Q., Loftin, R., Hofmann, K.: Better exploration with optimistic actor critic, pp. 1787–1798 (2019)

    Google Scholar 

  7. Cote, D.: Using machine learning in communication networks. IEEE/OSA J. Opt. Commun. Networking 10(10), D100–D109 (2018)

    Article  Google Scholar 

  8. Dabbaghjamanesh, M., Moeini, A., Kavousi-Fard, A.: Reinforcement learning-based load forecasting of electric vehicle charging station using Q-learning technique. IEEE Trans. Indust. Inform. V1, 1–9 (2020)

    Google Scholar 

  9. Defazio, A., Graepel, T.: A comparison of learning algorithms on the arcade learning environment. arXiv:1410.8620 [cs], October 2014

  10. Happ, D., Wolisz, A.: Limitations of the Pub/Sub pattern for cloud based IoT and their implications. In: Cloudification of the Internet of Things (CIoT), Paris, pp. 1–6. IEEE, November 2016

    Google Scholar 

  11. Koo, J., Mendiratta, V.B., Rahman, M.R., Walid, A.: Deep reinforcement learning for network slicing with heterogeneous resource requirements and time varying traffic dynamics. In: 2019 15th International Conference on Network and Service Management, pp. 1–5, October 2019

    Google Scholar 

  12. Kreutz, D., Ramos, F.M.V., Verissimo, P., Rothenberg, C.E., Azodolmolky, S., Uhlig, S.: Software-defined networking: a comprehensive survey. Proc. IEEE 103(1), 14–76 (2014)

    Article  Google Scholar 

  13. Latah, M., Toker, L.: Artificial intelligence enabled software-defined networking: a comprehensive overview. IET Netw. 8(2), 79–99 (2019)

    Article  Google Scholar 

  14. Liao, X., Wu, D., Wang, Y.: Dynamic spectrum access based on improved SARSA algorithm. IOP Conf. Ser. Mater. Sci. Eng. 768(7), 072015 (2020)

    Google Scholar 

  15. Machado, M.C., Bellemare, M.G., Talvitie, E., Veness, J., Hausknecht, M., Bowling, M.: Revisiting the arcade learning environment: evaluation protocols and open problems for general agents. J. Artif. Intell. Res. 61, 523–562 (2018)

    Article  MathSciNet  Google Scholar 

  16. Mahadevan, S.: Average reward reinforcement learning: foundations, algorithms, and empirical results. Mach. Learn. 22(1), 159–195 (1996)

    MATH  Google Scholar 

  17. Martins, J.S.B.: Towards smart city innovation under the perspective of software-defined networking, artificial intelligence and big data. Revista de Tecnologia da Informação e Comunicação 8(2), 1–7 (2018)

    Google Scholar 

  18. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  19. Moerland, T.M., Broekens, J., Jonker, C.M.: A framework for reinforcement learning and planning. Ph.D. thesis, TU Delft. June 2020

    Google Scholar 

  20. Mohammadi, M., Al-Fuqaha, A.: Enabling cognitive smart cities using big data and machine learning: approaches and challenges. IEEE Commun. Mag. 56(2), 94–101 (2018)

    Article  Google Scholar 

  21. Moraes, P.F., Martins, J.S.B.: A Pub/Sub SDN-integrated framework for IoT traffic orchestration. In: Proceedings of the 3rd International Conference on Future Networks and Distributed Systems, ICFNDS 2019, Paris, France, pp. 1–9 (2019)

    Google Scholar 

  22. Moraes, P.F., Reale, R.F., Martins, J.S.B.: A publish/subscribe QoS-aware framework for massive IoT traffic orchestration. In: Proceedings of the 6th International Workshop on ADVANCEs in ICT Infrastructures and Services (ADVANCE), Santiago, pp. 1–14, January 2018

    Google Scholar 

  23. Mukherjee, M., Shu, L., Wang, D.: Survey of fog computing: fundamental, network applications, and research challenges. IEEE Commun. Surv. Tutorials 20(3), 1826–1857 (2018)

    Google Scholar 

  24. Nassar, A., Yilmaz, Y.: Reinforcement learning for adaptive resource allocation in fog RAN for IoT with heterogeneous latency requirements. IEEE Access 7, 128014–128025 (2019)

    Article  Google Scholar 

  25. Nour, B., Sharif, K., Li, F., Yang, S., Moungla, H., Wang, Y.: ICN publisher-subscriber models: challenges and group-based communication. IEEE Netw. 33(6), 156–163 (2019)

    Article  Google Scholar 

  26. Ramani, D.: A short survey on memory based reinforcement learning. arXiv:1904.06736 [cs], April 2019

  27. Rathore, M.M., Ahmad, A., Paul, A., Rho, S.: Urban planning and building smart cities based on the Internet of Things using big data analytics. Comput. Netw. 101, 63–80 (2016)

    Article  Google Scholar 

  28. Rendon, O.M.C., et al.: Machine learning for cognitive network management. IEEE Commun. Mag. 1–9 (2018)

    Google Scholar 

  29. Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Technical report, TR 166, Cambridge University Engineering Department, Cambridge, England (1994)

    Google Scholar 

  30. Sampaio, L.S.R., Faustini, P.H.A., Silva, A.S., Granville, L.Z., Schaeffer-Filho, A.: Using NFV and reinforcement learning for anomalies detection and mitigation in SDN. In: 2018 IEEE Symposium on Computers and Communications (ISCC), pp. 00432–00437, June 2018

    Google Scholar 

  31. Santos, J., Wauters, T., Volckaert, B., De Turck, F.: Resource provisioning in fog computing: from theory to practice. Sensors (Basel, Switzerland) 19(10), 2238 (2019)

    Google Scholar 

  32. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

  33. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  34. Wang, J.H., Lu, P.E., Chang, C.S., Lee, D.S.: A reinforcement learning approach for the multichannel rendezvous problem. In: 2019 IEEE Globecom Workshops (GC Wkshps), pp. 1–5, December 2019

    Google Scholar 

  35. Wang, Y., Zou, S.: Finite-sample analysis of Greedy-GQ with linear function approximation under Markovian noise. In: Proceedings of Machine Learning Research. Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), vol. 124, pp. 1–26 (2020)

    Google Scholar 

  36. Xie, J., et al.: A survey of machine learning techniques applied to software defined networking (SDN): research issues and challenges. IEEE Commun. Surv. Tutorials 21(1), 393–430 (2019)

    Google Scholar 

  37. Zhang, X., Wang, Y., Lu, S., Liu, L., Xu, L., Shi, W.: OpenEI: an open framework for edge intelligence. In: 39th IEEE International Conference on Distributed Computing Systems (ICDCS), Dallas, US, pp. 1–12, July 2019

    Google Scholar 

  38. Zhao, L., Wang, J., Liu, J., Kato, N.: Routing for crowd management in smart cities: a deep reinforcement learning perspective. IEEE Commun. Mag. 57(4), 88–93 (2019)

    Article  Google Scholar 

Download references

Acknowledgments

Authors want to thanks CAPES (Coordination for the Improvement of Higher Education Personnel) for the master’s scholarship support granted.

Author information

Authors and Affiliations

  1. University of Paris-Saclay/IBISC Laboratory, Paris, France

    Nazim Agoulmine

  2. Salvador University - UNIFACS, Salvador, Brazil

    Carlos E. Arruda, Pedro F. Moraes & Joberto S. B. Martins

Authors
  1. Carlos E. Arruda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro F. Moraes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nazim Agoulmine
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joberto S. B. Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joberto S. B. Martins .

Editor information

Editors and Affiliations

  1. Laboratoire LIGM UMR 8049 CNRS, ESIEE Paris, Noisy-le-Grand, France

    Éric Renault

  2. CNAM/CEDRIC, Paris, France

    Selma Boumerdassi

  3. Inria/EVA Project, Paris, France

    Paul Mühlethaler

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Arruda, C.E., Moraes, P.F., Agoulmine, N., Martins, J.S.B. (2021). Enhanced Pub/Sub Communications for Massive IoT Traffic with SARSA Reinforcement Learning. In: Renault, É., Boumerdassi, S., Mühlethaler, P. (eds) Machine Learning for Networking. MLN 2020. Lecture Notes in Computer Science(), vol 12629. Springer, Cham. https://doi.org/10.1007/978-3-030-70866-5_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-70866-5_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-70865-8

  • Online ISBN: 978-3-030-70866-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation