Skip to main content
SpringerLink
Log in

Deep Reinforcement Learning for Continuous Control of Material Thickness

  • Conference paper
  • First Online:
Artificial Intelligence XL (SGAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14381))

  • 507 Accesses

Abstract

To achieve the desired quality standards of certain manufactured materials, the involved parameters are still adjusted by knowledge-based procedures according to human expertise, which can be costly and time-consuming. To optimize operational efficiency and provide decision support for human experts, we develop a general continuous control framework that utilizes deep reinforcement learning (DRL) to automatically determine the main control parameters, in situations where simulation environments are unavailable and traditional PID controllers are not viable options. In our work, we aim to automatically learn the key control parameters to achieve the desired outlet thickness of the manufactured material. We first construct a surrogate environment based on real-world expert trajectories obtained from the true underlining manufacturing process to achieve this. Subsequently, we train a DRL agent within the surrogate environment. Our results suggest a Proximal Policy Optimization (PPO) algorithm combined with a Multi-Layer Perceptron (MLP) surrogate environment to successfully learn a policy that continuously changes parameter configurations optimally, achieving the desired target material thickness within an acceptable range.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Andrychowicz, M., et al.: What matters for on-policy deep actor-critic methods? A large-scale study. In: International Conference on Learning Representations (2021)

    Google Scholar 

  2. Araki, M.: PID control. In: Control Systems, Robotics and Automation: System Analysis and Control: Classical Approaches II, 58–79 (2009)

    Google Scholar 

  3. Bennett, S.: Development of the PID controller. IEEE Control Syst. Mag. 13(6), 58–62 (1993)

    Article  Google Scholar 

  4. Brockman, G., et al.: OpenAI Gym (2016)

    Google Scholar 

  5. Deng, J., Sierla, S., Sun, J., Vyatkin, V.: Reinforcement learning for industrial process control: a case study in flatness control in steel industry. Comput. Ind. 143, 103748 (2022)

    Article  Google Scholar 

  6. Dornheim, J., Link, N., Gumbsch, P.: Model-free adaptive optimal control of episodic fixed-horizon manufacturing processes using reinforcement learning. Int. J. Control Autom. Syst. 18, 1593–1604 (2020)

    Article  Google Scholar 

  7. Gamal, O., Mohamed, M.I.P., Patel, C.G., Roth, H.: Data-driven model-free intelligent roll gap control of bar and wire hot rolling process using reinforcement learning. Int. J. Mech. Eng. Robot. Res. 10(7), 349–356 (2021)

    Article  Google Scholar 

  8. Guo, F., Zhou, X., Liu, J., Zhang, Y., Li, D., Zhou, H.: A reinforcement learning decision model for online process parameters optimization from offline data in injection molding. Appl. Soft Comput. 85, 105828 (2019)

    Article  Google Scholar 

  9. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)

    Article  MATH  Google Scholar 

  10. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  11. Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. 17(1), 1334–1373 (2016)

    MathSciNet  MATH  Google Scholar 

  12. Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37(4–5), 421–436 (2018)

    Article  Google Scholar 

  13. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

  14. Lin, L.J.: Reinforcement Learning for Robots Using Neural Networks. Carnegie Mellon University (1992)

    Google Scholar 

  15. Martínez, M.A., Sanchis, J., Blasco, X.: Multiobjective controller design handling human preferences. Eng. Appl. Artif. Intell. 19(8), 927–938 (2006)

    Article  Google Scholar 

  16. Mazgualdi, C.E., Masrour, T., Hassani, I.E., Khdoudi, A.: A deep reinforcement learning (DRL) decision model for heating process parameters identification in automotive glass manufacturing. In: Masrour, T., Cherrafi, A., El Hassani, I. (eds.) A2IA 2020. AISC, vol. 1193, pp. 77–87. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-51186-9_6

    Chapter  Google Scholar 

  17. Mirhoseini, A., et al.: A graph placement methodology for fast chip design. Nature 594(7862), 207–212 (2021)

    Article  Google Scholar 

  18. Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  19. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  20. Ruelens, F., Claessens, B.J., Quaiyum, S., De Schutter, B., Babuška, R., Belmans, R.: Reinforcement learning applied to an electric water heater: from theory to practice. IEEE Trans. Smart Grid 9(4), 3792–3800 (2016)

    Article  Google Scholar 

  21. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897. PMLR (2015)

    Google Scholar 

  22. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017)

    Google Scholar 

  23. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  Google Scholar 

  24. Stewart, G., Samad, T.: Cross-application perspectives: application and market requirements. Impact Control Technol. 95–100 (2011)

    Google Scholar 

  25. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  26. Wu, T., Zhao, H., Gao, B., Meng, F.: Energy-saving for a velocity control system of a pipe isolation tool based on a reinforcement learning method. Int. J. Precis. Eng. Manuf.-Green Technol. 9(1), 225–240 (2021). https://doi.org/10.1007/s40684-021-00309-8

    Article  Google Scholar 

  27. Yu, J., Guo, P.: Run-to-run control of chemical mechanical polishing process based on deep reinforcement learning. IEEE Trans. Semicond. Manuf. 33(3), 454–465 (2020)

    Article  Google Scholar 

  28. Zinn, J., Vogel-Heuser, B., Gruber, M.: Fault-tolerant control of programmable logic controller-based production systems with deep reinforcement learning. J. Mech. Des. 143(7), 072004 (2021)

    Article  Google Scholar 

  29. Zirngibl, C., Dworschak, F., Schleich, B., Wartzack, S.: Application of reinforcement learning for the optimization of clinch joint characteristics. Prod. Eng. Res. Devel. 16(2–3), 315–325 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Liverpool, Liverpool, UK

    Oliver Dippel, Alexei Lisitsa & Bei Peng

  2. Centre for Doctoral Training in Distributed Algorithms, Liverpool, UK

    Oliver Dippel

Authors
  1. Oliver Dippel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexei Lisitsa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bei Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oliver Dippel .

Editor information

Editors and Affiliations

  1. University of Portsmouth, Portsmouth, UK

    Max Bramer

  2. DFKI: German Research Center for Artificial Intelligence, Oldenburg, Germany

    Frederic Stahl

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dippel, O., Lisitsa, A., Peng, B. (2023). Deep Reinforcement Learning for Continuous Control of Material Thickness. In: Bramer, M., Stahl, F. (eds) Artificial Intelligence XL. SGAI 2023. Lecture Notes in Computer Science(), vol 14381. Springer, Cham. https://doi.org/10.1007/978-3-031-47994-6_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47994-6_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47993-9

  • Online ISBN: 978-3-031-47994-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation