Abstract
Data preprocessing is a commonly used method to improve the efficiency of neural network training algorithms. In this paper, we suggest an approach for organizing parallel computations that makes it possible to preprocess data against the background of neural network training. We assume that data preprocessing is performed on the processor using multiprocessing calculations, whereas training involves graphic processors. The proposed algorithms differ in the way of organizing parallelism and interprocess communication. The methods are implemented in Python and C++ and presented as a software library. We describe the results of comparing the efficiency of the methods with the implementation of parallel preprocessing within the PyTorch framework on various test problems. Also, we give some recommendations on the method choice depending on the dataset and the batch preprocessing algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Murray, D.G., Simsa, J., Klimovic, A., Indyk, I.: tf.data: a machine learning data processing framework (2021). arXiv:2101.12127
NVIDIA DALI Documentation. https://docs.nvidia.com/deeplearning/dali/. Accessed 29 Sept 2022
Mohan, J., Phanishayee, A., Raniwala, A., Chidambaram, V.: Analyzing and mitigating data stalls in DNN training. Proc. VLDB Endow. 14(5), 771–784 (2021). https://doi.org/10.14778/3446095.3446100
Sherstinsky, A.: Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018). Elsevier “Physica D: Nonlinear Phenomena” Journal 404 (2020). Special Issue on Machine Learning and Dynamical Systems. arXiv:1808.03314. https://doi.org/10.1016/j.physd.2019.132306
Nadarajah, S., Kotz, S.: Exact distribution of the max/min of two Gaussian random variables. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 16(2), 210–212 (2008). https://doi.org/10.1109/TVLSI.2007.912191
pybind11—Seamless operability between C++11 and Python. https://github.com/pybind/pybind11. Accessed 16 May 2022
Voevodin, V.V., et al.: Supercomputer Lomonosov-2: large scale, deep monitoring and fine analytics for the user community. J. Supercomput. Front. Innov. 6(2), 4–11 (2019). https://doi.org/10.14529/jsfi190201
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
O’Shea, K., Nash, R.: An introduction to convolutional neural networks (2015). arXiv:1511.08458
Acknowledgments
The study was partly funded by the Russian Foundation for Basic Research (research project № 20-07-01053). The research was carried out on shared HPC equipment at Lomonosov Moscow State University research facilities [7].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Buryak, D., Vakhrushev, V., Shubin, M., Popova, N., Khamitov, K., Ivanov, O. (2023). Parallel Data Preprocessing Library for Neural Network Training. In: Sokolinsky, L., Zymbler, M. (eds) Parallel Computational Technologies. PCT 2023. Communications in Computer and Information Science, vol 1868. Springer, Cham. https://doi.org/10.1007/978-3-031-38864-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-38864-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38863-7
Online ISBN: 978-3-031-38864-4
eBook Packages: Computer ScienceComputer Science (R0)