Abstract
Employing convolutional neural network models for large scale datasets represents a big challenge. Especially embedded devices with limited resources cannot run most state-of-the-art model architectures in real-time, necessary for many applications. This paper proves the applicability of shunt connections on large scale datasets and narrows this computational gap. Shunt connections is a proposed method for MobileNet compression. We are the first to provide results of shunt connections for the MobileNetV3 model and for segmentation tasks on the Cityscapes dataset, using the DeeplabV3 architecture, on which we achieve compression by 28%, while observing a 3.52 drop in mIoU. The training of shunt-inserted models are optimized through knowledge distillation. The full code used for this work will be available online.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
- 3.
TensorRT: https://developer.nvidia.com/Tensorrt.
- 4.
OpenVino: https://01.org/openvinotoolkit.
- 5.
- 6.
References
Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: optimal speed and accuracy of object detection (2020)
Chen, L., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. CoRR abs/1706.05587 (2017). http://arxiv.org/abs/1706.05587
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding (2016)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009 (2009)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)
Howard, A., et al.: Searching for mobilenetv3 (2019)
Kothandaraman, D., Nambiar, A., Mittal, A.: Domain adaptive knowledge distillation for driving scene semantic segmentation (2020)
Krizhevsky, A.: Learning multiple layers of features from tiny images (2009)
Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge distillation for semantic segmentation. CoRR abs/1903.04197 (2019). http://arxiv.org/abs/1903.04197
Park, S., Heo, Y.: Knowledge distillation for semantic segmentation using channel and spatial correlations and adaptive cross entropy. Sensors 20, 4616 (2020). https://doi.org/10.3390/s20164616
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. CoRR abs/1801.04381 (2018). http://arxiv.org/abs/1801.04381
Singh, B., Toshniwal, D., Allur, S.K.: Shunt connection: an intelligent skipping of contiguous blocks for optimizing MobileNet-V2. Neural Networks 118, 192–203 (2019). https://doi.org/10.1016/j.neunet.2019.06.006. http://www.sciencedirect.com/science/article/pii/S0893608019301790
Tao, A., Sapra, K., Catanzaro, B.: Hierarchical multi-scale attention for semantic segmentation (2020)
Veit, A., Wilber, M., Belongie, S.: Residual networks behave like ensembles of relatively shallow networks (2016)
Wess, M., Ivanov, M., Unger, C., Nookala, A., Wendt, A., Jantsch, A.: ANNETTE: accurate neural network execution time estimation with stacked models. IEEE Access 9, 3545–3556 (2021). https://doi.org/10.1109/ACCESS.2020.3047259
Acknowledgements
The financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development and the Christian Doppler Research Association is gratefully acknowledged. The computational results presented have been achieved [in part] using the Vienna Scientific Cluster (VSC).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 IFIP International Federation for Information Processing
About this paper
Cite this paper
Haas, B., Wendt, A., Jantsch, A., Wess, M. (2021). Neural Network Compression Through Shunt Connections and Knowledge Distillation for Semantic Segmentation Problems. In: Maglogiannis, I., Macintyre, J., Iliadis, L. (eds) Artificial Intelligence Applications and Innovations. AIAI 2021. IFIP Advances in Information and Communication Technology, vol 627. Springer, Cham. https://doi.org/10.1007/978-3-030-79150-6_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-79150-6_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79149-0
Online ISBN: 978-3-030-79150-6
eBook Packages: Computer ScienceComputer Science (R0)
