Skip to main content
SpringerLink
Log in

IR\(^2\)Net: information restriction and information recovery for accurate binary neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Weight and activation binarization can efficiently compress deep neural networks and accelerate model inference, but they cause severe accuracy degradation. Existing optimization methods for binary neural networks (BNNs) focus on fitting full-precision networks to reduce quantization errors and suffer from a tradeoff between accuracy and efficiency. In contrast, considering information loss and the mismatch between model capacity and input information quantity caused by network binarization, we propose Information Restriction and Information Recovery Network (IR\(^2\)Net) to stimulate the potential of BNNs and achieve improved network accuracy by restricting the input information and recovering feature information. The proposed approach includes (1) information restriction, which evaluates the feature information extracted from the input by a BNN, discards some of the information it cannot focus on, and reduces the amount of the input information to match the model capacity; and (2) information recovery: due to the information loss incurred during forward propagation, the extracted feature information of the network is not sufficient for supporting accurate classification. Shallow feature maps with richer information are selected, and these feature maps are fused with the final feature maps to recover the extracted feature information and further enhance the model capacity to match the amount of input information. In addition, the computational cost is reduced by streamlining the information recovery method to strike a better tradeoff between accuracy and efficiency. Experimental results demonstrate that our approach still achieves comparable accuracy even with a \(\sim\)10x floating-point operations (FLOPs) reduction for ResNet-18. The models and code are available at https://github.com/pingxue-hfut/IR2Net.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

All data generated or analyzed during this study are included in this article.

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  2. Wang Z, Lu J, Wu Z, Zhou J (2021) Learning efficient binarized object detectors with information compression. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3050464

    Article  Google Scholar 

  3. Tong Z, Xu P, Denoeux T (2021) Evidential fully convolutional network for semantic segmentation. Appl Intell 51(9):6376–6399. https://doi.org/10.1007/s10489-021-02327-0

    Article  Google Scholar 

  4. Ding Y, Ma Z, Wen S, Xie J, Chang D, Si Z, Wu M, Ling H (2021) AP-CNN: weakly supervised attention pyramid convolutional neural network for fine-grained visual classification. IEEE Trans Image Process 30:2826–2836. https://doi.org/10.1109/TIP.2021.3055617

    Article  Google Scholar 

  5. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  6. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  7. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, June 7–12, 2015, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  9. Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in neural information processing systems 27: annual conference on neural information processing systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp 1269–1277

  10. Wang P, Hu Q, Fang Z, Zhao C, Cheng J (2018) Deepsearch: a fast image search framework for mobile devices. ACM Trans Multim Comput Commun Appl 14(1):6–1622. https://doi.org/10.1145/3152127

    Article  Google Scholar 

  11. Singh P, Verma VK, Rai P, Namboodiri VP (2020) Acceleration of deep convolutional neural networks using adaptive filter pruning. IEEE J Sel Top Signal Process 14(4):838–847. https://doi.org/10.1109/JSTSP.2020.2992390

    Article  Google Scholar 

  12. Zhang Z, Kouzani AZ (2020) Implementation of dnns on iot devices. Neural Comput Appl 32(5):1327–1356. https://doi.org/10.1007/s00521-019-04550-w

    Article  Google Scholar 

  13. Ding G, Zhang S, Jia Z, Zhong J, Han J (2021) Where to prune: using LSTM to guide data-dependent soft pruning. IEEE Trans Image Process 30:293–304. https://doi.org/10.1109/TIP.2020.3035028

    Article  Google Scholar 

  14. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  15. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716

  16. Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C (2020) Ghostnet: more features from cheap operations. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp 1577–1586. https://doi.org/10.1109/CVPR42600.2020.00165

  17. Gao H, Wang Z, Cai L, Ji S (2021) Channelnets: compact and efficient convolutional neural networks via channel-wise convolutions. IEEE Trans Pattern Anal Mach Intell 43(8):2570–2581. https://doi.org/10.1109/TPAMI.2020.2975796

    Article  Google Scholar 

  18. Li X, Li S, Omar B, Wu F, Li X (2021) Reskd: residual-guided knowledge distillation. IEEE Trans Image Process 30:4735–4746. https://doi.org/10.1109/TIP.2021.3066051

    Article  Google Scholar 

  19. Gou J, Yu B, Maybank SJ, Tao D (2021) Knowledge distillation: a survey. Int J Comput Vis 129(6):1789–1819. https://doi.org/10.1007/s11263-021-01453-z

    Article  Google Scholar 

  20. Tan C, Liu J, Zhang X (2021) Improving knowledge distillation via an expressive teacher. Knowl Based Syst 218:106837. https://doi.org/10.1016/j.knosys.2021.106837

    Article  Google Scholar 

  21. Tung F, Mori G (2020) Deep neural network compression by in-parallel pruning-quantization. IEEE Trans Pattern Anal Mach Intell 42(3):568–579. https://doi.org/10.1109/TPAMI.2018.2886192

    Article  Google Scholar 

  22. Huang C, Liu P, Fang L (2021) MXQN: mixed quantization for reducing bit-width of weights and activations in deep convolutional neural networks. Appl Intell 51(7):4561–4574. https://doi.org/10.1007/s10489-020-02109-0

    Article  Google Scholar 

  23. Sakai Y, Tamiya Y (2021) S-dfp: shifted dynamic fixed point for quantized deep neural network training. Neural Comput Appl 1–8

  24. Hu S, Qiao GC, Chen TP, Yu Q, Liu Y, Rong LM (2021) Quantized stdp-based online-learning spiking neural network. Neural Comput Appl 33(19):12317–12332. https://doi.org/10.1007/s00521-021-05832-y

    Article  Google Scholar 

  25. Gong R, Liu X, Jiang S, Li T, Hu P, Lin J, Yu F, Yan J (2019) Differentiable soft quantization: bridging full-precision and low-bit neural networks. In: 2019 IEEE/CVF international conference on computer vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp 4851–4860. https://doi.org/10.1109/ICCV.2019.00495

  26. Zagoruyko S, Komodakis N (2017) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference track proceedings

  27. Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: Computer vision - ECCV 2016 - 14th European conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pp 525–542. https://doi.org/10.1007/978-3-319-46493-0_32

  28. Liu Z, Luo W, Wu B, Yang X, Liu W, Cheng K (2020) Bi-real net: binarizing deep network towards real-network performance. Int J Comput Vis 128(1):202–219. https://doi.org/10.1007/s11263-019-01227-8

    Article  Google Scholar 

  29. Zhuang B, Shen C, Tan M, Liu L, Reid ID (2019) Structured binary neural networks for accurate image classification and semantic segmentation. In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019, pp 413–422. https://doi.org/10.1109/CVPR.2019.00050

  30. Bethge J, Bartz C, Yang H, Chen Y, Meinel C (2021) Meliusnet: an improved network architecture for binary neural networks. In: IEEE winter conference on applications of computer vision, WACV 2021, Waikoloa, HI, USA, January 3–8, 2021, pp 1438–1447. https://doi.org/10.1109/WACV48630.2021.00148

  31. Qin H, Gong R, Liu X, Shen M, Wei Z, Yu F, Song J (2020) Forward and backward information retention for accurate binary neural networks. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13–19, 2020, pp 2247–2256. https://doi.org/10.1109/CVPR42600.2020.00232

  32. Shen M, Liu X, Gong R, Han K (2020) Balanced binary neural networks with gated residual. In: 2020 IEEE International conference on acoustics, speech and signal processing, ICASSP 2020, Barcelona, Spain, May 4–8, 2020, pp 4197–4201. https://doi.org/10.1109/ICASSP40776.2020.9054599

  33. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2016) Binarized neural networks. In: Annual conference on neural information processing systems 2016, December 5–10, 2016, Barcelona, Spain, pp 4107–4115

  34. Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with unsupervised feature learning

  35. Torralba A, Fergus R, Freeman WT (2008) 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Trans Pattern Anal Mach Intell 30(11):1958–1970. https://doi.org/10.1109/TPAMI.2008.128

    Article  Google Scholar 

  36. Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848

  37. Bulat A, Tzimiropoulos G (2019) Xnor-net++: improved binary neural networks. In: 30th British machine vision conference 2019, BMVC 2019, p 62 Cardiff, UK, September 9–12, 2019

  38. Martínez B, Yang J, Bulat A, Tzimiropoulos G (2020) Training binary neural networks with real-to-binary convolutions. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

  39. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  40. Lin M, Ji R, Xu Z, Zhang B, Wang Y, Wu Y, Huang F, Lin C (2020) Rotated binary neural network. In: Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual

  41. Lin X, Zhao C, Pan W (2017) Towards accurate binary convolutional neural network. In: Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 345–353

  42. Pouransari H, Tu Z, Tuzel O (2020) Least squares binary quantization of neural networks. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR workshops 2020, Seattle, WA, USA, June 14–19, 2020, pp 2986–2996. https://doi.org/10.1109/CVPRW50498.2020.00357

  43. Liu C, Ding W, Hu Y, Xia X, Zhang B, Liu J, Doermann D (2020) Circulant binary convolutional networks for object recognition. IEEE J Sel Top Signal Process 14(4):884–893. https://doi.org/10.1109/JSTSP.2020.2969516

    Article  Google Scholar 

  44. Zhu S, Dong X, Su H (2019) Binary ensemble neural network: more bits per network or more networks per bit? In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019, pp 4923–4932. https://doi.org/10.1109/CVPR.2019.00506

  45. Zhang T, Qi G, Xiao B, Wang J (2017) Interleaved group convolutions. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp 4383–4392. https://doi.org/10.1109/ICCV.2017.469

  46. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21–26, 2017, pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  47. Liu Z, Shen Z, Savvides M, Cheng K (2020) Reactnet: Towards precise binary neural network with generalized activation functions. In: Computer vision - ECCV 2020 - 16th European conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV, pp 143–159. https://doi.org/10.1007/978-3-030-58568-6_9

  48. Tishby N, Pereira FCN, Bialek W (2000) The information bottleneck method. arXiv:physics/0004057

  49. Zhang D, Yang J, Ye D, Hua G (2018) Lq-nets: learned quantization for highly accurate and compact deep neural networks. In: Computer vision - ECCV 2018 - 15th European conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, pp 373–390. https://doi.org/10.1007/978-3-030-01237-3_23

  50. Bengio Y, Léonard N, Courville AC (2013) Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432

  51. Bulat A, Tzimiropoulos G, Kossaifi J, Pantic M (2019) Improved training of binary networks for human pose estimation and image recognition. arXiv:1904.05868

  52. Ding R, Chin T, Liu Z, Marculescu D (2019) Regularizing activation distribution for training binarized deep networks. In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019, pp 11408–11417. https://doi.org/10.1109/CVPR.2019.01167

  53. Kim H, Kim K, Kim J, Kim J (2020) Binaryduo: reducing gradient mismatch in binary activation network by coupling binary activations. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020

  54. Wang Z, Lu J, Zhou J (2021) Learning channel-wise interactions for binary convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 43(10):3432–3445. https://doi.org/10.1109/TPAMI.2020.2988262

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the Anhui Provincial Key Research and Development Program under Grant 202004a05020040, in part by the National Key Research and Development Program under Grant 2018YFC0604404, in part by Intelligent Network and New Energy Vehicle Special Project of Intelligent Manufacturing Institute of HFUT under Grant IMIWL2019003, and in part by Fundamental Research Funds for the Central Universities under Grant PA2021GDGP0061.

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230009, China

    Ping Xue, Yang Lu, Jingfei Chang, Xing Wei & Zhen Wei

  2. Anhui Mine IOT and Security Monitoring Technology Key Laboratory, Hefei, 230009, China

    Yang Lu, Xing Wei & Zhen Wei

  3. Engineering Research Center of Safety Critical Industrial Measurement and Control Technology, Ministry of Education, Hefei University of Technology, Hefei, 230009, China

    Yang Lu & Zhen Wei

  4. Intelligent Manufacturing Institute of Hefei University of Technology, Hefei, 230009, China

    Xing Wei

Authors
  1. Ping Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingfei Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xing Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Lu.

Ethics declarations

Conflict of interest

All authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, P., Lu, Y., Chang, J. et al. IR\(^2\)Net: information restriction and information recovery for accurate binary neural networks. Neural Comput & Applic 35, 14449–14464 (2023). https://doi.org/10.1007/s00521-023-08495-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08495-z

Keywords

Search

Navigation