Skip to main content
SpringerLink
Log in

Intellectual property protection for deep semantic segmentation models

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Deep neural networks have achieved great success in varieties of artificial intelligent fields. Since training a good deep model is often challenging and costly, such deep models are of great value and even the key commercial intellectual properties. Recently, deep model intellectual property protection has drawn great attention from both academia and industry, and numerous works have been proposed. However, most of them focus on the classification task. In this paper, we present the first attempt at protecting deep semantic segmentation models from potential infringements. In details, we design a new hybrid intellectual property protection framework by combining the trigger-set based and passport based watermarking simultaneously. Within it, the trigger-set based watermarking mechanism aims to force the network output copyright watermarks for a pre-defined trigger image set, which enables black-box remote ownership verification. And the passport based watermarking mechanism is to eliminate the ambiguity attack risk of trigger-set based watermarking by adding an extra passport layer into the target model. Through extensive experiments, the proposed framework not only demonstrates its effectiveness upon existing segmentation models, but also shows strong robustness to different attack techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778

  2. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd International Conference on Learning Representations. 2014

  3. Nassif A B, Shahin I, Attili I, Azzeh M, Shaalan K. Speech recognition using deep neural networks: A systematic review. IEEE Access, 2019, 7: 19143–19165

    Article  Google Scholar 

  4. Adi Y, Baum C, Cisse M, Pinkas B, Keshet J. Turning your weakness into a strength: watermarking deep neural networks by backdooring. In: Proceedings of the 27th USENIX Conference on Security Symposium. 2018, 1615–1631

  5. Chen L C, Zhu Y, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 833–851

  6. Rouhani B D, Chen H, Koushanfar F. Deepsigns: a generic watermarking framework for IP protection of deep learning models. IACR Cryptology ePrint Archive, 2018, 2018: 311

    Google Scholar 

  7. Uchida Y, Nagai Y, Sakazawa S, Satoh S. Embedding watermarks into deep neural networks. In: Proceedings of 2017 ACM on International Conference on Multimedia Retrieval. 2017, 269–277

  8. Zhang J, Chen D, Liao J, Zhang W, Feng H, Hua G, Yu N. Deep model intellectual property protection via deep watermarking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, DOI: https://doi.org/10.1109/TPAMI.2021.3064850

  9. Zhang J, Gu Z, Jang J, Wu H, Stoecklin M P, Huang H, Molloy I. Protecting intellectual property of deep neural networks with watermarking. In: Proceedings of the 2018 on Asia Conference on Computer and Communications Security. 2018, 159–172

  10. Zhang J, Chen D, Liao J, Zhang W, Hua G, Yu N. Passport-aware normalization for deep model protection. In: Proceedings of the 34th Conference on Neural Information Processing Systems. 2020, 22619–22628

  11. Fan L, Ng K W, Chan C S. Rethinking deep neural network ownership verification: embedding passports to defeat ambiguity attacks. In: Proceedings of the 33rd Conference on Neural Information Processing Systems. 2019, 4716–4725

  12. Chen L C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. 2017, arXiv preprint arXiv: 1706.05587

  13. Chen H, Rohani B D, Koushanfar F. Deepmarks: a digital fingerprinting framework for deep neural networks. IACR Cryptology ePrint Archive, 2018, 2018: 322

    Google Scholar 

  14. Zhang J, Chen D, Liao J, Fang H, Zhang W, Zhou W, Cui H, Yu N. Model watermarking for image processing networks. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 12805–12812

  15. Lim J H, Chan C S, Ng K W, Fan L X, Yang Q. Protect, show, attend and tell: empowering image captioning models with ownership protection. 2020, arXiv preprint arXiv: 2008.11009

  16. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I J, Fergus R. Intriguing properties of neural networks. In: Proceedings of the 2nd International Conference on Learning Representations. 2014

  17. Carlini N, Wagner D. Towards evaluating the robustness of neural networks. In: Proceedings of 2017 IEEE Symposium on Security and Privacy. 2017, 39–57

  18. Dong X, Chen D, Bao J, Qin C, Yuan L, Zhang W, Yu N H, Chen D. Greedyfool: distortion-aware sparse adversarial attack. In: Proceedings of the 34th Conference on Neural Information Processing Systems. 2020

  19. Dong X, Han J, Chen D, Liu J, Bian H, Ma Z, Li H, Wang X, Zhang W, Yu N. Robust superpixel-guided attentional adversarial attack. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 12892–12901

  20. Kurakin A, Goodfellow I J, Bengio S. Adversarial examples in the physical world. In: Proceedings of the 5th International Conference on Learning Representations. 2017

  21. Poursaeed O, Katsman I, Gao B, Belongie S. Generative adversarial perturbations. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 4422–4431

  22. Han J, Dong X, Zhang R, Chen D, Zhang W, Yu N, Luo P, Wang X. Once a man: towards multi-target attack via learning multi-target adversarial network once. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 5157–5166

  23. Zhou H, Chen D, Liao J, Chen K, Dong X, Liu K, Zhang W, Hua G, Yu N. LG-GAN: label guided adversarial network for flexible targeted attack of point cloud based deep networks. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 10353–10362

  24. Xie C, Wang J, Zhang Z, Zhou Y, Xie L, Yuille A. Adversarial examples for semantic segmentation and object detection. In: Proceedings of 2017 IEEE International Conference on Computer Vision. 2017, 1378–1387

  25. Razavian A S, Azizpour H, Sullivan J, Carlsson S. CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014, 512–519

  26. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations. 2015

  27. Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 3320–3328

  28. See A, Luong M T, Manning C D. Compression of neural machine translation models via pruning. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 2016, 291–301

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 61872189, 41975183, 61825601), in part by the Natural Science Foundation of Jiangsu Province (BK20191397).

Author information

Authors and Affiliations

  1. B-DAT, CICAEET, Nanjing University of Information Science & Technology, Nanjing, 211800, China

    Hongjia Ruan, Huihui Song, Yong Cheng & Qingshan Liu

  2. JD Finance America Corporation, Mountain View, 94089, USA

    Bo Liu

Authors
  1. Hongjia Ruan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huihui Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qingshan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huihui Song.

Additional information

Hongjia Ruan is currently pursuing the MS degree with the School of Automation, Nanjing University of Information Science and Technology, China. His current research interests include image/vedio super-resolution algorithms, model watermarking.

Huihui Song is a Professor with the Jiangsu Key Laboratory of Big Data Analysis Technology, Nanjing University of Information Science and Technology, China. She received her BS degree in technology and science of electronic information from Ocean University of China, China in 2008, Master’s degree in communication and information system from University of Science and Technology of China, China in 2011, and PhD degree in geography and resource management from the Chinese University of Hong Kong, China in 2014. Her research interests include remote sensing image processing and image fusion.

Bo Liu is a research scientist at JD Finance America Corporation, USA. His current research focuses on machine learning, computer vision and data analytics. He received PhD degree from the Computer Science Department, Rutgers, The State University of New Jersey, USA in 2018. Before that he worked as a research staff at The Hong Kong Polytechnic University, China. His other previous employments include Siemens Healthineers, GE Global Research and Microsoft Research Asia.

Yong Cheng received the PhD degree from the School of Computer, Wuhan University, China in 2009. Since 2010, he has been with the Nanjing University of Information Science and Technology, China. His current research interests are deep learning, computing for sensor networks, Internet of Things, and cyber-physical systems.

Qingshan Liu is a Professor with the School of Information and Control, Nanjing University of Information Science and Technology, China. He received the PhD degree from the National Laboratory of Pattern Recognition, Chinese Academic of Science, China in 2003, and the MS degree from the Department of Auto Control, Southeast University, China in 2000. He was an Assistant Research Professor with the Department of Computer Science, Computational Biomedicine Imaging and Modeling Center, Rutgers, The State University of New Jersey, USA from 2010 to 2011. Before he joined Rutgers University, he was an Associate Professor with the National Laboratory of Pattern Recognition, Chinese Academic of Science, and an Associate Researcher with the Multimedia Laboratory, Chinese University of Hong Kong, China from 2004 and 2005. He was a recipient of the President Scholarship of the Chinese Academy of Sciences, China in 2003. His current research interests are image and vision analysis, including face image analysis, graph and hypergraph-based image and video understanding, medical image analysis, and event-based video analysis.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ruan, H., Song, H., Liu, B. et al. Intellectual property protection for deep semantic segmentation models. Front. Comput. Sci. 17, 171306 (2023). https://doi.org/10.1007/s11704-021-1186-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-021-1186-y

Keywords

Search

Navigation