Abstract
With the rise of realistic face forgery techniques, the threat of identity fraud is more significant than ever. Several research works have focused on detecting such forgeries, but they extract forgery clues as a preprocessing step to the feature extraction phase of deep neural networks. A novel DenseTrace-Net architecture is designed in this manuscript to extract more comprehensive and refined face tampering traces locally and globally. Specifically, DenseTrace-Net extracts attentional multi-level tampering traces from facial images. A novel ‘Local Attentional Tamper Trace Extractor’ (LATTE) module extracts face tampering traces locally at the block level. A novel ‘Global Attentional Tamper Trace Extractor’ (GATTE) module aggregates multi-scale tampering traces globally. The LATTE and GATTE modules use visual depth attention to enhance their feature representation capability. Additionally, the proposed DenseTrace-Net is computationally lightweight with just 1.378 million parameters. DenseTrace-Net is evaluated on three benchmark datasets, the FF + + , CelebDF and DFDC datasets, achieving AUC scores of 0.9784, 0.9843 and 0.9916, respectively. These excellent scores allow the DenseTrace-Net to outperform the existing state-of-the-art face forgery detection methods comfortably.
Similar content being viewed by others
Data availability
The datasets generated during and/or analyzed during the current study are available through an online web repository via the following web links: CelebDF—https://cse.buffalo.edu/~siweilyu/celeb-deepfakeforensics.html. DFDC—https://ai.facebook.com/datasets/dfdc/. FaceForensics + +—https://github.com/ondyari/FaceForensics.
References
“DeepFakes,” GitHub, 14 August 2020. [Online]. https://github.com/deepfakes/faceswap. Accessed 08 July 2022
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. Commun. ACM 62(1), 96–104 (2019)
“DeepFaceLab,” GitHub, 18 March 2020. [Online]. https://github.com/iperov/DeepFaceLab. Accessed 08 July 2022
Wang, G., Jiang, Q., Jin, X., Li, W., Cui, X.: MC-LCR: Multimodal contrastive classification by locally correlated representations for effective face forgery detection. Knowl. Based Syst. 250, 109114 (2022)
Hu, Z., Duan, Q., Zhang, P., Tao, H.: An attention-erasing stripe pyramid network for face forgery detection. Signal Image Video Process. (2023). https://doi.org/10.1007/s11760-023-02644-6
Masud, U., Sadiq, M., Masood, S., Ahmad, M., El-Latif, A.A.A.: LW-DeepFakeNet: a light-weight time distributed CNN-LSTM network for real-time DeepFake video detection. Signal Image Video Process. (2023). https://doi.org/10.1007/s11760-023-02633-9
Shang, Z., Xie, H., Zha, Z., Yu, L., Li, Y., Zhang, Y.: PRRNet: pixel-region relation network for face forgery detection. Pattern Recognit (2021). https://doi.org/10.1016/j.patcog.2021.107950
Xu, Z., Liu, J., Lu, W., Xu, B., Zhao, X., Li, B., Huang, J.: Detecting facial manipulated videos based on set convolutional neural networks. J. Vis. Commun. Image Represent. (2021). https://doi.org/10.1016/j.jvcir.2021.103119
Chen, Z., Yang, H.: Attentive semantic exploring for manipulated face detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto (2021)
Heo, Y.J., Yeo, W.-H., Kim, B.-G.: DeepFake detection algorithm based on improved vision transformer. Appl. Intell. 53, 7512–7527 (2023)
Bonomi, M., Pasquini, C., Boato, G.: Dynamic texture analysis for detecting fake faces in video sequences. J. Vis. Commun. Image Represent. (2021). https://doi.org/10.1016/j.jvcir.2021.103239
Yang, J., Xiao, S., Li, A., Lan, G., Wang, H.: Detecting fake images by identifying potential texture difference. Futur. Gener. Comput. Syst. 125, 127–135 (2021)
Yang, J., Li, A., Xiao, S., Lu, W., Gao, X.: MTD-Net: Learning to Detect Deepfakes Images by Multi-Scale Texture Difference. IEEE Trans. Inf. Forensics Secur. 16, 4234–4245 (2021)
Caldelli, R., Galteri, L., Amerini, I., Bimbo, A.D.: Optical Flow based CNN for detection of unlearnt deepfake. Pattern Recogn. Lett. 146, 31–37 (2021)
Kohli, A., Gupta, A.: Detecting DeepFake, FaceSwap and Face2Face facial forgeries using frequency CNN. Multimedia Tools and Applications 80, 18461–18478 (2022)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16×16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR), Austria (2021)
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: CvT: Introducing convolutions to vision transformers. In: International Conference on Computer Vision (ICCV), Montreal (2021)
Heo, Y.J., Yeo, W.-H., Kim, B.-G.: DeepFake detection algorithm based on improved vision transformer. Appl. Intell. (2022)
Guo, Z., Yang, G., Chen, J., Sun, X.: Fake face detection via adaptive manipulation traces extraction network. Comput. Vis. Image Underst. (2021). https://doi.org/10.1016/j.cviu.2021.103170
Amerini, I., Galteri, L., Caldelli, R., Bimbo, A.D.: Deepfake video detection through optical flow based CNN. In: IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul (2019)
Caldelli, R., Galteri, L., Amerini, I., Bimbo, A.D.: Optical flow based CNN for detection of unlearnt deepfake manipulations. Pattern Recogn. Lett. 146, 31–37 (2021)
Hu, J., Liao, X., Wang, W., Qin, Z.: Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Trans. Circuits Syst. Video Technol. (2021). https://doi.org/10.1109/TCSVT.2021.3074259
Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.K.: On the detection of digital face manipulation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle (2020)
Choi, D.H., Lee, H.J., Lee, S., Kim, J.U., Ro, Y.M.: Fake video detection with certainty-based attention network. In: IEEE International Conference on Image Processing (ICIP), Abu Dhabi (2020)
Lu, C., Liu, B., Zhou, W., Chu, Q., Yu, N.: Deepfake video detection using 3D-attentional inception convolutional neural network. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas (2016)
Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European Conference on Computer Vision, Amsterdam (2016)
Huang, G., Liu, Z., Maaten, L.V.D., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu (2017)
J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018
Park, J., Woo, S., Lee, J.-Y., Kweon, I.S.: BAM: bottleneck attention module (2018). https://arxiv.org/pdf/1807.06514.pdf
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: European Conference on Computer Vision, Munich, Germany (2018)
Zhou, Y., Luo, A., Kang, X., Lyu, S.: Face forgery detection based on segmentation network. In: IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA (2021)
Zhang, J., Ni, J., Xie, H.: DeepFake videos detection using self-supervised decoupling network. In: IEEE International Conference on Multimedia and Expo (ICME), Shenzhen (2021)
Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for DeepFake forensics. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (2020)
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Ferrer, C.C.: The deepfake detection challenge (DFDC) preview dataset (2019). https://arxiv.org/abs/1910.08854
Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: FaceForensics++: learning to detect manipulated facial images. In: IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South) (2019)
Yang, G., Wei, A., Fang, X., Zhang, J.: FDS_2D: rethinking magnitude-phase features for DeepFake detection. Multimed. Syst. (2023). https://doi.org/10.1007/s00530-023-01118-6
Guo, Z., Yang, G., Zhang, D., Xia, M.: Rethinking gradient operator for exposing AI-enabled face forgeries. Expert Syst. Appl. (2023). https://doi.org/10.1016/j.eswa.2022.119361
Xu, K., Yang, G., Fang, X., Zhang, J.: Facial depth forgery detection. Multimed. Tools Appl. (2023). https://doi.org/10.1007/s11042-023-14626-4
Lin, H., Huang, W., Luo, W., Lu, W.: DeepFake detection with multi-scale convolution and vision transformer. Digit. Signal Process. (2023). https://doi.org/10.1016/j.dsp.2022.103895
Yang, G., Xu, K., Fang, X., Zhang, J.: Video face forgery detection via facial motion-assisted capturing dense optical flow truncation. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02683-z
Nirkin, Y., Wolf, L., Keller, Y., Hassner, T.: DeepFake detection based on discrepancies between faces and their context. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6111–6121 (2022)
Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA (2021)
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., Yu, N.: Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA (2021)
Hu, J., Liao, X., Wang, W., Qin, Z.: Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1089–1102 (2021)
Chen, H.-S., Rouhsedaghat, M., Ghani, H., Hu, S., You, S., Kuo, C.-C.J.: DefakeHop: a light-weight high-performance deepfake detector. In: IEEE International Conference on Multimedia and Expo (ICME), Shenzhen (2021)
Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., Zhao, G.: Searching central difference convolutional networks for face anti-spoofing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA (2020)
Baek, J.-Y., Yoo, Y.-S., Bae, S.-H.: Generative adversarial ensemble learning for face forensics. IEEE Access 8, 45421–45431 (2020)
Zi, B., Chang, M., Chen, J., Ma, X., Jiang, Y.-G.: WildDeepfake: a challenging real-world dataset for deepfake detection. In: 28th ACM International Conference on Multimedia (2020)
Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network. In: IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, China (2018)
Mohiuddin, S., Sheikh, K.H., Malakar, S., Velásquez, J.D., Sarkar, R.: A hierarchical feature selection strategy for deepfake video detection. Neural Comput. Appl. (2023). https://doi.org/10.1007/s00521-023-08201-z
Deng, L., Wang, J., Liu, Z.: Cascaded network based on EfficientNet and transformer for deepfake video detection. Neural. Process. Lett. (2023). https://doi.org/10.1007/s11063-023-11249-6
Asha, S., Vinod, P., Menon, V.G.: A defensive framework for deepfake detection under adversarial settings using temporal and spatial features. Int. J. Inf. Secur. (2023). https://doi.org/10.1007/s10207-023-00695-x
Ke, J., Wang, L.: DF-UDetector: An effective method towards robust deepfake detection via feature restoration. Neural Netw. 160, 216–226 (2023)
Guo, Z., Yang, G., Wang, D., Zhang, D.: A data augmentation framework by mining structured features for fake face image detection. Comput. Vis. Image Underst. (2023). https://doi.org/10.1016/j.cviu.2022.103587
Zhao, C., Wang, C., Hu, G., Chen, H., Liu, C., Tang, J.: ISTVT: interpretable spatial-temporal video transformer for deepfake detection. IEEE Trans. Inf. Forensics Secur. 18, 1335–1348 (2023)
Yu, Y., Zhao, X., Ni, R., Yang, S., Zhao, Y., Kot, A.C.: Augmented multi-scale spatiotemporal inconsistency magnifier for generalized DeepFake detection. IEEE Trans. Multimed. 99, 1–13 (2023)
Yang, Z., Liang, J., Xu, Y., Zhang, X.-Y., He, R.: Masked relation learning for DeepFake detection. IEEE Trans. Inf. Forensics Secur. 18, 1696–1708 (2023)
Ganguly, S., Ganguly, A., Mohiuddin, S., Malakar, S., Sarkar, R.: ViXNet: vision transformer with xception network for deepfakes based video and image forgery detection. Expert Syst. Appl. (2022). https://doi.org/10.1016/j.eswa.2022.118423
Ganguly, S., Mohiuddin, S., Malakar, S., Cuevas, E., Sarkar, R.: Visual attention-based deepfake video forgery detection. Pattern Anal. Appl. 25, 981–992 (2022)
Nadimpalli, A.V., Rattani, A.: On improving cross-dataset generalization of deepfake detectors. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA (2022)
Li, G., Cao, Y., Zhao, X.: Exploiting facial symmetry to expose deepfakes. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., Guo, B.: Face X-ray for more general face forgery detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle (2020)
Luo, Z., Kamata, S.-I., Sun, Z.: Transformer and node-compressed Dnn based dual-path system for manipulated face detection. In: IEEE International Conference on Image Processing (ICIP), Anchorage (2021)
Qi, H., Guo, Q., Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., Zhao, J.: DeepRhythm: exposing DeepFakes with attentional visual heartbeat rhythms. In: 28th ACM International Conference on Multimedia, Lisboa (2020)
Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., Lu, Q.: Sharp multiple instance learning for DeepFake video detection. In: Proceedings of the 28th ACM International Conference on Multimedia, Seattle WA USA (2020)
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emotions don’t lie: an audio-visual deepfake detection method using affective cues. In: 28th ACM International Conference on Multimedia, Lisboa (2020)
Montserrat, D.M., Hao, H., Yarlagadda, S.K., Baireddy, S., Shao, R., Horváth, J., Bartusiak, E., Yang, J., Güera, D., Zhu, F., Delp, E.J.: Deepfakes detection with automatic face weighting. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle (2020)
Chugh, K., Gupta, P., Dhall, A., Subramanian, R.: Not made for each other-audio-visual dissonance-based deepfake detection and localization. In: 28th ACM International Conference on Multimedia, Lisboa (2020)
Li, G., Zhao, X., Cao, Y.: Forensic symmetry for DeepFakes. IEEE Trans. Inf. Forensics Secur. 18, 1095–1110 (2023)
Qian, Y., Yin, G., Sheng, L., Chen, Z., Shao, J.: Thinking in frequency: face forgery detection by mining frequency-aware clues. In: European Conference on Computer Vision (2020)
Funding
No funding was received to assist with the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
Ankit Yadav: Software, Validation, Investigation, Data Curation, Writing – Original Draft, Visualization. Dhruv Gupta: Software, Validation, Investigation, Data Curation, Writing – Original Draft, Visualization. Dinesh Kumar Vishwakarma: Conceptualization, Methodology, Formal Analysis, Resources, Writing – Review & Editing, Supervision, Project Administration, Funding Acquisition.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yadav, A., Gupta, D. & Vishwakarma, D.K. Uncovering visual attention-based multi-level tampering traces for face forgery detection. SIViP 18, 1259–1272 (2024). https://doi.org/10.1007/s11760-023-02774-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-023-02774-x