Skip to main content
SpringerLink
Log in

Machine to brain: facial expression recognition using brain machine generative adversarial networks

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

The human brain can effectively perform Facial Expression Recognition (FER) with a few samples by utilizing its cognitive ability. However, unlike the human brain, even the well-trained deep neural network is data-dependent and lacks cognitive ability. To tackle this challenge, this paper proposes a novel framework, Brain Machine Generative Adversarial Networks (BM-GAN), which utilizes the concept of brain’s cognitive ability to guide a Convolutional Neural Network to generate LIKE-electroencephalograph (EEG) features. More specifically, we firstly obtain EEG signals triggered from facial emotion images, then we adopt BM-GAN to carry out the mutual generation of image visual features and EEG cognitive features. BM-GAN intends to use the cognitive knowledge learnt from EEG signals to instruct the model to perceive LIKE-EEG features. Thereby, BM-GAN has a superior performance for FER like the human brain. The proposed model consists of VisualNet, EEGNet, and BM-GAN. More specifically, VisualNet can obtain image visual features from facial emotion images and EEGNet can obtain EEG cognitive features from EEG signals. Subsequently, the BM-GAN completes the mutual generation of image visual features and EEG cognitive features. Finally, the predicted LIKE-EEG features of test images are used for FER. After learning, without the participation of the EEG signals, an average classification accuracy of 96.6 % is obtained on Chinese Facial Affective Picture System dataset using LIKE-EEG features for FER. Experiments demonstrate that the proposed method can produce an excellent performance for FER.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Availability of data and materials

EEG signals data will be made available upon reasonable request for academic use and within the limitations of the provided informed consent by the corresponding author upon acceptance. Further more, EEG signals data will be made available upon reasonable academic request within the limitations of informed consent by the corresponding author upon acceptance.

References

  • Aouayeb M, Hamidouche W, Soladié C, Kpalma K, Séguier R (2021)“Learning vision transformer with squeeze and excitation for facial expression recognition,” CoRR, vol. abs/2107.03107

  • Assari MA, Rahmati M (2011) “Driver drowsiness detection using face expression recognition,” in 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pp. 337–341, IEEE

  • Barkau RL (1996)“Unet: One-dimensional unsteady flow through a full network of open channels. user’s manual,” tech. rep., Hydrologic Engineering Center Davis CA

  • Bartneck C, Lyons MJ, (2007)“Hci and the face: Towards an art of the soluble,” in International Conference on Human-computer Interaction, pp. 20–29, Springer

  • Belgiu M, Drăguţ L (2016) Random forest in remote sensing: A review of applications and future directions. ISPRS journal of photogrammetry and remote sensing 114:24–31

    Article  Google Scholar 

  • Chatrian GE, Lettich E, Nelson PL (1985) Ten percent electrode system for topographic studies of spontaneous and evoked eeg activities. American Journal of EEG technology 25(2):83–92

    Article  Google Scholar 

  • Chen Z, Liu G (2010) Application of eeg signal in emotion recognition. Computer Engineering 36(9):168–170

    Google Scholar 

  • Cogswell M, Ahmed F, Girshick R, Zitnick L, Batra D (2015) “Reducing overfitting in deep networks by decorrelating representations,” arXiv preprint arXiv:1511.06068

  • Darwin C (2015) The expression of the emotions in man and animals. University of Chicago press

  • Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee

  • Dhavalikar AS, Kulkarni R (2014) “Face detection and facial expression recognition system,” in 2014 International Conference on Electronics and Communication Systems (ICECS), pp. 1–7, IEEE

  • Dornaika F, Raducanu B, (2007)“Efficient facial expression recognition for human robot interaction,” in International Work-Conference on Artificial Neural Networks, pp. 700–708, Springer

  • Fix E, Hodges JL (1989) Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique 57(3):238–247

    Google Scholar 

  • Fong RC, Scheirer WJ, Cox DD (2018) Using human brain activity to guide machine learning. Scientific reports 8(1):1–10

    Article  Google Scholar 

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) “Generative adversarial nets,” Advances in neural information processing systems, vol. 27,

  • Happy S, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE transactions on Affective Computing 6(1):1–12

    Article  Google Scholar 

  • Harper ER, Rodden T, Rogers Y, Sellen A, Human B, et al. (2008) “Human-computer interaction in the year 2020,”

  • Hartmann KG, Schirrmeister RT, Ball T (2018) “Eeg-gan: Generative adversarial networks for electroencephalograhic (eeg) brain signals,”

  • He K, Zhang X, Ren S, Sun J (2016) “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

  • Horlings R, Datcu D, Rothkrantz LJ (2008)“Emotion recognition using brain activity,” in Proceedings of the 9th international conference on computer systems and technologies and workshop for PhD students in computing, pp. II–1

  • Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al. (2019) “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324

  • Hwang S, Hong K, Son G, Byun H (2019) “Ezsl-gan: Eeg-based zero-shot learning approach using a generative adversarial network,” in 2019 7th International Winter Conference on Brain-Computer Interface (BCI)

  • Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016)“Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,” arXiv preprint arXiv:1602.07360

  • Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) “Densenet: Implementing efficient convnet descriptor pyramids,” arXiv preprint arXiv:1404.1869

  • Jebelli H, Hwang S, Lee S (2018) Eeg signal-processing framework to obtain high-quality brain waves from an off-the-shelf wearable eeg device. Journal of Computing in Civil Engineering 32(1):04017070

    Article  Google Scholar 

  • Khosla A, Khandnor P, Chand T (2020) A comparative analysis of signal processing and classification methods for different applications based on eeg signals. Biocybernetics and Biomedical Engineering 40(2):649–690

    Article  Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25:1097–1105

    Google Scholar 

  • Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces. Journal of neural engineering 15(5):056013

    Article  PubMed  Google Scholar 

  • Lebedev MA, Nicolelis MA (2006) Brain-machine interfaces: past, present and future. TRENDS in Neurosciences 29(9):536–546

    Article  CAS  PubMed  Google Scholar 

  • LeCun Y, Bengio Y, Hinton G (2015) “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444

  • Li S, Deng W, (2020)“Deep facial expression recognition: A survey,” IEEE transactions on affective computing

  • Ling W, Kong W, Long Y, Yang C, Jin X, (2019)“Facial emotion recognition based on brain and machine collaborative intelligence,” in 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), pp. 1–5, IEEE

  • Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) “Coding facial expressions with gabor wavelets,” in Proceedings Third IEEE international conference on automatic face and gesture recognition, pp. 200–205, IEEE

  • Ma N, Zhang, Zheng HT, Sun J (2018) “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in Proceedings of the European conference on computer vision (ECCV), pp. 116–131

  • Minaee S, Minaei M, Abdolrashidi A (2021) “Deep-emotion: Facial expression recognition using attentional convolutional network,” Sensors, vol. 21, no. 9

  • Money AG, Agius H (2010) “Elvis: entertainment-led video summaries,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 6, no. 3, pp. 1–30

  • Moridian P, Shoeibi A, Khodatars M, Jafari M, Pachori RB, Khadem A, Alizadehsani R, Ling SH (2022) Automatic diagnosis of sleep apnea from biomedical signals using artificial intelligence techniques: Methods, challenges, and future works. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12(6):e1478

    Google Scholar 

  • Mourão A, Magalhães J, (2013)“Competitive affective gaming: winning with a smile,” in Proceedings of the 21st ACM international conference on Multimedia, pp. 83–92

  • Nishida S, Nakano Y, Blanc A, Maeda N, Kado M, Nishimoto S (2020) Brain-mediated transfer learning of convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 34:5281–5288

    Article  Google Scholar 

  • Palazzo S, Rundo F, Battiato S, Giordano D, Spampinato C (2020) “Visual saliency detection guided by neural signals,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 525–531, IEEE

  • Pramerdorfer C, Kampel M, (2016) “Facial expression recognition using convolutional neural networks: State of the art,” CoRR, vol. abs/1612.02903

  • Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation 29(9):2352–2449

    Article  PubMed  Google Scholar 

  • Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K (2020) Human behavior understanding in big multimedia data using cnn based facial expression recognition. Mobile networks and applications 25(4):1611–1621

    Article  Google Scholar 

  • Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520

  • Schulz M-A, Yeo BT, Vogelstein JT, Mourao-Miranada J, Kather JN, Kording K, Richards B, Bzdok D (2020) Different scaling of linear models and deep learning in ukbiobank brain images versus machine-learning datasets. Nature communications 11(1):1–15

    Article  Google Scholar 

  • Shoeibi A, Ghassemi N, Khodatars M, Moridian P, Khosravi A, Zare A, Gorriz JM, Chale-Chale AH, Khadem A, Rajendra Acharya U (2022) “Automatic diagnosis of schizophrenia and attention deficit hyperactivity disorder in rs-fmri modality using convolutional autoencoder model and interval type-2 fuzzy regression,” Cognitive Neurodynamics, pp. 1–23

  • Shoeibi A, Rezaei M, Ghassemi N, Namadchian Z, Zare A, Gorriz JM (2022)“Automatic diagnosis of schizophrenia in eeg signals using functional connectivity features and cnn-lstm model,” in International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 63–73, Springer

  • Simonyan K, Zisserman A (2014) “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556

  • Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M (2016) Analysis of eeg signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing 7(1):17–28

    Article  Google Scholar 

  • Soleymani M, Kierkels JJ, Chanel G, Pun T (2009) “A bayesian framework for video affective representation,” in 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pp. 1–7, IEEE

  • Soleymani M, Pantic M, (2013) “Multimedia implicit tagging using eeg signals,” in 2013 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, IEEE

  • Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M (2017) “Deep learning human mind for automated visual classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6809–6817

  • Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

  • Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) “Mnasnet: Platform-aware neural architecture search for mobile,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820–2828

  • Tian Y-I, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence 23(2):97–115

    Article  PubMed  PubMed Central  Google Scholar 

  • van der Maaten L, Hinton G (2008) Visualizing data using t-sne. Journal of Machine Learning Research 9(86):2579–2605

    Google Scholar 

  • Xie S, Girshick R, Dollár P, Tu Z, He K (2017) “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500

  • Xinying L, Xuebing L et al (2006) Effects of emotion on cognitive processing: Series of event-related potentials study. Advances in Psychological Science 14(04):505

    Google Scholar 

  • Yang H, Ciftci U, Yin L, (2018) “Facial expression recognition by de-expression residue learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2168–2177

  • Yolcu G, Oztel I, Kazan S, Oz C, Palaniappan K, Lever TE, Bunyak F (2017) “Deep learning-based facial expression recognition for monitoring neurological disorders,” in 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1652–1657, IEEE

  • Zagoruyko S, Komodakis N (2016) “Wide residual networks,” arXiv preprint arXiv:1605.07146

  • Zhan C, Li W, Ogunbona P, Safaei F (2008) “A real-time facial expression recognition system for online games,” International Journal of Computer Games Technology, vol. 2008

  • Zhu JY, Park T, Isola P, Efros AA, (2017)“Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (U20B2074), Key Research and Development Project of Zhejiang Province (2023C03026, 2021C03001, 2021C03003), and supported by Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province (2020E10010).

Author information

Authors and Affiliations

  1. School of Computer Science, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China

    Dongjun Liu, Jin Cui, Zeyu Pan, Hangkui Zhang & Wanzeng Kong

  2. Graduate School of Engineering, Saitama Institute of Technology, Saitama, 369-0293, Japan

    Jianting Cao

  3. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, 310018, Zhejiang, China

    Wanzeng Kong

Authors
  1. Dongjun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zeyu Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hangkui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianting Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wanzeng Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wanzeng Kong.

Ethics declarations

Ethics approval

The study was reviewed and approved by the institutional Ethical Review Committee at Saitama Institute of Technology and the protocol number is 2018-01. Signed informed consent was obtained from each participant.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, D., Cui, J., Pan, Z. et al. Machine to brain: facial expression recognition using brain machine generative adversarial networks. Cogn Neurodyn (2023). https://doi.org/10.1007/s11571-023-09946-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11571-023-09946-y

Keywords

Search

Navigation