Skip to main content
SpringerLink
Log in

An Advanced Data Augmentation Scheme on Limited EEG Signals for Human Emotion Recognition

  • Conference paper
  • First Online:
Proceeding of the 3rd International Conference on Electronics, Biomedical Engineering, and Health Informatics

Part of the book series: Lecture Notes in Electrical Engineering ((LNEE,volume 1008))

Abstract

EEG signals are the modality that is widely used to recognize human emotions. However, the limited data on EEG signals remains challenging because of the small recording participants, the need for an expert to interpret EEG signals, and the expensive cost of tools to record EEG signals. This research proposed the data augmentation schemes on the EEG datasets to overcome the limited available data problem. Augmenting the data will help the generalizability of the emotion recognition model. The EEG signals on the DEAP and SEED datasets are transformed into image samples using a recurrence plot and spectrogram. Then, the artificial recurrence plot and the artificial spectrogram samples are generated using Pix2pix. This research used these artificial samples to conduct the data augmentation process. LeNet5, ResNet50, MobileNet, and DenseNet121 are used to conduct the classification. The best four data augmentation schemes are as follows: Appending 20,000 artificial recurrence plot samples to DEAP and SEED training datasets, appending 20,000 artificial spectrogram samples to the DEAP training dataset, and appending 15,000 artificial spectrogram samples to the SEED training dataset. The kappa coefficient for each classification model based on the best data augmentation schemes is computed. It is found that among the compared classifiers, LeNet5 achieved the best accuracy in both SEED (98.58%) and DEAP (86.12%) datasets when spectrogram was used. Therefore, LeNet5 trained on the spectrogram samples is a reliable and robust classification model. This finding implies that the use of spectrogram is more promising than the recurrence plot in human emotion recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.eecs.qmul.ac.uk/mmv/datasets/deap/index.html.

  2. 2.

    https://bcmi.sjtu.edu.cn/home/seed/seed.html.

References

  1. Ren F, Dong Y, Wang W (2019) Emotion recognition based on physiological signals using brain asymmetry index and echo state network. Neural Comput Appl 31:4491–4501. https://doi.org/10.1007/s00521-018-3664-1

    Article  Google Scholar 

  2. Ziye L, Yan W, Derui S, Yiwen Q, Xiaofei S (2021) Speech emotion recognition based on transfer learning of spectrogram. In: Lecture Notes in Electrical Engineering. pp 1963–1967. Springer Singapore, Singapore. https://doi.org/10.1007/978-981-15-8411-4_259

  3. Wang Y, Wang X, He C (2021) Speech emotion recognition algorithm for school bullying detection based on MFCC-PCA-SVM classification. In: Lecture Notes in Electrical Engineering. pp 1088–1093. Springer Singapore, Singapore. https://doi.org/10.1007/978-981-15-8411-4_143

  4. Gu H, Chen Q, Xing X, Zhao J, Li X (2019) Facial emotion recognition in deaf children: Evidence from event-related potentials and event-related spectral perturbation analysis. Neurosci Lett 703:198–204. https://doi.org/10.1016/j.neulet.2019.01.032

    Article  Google Scholar 

  5. Abhang PA, Gawali BW, Mehrotra SC (2016) Introduction to EEG and Speech-based emotion recognition. Elsevier. https://doi.org/10.1016/C2015-0-01959-1

    Article  Google Scholar 

  6. Song T, Zheng W, Lu C, Zong Y, Zhang X, Cui Z (2019) MPED: A Multi-Modal physiological emotion database for discrete emotion recognition. IEEE Access. 7:12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579

    Article  Google Scholar 

  7. Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F (2018) A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng 15:031005. https://doi.org/10.1088/1741-2552/aab2f2

    Article  Google Scholar 

  8. Harada S, Hayashi H, Uchida S (2018) Biosignal Data augmentation based on generative adversarial networks. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. pp 368–371. https://doi.org/10.1109/EMBC.2018.8512396.

  9. Chai X, Wang Q, Zhao Y, Liu X, Liu D, Bai O (2018) Multi-subject subspace alignment for non-stationary EEG-based emotion recognition. Technol Heal Care 26:327–335. https://doi.org/10.3233/THC-174739

    Article  Google Scholar 

  10. Luo Y, Zhu L-Z, Lu B-L (2019) A GAN-Based data augmentation method for multimodal emotion recognition. In: Advances in Neural Networks—ISNN 2019. ISNN 2019. Lecture Notes in Computer Science. pp 141–150. Springer, Cham, Shanghai. https://doi.org/10.1007/978-3-030-22796-8_16

  11. Zhao W, Zhao Z, Li C (2018) Discriminative-CCA promoted By EEG signals for physiological-based emotion recognition. In: 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia). IEEE, College of Computer and Information Engineering, Tianjin Normal University, Tianjin, China, pp 1–6. https://doi.org/10.1109/ACIIAsia.2018.8470373.

  12. Ma, X, Liu Z, Jiang T, Zhang X (2019) Study of the algorithm for the classification of brain waves. In: Lecture Notes in Electrical Engineering. Springer Verlag, China, pp 2325–2331. https://doi.org/10.1007/978-981-10-6571-2_283

  13. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press

    Google Scholar 

  14. Wang F, Zhong S, Peng J, Jiang J, Liu Y (2018) Data augmentation for EEG-Based emotion recognition with deep convolutional neural networks. In: Lecture Notes in Computer Science. Springer, Cham, Shenzhen, pp 82–93. https://doi.org/10.1007/978-3-319-73600-6_8

  15. Luo Y, Lu B-L (2018) EEG data augmentation for emotion recognition using a conditional wasserstein GAN. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, Shanghai, pp 2535–2538. https://doi.org/10.1109/EMBC.2018.8512865

  16. Zhang A, Su L, Zhang Y, Fu Y, Wu L, Liang S (2022) EEG data augmentation for emotion recognition with a multiple generator conditional Wasserstein GAN. Complex Intell Syst 8:3059–3071. https://doi.org/10.1007/s40747-021-00336-7

    Article  Google Scholar 

  17. Luo Y, Zhu L-Z, Wan Z-Y, Lu B-L (2020) Data augmentation for enhancing EEG-based emotion recognition with deep generative models. J Neural Eng 17:056021. https://doi.org/10.1088/1741-2552/abb580

    Article  Google Scholar 

  18. Klonowski W (2009) Everything you wanted to ask about EEG but were afraid to get the right answer. Nonlinear Biomed Phys 3:2. https://doi.org/10.1186/1753-4631-3-2

    Article  Google Scholar 

  19. Shankar A, Khaing HK, Dandapat S, Barma S (2021) Analysis of epileptic seizures based on EEG using recurrence plot images and deep learning. Biomed Signal Process Control 69:102854. https://doi.org/10.1016/j.bspc.2021.102854

    Article  Google Scholar 

  20. Vrbancic G, Podgorelec V (2018) Automatic Classification of motor impairment neural disorders from EEG signals using deep convolutional neural networks. Elektron ir Elektrotechnika 24:1–7. https://doi.org/10.5755/j01.eie.24.4.21469

    Article  Google Scholar 

  21. Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16:031001. https://doi.org/10.1088/1741-2552/ab0ab5

    Article  Google Scholar 

  22. Koelstra S, Muhl C, Soleymani M, Jong-Seok Lee, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: A database for emotion analysis ;using physiological signals. IEEE Trans Affect Comput 3, 18–31. https://doi.org/10.1109/T-AFFC.2011.15

  23. Zheng W, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-Based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7:162–175. https://doi.org/10.1109/TAMD.2015.2431497

    Article  Google Scholar 

  24. Eckmann J-P, Kamphorst SO, Ruelle D (1987) Recurrence plots of dynamical systems. Europhys Lett 4:973–977. https://doi.org/10.1209/0295-5075/4/9/004

    Article  Google Scholar 

  25. Sanei S, Chambers JA (2007) EEG signal processing. John Wiley & Sons Ltd, West Sussex, England. https://doi.org/10.1002/9780470511923

  26. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-Image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 5967–5976. IEEE. https://doi.org/10.1109/CVPR.2017.632

  27. Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015. MICCAI 2015. Springer, Cham. pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

  28. Generative Adversarial Networks (GANs) | Coursera, https://www.coursera.org/specializations/generative-adversarial-networks-gans, last accessed 2022/07/19.

  29. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20:37–46. https://doi.org/10.1177/001316446002000104

    Article  Google Scholar 

  30. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159. https://doi.org/10.2307/2529310

    Article  MATH  Google Scholar 

  31. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  32. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  33. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv Prepr. arXiv1704.04861

    Google Scholar 

  34. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  35. McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157. https://doi.org/10.1007/BF02295996

    Article  Google Scholar 

  36. Hall MA, Holmes G (2003) Benchmarking attribute selection techniques for discrete class data mining. IEEE Trans Knowl Data Eng 15:1437–1447. https://doi.org/10.1109/TKDE.2003.1245283

    Article  Google Scholar 

  37. Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X (2016) Improved techniques for training GANs. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. Curran Associates Inc., Barcelona, Spain. pp 2234–2242

    Google Scholar 

  38. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc., California. pp 6629–6640

    Google Scholar 

  39. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  40. Borji A (2019) Pros and cons of GAN evaluation measures. Comput Vis Image Underst 179:41–65. https://doi.org/10.1016/j.cviu.2018.10.009

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by an Indonesian Doctoral Dissertation Research Grant from DIKTI. The authors would like to thank the Intelligent Systems Research Group at the Department of Electrical and Information Engineering for contributing to stimulating discussion and inspiration.

Author information

Authors and Affiliations

  1. Department of Electrical and Information Engineering, Faculty of Engineering, Universitas Gadjah Mada, Yogyakarta, Indonesia

    Dwi Wahyu Prabowo, Hanung Adi Nugroho, Noor Akhmad Setiawan & Johan Debayle

  2. Department of Information System, Faculty of Computer Science, Universitas Darwan Ali, Sampit, Indonesia

    Dwi Wahyu Prabowo

  3. UMR 5307 LGF, Centre SPIN, MINES Saint-Etienne, CNRS, Saint-Etienne, France

    Johan Debayle

Authors
  1. Dwi Wahyu Prabowo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanung Adi Nugroho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Noor Akhmad Setiawan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Johan Debayle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanung Adi Nugroho .

Editor information

Editors and Affiliations

  1. Department of Electromedical Engineering, Poltekkes Kemenkes Surabaya, Surabaya, Indonesia

    Triwiyanto Triwiyanto

  2. School of Electrical Engineering, Telkom University, Indonesia, Indonesia

    Achmad Rizal

  3. Faculty of Integrated Technologies, Universiti Brunei Darussalam, Gadong, Brunei Darussalam

    Wahyu Caesarendra

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Prabowo, D.W., Nugroho, H.A., Setiawan, N.A., Debayle, J. (2023). An Advanced Data Augmentation Scheme on Limited EEG Signals for Human Emotion Recognition. In: Triwiyanto, T., Rizal, A., Caesarendra, W. (eds) Proceeding of the 3rd International Conference on Electronics, Biomedical Engineering, and Health Informatics. Lecture Notes in Electrical Engineering, vol 1008. Springer, Singapore. https://doi.org/10.1007/978-981-99-0248-4_27

Download citation

Publish with us

Policies and ethics

Search

Navigation