Skip to main content
SpringerLink
Log in

A Novel DAAM-DCNNs Hybrid Approach to Facial Expression Recognition to Enhance Learning Experience

  • Conference paper
  • First Online:
Computational Science – ICCS 2023 (ICCS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 10476))

Included in the following conference series:

  • 495 Accesses

Abstract

Many machine learning models are applied on facial expression classification and there are three main issues affecting the performance of any algorithms in classifying emotions based on facial expressions, and these issues include image illumination, image quality and partial features recognition. Many approaches have been proposed to handle these issues. Unfortunately, one of the main challenges in detecting and classifying facial expression process is minimal differences of features between different types of emotions that can be used to differentiate these different types of emotions. Thus, there is a need to enrich each type of emotion with more relevant extracted features by having a more effective approach to extract features that can be used to represent each type of emotions more effectively and efficiently. This work addresses the issue of improving the emotion recognition accuracy by introducing a novel hybrid approach that combines the Depth Active Appearance Model (DAAM) and Deep Convolutional Neural Networks (DCNNs). The proposed DAAM and DCNNs model can be used to assist one in identifying emotions and classify learner involvement and interest in the topic which are plotted as feedback to the instructor to improve learner experience. The proposed method is evaluated on two publicly available datasets namely, JAFFE and CK+ and the results are compared to the state-of-the-art results. The empirical study showed that the proposed DAMM-CNNs hybrid method managed to perform the face expression recognition with 97.4% for the JAFFE dataset and 96.9% for the CK+ dataset.

Supported by Universiti Malaysia Sabah under Grant No: SDN0057-2019.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)

    Article  Google Scholar 

  2. Huang, G., Cui, J., Alam, M., Wong, K.H.: Experimental analysis of the facial expression recognition of male and female. In: Proceedings of the 3rd International Conference on Computer Science and Application Engineering, pp. 1–5, October 2019

    Google Scholar 

  3. Tyng, C.M., Amin, H.U., Saad, M.N., Malik, A.S.: The influences of emotion on learning and memory. Front. Psychol. 1454 (2017)

    Google Scholar 

  4. Sathik, M., Jonathan, S.G.: Effect of facial expressions on student’s comprehension recognition in virtual educational environments. Springerplus 2, 1–9 (2013)

    Article  Google Scholar 

  5. Kerkeni, L., Serrestou, Y., Mbarki, M., Raoof, K., Mahjoub, M.A., Cleder, C.: Automatic speech emotion recognition using machine learning (2019)

    Google Scholar 

  6. Sun, L., Zou, B., Fu, S., Chen, J., Wang, F.: Speech emotion recognition based on DNN-decision tree SVM model. Speech Commun. 115, 29–37 (2019)

    Article  Google Scholar 

  7. Marechal, C., et al.: Survey on AI-based multimodal methods for emotion detection. High-Perform. Model. Simul. Big Data Appl. 11400, 307–324 (2019)

    Google Scholar 

  8. Alarcao, S.M., Fonseca, M.J.: Emotions recognition using EEG signals: a survey. IEEE Trans. Affect. Comput. 10(3), 374–393 (2017)

    Article  Google Scholar 

  9. Shen, Z., Cheng, J., Hu, X., Dong, Q.: Emotion recognition based on multi-view body gestures. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 3317–3321. IEEE, September 2019

    Google Scholar 

  10. Nafisi, J.S.A.: Gesture and body-movement as teaching and learning tools in western classical singing (Doctoral dissertation, Monash University) (2013)

    Google Scholar 

  11. Ko, B.C.: A brief review of facial emotion recognition based on visual information. Sensors 18(2), 401 (2018)

    Google Scholar 

  12. Valstar, M.F., Mehu, M., Jiang, B., Pantic, M., Scherer, K.: Meta-analysis of the first facial expression recognition challenge. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 42(4), 966–979 (2012)

    Google Scholar 

  13. Ringeval, F., et al.: Avec 2017: real-life depression, and affect recognition workshop and challenge. In: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, pp. 3–9, October 2017

    Google Scholar 

  14. Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Cowie, R., Pantic, M.: Summary for AVEC 2016: depression, mood, and emotion recognition workshop and challenge. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 1483–1484, October 2016

    Google Scholar 

  15. Smolyanskiy, N., Huitema, C., Liang, L., Anderson, S.E.: Real-time 3D face tracking based on active appearance model constrained by depth data. Image Vis. Comput. 32(11), 860–869 (2014)

    Article  Google Scholar 

  16. Wang, L., Li, R., Wang, K.: A novel automatic facial expression recognition method based on AAM. J. Comput. 9(3), 608–617 (2014)

    Article  Google Scholar 

  17. Cheng, R., et al.: Active appearance model and deep learning for more accurate prostate segmentation on MRI. In: Medical Imaging 2016: Image Processing, vol. 9784, pp. 678–686. SPIE (2016)

    Google Scholar 

  18. Krithika, L.B., GG, L.P.: Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Comput. Sci. 85, 767–776 (2016)

    Google Scholar 

  19. Agrawal, A., Mittal, N.: Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2020)

    Article  Google Scholar 

  20. Alfred, R., Obit, J.H., Chin, C.P.-Y., Haviluddin, H., Lim, Y.: Towards paddy rice smart farming: a review on big data, machine learning, and rice production tasks. IEEE Access 9, art. no. 9389541, 50358–50380 (2021). https://doi.org/10.1109/ACCESS.2021.3069449

  21. Alfred, R., Obit, J.H.: The roles of machine learning methods in limiting the spread of deadly diseases: a systematic review. Heliyon 7(6), art. no. e07371 (2021). https://doi.org/10.1016/j.heliyon.2021.e07371

  22. Pedraza, A., Gallego, J., Lopez, S., Gonzalez, L., Laurinavicius, A., Bueno, G.: Glomerulus classification with convolutional neural networks. In: Valdés Hernández, M., González-Castro, V. (eds.) MIUA 2017. CCIS, vol. 723, pp. 839–849. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60964-5_73

    Chapter  Google Scholar 

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  24. Jun, H., Shuai, L., Jinming, S., Yue, L., Jingwei, W., Peng, J.: Facial expression recognition based on VGGNet convolutional neural network. In: 2018 Chinese Automation Congress (CAC), pp. 4146–4151. IEEE, November 2018

    Google Scholar 

  25. Gopalakrishnan, K., Khaitan, S.K., Choudhary, A., Agrawal, A.: Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 157, 322–330 (2017)

    Article  Google Scholar 

  26. Ning, C., Zhou, H., Song, Y., Tang, J.: Inception single shot multibox detector for object detection. In: 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 549–554. IEEE (2017)

    Google Scholar 

  27. Chen, X., Yang, X., Wang, M., Zou, J.: Convolution neural network for automatic facial expression recognition. In: 2017 International Conference on Applied System Innovation (ICASI), pp. 814–817. IEEE (2017)

    Google Scholar 

  28. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  29. Li, J., et al.: Facial expression recognition with faster R-CNN. Procedia Comput. Sci. 107, 135–140 (2017)

    Article  Google Scholar 

  30. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28 (2015)

    Google Scholar 

  31. Talele, K., Shirsat, A., Uplenchwar, T., Tuckley, K.: Facial expression recognition using general regression neural network. In: 2016 IEEE Bombay Section Symposium (IBSS), pp. 1–6. IEEE (2016)

    Google Scholar 

  32. Arriaga, O., Valdenegro-Toro, M., Plöger, P.: Real-time convolutional neural networks for emotion and gender classification. arXiv preprint arXiv:1710.07557. (2017)

  33. Fan, Y., Lam, J.C., Li, V.O.: Video-based emotion recognition using deeply-supervised neural networks. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 584–588 (2018)

    Google Scholar 

  34. Li, T.H.S., Kuo, P.H., Tsai, T.N., Luan, P.C.: CNN and LSTM based facial expression analysis model for a humanoid robot. IEEE Access 7, 93998–94011 (2019)

    Article  Google Scholar 

  35. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): S complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  36. Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), pp. 123–128. IEEE (2017)

    Google Scholar 

  37. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205. IEEE (1998)

    Google Scholar 

  38. Yang, H., Yin, L.: CNN based 3D facial expression recognition using masking and landmark features. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 556–560. IEEE (2017)

    Google Scholar 

  39. Mayya, V., Pai, R.M., Pai, M.M.: Automatic facial expression recognition using DCNN. Procedia Comput. Sci. 93, 453–461 (2016)

    Article  Google Scholar 

  40. Jain, A.K.: Fundamentals of Digital Image Processing. Prentice-Hall, Inc., Hoboken (1989)

    Google Scholar 

  41. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0054760

    Chapter  Google Scholar 

  42. Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)

    Article  Google Scholar 

  43. Zhao, X., Zhang, S.: Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10), 9573–9588 (2011)

    Article  Google Scholar 

  44. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)

    Google Scholar 

  45. Mlakar, U., Potočnik, B.: Automated facial expression recognition based on histograms of oriented gradient feature vector differences. SIViP 9(1), 245–253 (2015). https://doi.org/10.1007/s11760-015-0810-4

    Article  Google Scholar 

  46. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Google Scholar 

  47. Jeni, L.A., Takacs, D., Lorincz, A.: High quality facial expression recognition in video streams using shape related information only. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2168–2174. IEEE (2011)

    Google Scholar 

  48. Kahou, S.E., Froumenty, P., Pal, C.J.: Facial expression analysis based on high dimensional binary features. In: ECCV Workshops (2), pp. 135–147 (2014)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Universiti Malaysia Sabah internal grant no. SDN0057-2019 (Biometric Patient Authentication System Using Face Recognition Approach for Smart Hospital).

Author information

Authors and Affiliations

  1. Creative Advanced Machine Intelligence Research Centre, Faculty of Computing and Informatics, Universiti Malaysia Sabah, Kota Kinabalu, Malaysia

    Rayner Alfred, Rayner Henry Pailus & Joe Henry Obit

  2. School of Information Science, Japan Advanced Institute of Science and Technology, Access 1-1 Asahidai, Nomi, Ishikawa, 923-1292, Japan

    Yuto Lim

  3. Department of Informatics, Universitas Mulawarman, Samarinda, Kalimantan Timur, Indonesia

    Haviluddin Sukirno

Authors
  1. Rayner Alfred
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rayner Henry Pailus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joe Henry Obit
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuto Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haviluddin Sukirno
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rayner Alfred .

Editor information

Editors and Affiliations

  1. Czech Technical University in Prague, Prague, Czech Republic

    Jiří Mikyška

  2. University of Amsterdam, Amsterdam, The Netherlands

    Clélia de Mulatier

  3. AGH University of Science and Technology, Krakow, Poland

    Maciej Paszynski

  4. University of Amsterdam, Amsterdam, The Netherlands

    Valeria V. Krzhizhanovskaya

  5. University of Tennessee at Knoxville, Knoxville, TN, USA

    Jack J. Dongarra

  6. University of Amsterdam, Amsterdam, The Netherlands

    Peter M.A. Sloot

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alfred, R., Pailus, R.H., Obit, J.H., Lim, Y., Sukirno, H. (2023). A Novel DAAM-DCNNs Hybrid Approach to Facial Expression Recognition to Enhance Learning Experience. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 10476. Springer, Cham. https://doi.org/10.1007/978-3-031-36027-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36027-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36026-8

  • Online ISBN: 978-3-031-36027-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation