Skip to main content
SpringerLink
Log in

Multimodal Feature Evaluation and Fusion for Emotional Well-Being Monitorization

  • Conference paper
  • First Online:
Pattern Recognition and Image Analysis (IbPRIA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13256))

Included in the following conference series:

  • 1461 Accesses

Abstract

Mental health is a global issue that plays an important roll in the overall well-being of a person. Because of this, it is important to preserve it, and conversational systems have proven to be helpful in this task. This research is framed in the MENHIR project, which aims at developing a conversational system for emotional well-being monitorization. As a first step for achieving this purpose, the goal of this paper is to select the features that can be helpful for training a model that aims to detect if a patient suffers from a mental illness. For that, we will use transcriptions extracted from conversational information gathered from people with different mental health conditions to create a data set. After the feature selection, the constructed data set will be fed to supervised learning algorithms and their performance will be evaluated. Concretely we will work with random forests, neural networks and BERT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Auto classes. https://huggingface.co/docs/transformers/model_doc/auto. Accessed 20 Jan 2022

  2. Mental health monitoring through interactive conversations. https://cordis.europa.eu/project/id/823907. Accessed 15 Jan 2022

  3. Modelcheckpoint. https://keras.io/api/callbacks/model_checkpoint/. Accessed 20 Jan 2022

  4. Using uh and um in spontaneous speaking: Cognition 84(1), 73–111 (2002). https://doi.org/10.1016/S0010-0277(02)00017-3

  5. Home: Action mental health (Dec 2021). https://www.amh.org.uk/. Accessed 15 Jan 2022

  6. Alaparthi, S., Mishra, M.: BERT: a sentiment analysis odyssey. J. Mark. Anal. 9(2), 118–126 (2021). https://doi.org/10.1057/s41270-021-00109-8

    Article  Google Scholar 

  7. Alghowinem, S., Gedeon, T., Goecke, R., Cohn, J., Parker, G.: Interpretation of depression detection models via feature selection methods. IEEE Trans. Affect. Comput. 1 (2020). https://doi.org/10.1109/TAFFC.2020.3035535

  8. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017). https://doi.org/10.1162/tacl_a_00051

    Article  Google Scholar 

  9. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  10. Bueno-Notivol, J., Gracia-García, P., Olaya, B., Lasheras, I., López-Antón, R., Santabárbara, J.: Prevalence of depression during the COVID-19 outbreak: a meta-analysis of community-based studies. Int. J. Clin. Health Psychol. 21(1), 100196 (2021). https://doi.org/10.1016/j.ijchp.2020.07.007

  11. Callejas, Z., et al.: Towards conversational technology to promote, monitor and protect mental health. In: IberSPEECH 2021, 24–25 March 2021, Valladolid, Spain, Proceedings. ISCA (2021)

    Google Scholar 

  12. Chen, R.-C., Dewi, C., Huang, S.-W., Caraka, R.E.: Selecting critical features for data classification based on machine learning methods. J. Big Data 7(1), 1–26, 100196 (2020). https://doi.org/10.1186/s40537-020-00327-4

  13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, June 2019. https://doi.org/10.18653/v1/N19-1423

  14. Esposito, A., Esposito, A., Likforman-Sulem, L., Maldonato, N., Vinciarelli, A.: On the Significance of Speech Pauses in Depressive Disorders: Results on Read and Spontaneous Narratives, pp. 73–82, 01 2016. https://doi.org/10.1007/978-3-319-28109-4_8

  15. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. 226–231. KDD 1996. AAAI Press (1996)

    Google Scholar 

  16. Fitzpatrick, K.K., Darcy, A., Vierhile, M.: Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment. Health 4(2), e19, June 2017. https://doi.org/10.2196/mental.7785

  17. He, L., et al.: Deep learning for depression recognition with audiovisual cues: a review. Inf. Fus. 80, 56–86, 100196 (2022). https://doi.org/10.1016/j.inffus.2021.10.012

  18. Inkster, B., Sarda, S., Subramanian, V.: A real-world mixed methods data evaluation of an empathy-driven, conversational artificial intelligence agent for digital mental wellbeing. JMIR mHealth and uHealth 6 (2018). https://doi.org/10.2196/12106

  19. Joerin, A., Rauws, M., Ackerman, M.: Psychological artificial intelligence service, tess: delivering on-demand support to patients and their caregivers: Technical report. Cureus 11, 01 2019. https://doi.org/10.7759/cureus.3972

  20. Liu, Z., Kang, H., Feng, L., Zhang, L.: Speech pause time: a potential biomarker for depression detection, pp. 2020–2025, 11 2017. https://doi.org/10.1109/BIBM.2017.8217971

  21. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

  22. MacQueen, J.B.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297 (1967)

    Google Scholar 

  23. Miao, J., Niu, L.: A survey on feature selection. Proc. Comput. Sci. 91, 919–926 (2016). https://doi.org/10.1016/j.procs.2016.07.111

  24. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), May 2018

    Google Scholar 

  25. Pampouchidou, A., et al.: Automatic assessment of depression based on visual cues: a systematic review. IEEE Trans. Affect. Comput. 10(4), 445–470, 100196 (2019). https://doi.org/10.1109/TAFFC.2017.2724035

  26. Rubenstein, H., Goodenough, J.: Contextual correlates of synonymy. Commun. ACM 8, 627–633 (1965). https://doi.org/10.1145/365628.365657

  27. Madabushi, H.T., Kochkina, E., Castelle, M.: Cost-sensitive BERT for generalisable sentence classification on imbalanced data. In: Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pp. 125–134. Association for Computational Linguistics, November 2019. https://doi.org/10.18653/v1/D19-5018

  28. Tennant, R., et al.: The Warwick-Edinburgh mental well-being scale (WEMWBS): development and UK validation. Health Qual. Life Outcomes 5(1) (2007). https://doi.org/10.1186/1477-7525-5-63

  29. Toshevska, M., Stojanovska, F., Kalajdjieski, J.: Comparative analysis of word embeddings for capturing word similarities. In: 6th International Conference on Natural Language Processing (NATP 2020), April 2020. https://doi.org/10.5121/csit.2020.100402

  30. Zhu, Y., Shang, Y., Shao, Z., Guo, G.: Automated depression diagnosis based on deep networks to encode facial appearance and dynamics. IEEE Trans. Affect. Comput. 9(4), 578–584 (2018). https://doi.org/10.1109/TAFFC.2017.2650899

  31. Zulfiker, M.S., Kabir, N., Biswas, A.A., Nazneen, T., Uddin, M.S.: An in-depth analysis of machine learning approaches to predict depression. Current Res. Behav. Sci. 2, e100044–e100044, 100196 (2021). https://doi.org/10.1016/j.crbeha.2021.100044

Download references

Acknowledgements

This work was partially funded by the European Commission, grant number 823907 and the Spanish Ministry of Science under grant TIN2017-85854-C4-3-R.

Author information

Authors and Affiliations

  1. University of the Basque Country, Sarriena no number, 48940, Leioa, Spain

    Irune Zubiaga & Raquel Justo

Authors
  1. Irune Zubiaga
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raquel Justo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Irune Zubiaga .

Editor information

Editors and Affiliations

  1. University of Aveiro, Aveiro, Portugal

    Armando J. Pinho

  2. University of Aveiro, Aveiro, Portugal

    Petia Georgieva

  3. University of Porto, Porto, Portugal

    Luís F. Teixeira

  4. Universitat Politècnica de València, Valencia, Spain

    Joan Andreu Sánchez

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zubiaga, I., Justo, R. (2022). Multimodal Feature Evaluation and Fusion for Emotional Well-Being Monitorization. In: Pinho, A.J., Georgieva, P., Teixeira, L.F., Sánchez, J.A. (eds) Pattern Recognition and Image Analysis. IbPRIA 2022. Lecture Notes in Computer Science, vol 13256. Springer, Cham. https://doi.org/10.1007/978-3-031-04881-4_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04881-4_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04880-7

  • Online ISBN: 978-3-031-04881-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation