Abstract
This paper describes a multimodal assistive technology, called NJOY, to help students with multiple disabilities or with some type of special needs in terms of mobility, communication, and learning impairments. The basis of the entire system is an App that runs on a Tablet with a multimodal interaction system that includes a joystick that was developed for this purpose, a symbol-based communication system that aims to increase or compensate for communication, language, and learning difficulties with audio feedback. The system also integrates with a home automation system, which can be controlled using the same App and the same interaction mechanisms, and an SMS module that allows the read, through text-to-speech technology, of the received SMS, and send predefined SMS messages to predefined users. The system was tested by a sixteen-year-old student with degenerative disease of the central nervous system including severe motor, articulation, visual impairment, and limitations at a cognitive level, which greatly compromises its functionality, depending on adults to carry out the basic tasks of daily life. The results were very promising with the user being able to use the system without any major difficulties.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Orelove, F.P., Sobsey, D., Silberman, R.: Educating Children with Multiple Disabilities: A Collaborative Approach, 4th edn. Brookes Publishing Company, Baltimore (2004)
Mansell, J.: Raising our sights: services for adults with profound intellectual and multiple disabilities. Tizard Learn. Disabil. Rev. (2010). https://doi.org/10.5042/tldr.2010.0399
Downing, J.E., Eichinger, J.: Educating students with diverse strengths and needs together: rationale for inclusion. In: Including Students with Severe and Multiple Disabilities in Typical Classrooms: Practical Strategies for Teachers, p. 19. Paul H. Brookes Publishing Co., P.O. Box 10624, Baltimore, MD, pp. 21285–0624 (2008)
Sharma, N., Doherty, I., Dong, C.: Adaptive learning in medical education: the final piece of technology enhanced learning? Ulster Med. J. 86, 198–200 (2017)
Harding, C., Lindsay, G., O’Brien, A., Dipper, L., Wright, J.: Implementing AAC with children with profound and multiple learning disabilities: a study in rationale underpinning intervention. J. Res. Spec. Educ. Needs. (2011). https://doi.org/10.1111/j.1471-3802.2010.01184.x
Fossett, B., Mirenda, P.: Augmentative and alternative communication. In: Handbook of Developmental Disabilities. The Guilford Press, New York (2007). https://doi.org/10.1007/s12453-010-0015-0
Strangman, N., Dalton, B.: Using technology to support struggling readers: a review of the research. In: Edyburn, D., Higgings, K., Boone, R. (eds.) Handbook of Special Education: Research and Practice, pp. 325–334 (2005). Whitefish Bay, Wis.: Knowledge by Design
Graham, S.: The role of text production skills in writing development: a special issue. Learn. Disabil. Q. (1999). https://doi.org/10.2307/1511267
Pino, A.: Augmentative and alternative communication systems for the motor disabled. In: Disability Informatics and Web Accessibility for Motor Limitations, pp. 105–152. IGI Global (2013). https://doi.org/10.4018/978-1-4666-4442-7.ch004.
Aragonese Center of Augmentative and Alternative Communication (ARASAAC). https://arasaac.org/. Accessed 5 Nov 2020
Bliss, C.K.: Semantography - Blissymbolics, 3rd enlarged edn. Semantography-Blissymbolics Publications, Sydney (1978)
Grove, N., Walker, M.: The makaton vocabulary: using manual signs and graphic symbols to develop interpersonal communication. Augment. Altern. Commun. 6, 15–28 (1990). https://doi.org/10.1080/07434619012331275284
Mulberry Symbols. https://mulberrysymbols.org/. Accessed 18 Nov 2020
Mayer-Johnson: PCSTM Symbols - Picture Communication Symbols. https://goboardmaker.com/. Accessed 18 Nov 2020
Sclera NPO. https://sclera.be. Accessed 18 Nov 2020
Karpov, A., Ronzhin, A.: A universal assistive technology with multimodal input and multimedia output interfaces. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 369–378. Springer (2014). https://doi.org/10.1007/978-3-319-07437-5_35
Bissoli, A.L.C., Coelho, Y.L., Bastos-Filho, T.F.: A system for multimodal assistive domotics and augmentative and alternative communication. In: ACM International Conference Proceeding Series. Association for Computing Machinery (2016). https://doi.org/10.1145/2910674.2910733
Bourguet, M.-L.: Designing and prototyping multimodal commands. In: Human-Computer Interaction (INTERACT 2003), pp. 717–720 (2003)
Martin, J.C., Veldman, R., Béroule, D.: Developing multimodal interfaces: a theoretical framework and guided propagation networks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (1998). https://doi.org/10.1007/bfb0052318
Ferri, F., Paolozzi, S.: Analyzing multimodal interaction. In: Grifoni, P. (ed.) Multimodal Human Computer Interaction and Pervasive Services, pp. 19–33. IGI Global (2009). https://doi.org/10.4018/978-1-60566-386-9.ch002
Silva, S., Almeida, N., Pereira, C., Martins, A.I., Rosa, A.F., e Silva, M.O., Teixeira, A.: Design and development of multimodal applications: a vision on key issues and methods. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction. Access to Today’s Technologies. UAHCI 2015. Lecture Notes in Computer Science, pp. 109–120. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20678-3_11
Rafael, S.: Multimodality, naturalness and transparency in affective computing for HCI. In: Marcus, A., Rosenzweig, E. (eds.) Design, User Experience, and Usability. Interaction Design. HCII 2020. Lecture Notes in Computer Science, vol. 12200, pp. 521–531. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49713-2_36
DFRobot: Bluno Series. https://www.dfrobot.com/bluno.html. Accessed 09 Nov 2020
Arduino. https://www.arduino.cc. Accessed 09 Nov 2020
Realinho, V.: Low cost domotic system based on open hardware and software. In: Proceeding of The Eighth International Conference on Advances in Human-Oriented and Personalized Mechanisms, Technologies, and Services-oriented and Personalized Mechanisms, Technologies, and Services (CENTRIC 2015), pp. 13–16. IARIA XPS Press, Barcelona (2015). https://doi.org/10.13140/RG.2.1.2061.7687
Pelizaeus, F.: Ueber eine eigenthümliche Form spastischer Lähmung mit Cerebralerscheinungen auf hereditärer Grundlage. (Multiple Sklerose). Arch. Psychiatr. Nervenkr. 16, 698–710 (1885). https://doi.org/10.1007/BF02057569
Merzbacher, L.: Eine eigenartige familiär-hereditäre erkrankungsform (Aplasia axialis extracorticalis congenita) (1910). https://doi.org/10.1007/BF02893591
Acknowledgements
This work was funded in part by Fundação Ilídio Pinho through the award Ciência na Escola, in which it won the 3rd national place. The authors would like to thank the valuable collaboration of students and teachers of the Polytechnic Institute of Portalegre and Escola Secundária Mouzinho da Silveira involved in this project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Realinho, V., Baptista, L., Dias, R., Marmelo, D., Páscoa, P., Mourato, J. (2021). Multimodal Assistive Technology for the Support of Students with Multiple Disabilities. In: Rocha, Á., Adeli, H., Dzemyda, G., Moreira, F., Ramalho Correia, A.M. (eds) Trends and Applications in Information Systems and Technologies. WorldCIST 2021. Advances in Intelligent Systems and Computing, vol 1365. Springer, Cham. https://doi.org/10.1007/978-3-030-72657-7_45
Download citation
DOI: https://doi.org/10.1007/978-3-030-72657-7_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72656-0
Online ISBN: 978-3-030-72657-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)