Deploying Serious Games for Cognitive Rehabilitation

Abstract

The telerehabilitation of patients with neurological lesions has recently assumed significant importance due to the COVID-19 pandemic, which has reduced the possibility of access to healthcare facilities by patients. Therefore, the possibility of exercise for these patients safely in their own homes has emerged as an essential need. Our efforts aim to provide an easy-to-implement and open-source methodology that provides doctors with a set of simple, low-cost tools to create and manage patient-adapted virtual reality telerehabilitation batteries of exercises. This is particularly important because many studies show that immediate action and appropriate, specific rehabilitation can guarantee satisfactory results. Appropriate therapy is based on crucial factors, such as the frequency, intensity, and specificity of the exercises. Our work’s most evident result is the definition of a methodology that allows the development of rehabilitation exercises with a limited effect in both economic and implementation terms, using software tools accessible to all.

1. Introduction

Virtual reality is a powerful tool for motor telerehabilitation due to neurological lesions [1,2]; therefore, games for non-recreational purposes, serious games, are powerful rehabilitation tools that have given relevant results [3,4]. Cognitive rehabilitation is a subset of rehabilitative treatments that is dedicated to all patients who require ongoing cognitive function training due to injury (traumatic brain injury, stroke, cerebral palsy, etc.) or pathology (Alzheimer’s disease, multiple sclerosis, etc.) affecting the central nervous system. Stroke is the second-leading cause of death and disability in the world. It is a disease whose timely access to treatment is crucial to allow important improvements to recover motor and cognitive skills [5]. With the advent of the pandemic, which has made access to rehabilitation centres more problematic for patients with cognitive rehabilitation needs (such as those who have suffered a stroke), the importance of the possibility of exercise for these patients safely in their own homes has emerged strongly [6]. There is no doubt that at such a problematic time, poor digital organisation and not being used to organising processes with the principles of digital sustainability have been a major detriment to the quality of life of citizens in many countries around the world.

Many studies show that immediate action and appropriate, specific rehabilitation can guarantee satisfactory results [7]. Appropriate therapy is based on key factors to be taken into account such as the frequency, intensity, and specificity of the exercises.

Our ultimate objective is to create a modality for developing open-source digital goods that may provide access to a virtual environment at any time. This allows patients who have experienced cognitive limitations to exercise and recover all or part of these skills in the least amount of time.

In view of the spread of IoT devices capable of easily monitoring various vital parameters, we propose with our system a low-cost and very efficient solution that can provide the doctor not only with quantitative data on the exercises performed (number and type of exercises, time spent, results obtained), but also an overview of vital parameters, so as to observe any states of agitation or excessive effort in completing the exercise. This information is provided to the doctor through a CSV file. This file contains the measured values of the patient’s parameters collected from the start to the end of the exercise.

This article is divided as follows: in Section 2, we discuss the articles and manuscripts that have addressed this issue and represent the state of academic research today. Section 3 describes the tools, the software, and the hardware used during the analysis. Section 4 discusses the main exercises developed for the cognitive rehabilitation. Section 5 reports the conclusions and results obtained during our analysis, and possible future developments are also highlighted.

2. Related Works

In 1970, Abt coined the phrase “serious game” [8], which refers to games with a stated and thoroughly thought-out educational aim and which are not intended to be played merely for entertainment. Currently, “serious games” is a widespread and specific phrase that refers to any video-game-based learning and instruction (e.g. business, military, medical, marketing). Serious games can be placed in both professional and casual contexts, and their target audiences range in age [9,10].

An increasing number of researchers and educators have worked to integrate serious games into schools [11,12]. For years, attempts have been made to establish the potential effects of this type of activity [13]: the question is how far games may be utilised for educational reasons [14,15]. Serious games provide a virtually immersive environment in which students can experiment and repeat tasks that are rare in their ordinary lives [16]. In fact, it is argued that their application in education can increase the likelihood of providing students with a real and deep form of learning [17,18]. The use of virtual reality and augmented reality seems to have a profound positive influence on activating specific areas of the brain; this gave rise to the idea of stimulating specific nerve centres to recover functionality that had gradually been lost due to illnesses or traumatic events [19]. Therefore, in recent years, there has been an increase in interest in bringing healthcare services that can be operated remotely. The augmented demand for healthcare services drove the development of easy and quick telerehabilitation solutions in various sectors. These services were designed to be reliable and routine (rather than just pilot experiments), and they are now even more vital in providing continuous and efficient healthcare [20,21].

Telerehabilitation was initially used in motor rehabilitation, but it was later discovered to be beneficial in cognitive rehabilitation [22,23]. Specific cognitive capacities can be severely impaired due to various conditions and events, the majority of which are age-related issues such as dementia, circulatory difficulties, head traumas, persistent mental diseases, or brain pathology [24,25,26].

Following a thorough neuro-psychological examination that reveals the presence of cognitive deficiencies, neurological rehabilitation cycles can be carried out to enhance cognitive functioning, stabilise deficits, boost residual cognitive capacities, and reduce the path of decay. Currently, there are three theoretical–practical schools of thought underlying the rehabilitation of neurological deficits.

The first one proposes to address the rehabilitation of cognitive processes through non-specific stimulation. The second, on the contrary, suggests that recovery must necessarily pass through specific stimulation of the disorder; this type of intervention can be differentiated into a restorative approach (recovery) or a substitution approach (compensation). The third does not deal with the specific disorder, but proposes stimulating the residual abilities to overcome the difficulties and inhibitory effects, guaranteeing the most significant possible autonomy.

Our methodology implements the third approach by creating exercises that, by stimulating the patient’s neurological activity, allow him/her to recover functionality and find alternative neurological pathways to the traditional ones damaged by the trauma suffered. The creation of virtual environments designed ad hoc, enriched with visual stimuli and evocative content that arouse the patient’s interest, has the function of accelerating and stimulating this rehabilitation process.

Virtual-reality-based telerehabilitation systems use three-dimensional virtual environments to stimulate and obtain specific movements from the patient. The patient can view the virtual environment on the computer screen or utilise fully immersive devices, such as a head-mounted display, 3D movement sensors, and haptic devices [27,28,29].

The effectiveness of the exercise will be directly proportional to the degree of immersiveness the user experiences.

Virtual reality technologies benefit rehabilitation because they can construct virtual worlds and employ activity repetition, feedback, and motivation to encourage learning of lost abilities and subsequently convert them into the real world [30,31]. Video-game-based virtual reality telerehabilitation systems are among those available. Thanks to the development of technology, currently, one can experience a virtual reality game very easily, given the availability of robust and low-cost devices. The created virtual environments reach such a level of realism and immersion that they positively affect telerehabilitation [32,33]. The usage of serious games enables patients with various infirmities to conduct exercises in a highly engaging and non-intrusive manner, based on multiple virtual reality situations, helping to increase motivation throughout the rehabilitation process [34].

We enter a world particularly rich in proposals, both playful–recreational and educational. As regards the latter, we can say that numerous solutions have been created, beginning with educational authoring tools, such as presentation, interactive exercises, algorithm exploration, problem-solving activities, and, more recently, social and educational games.

The Internet of Things (IoT) can be considered the dominant technology for integrating all of the medical resources of the community-based innovative and intelligent rehabilitation system in a network [35,36]. The IoT refers to technologies that allow a wide range of devices to interact and communicate with one another via networking technologies [37]. The monitoring application’s front-end, which functions similarly to a network manager, manages the storage, aggregates, integrates, analyses, and models data received. Such systems in healthcare employ a collection of networked devices to build an IoT network targeted at health assessment, including patient monitoring and automated identification of circumstances requiring medical action [38,39]. The entire network receives data from various sensing devices via software, which offers the necessary interoperability and security in the IoT paradigm for healthcare.

Moreover, the wide diffusion of wearable devices or beacon–sensor systems [40] has considerably increased the possibility of monitoring bio-signals, both for diagnostic and therapeutic purposes [41].

In our case, the IoT monitors the patients’ vital parameters and provides the doctor with a wide range of information regarding the patients’ emotional state and physical condition.

3. The System Architecture

The system we developed (the system is available at the URL https://ogervasi.unipg.it/seriousGames/index.html, including the source code, accessed date: 1 May 2022) requires the patient to use a mobile or fixed device to access a web app containing a set of exercises planned by the doctor according to the type and severity of the patient’s cognitive deficit. The outstanding feature of the proposed system is to identify a set of open-access technologies (we used Blender, which is open-source, and recommend it for creating objects, and Unity, which can be used freely even if the app has revenues up to USD 100,000), which developers can use to produce serious games with a relatively low investment of time and energy.

In fact, the exercises are characterised by being placed in a very poor virtual context from the realistic rendering of the virtual world. The setting of the exercises does not require a high level of realism and graphic detail, as the user must be focused on the exercise and not risk being distracted from the assigned task. Instead, it is vital that whomever designs the virtual world does so while leaving the doctor free to include as many elements as possible relating to the patient’s life and history (images and objects related to his or her daily life before the trauma occurred).

The use of immersive devices per se is not indispensable, but their use allows the patient to have a much more intense experience during rehabilitation practice. Moreover, the availability of high-end hardware allows the patient to perform much more natural movements, facilitating the success of the rehabilitation practice. When designing the virtual environment surrounding the rehabilitation exercise, the developer must take great care to set up areas of the scene where the doctor can dynamically insert either images or virtual models of objects that can stimulate the patient’s memory and attention. The importance of contextualising the exercise in terms of images or objects familiar to the patient is well documented in the literature [42]. It is vital to enable the doctor to easily set up the exercise, including pictures of people or landscapes and objects or other things that evoke memories in the patient to make the exercise even more effective and engaging.

In Figure 1, the diagram of the steps the doctor can follow is shown. The last three blocks can be iterated until the patient rehabilitation is completed.

3.1. Use of IoT Devices and Edge Computing

The use of the IoT enables the doctor to monitor several vital parameters of the patient in order to be able to track the state of anxiety and performance stress and provide further helpful information. The large quantity of data collected may be analysed by computational resources located close to the patient to minimise the data transmitted to the web app server, following the paradigms of edge computing [43]. We used the Fitbit Sense smartwatch device to keep some of the patient’s vital parameters under surveillance, such as heart rate, tissue oxygenation (SpO2), respiratory rate, skin temperature, and atrial fibrillation (aFib) signals. The monitoring we carried out takes place in the few minutes before the exercise (value set by the doctor), during the exercise, and a few minutes after the exercise. In this way, we can obtain data to compare the trends of the parameters in a state of calm and a state of concentration.

3.2. The Web App

The web app consists of the front-end environment accessed by the patient, which contains the daily exercise list, and the environment reserved for doctors, where the exercise program for the specific patient is set up. Figure 2 shows the doctor’s graphical user interface to choose and set one of the several types of exercises. The Linux server that delivers the web app is based on a Debian distribution, within which, a Docker container orchestration service runs (https://www.docker.com/, accessed on 1 June 2022). One of the features we focused on most was the compatibility of our applications with the most significant number of devices on the market. Indeed, we tried to create a software environment that did not require an exceptionally high computational power.

3.3. The Exercises

The exercises we propose were created mainly using open-source software. Blender was used to create the objects with which the virtual world was made, while Unity was used to assemble the virtual interactive environment (Unity is not open-source. However, its usage is free for environments that produce an income of fewer than USD 100,000, so it is free for medical applications.). The open-source OpenCV libraries were used to offline pre-process images and to implement the exercise connect the dots. The open-source JavaScript language and Python were used for creating the script programs.

4. Implementing the Case Studies

The innovative aspects of this work are related to the definition of a methodology that allows the production release of tele-rehabilitation exercises in a relatively simple and low-cost manner. In order to clarify the implementation aspects and the various problems, we developed three use cases with different characteristics. The first concerns the solution of a puzzle, for which the doctor will determine the level of difficulty and the type of image according to the patient’s characteristics. The second one consists of joining numbered dots to draw a figure, so that the doctor can produce the exercise by customising it according to the patient’s needs. The third is related to inserting a key into a lock and opening it: the doctor can personalise the virtual environment with objects or images that the patient loved.

Thanks to these practices, the patient can recover necessary skills and alleviate himself/herself of the inability to perform essential daily routine activities.

4.1. Solving a Puzzle

In this type of exercise, the patient ought to be able to determine the right order and position of pieces obtained by dividing a photo into several parts. This practice is important for stimulating patient’s visual and motor coordination. The patient must also coordinate hand and finger movements to select the puzzle pieces and position them correctly. In order to make the exercise more enjoyable, the doctor selects images that have an emotional connection to the patient’s life history, e.g., photos of the family, pets with which there is a strong emotional bond, or pictures related to childhood.

The application has two separate interfaces: one for the doctor and the other for the patient. The doctor has the task of setting up the exercise and selecting a suitable image and the number of pieces that will make up the puzzle. After that, the software breaks down the image into the proper numbers of pieces and then randomly shuffles them on the screen. Next, the server generates a public URL, which can be sent to the patient. When the patient opens the URL, he or she will immediately find the exercise ready to be performed, with a very short loading time, as the software is entirely realised in HTML5, JavaScript, and CSS. The additional components that the user’s browser will have to download are a JavaScript script and a JPEG image.

The doctor can also choose the shape of the puzzle pieces. The choice can be made between rectangular pieces that are all the same shape, as shown in Figure 3a, or interlocking pieces, which must be fit together, as shown in Figure 3b. During the exercise, the time taken is recorded and displayed.

Figure 4 shows the doctor’s graphical user interface to manage the various parameters for the puzzle training exercise.

The timer turns on automatically as soon as the puzzle is started. After a congratulatory message, the time taken will be saved in the database and displayed when the puzzle is finished. In the doctor’s interface, a report will show the exercise time for each image used, providing the opportunity to analyse the patient’s progress and monitor it over time.

Figure 5 shows a patient performing the puzzle exercise.

4.2. Connect the Dots

The second exercise is called connect the dots and has a slightly higher degree of difficulty than the previous one. To complete the exercise, one must connect the dots identified by integer numbers, starting with the number 1 and continuing in ascending order up to the highest number shown. This exercise aims to stimulate the patient’s neurological skills. In fact, he/she must identify the numbers and the grapheme, try to deduce the final figure, and then, use visual and motor coordination to move the finger across the screen until the exercise is completed. There are also two graphical interfaces in this exercise; the doctor’s interface and the patient’s interface relating to the execution of the exercise.

The doctor’s interface allows selecting an image on which to create the series of dots to be joined to reproduce the main lines.

Images suitable for this type of exercise are predominantly black and white, with a low level of detail and graphic complexity. The shapes we recommend be simple and easily understood, such as regular polygons, animal drawings, or children’s colouring figures.

Figure 6a shows an image with dots to be joined. Figure 6b shows the image with the dots connected.

Once the doctor has selected one of the predefined images or has uploaded an image of his/her preference, a JavaScript script will be executed. After analysing the image, it will extract features using the OpenCV graphics library. In fact, comma filters are applied to bring out the contours and the most defined parts of the image. Then, the result obtained is processed in such a way as to eliminate the segments and replace them with a sequence of dots to which labels characterised by integer numbers are associated. The doctor will then see the result obtained and decide whether the complexity level is acceptable. We can now suppose the image obtained is evaluated as too simple or too complicated. In that case, the doctor can adjust the number of dots shown on the exercise using a stepped slider. These adjustments allow finely calibrating the exercise. Following each adjustment, a function associated with the onChange event of the slider is executed. The function executes the script again, which analyses the image and determines its representation in points and segments, adjusting the complexity according to the doctor’s preference. Once the doctor has finished setting up the exercise, the system will provide a URL that can be given to the patient.

Figure 7 shows the menu to choose shapes and their parameters in the “connect the dots” exercise.

Once the patient has received the URL and clicked on it, the exercise page, which has already been set up correctly, will open. The exercise page has a timer, a picture with the dots to be joined, and a button to stop the exercise if the patient does not want to finish it. The timer starts automatically as soon as the patient connects the first two dots. The web page has a very short loading time as it consists of a JavaScript code, which manages the timer and the detection of finger touch input, and a canvas on which there is a monochrome JPEG image. The exercise can be stopped in two cases: firstly, the patient completes the joining of all the dots present in the image; secondly, the patient prematurely interrupts the exercise by pressing the exit button presented below the image. The interruption of the exercise involves sending the result achieved by the patient encoded via a JSON list to the central server, which is stored in the database. The doctor can then see the level reached by the patient, how many dots have been connected correctly, and the time taken. If the exercise has been performed several times, the records of previous attempts are also reported to show any progress.

Figure 8 shows a patient performing the exercise of connecting the dots on his tablet.

4.3. Key Turning in a Lock

The exercise that is now presented was realised by exploiting immersive virtual reality techniques. This exercise involves the patient wearing a VR visor and using hand controllers capable of transforming the wrist’s movement into actions that are performed in the simulated environment. The patient will be in a room, have to pick up a key, locate the door and its lock, and finally, after approaching it, insert the key into the lock and rotate it. The door will open at the end of the rotation, and the exercise will be finished. Using Unity (https://unity.com, accessed on 1 June 2022) and the open-source software Blender (https://www.blender.org, accessed on 1 June 2022), a domestic scenario was created, with a room and various objects in it. All objects in the scenario were created using Blender.

The objects were then arranged in an interactive virtual environment using the Unity software. The software was configured to exploit the potential offered by immersive visors, such as the HTC Vive. This exercise is designed for patients who have neuro-muscular problems and need rehabilitation to recover motor coordination of the hand. In Figure 9, the main step executed by a patient performing this exercise can be observed.

5. Conclusions and Future Works

This study aimed to provide patients with cognitive impairments the option of practising serious games from home using a tablet, smartphone, or desktop computer. This was accomplished by providing the doctor with an environment easily extendible to new exercises and customizable with media content that may stimulate the patient during their performance. The environment we created is a good example of the sustainability of digital approaches, in line with the EU 2030 agenda.

The patient must be able to exercise consistently and be monitored for his/her progress while performing the tasks assigned by the doctor. Moreover, the doctor ought to control patients’ vital parameters to be able to quantify the emotional stress and any difficulties arising during the exercises. With the actual development of technology, cost-effective devices connected to smartphones and computers, which can monitor an increasing number of patient’s parameter, the doctor can gain an increasingly detailed picture of the person’s physical and emotional condition while performing the exercises. However, it must be emphasised that although this software is easily accessible as it is open-source, the implementation of exercises in accordance with our methodology still requires personnel experience in virtual reality, and this is undoubtedly a limitation of our approach.

An advantage of the proposed methodology is its versatility to adapt to the devices possessed by the user. Immersive hardware will undoubtedly provide a more immersive experience, but it is not indispensable for rehabilitation purposes.

The proposed approach is of enormous benefit in difficult times, such as we have experienced with the COVID-19 pandemic. There is no doubt that it would be essential to use everything we have learned over the past two years while living in absolutely precarious conditions and often unable to reach the designated places of care.

Our future work will extend the types of sample practice tasks, further optimising the user experience and increasing the accessibility of methods for impaired people. Moreover, we realise that it is necessary to enhance communication among some parts of the application currently delivered offline to implement a flow of information in real-time that streams automatically across the various components. That mainly concerns the transmission of data related to the vital parameters, from the patient to the doctor, and the images’ preprocessing phase with OpenCV. This leads to the issuance of the medical evaluation about the quality of the exercise performed by the patient. Furthermore, the design and implementation for demonstration purposes of serious games with increasing complexity will be performed. Patients in the advanced stages of rehabilitation or with moderate neurological problems will have priority for involvement. The complexity of the practice will be inversely proportional to the extent of the cognitive deficit.