Development of a home-based wrist range-of-motion training system for children with cerebral palsy

2.1 Used therapeutic movements

Supplementary to literature research on hand function and therapeutic goals in CCP, a workshop in the “Practice for Neurological Rehabilitation” was conducted as part of the user-centered design approach. 13 physiotherapists discussed hand and arm exercises for fictive patients to identify and structure therapeutic goals and training movements with emphasis on the importance for the child’s daily life. Semi-structured interviews with physiotherapist and pediatric experts completed the process, resulting in a selection of movements to be used for the game, which are not only therapeutically well-founded but also well detectable by IMUs.

Beyond the importance of arm movements, therapists underlined the importance of wrist movements for the motoric development. Several measuring options were found to be inadequate or intolerable for patients, e. g., the use of gloves with spastic hands or the necessity of attaching something to the hand. This led to the development of a technical system using wristbands with IMUs complemented by a cylindrical tangible device that contains an IMU as well. The advantages of robustness and child-friendly use face the disadvantage that, in comparison to a hand mounted IMU, only the orientation of the cylinder’s longitudinal axis can be approximated.

Interviews and observation of the therapeutic work revealed the importance of individual adjustments to the abilities of each child and its temporary performance (e. g., the presence of fatigue). The therapist motivates the child to go beyond his or her limits but without overstraining it. In the development of the serious game for CCP, we try to adopt this and transfer the mode of operation of the therapist into the game’s guidelines and requirements. We aim for the game to be adjustable to the individual and temporary ability of the child and to be challenging without being overdemanding, because we expect a positive gaming experience to be the key contribution to ensure voluntary use at home.

2.2 Wrist range of motion

An algorithm which calculates the wrist angles for the flexion/extension (F/E) and abduction/adduction (A/A) movement as well as the resulting wrist RoM is needed to provide the real-time game input data as well as offline motion and performance assessment for therapists.

IMUs can be used to determine the orientation of a rigid body with respect to an inertial frame of reference that neither rotates nor accelerates and is aligned with Earth’s gravitational and magnetic fields. This is achieved by algorithms that use the gyroscope for orientation strap-down integration and compensate integration drift by weighted corrections based on accelerometer and magnetometer readings. Orientations are commonly represented by unit quaternions or rotation matrices, and existing algorithms include extended Kalman filter and particle filter implementations, among others. We use the quaternion-based algorithm proposed in [15]. Quaternions are numbers with three imaginary parts i, j, k defined by i2+j2+k2=ijk=−1.

They are conveniently represented by a four-dimensional vector

The conjugate q∗ of a unit quaternion coincides with its inverse and is obtained by flipping the signs of all imaginary parts. Quaternion multiplication ⊗ follows from the aforementioned definition. If unit quaternions are associated with a rotation axis j=[jxjyjz]⊤ and rotation angle α via the definition

then for any vector [AxAyAz]T given in some coordinate system A, the transformation

yields the vector’s coordinates in a frame B that is rotated with respect to the frame A by the angle α around the axis j. In the following, BAq describes the orientation of the frame A with respect to B, and ABq∗=BAq describes the orientation of the frame B with respect to A. Finally, quaternion multiplication corresponds to a concatenation of rotations. Given BAq and CBq, the orientation of A with respect to C is

In the present setup, two body frames and two sensor frames are considered. The forearm coordinate system is defined according to the International Society of Biomechanics (ISB) recommendations [23] as illustrated in Figure 1. The forearm IMU is mounted to the wristband such that its sensing axes coincide with the axes of this coordinate system. Due to the cylindrical shape of the tangible device, its longitudinal axis is approximately aligned with the mediolateral axis of the hand, and the IMU is embedded into the tangible such that its z-axis coincides with the longitudinal axis. However, also due to the cylindrical shape, the tangible-to-hand orientation around the longitudinal axis is not known.

The proposed algorithm for wrist RoM estimation can be described in three main steps:

Step 1: estimate the relative joint orientation.

The quaternion-based orientation estimation algorithm [15] yields the orientations of the forearm sensor EFqt and the tangible sensor ETqt with respect to a common inertial reference frame E.

The orientation of the tangible sensor with respect to the forearm coordinate system is then obtained by:

and this quaternion is used to transform the tangible coordinate axes Tx=100⊤, Ty=010⊤, Tz=001⊤ into the forearm coordinate system. For Tx the calculation reads as follows:

In the further description, all vectors are presented in the forearm frame F.

Step 2: calculate angles that describe the motion.

When the wrist moves, the coordinates of the tangible axes in the forearm coordinate system change. For the A/A estimation, we project the tangible z-axis Fz=[FzxFzyFzz]⊤ into the yz-forearm-plane by omitting its x-coordinate and determine the phase angle in that plane using the four-quadrant arctangent function atan2 with unwrap:

For F/E, we project the tangible x-axis Fx=[FxxFxyFxz]⊤ into the xy-forearm plane and likewise determine its phase angle

Since the orientation of the tangible with respect to the hand depends on the grasp and is not completely known, the determined angles differ from the true wrist joint angles by an unknown offset. This is a major challenge that results from the robust and easy-to-use design of the system. Nevertheless, if the subject moves back and forth between some minimum and maximum F/E or A/A angle, we can determine the corresponding RoM.

Step 3: define envelope and estimate RoM.

A dynamically adjusting envelope is used to determine the minimum and maximum value that both angles reach during a performed motion. To account for fatigue, the envelope shrinks automatically by a rate r=1∘/s whenever the boundary values are not reached or exceeded. For A/A, the formalism is

The algorithm should automatically adjust to individual children with different RoM. Therefore, at each time instant, the joint angles are scaled to the current RoM as follows:

In summary, the presented algorithm determines the wrist angles for a F/E and A/A movement from two IMUs. Then the angles are scaled to the current RoM (α˜ and ε˜) to obtain one parameter for each movement that can be used as game input. The children’s individual RoM can provide feedback for the therapists.

2.3 Range of motion feasibility evaluation

The presented algorithm automatically determines the individual RoM from the measurements of a forearm IMU and an IMU inside a tangible device. As mentioned above, this setup was chosen to improve usability and avoid unnecessary donning effort. We now investigate whether this approach yields reliable RoM estimates in comparison to the more complex approach of attaching the second IMU directly onto the back of the hand. Moreover, the hand-mounted IMU serves as an approximate ground truth, since the accuracy of orientation and joint angle measurements from body-worn IMUs has been demonstrated to lie in the range of a few degrees in numerous studies [13], [14], [21], [22].

Another crucial objective of the algorithm is to be adaptable to changing RoM, which may occur due to fatigue or be dependent on the severity of the patient’s impairment. Therefore, the algorithm is evaluated with respect to the following three key questions, answered by experimental procedures:

1. Which accuracy can be achieved with the tangible in comparison to a hand-mounted IMU?

2. Does the algorithm automatically adapt to changing RoM?

3. Can the algorithm be used to track wrist movements of CCP?

1. grasping, max. F/E (5 times), release

2. grasping, max. A/A (5 times), release.

Question (3) is addressed by investigating the applicability of the presented algorithm in CCP using two different sets of data from atypical therapeutic wrist exercise called “feed the monster”. The exercise is composed of a complex, task-oriented arm and hand movement: grasp a thin cylindrical card, move the hand under a horizontal bar and release the card through a horizontal resp. vertical slit (Figure 2 shows the exercise presented by a therapist), each version executed six times.

Figure 2

Therapy exercise “feed the monster”.

Data was sampled and recorded a) from a therapist during a workshop, presenting “healthy” movements and b) from a child with CP in the course of a longer movement analysis, including the Assisting Hand Assessment (AHA) [8]. The therapist and the child wore the aforementioned IMUs on each hand and both forearms (see Figure 2). The sampling frequency was set to 20 Hz in the therapist workshop and 60 Hz in the assessment with the child to get a higher resolution for further analysis. For this paper the data from the forearm and hand IMU was analyzed during the “feed the monster” exercise with the proposed algorithm.

2.4 Gamification concept and implementation

In addition to technical feasibility, game concepts for CCP must meet therapeutic requirements such as adaptability. Further impairments or reduced concentration due to brain injury need to be considered during the design of the optical game appearance. The motivational aspect must include different challenges and adaption to fatigue.

In the game ideation phase, the core mechanic of the game was chosen to be an item collecting task in the genre of role playing with an aquatic turtle as avatar. Different approaches to cognitive challenges were developed, and these as well as the motoric challenges were sorted by difficulty. The wrist F/E and A/A movement can be trained separately or in combination. Therefore, two different concepts with different motoric challenges were implemented.

The first game concept (Figure 3, left side) considers only the F/E movement. The swimming turtle moves up and down upon extension and flexion respectively. Up to three cognitive levels were developed for motivation. Whereas in the first level the task is to collect the stars, in the second level the child must additionally avoid water plants. Plants grow at the bottom, and the child must perform a dorsal extension to avoid them. In the third level, more obstacles in the form of swimming fish need to be avoided.

Figure 3

Game concepts for F/E and A/A movement.

The second concept (Figure 3, right side) uses the scaled F/E angle to move the turtle up and down and the scaled A/A angle to move the turtle to the sides. This game is controlled by 2D movements and has a higher motoric difficulty level. In the first level of the game, the stars will appear stationary on the outer boundaries of the screen. In the second level the stars are moving, and in the third level obstacles in the form of swimming fish need to be avoided.

The described concepts were implemented with the following technical realization. The raw data, consisting of three-dimensional accelerometer, gyroscope and magnetometer data from the forearm IMU and the tangible IMU, are transmitted wirelessly to a Linux laptop, which is executing code that was generated from a custom-made Simulink diagram and sends the game variables and parameters to a web socket. The game itself was implemented in GDevelop, an open-source and easy to use game engine. GDevelop retrieves the data from the web socket.

After implementation, both games were introduced to two experienced therapists for feedback as basis of a successive user-centered validation. A semi-structured interview has been conducted to discuss the following topics: therapeutic goals, target group, game parameter settings, motivation and design aspects.

3.1 RoM feasibility

First the differences between hand movement and tangible device as well as the adaptation to different RoM were analyzed. Figure 4 shows the result of the experimental procedure from a healthy subject, performing an F/E movement with different RoMs. The first subplot illustrates the envelope adaption, which shows a fast reaction to an increasing envelope and slower adaption to the decreasing RoM. Because in a game situation the CCP obtains a reward for wider RoM movement, the second subplot marks with crosses, where the current angle was in the lower or upper 20 %, corresponding to a scaled angle below 0.2 or above 0.8. A star marks that the current RoM was exceeded. In the third subplot the A/A and F/E RoM are shown as determined by the proposed method and the approximated ground truth determined from the hand mounted IMU. For the F/E movement, the mean RoM difference between hand and tangible device is 7 ± 3 deg for F/E and 12 ± 5 deg for A/A. For the A/A movement, the values amount to 10 ± 12 deg (A/A) and 5 ± 17 deg (F/E).

Figure 4

Flexion and extension movement from a healthy subject with a tangible device.

In a second step the algorithm was used to analyze the wrist movement of a nine-year-old girl with a right side unilateral cerebral palsy. She reached an AHA score of 84 out of 100, so her hemiparesis is mild. Wrist F/E and A/A angles were determined from the forearm and hand IMUs. The current angles for F/E and A/A as well as the RoM of the “feed the monster” exercise with horizontal slot are shown in Figure 5. When grasping a card, ε is minimal and α is maximal, i. e., during grasping the hand is in palmar flexion and abduction. Each time the child releases the card, her individual maximum dorsal extension is reached. The fourth card was dropped on the table, picked up and the way of grasping was corrected by the therapist, which explains the irregularity in the curve.

Figure 5

Exercise “feed the monster” performed by a child with CP.

Table 1 shows the average RoM for the three healthy probands, the therapist and the child with CP. For the F/E movement, the therapist and the child have a similar RoM. When the healthy subjects were asked to perform a maximum movement, their F/E RoM is about 30–40 degrees higher. For the A/A movement, the patient shows a higher RoM than the therapist or the healthy subjects during the F/E movement. The maximum A/A RoM of the healthy subjects corresponds to the child’s A/A RoM.

The child needs about 55 seconds for 6 repetitions with the horizontal slot and 40 seconds with the vertical slot, the therapist needs about 22 seconds and 23 seconds, respectively.

These different experimental settings demonstrate (1) the feasibility of the IMU-based RoM estimation with a tangible device, (2) the automatic adaptation to time-variant RoM, as well as (3) the applicability of the system for CCP.

3.2 Implementation and therapist evaluation

The implemented program calculates the scaled angles (α˜ and ε˜) according to the proposed algorithm and sends the calculated angles in real-time to a web socket. The flow experience of the implemented game was good without noticeable time delay. The presented algorithm adapts the game parameters α˜ and ε˜ to the child’s individual performance by a dynamic envelope. The child’s individual baseline RoM is determined by a calibration movement at the beginning of each play session. The shrinking-envelope feature assures that the subject can continue playing even in the presence of fatigue.

A simple design without much distraction was realized in GDevelop, which helps the child to focus on the movement task. A primary goal of the game is to motivate a child to perform repeated motions at large amplitudes, which corresponds to α˜ or ε˜ regularly exceeding 0.8 or falling below 0.2. The F/E game target is to collect 15 stars, which will appear at the top and bottom of the water corresponding to scaled epsilon ranges of 0–0.2 and 0.8–1. When the child gets “lazy” and only collects the lower stars without moving the hand, the bottom stars will disappear and the child has to collect stars at the top. This encourages continuous F/E movement.

The second game uses movements in two directions. In the first level, the stars will appear stationary on the outer boundaries from 0–0.2 and 0.8–1 at both dimensions.

In addition to these automatic adaptions, speed levels and number of obstacles may be adapted to the child’s capability by the therapist, such that the game is perceived as a challenge without being overly demanding.

The three cognitive levels aim to motivate the child to play for a longer time and therefore perform more repetitions.

The presented games were evaluated in the categories therapeutic goals, target group, setting parameters, motivational and design aspects by two experienced physiotherapists who very often train the wrist dorsal extension with their patients. Their affirmative first impression of the game was supported by positively rated details such as: (1) The turtle as a slow but amiable animal offers a “role model” for the children, whose movements are slow as well. (2) The clear design offers only little distraction from the concentration on the game task. (3) In addition to wrist movement, other therapeutic goals such as concentration and reaction, eye-hand-coordination and movement planning are trained. (4) The possibility to play with different cognitive levels offers not only chances for children with different grades of severity, but also potential for fast learners to be increasingly challenged with regard to reaction time and movement planning due to moved elements. (5) The therapists approve of their options to adapt the difficulty level by game parameters such as turtle speed, number of obstacles or size of stars as well as the chance to get feedback on the child’s RoM.

They conclude that the game is feasible for all severity levels and CCP of three years or older, provided that they can control their wrist movement, understand the connection between hand movement and turtle movement and can hold the tangible during game flow.