Use of mixed reality for the training of operators of mobile elevating work platforms with the aim of increasing the level of health and safety at work and reducing training costs

Introduction

Virtual and Mixed Reality devices can provide realistic immersive experiences and are spreading in various industrial sectors (Greci, 2022), showing their true potential either in the production processes in combination with other technologies of the Industry 4.0 paradigm, such as collaborative robotics and rapid prototyping (Badia, et al., 2022; Schroeter & Gerber, 2018), or in interactive simulation applications for educational, training and remote maintenance purposes (Casini,2022; Mourtzis et al 2020, Xie, et al., 2021).

In the field of skill training, an important asset of Virtual and Mixed Realities is the ability of providing a multimodal environment (Bergamasco et al., 2012) because training protocols using multisensory stimulation are closer approximations of real situations and facilitate multisensory learning (Shams & Seitz, 2008). The positive impact of multisensory experiences on training performance has been reported in several systematic reviews of the scientific literature about multimodal Virtual Environments (Melo et al., 2020; Martin et al., 2022). The capability of skill transfer from multimodal training simulators to real scenarios has also been proven for complex tasks, in particular involving vision, audition and haptic modalities (Gopher, 2012; Philippe et al., 2020). A realistic haptic feedback, i.e., a realistic sensation of touch and proprioception, is an important factor in training simulators because it affects user’s presence and immersion in the Virtual Environment, and has an impact on the transfer to the real task (Melo et al., 2020; Cooper et al., 2021).

Simulators used for the training of operators that have to work with complex and dangerous machines are a common example (Garcia et al., 2019), but their application can be further extended as a specific tool to create training programs that simulate situations aimed at understanding and adopting the skills and abilities needed to prevent accidents and injuries at work.

Lifting machines represent an interesting case study in this context, due to their diffusion, their challenging work environments, the prevalence and causes of related accidents, and to the impact that effective training courses can have on occupational safety and health in that sector (OSH).

Usually, operators on lifting equipment of significant complexity are trained by working alongside an operator with extensive operational experience. However, this training methodology is particularly onerous, as it involves an expert operator for training purposes and, most of all, a lifting vehicle that has significant purchase and maintenance costs and whose actual amortization depends on the possibility of use in operational activities.

The use of simulation technologies is able to provide an effective response to this problem with particular reference to situations of danger for people and equipment. In fact, Virtual Reality offers a unique way of delivering training for safety procedures (Norris et al., 2019), including confronting with risky operations, in a safe environment, with the possibility to provide training analytics (Grassini & Laumann, 2020). Simulators based on Virtual Reality technologies have also been proposed for promoting occupational reintegration of disabled persons (Gasparello et al., 2014).

Accidents in operations largely result from an incorrect execution of positioning or stabilization maneuvers, from collapse or collisions, or from a limited ability to manage anomalous situations or partial failure of the vehicle in use (IPAF, 2022): these kinds of dynamics highlight significant gaps in the training courses for maneuver operators that it is important to try to fill.

In the case of machinery for lifting persons, with particular reference to self-propelled elevating work platforms, there are also dynamics of major accidents and injuries in handling operations, for example due to dangerous situations caused by failure of the main parts of the extensible structure. Other particular risk conditions refer to overturning, collision with other moving vehicles and with fixed structures, entrapment between the base and the machine structure, uncontrolled descent due to component failure, entrapment with moving parts, falls from the platform, electrocution due to contact with live power lines.

In the following sections we present an immersive VR-based simulator designed for training operators in charge of running and maintaining self-propelled mobile elevating work platforms (MEWPs) with an extendable articulated boom. The simulator is developed together with INAIL (the Italian Workers' Compensation Authority) as part of a collaborative research program for the application of innovative technologies and systems to the management of health and safety at work.

We argue this system is capable of demonstrating how Extended Reality (XR) can be an effective and flexible tool to implement qualifying paths for operators and verifiers designed to prevent accidents and promote safety at work in a sector, that of aerial work platforms for lifting people, which has well-defined risk factors and criticalities, taken into account in the design of the operating scenario simulated in the system and in the methods for its deployment.

System description

The simulator we are presenting reproduces all the aspects that contribute to a complete driving experience of the simulated machine in an artificial working environment that replicates typical and atypical operations and hazards involved in the use of self-propelled mobile elevating work platforms with an extendable articulated boom.

The simulator consists of three main elements: the Aerial Platform Station, the Ground Station and the Operating Scenario (Figure 1).

FIGURE 1. The simulator main elements: (A) Aerial platform simulation station; (B) Ground simulation station and computers controlling the simulation; (C) Operating scenario of the simulation.

The simulation system includes two work areas so that two operators can be trained simultaneously and collaboratively, that is both the main operator who is in the aerial platform and performs work at height and the operator on the ground who operates from a safety control panel on the ground by the side of the vehicle. In fact, mainly for safety reasons, MEWPs have a second machine control station located on the side of the main body of the vehicle, which can be operated by a second operator on the ground. The ground operator intervenes when the operator in the platform is unable to continue the maneuvers due to serious failures, or, even worse, in case of illness at height. The ground operator’s task is therefore very critical and essential to provide quick assistance to the operator on the platform.

Different components constitutes and control the simulation system (Figure 2), which also includes the possibility to display the ongoing simulation experience in real-time to a group of trainers in a class.

FIGURE 2. The simulation system architecture.

The Aerial Platform Station

The first station of the simulation is the one for training the operator who has to execute the work at height. It has been realized starting from a real MEWP aerial platform complete with railings and the original dashboard. From this station the operator can drive the MEWP inside the work area (i.e., the virtual operating scenario) and use all the functions of the articulated boom to reach the desired working point. In order to make the simulation immersive and so that each command corresponds to a movement of the aerial platform, it has been firmly anchored to a mobile platform with 3 degrees of freedom (roll, pitch, heave) so as to stimulate the vestibular apparatus of the operator (the 3DoF motion platform in Figure 2). That is done according to the correct acceleration values obtained from the physics engine of the simulator. This mechanism allows the operator to feel realistic sensations related to acceleration, braking, curves, bumps and slopes while driving, and to perceive every single movement and balance when using the articulated boom.

A feature that makes the use of the simulator very faithful to a real machine is that the operator is trained directly with the original control panel. It comes directly from the MEWP manufacturer together with the aerial platform. All the buttons and levers have been kept, while the entire original electronics have been replaced with a dedicated microcontroller programmed to manage all the various commands through appropriate digital or analog input/output ports. In turn, the microcontroller interfaces with the PC running the simulation software via USB connection and HID joystick driver. In this way, in addition to managing the inputs of the control panel, the simulation software can keep track of what the operator is doing during the exercise and can display it to the teacher, instructor or the supervisor of the simulation in real time.

Another important contribution to the degree of immersivity reached by the simulator is given by the audio-visual hardware. In fact, the use of XR technology requires the user to wear an Head Mounted Display (HMD) viewer with integrated headphones that reproduces the virtual environment in 360°. The virtual environment developed for the project has been realized with the Godot open-source engine. It manages all the graphical and acoustic elements, all the hardware devices and also takes care of the physical interaction between the aerial platform and the various virtual elements composing the scenario. In other terms, it defines the temporal evolution of the logic of the exercise.

However, the multisensory system described presents a potential critical issue: since the user wears a HMD, s/he cannot directly see the real objects around and that s/he has to interact with, namely the structure of the platform and the control panel. To correctly integrate the tactile feelings of all the mechanical components reachable by the operator, they have been reproduced as 3D models and inserted within the virtual environment. Then, thanks to special tracking devices firmly fixed to the platform, the real mechanical parts are dynamically localized and synchronized in the space of the virtual environment so that the operator can see an artificial version of them through the viewer, positioned exactly at the same place as the real counterpart. In order to improve the user’s sense of presence within the virtual environment, it is also possible to bring up a reconstructed version of her/his hands. In fact, they are not directly visible due to the presence of the HMD, but thanks to two small cameras placed on the top of it, it is possible to compute in real time the position of the hands and the pose of each finger through an algorithm based on artificial intelligence that tracks up to 21 points in 3D for each hand.

The Ground Station

The ground operator training station is completely identical to the one of the aerial platform, except for the absence of the underlying motion platform. In fact, it is a station where the user is using the same XR technology described above and operates standing near the ground control panel. In a real MEWP the ground control panel is mounted on the side of the machine. Since the simulator does not contain the entire vehicle body, only the original panel has been employed, supported by a dedicated mechanical stand. As for the aerial control panel, it has been sensorized, interfaced with the PC and has been equipped with a tracking device. It allows the user, who wears an HMD, to see an artificial version of the ground control panel in the virtual environment perfectly integrated with the 3D model of a complete MEWP.

Since the aerial platform operator can translate the MEWP during the training maneuvers, the ground control station can consequently move in the virtual environment. To ensure that the operator on the ground can reach out her/his control panel, her/his avatar should also follow in real time all the movements of the MEWP. Although this solution allows the operator on the ground to always have the real panel in sync with the virtual one, the authors found that s/he might suffer a form of sickness during the movements of the MEWP. In fact the user’s sensory system is deceived because through the HMD s/he sees her-/himself navigate the virtual environment while actually s/he is not walking at all. That is why the actual solution has been to completely interrupt the synchronization of the ground control panel during the translation of the MEWP. In other words, the avatar of the ground operator does not move together with the MEWP in the virtual environment. Under this assumption, during the translation tasks the ground operator sees the virtual control panel move away from her/him with the MEWP and thus s/he is no longer able to reach it. S/He can operate again only after the MEWP terminates the translation and the vehicle is left stationary for at least 10 s. After this happens the synchronization is restored and the operator is teleported near the virtual ground panel so that it appears in the same position where the real one and the operator can then touch it again.

Both the simulation stations communicate with each other via an ethernet connection and allow the two operators to perform the various tasks of the exercise collaboratively and in real time. In order to feel each other’s presence within the virtual environment both the operators can see each other’s avatars.

The operating scenario

The scenario developed for the simulator includes a detailed model of a common Genie Z45-XC MEWP. In addition to faithfully reproducing the translation and the movement of the articulated arm, it pays particular attention to the safety-related mechanisms. In fact the simulation includes the presence detector, the inclinometer, the overload sensor, all the various warning and error lights and acoustic alarms and it is possible to perform the entire pre-operational check procedure as described in the official manual of the machine.

The simulated operating scenario is composed of five main areas, each one presenting different types of difficulties in reaching hypothetical work sites. They can be a window, a roof, a balcony or a streetlamp. The central area is constantly crossed by virtual pedestrians, and gives access to the other four areas respectively called area A, area B, area C and area D. Each of them contains at least one building and various streetlamps and their access is granted by varying degrees of challenging tasks. A view of one of the operating scenario areas is depicted in panel (C) of Figure 1.

Area A is characterized by the presence of humps and bumps along the access roads. On the other hand, area B has limited access due to bottlenecks and road construction sites. Area C contains a truck typically used for the parking and the long-distance transport of MEWPs. It allows the training of the operator for the tasks of getting on and off the access ramp. In area C there is also a building whose access is obstructed by poorly parked vehicles and crossing pedestrians. Finally, area D can only be reached via three ramps with different inclinations and the access to the upper floors of the building is hindered for the presence of high voltage cables. In all areas it is possible to change the weather conditions in real time, in fact events such as rain, fog or wind are simulated. The wind can be detected by the operator thanks to the presence of wind cones typically used for measuring the intensity and direction of the wind that are deployed in various strategic points of the virtual environment.

The instructor station

The instructor sits at a workstation and can supervise the training session on a dedicated monitor by which s/he can guide the trainee through the various areas. S/He can suggest one or more tasks as explained in the previous section in order to evaluate the performance of the training session. The instructor has a complete view of how the operator is using the controls in real time. S/He can also autonomously navigate the virtual environment in order to observe the exercise from the most suitable point of view and can intervene at any time on changes of atmospheric conditions. The instructor’s work session can be projected with an additional projector or a monitor to allow any audience or the rest of the class to attend the exercise.

Discussion

Occupational safety and health courses for workers are usually based on lectures accompanied by audio-visual presentations. Learning-by-doing, which is the approach ensuring higher retention rates of learners, particularly when the operation of machines is involved, is rarely possible. This is especially true for lifting machines like the aerial work platforms that have large amortization costs and are complex and risky to be handled by non-expert operators. Additionally, many hazards and critical situations that could be faced in real working scenarios cannot be replicated in traditional training sessions because they are too risky (e.g., an overturning or collision of the machine) or cannot be artificially created in the real world (e.g., a windy and foggy workspace).

XR technology, combining immersive VR-based simulation with the physical parts of the real machine interacting with the operator, can offer a learning-by-doing training method leveraging multisensory learning and a safe and controlled context without the limits of simulation found in the real world.

The ability of multimodal Virtual Environments to train complex tasks and transfer knowledge to the real world performance has been demonstrated in different fields (Gopher, 2012) and with different types of VR-based experience, encompassing simulation, collaborative environments for knowledge transmission and serious games (Philippe, et al., 2020).

Virtual Reality is quite widely used for training in industry (Xie et al., 2021) and VR-based training for safety, although to a more limited extent, has been applied in several sectors spanning chemical and construction industries (Patle et al., 2019; Zhao & Lucas, 2015), underground mines (Isleyen & Duzgun, 2019) and first responders context (Haskins et al., 2020).

The system for MEWP safety training presented in this manuscript has been designed as a high-TRL prototype for demonstrating the feasibility of a learning-by-doing training of operators and verifiers based on a mixed-reality simulator.

The simulator has been designed following the analysis of the most critical working routines with aerial work platforms and using physical components of a real MEWP. This work has been carried out by a multidisciplinary team joining experts in Virtual Environments and Mechatronics, Occupational Medicine and training for Occupational Safety and Health, and researchers from the Italian Workers’ Compensation Authority.

The simulator provides a multimodal training environment that is able to integrate for the operator the tactile feelings coming from the interaction with the real MEWP commands and the vestibular feedback through the motion platform, in order to exploit the potential of multisensory learning on skill acquisition and transfer.

New simulated working scenarios, with ad hoc characteristics, can be smoothly developed for the simulation and the same concept can be clearly applied to different working sectors where machines of different kinds are used (e.g., harbors, logistics).

Next crucial steps will be to design and test a training protocol that effectively integrate the XR-based simulator in the occupational health and safety training of workers and to assess the resulting users’ experience and the achieved learning outcomes.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding authors.

Author contributions

PSG contributed to the whole article organization and focused on System Description with the inputs of GF, AN, and FP. His major contribution was on the software components of the simulator. GF contributed to the system architecture presentation and prepared the figures for the manuscript. FV wrote the Introduction and Discussion sections with the inputs of all co-authors. AN and FP contributed to the description of the hardware components of the simulator. LM, SA, and AC contributed to the Introduction and Discussion sections with knowledge about the working routines and accidents with MEWP. MB was the PI supervising the work and its presentation in the manuscript.

Funding

The work presented in this manuscript is co-funded by INAIL (the Italian Workers' Compensation Authority) as part of the BRiC 2019 collaborative research program, Project ID 35/1-2019.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References