Psychomotor skills assessment in medical training based on virtual reality using a Weighted Possibilistic approach
Introduction
The advent of Virtual Reality (VR) brought new possibilities and applications to computer systems. The environments created by VR techniques include immersive and interactive features that allow users taking part of virtual simulations having their senses explored by special devices. Thus, these users can act and receive information from the system in a similar way this could be performed in the real world. An important focus of Virtual Reality simulators are risky tasks, that can cause injures, or those related to expensive procedures. In some areas, VR systems for training have been used to provide metrics to a proficiency criterion of learning, as in commercial aviation [16].
Assessment systems have been used with VR simulators to allow objectively determine the user’s performance. In some areas, it is done by an expert, which observes user actions and movements, and emits an assessment afterwards. In others, the assessment is based on videotapes post-analysis [28]. Despite the informality of many assessment methodologies, Carter [4] cites the reliability as an important factor related to the assessment since it indicates that an assessment tool must provide consistent results with minimal errors of measurement. In the same direction, Lammers et al. [13] indicate that the need of methods to assess technical competence with minimal errors is a consensus.
The risk present in several medical procedures make them a potential area for VR simulators and they have been pointed as an important learning tool in the educational process of new physicians [1], [2], [26]. The use of realistic simulators is based on the constructivism theory in which users can have active experiences, acting and receiving feedback. However, a common point of reflection rests on the methodology that can be used to objectively assess the user’s psychomotor skills in these simulators [4].
Although the benefits of training based on VR be known [27], [30], one of the main concerns about assessment is related to the definition of metrics that allow defining the steps and factors necessary to guarantee the assessment of the user’s psychomotor skills, as well as which variables are relevant in the assessment process. At this point, the lack of well defined and accepted metrics is still a problem [32]. Some authors have pointed out specific variables which can contribute for user’s assessment in medical training, as hand-motion [10], applied forces [8], [14] and elapsed time [12], among others [11]. Those variables allow measuring differences between experts and novices surgeons [3], [24].
However, the assessment problem can be solved by a decision support system (DSS) coupled to the VR simulator. The main idea behind DSS rests on the automatic composition of performance classes. Thus, the experts do not need to verbalize what is wrong or right in the procedure, but perform the simulation in the VR system and have their actions monitored by a DSS to compose a knowledge model to be used further by users in training [16]. Thus, the assessment system is a knowledge-based system which analyze data from users’ interactions in a training system based on VR and provides a decision making support on the users’ proficiency for performance in real procedures [21].
Since 90’s, several Single User’s Assessment System (SUAS) have been proposed [16], [18], [21], [23], [25], [29] for VR simulators, mainly for medical training. With continuous advances on capacity of computers, SUAS evolved too. Nowadays, a SUAS must continuously monitor all user interactions on the VR environment and compare their performance with pre-defined expert’s classes of performance to recognize user’s level of training. Basically, there are two types of SUAS: off-line and on-line. Off-line SUAS can be defined as methods coupled or not to VR systems, whose assessment results are provided some time (which can be minutes, hours or days) after the end of the training. On the other hand, on-line SUAS are coupled to the training system and collect user data to provide a result about their performance at the end of the simulation [16], [19], which is characterized by time of response lower than one second. Because many processes are running simultaneously in a VR system, an on-line SUAS must have low complexity to does not compromise the simulation performance. However, it also must have high accuracy to does not compromise the assessment results.
Recent researches focused on finding good methodologies for assessment that could be integrated to simulation systems based on VR. The main concern has been related to the quality of the assessment, which is relevant when the simulations are used for training purposes. This way, alternative methods using interval-based information are welcome to improve assessment accuracy and provide a more comprehensive use of the interaction data.
This paper presents a new training assessment system based on weighted Possibility and Necessity Measures. This method uses fuzzy measures to construct an interval-based estimator to provide users’ assessment support based on their actions in VR simulator and takes into account the relative weighting of variables in the simulation. It is an innovative idea, once a simple interval-based approach was proposed recently [17], [20] but it did not use any weighting on assessment variables. From the user’s point of view, it is expected that an objective assessment of a procedure could be more adequate and accurate when it takes into account weighting of variables relevance in that procedure.
This paper is organized as follows: the next section presents some requirements of VR training simulators. Section 3 brings some theoretical aspects of the assessment methodology proposed on Section 4. Results obtained, as well as their analysis, are presented in Section 5, followed by some considerations in Section 6.
Section snippets
Training simulators based on VR
Training simulators based on VR are real time computer systems used to provide realistic environments for the practice of activities. The specificity of the activity simulated will determine the features of the system and also the variables necessary to be monitored by the system. In VR systems is possible to deal with details and particular issues and also augment features related to the procedure, exploring human senses as sight, hearing and touch, to provide interactive and immersive virtual
Theoretical aspects
For better understanding of the assessment method proposed, some considerations and definitions need to be previously provided. Firstly, it is defined the concept of fuzzy sets, followed by definitions of measures of possibility and necessity. Finally, it is presented the assessment method. Definition 1 Let X be a universal set, in which each element is denoted by x. Then, a fuzzy set A in X is given by , , where is called the membership function, which provides a grade of membership of a
Training assessment
A SUAS based on a Weighted Possibilistic approach can be coupled to a VR simulator. To test this SUAS, a bone marrow harvest simulator [15] was considered. This simulator was designed to support the learning of bone marrow harvest in humans, a procedure necessary for bone marrow transplant. The simulator is composed by three modules corresponding to the steps considered important in the execution of a real procedure: visualization (1) of the anatomy, palpation (2) to identify the bone under the
Results and analysis
The calibration of the SUAS based on Weighted Possibilistic approach was made before any assessment of training. For that, an expert executed the procedure several times, where each execution was labeled according to one class of performance. For the tests, classes of performance were defined by the expert: (1) “correct procedure”, (2) “acceptable procedure” and (3) “badly executed procedure”. After, for a controlled and impartial analysis, several users used the system and 300 training
Final considerations
This paper presented a new methodology for SUAS of medical psychomotor skills in VR simulators. This approach allows integration of a DSS for objective assessement and uses experts’ knowledge to compose a knowledge model to be applied in the assessment of users. Data acquired from experts’ interactions compose this model. That methodology is based on a Weighted Possibilistic approach, which use Possibility and Necessity Measures as a kernel of the DSS. Those measures provide an interval-based
Acknowledgements
This project is partially supported by Grants 310561/2012-4 and 310470/2012-9 of the National Council for Scientific and Technological Development (CNPq) and is related to the National Institute of Science and Technology “Medicine Assisted by Scientific Computing” (181813/2010-6) also supported by CNPq.
References (33)
- et al.
Weighted fuzzy pattern matching
Fuzzy Sets Syst.
(1988) Quantification of motion characteristics and forces applied to tissues during suturing
Am. J. Surg.
(2005)- et al.
Cognitive simulators for medical education and training
J. Biomed. Inform.
(2009) - et al.
Intelligent assessment based on beta regression for realistic training on medical simulators
Knowl.-Based Syst.
(2012) Psychomotor testing predicts rate of skill acquisition for proficiency-based laparoscopic skills training
Surgery
(2006)Fuzzy sets as a basis for a theory of possibility
Fuzzy Sets Syst.
(1978)- et al.
Virtual intensive care unit (ICU): real-time simulation environment applying hybrid approach using dynamic Bayesian networks and ODEs
Stud. Health Technol. Inform.
(2009) An aneurysm clipping training module for the neurosurgical training simulator NeuroSim
Stud. Health Technol. Inform.
(2012)Trocar insertion: the neglected task of VR simulation
Stud. Health Technol. Inform.
(2008)- et al.
Consensus guidelines for validation on virtual reality surgical simulators
Surg. Endosc.
(2005)