Elsevier

Knowledge-Based Systems

Volume 70, November 2014, Pages 97-102
Knowledge-Based Systems

Psychomotor skills assessment in medical training based on virtual reality using a Weighted Possibilistic approach

https://doi.org/10.1016/j.knosys.2014.05.006Get rights and content

Abstract

Virtual reality has been used to provide training systems in several areas, particularly in medicine. In that area, user’s interactions in a virtual environment are modeled and compared with predefined classes of performance to know how much users are prepared to perform that procedure on human beings. In this paper is proposed a new approach for online Single User’s Assessment System (SUAS) using Weighted Possibility and Necessity measures. Those fuzzy measures provide an interval-based estimator to check the compatibility between interactions of an user and previous stored classes of performance. This approach integrates the kernel of a decision support system and it contributes to improve the user’s assessment taking into account the individual relevance of variables in a procedure. It was verified in performance tests, when this new SUAS approach achieved better results, according to Kappa Coefficient, when compared with other previous approaches.

Introduction

The advent of Virtual Reality (VR) brought new possibilities and applications to computer systems. The environments created by VR techniques include immersive and interactive features that allow users taking part of virtual simulations having their senses explored by special devices. Thus, these users can act and receive information from the system in a similar way this could be performed in the real world. An important focus of Virtual Reality simulators are risky tasks, that can cause injures, or those related to expensive procedures. In some areas, VR systems for training have been used to provide metrics to a proficiency criterion of learning, as in commercial aviation [16].

Assessment systems have been used with VR simulators to allow objectively determine the user’s performance. In some areas, it is done by an expert, which observes user actions and movements, and emits an assessment afterwards. In others, the assessment is based on videotapes post-analysis [28]. Despite the informality of many assessment methodologies, Carter [4] cites the reliability as an important factor related to the assessment since it indicates that an assessment tool must provide consistent results with minimal errors of measurement. In the same direction, Lammers et al. [13] indicate that the need of methods to assess technical competence with minimal errors is a consensus.

The risk present in several medical procedures make them a potential area for VR simulators and they have been pointed as an important learning tool in the educational process of new physicians [1], [2], [26]. The use of realistic simulators is based on the constructivism theory in which users can have active experiences, acting and receiving feedback. However, a common point of reflection rests on the methodology that can be used to objectively assess the user’s psychomotor skills in these simulators [4].

Although the benefits of training based on VR be known [27], [30], one of the main concerns about assessment is related to the definition of metrics that allow defining the steps and factors necessary to guarantee the assessment of the user’s psychomotor skills, as well as which variables are relevant in the assessment process. At this point, the lack of well defined and accepted metrics is still a problem [32]. Some authors have pointed out specific variables which can contribute for user’s assessment in medical training, as hand-motion [10], applied forces [8], [14] and elapsed time [12], among others [11]. Those variables allow measuring differences between experts and novices surgeons [3], [24].

However, the assessment problem can be solved by a decision support system (DSS) coupled to the VR simulator. The main idea behind DSS rests on the automatic composition of performance classes. Thus, the experts do not need to verbalize what is wrong or right in the procedure, but perform the simulation in the VR system and have their actions monitored by a DSS to compose a knowledge model to be used further by users in training [16]. Thus, the assessment system is a knowledge-based system which analyze data from users’ interactions in a training system based on VR and provides a decision making support on the users’ proficiency for performance in real procedures [21].

Since 90’s, several Single User’s Assessment System (SUAS) have been proposed [16], [18], [21], [23], [25], [29] for VR simulators, mainly for medical training. With continuous advances on capacity of computers, SUAS evolved too. Nowadays, a SUAS must continuously monitor all user interactions on the VR environment and compare their performance with pre-defined expert’s classes of performance to recognize user’s level of training. Basically, there are two types of SUAS: off-line and on-line. Off-line SUAS can be defined as methods coupled or not to VR systems, whose assessment results are provided some time (which can be minutes, hours or days) after the end of the training. On the other hand, on-line SUAS are coupled to the training system and collect user data to provide a result about their performance at the end of the simulation [16], [19], which is characterized by time of response lower than one second. Because many processes are running simultaneously in a VR system, an on-line SUAS must have low complexity to does not compromise the simulation performance. However, it also must have high accuracy to does not compromise the assessment results.

Recent researches focused on finding good methodologies for assessment that could be integrated to simulation systems based on VR. The main concern has been related to the quality of the assessment, which is relevant when the simulations are used for training purposes. This way, alternative methods using interval-based information are welcome to improve assessment accuracy and provide a more comprehensive use of the interaction data.

This paper presents a new training assessment system based on weighted Possibility and Necessity Measures. This method uses fuzzy measures to construct an interval-based estimator to provide users’ assessment support based on their actions in VR simulator and takes into account the relative weighting of variables in the simulation. It is an innovative idea, once a simple interval-based approach was proposed recently [17], [20] but it did not use any weighting on assessment variables. From the user’s point of view, it is expected that an objective assessment of a procedure could be more adequate and accurate when it takes into account weighting of variables relevance in that procedure.

This paper is organized as follows: the next section presents some requirements of VR training simulators. Section 3 brings some theoretical aspects of the assessment methodology proposed on Section 4. Results obtained, as well as their analysis, are presented in Section 5, followed by some considerations in Section 6.

Section snippets

Training simulators based on VR

Training simulators based on VR are real time computer systems used to provide realistic environments for the practice of activities. The specificity of the activity simulated will determine the features of the system and also the variables necessary to be monitored by the system. In VR systems is possible to deal with details and particular issues and also augment features related to the procedure, exploring human senses as sight, hearing and touch, to provide interactive and immersive virtual

Theoretical aspects

For better understanding of the assessment method proposed, some considerations and definitions need to be previously provided. Firstly, it is defined the concept of fuzzy sets, followed by definitions of measures of possibility and necessity. Finally, it is presented the assessment method.

Definition 1

Let X be a universal set, in which each element is denoted by x. Then, a fuzzy set A in X is given by A={x,μA(x)}, xX, where μA(x) is called the membership function, which provides a grade of membership of a

Training assessment

A SUAS based on a Weighted Possibilistic approach can be coupled to a VR simulator. To test this SUAS, a bone marrow harvest simulator [15] was considered. This simulator was designed to support the learning of bone marrow harvest in humans, a procedure necessary for bone marrow transplant. The simulator is composed by three modules corresponding to the steps considered important in the execution of a real procedure: visualization (1) of the anatomy, palpation (2) to identify the bone under the

Results and analysis

The calibration of the SUAS based on Weighted Possibilistic approach was made before any assessment of training. For that, an expert executed the procedure several times, where each execution was labeled according to one class of performance. For the tests, M=3 classes of performance were defined by the expert: (1) “correct procedure”, (2) “acceptable procedure” and (3) “badly executed procedure”. After, for a controlled and impartial analysis, several users used the system and 300 training

Final considerations

This paper presented a new methodology for SUAS of medical psychomotor skills in VR simulators. This approach allows integration of a DSS for objective assessement and uses experts’ knowledge to compose a knowledge model to be applied in the assessment of users. Data acquired from experts’ interactions compose this model. That methodology is based on a Weighted Possibilistic approach, which use Possibility and Necessity Measures as a kernel of the DSS. Those measures provide an interval-based

Acknowledgements

This project is partially supported by Grants 310561/2012-4 and 310470/2012-9 of the National Council for Scientific and Technological Development (CNPq) and is related to the National Institute of Science and Technology “Medicine Assisted by Scientific Computing” (181813/2010-6) also supported by CNPq.

References (33)

  • J. Cohen

    A coefficient of agreement for nominal scales

    Edu. Psychol. Measure.

    (1960)
  • D. Dubois et al.

    Possibility Theory

    (1988)
  • T. Flaminio et al.

    Possibilistic states over finitely valued Lukasiewicz events

    J. Logic Comput.

    (2011)
  • E.D. Grober

    Validation of novel and objective measures of microsurgical skill: hand-motion analysis and stereoscopic visual acuity

    Microsurgery

    (2003)
  • C. Ikehara

    Evaluating a virtual reality motor-skills simulator

    Stud. Health Technol. Inform.

    (2009)
  • R. Lammers et al.

    Teaching and assessing procedural skills using simulation: metrics and methodology

    Acad. Emerg. Med.

    (2008)
  • Cited by (0)

    View full text