Face, Content, and Construct Validation of the da Vinci Skills Simulator

doi:10.1016/j.urology.2012.01.028

Urology

Volume 79, Issue 5, May 2012, Pages 1068-1072

https://doi.org/10.1016/j.urology.2012.01.028 Get rights and content

Objective

To report on assessments of face, content, and construct validity for the commercially available da Vinci Skills Simulator (dVSS).

Methods

A total of 38 subjects participated in this prospective study. Participants were classified as novice (0 robotic cases performed), intermediate (1-74 robotic cases), or expert (≥75 robotic cases). Each subject completed 5 exercises. Using the metrics available in the simulator software, the performances of each group were compared to evaluate construct validation. Immediately after completion of the exercises, each subject completed a questionnaire to evaluate face and content validation.

Results

The novice group consisted of 18 medical students and 1 resident. The intermediate group included 6 residents, 1 fellow, and 2 faculty urologist. The expert group consisted of 2 residents, 1 fellow, and 7 faculty surgeons. The mean number of robotic cases performed by the intermediate and expert groups was 29.2 and 233.4, respectively. An overall significant difference was observed in favor of the more experienced group in 4 skill sets. When intermediates and experts were combined into a single “experienced” group, they significantly outperformed novices in all 5 exercises. Intermediates and experts rated various elements of the simulators realism at an average of 4.1/5 and 4.3/5, respectively. All intermediate and expert participants rated the simulator's value as a training tool as 4/5 or 5/5.

Conclusion

Our study supports the face, content, and construct validation attributed to the dVSS. These results indicate that the simulator may be most useful to novice surgeons seeking basic robot skills acquisition.

Section snippets

Material and Methods

A convenience cohort of 38 medical students, residents, and attending physicians from 5 services (urology, otolaryngology, cardiac surgery, thoracic surgery, and gynecology) participated in this prospective, institutional review board–approved study. Participants were classified as novice (no past performance of robotic cases), intermediate (past performance of 1-74 robotic cases) and expert (past performance of ≥75 robotic cases) according to review of a centralized case log. None of the

Results

The novice group (n = 19) consisted of 18 medical students and 1 resident. All participants in the novice group had 0 robotic cases. The intermediate group (n = 8) included 6 residents, 1 fellow, and 1 attending. Of note, all participants classified as intermediate had between 1 and 25 cases except the attending physician who had 130 cases but had not been on the console in more than 5 years. It was because of this lack of recent robotic surgical experience and unfamiliarity with the newer

Comment

The field of robotic surgery has seen sustained growth since the early 2000s. In 2010, an estimated 98 000 da Vinci prostatectomy procedures and 110 000 da Vinci hysterectomy procedures were preformed, both significant increases over 2009, which saw 90,000 and 69,000 performed, respectively. Overall, an estimated 278 000 da Vinci procedures were performed in 2010, compared with 205,000 in 2009.11, 12 The growth of robotics has been in both numbers and scope. With more specialties adopting

Conclusions

Our study confirms face, content, and, construct validation of the commercially available dVSS. The eventual role of VR simulation in robotic surgical training, credentialing, and proctoring has yet to be determined and should be evaluated in larger, multiinstitutional studies.

Acknowledgments

This work is supported by the US Army Medical Research and Materiel Command under Contract No W81XWH-10-1-1058.

The views, opinions and/or findings contained manuscript are those of the author(s) and should not be construed as an official Department of the Army position, policy or decision unless so designated by other documentation.

References (12)

S.D. Herrell et al.
Robotic-assisted laparoscopic prostatectomy: what is the learning curve?
Urology
(2005)
M. Menon et al.
Prospective comparison of radical retropubic prostatectomy and robot-assisted anatomic prostatectomy: the Vattikuti Urology Institute experience
Urology
(2002)
P.L. Steinberg et al.
The cost of learning robotic-assisted prostatectomy
Urology
(2008)
J.H. Peters et al.
Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery
Surgery
(2004)
P.A. Kenney et al.
Face, content, and construct validity of dV-Trainer, a novel virtual reality simulator for robotic surgery
Urology
(2009)
A.J. Hung et al.
Face, content and construct validity of a novel robotic surgery simulator
J Urol
(2011)

There are more references available in the full text version of this article.

Cited by (92)

Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum
2024, Journal of Surgical Education
Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks.
Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity.
This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital.
Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance.
Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels.
This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.
Augmented and virtual reality in spine surgery
2023, Journal of Orthopaedics
Augmented Reality (AR) and Virtual Reality (VR) have developed unprecedentedly in recent years, providing interesting opportunities for medical applications. Their integration into clinical assessment, surgical workflow, and training has shown tremendous potential to improve daily life activity in spine surgery. The paper explores the utilization of VR and AR in spine surgery, with their applications, benefits, challenges, and forthcoming prospects.
Virtual Reality Simulation Has Weak Correlation with Overall Trainee Robot-Assisted Laparoscopic Hysterectomy Performance
2022, Journal of Minimally Invasive Gynecology
Both simulator practice and intraoperative performance serve to inform surgical trainee training, but the skill transfer from simulation to the intraoperative setting remains unclear. This study evaluates the correlation between trainee performance on virtual reality simulation and (1) overall intraoperative performance during robotic-assisted laparoscopic hysterectomy (RALH) procedures and (2) suturing performance during vaginal cuff closure portion of the case.
Retrospective subgroup analysis of randomized controlled trial.
Academic hospital.
Patients with RALH (N = 29).
Gynecological trainees (N = 21) performed simulation tasks using the da Vinci skills simulator on the day of surgery before performing RALH. Attending surgeons assessed participants’ intraoperative performance using Global Evaluative Assessment of Robotic Skills (GEARS). Performance of the vaginal cuff closure step was subsequently assessed using GEARS scoring of anonymized videos. Spearman's correlation was used to quantify the relationship between simulation and intraoperative performances.
Trainees achieved a median intraoperative GEARS score of 18.5/30 (interquartile range: 17−22) and a median total simulator score of 84.4/100 (interquartile range: 78.1−87.5). More advanced residents exhibited worse overall simulator performance (median score 86.6/100 compared with 78.8/100, p = .03) and similar intraoperative GEARS scores during overall RALH and vaginal cuff closure compared with less experienced trainees. Total simulation performance score was negatively correlated with GEARS Bimanual Dexterity (ρ = -0.46, p = .02) and Force Sensitivity subscores (ρ = -0.39, p = .05). There was no correlation between total GEARS intraoperative vaginal cuff closure scores and overall simulation performances; however, total Tubes simulation score was correlated with higher GEARS Force Sensitivity subscore (ρ = 0.73, p = .05).
In this study, there was limited correlation between simulation score metrics and trainees’ overall intraoperative performance. Furthermore, we identified that GEARS scores could not distinguish between similar trainee skill levels. These findings underscore the need to develop intraoperative assessment tools that can better discriminate different but similar skill levels.
Simulators
2022, Endorobotics: Design, R and D and Future Trends
This chapter describes virtual reality (VR) simulators for robot-assisted surgery (RAS). It starts with the definition of surgical errors and shows how they are related to the level of technical skills of surgeons. Relevant skills and human factors for minimal access surgery are discussed. Then, the different types of surgical simulators are described. VR simulators are then presented as tools for the objective assessment of technical skills based on the paradigm in long established use for VR flight simulators. The features of the available VR simulators for RAS are presented, including their assessment system. A critical discussion is reported on evidence from the published literature on validation studies of the VR simulators, and approved curricula. The chapter concludes with areas of interests where future research should be conducted.
Robotic Surgery Training: Current Trends and Future Directions
2021, Urologic Clinics of North America
Haptics and virtual reality for oral and maxillofacial surgery
2021, Computer-Aided Oral and Maxillofacial Surgery: Developments, Applications, and Future Perspectives
The research of haptic technology involves computer science, psychology, cognitive science, neuroscience, mechanics and control, computers, and other disciplines. Although it has received intensive research over the last few decades, it is still a great challenge today and the continuous demand for the fidelity of force tactile interaction will continuously promote the progress of haptic interaction technology.

View all citing articles on Scopus

View full text

Laparoscopy and RoboticsFace, Content, and Construct Validation of the da Vinci Skills Simulator

Objective

Methods

Results

Conclusion

Section snippets

Material and Methods

Results

Comment

Conclusions

Acknowledgments

Urology

Urology

Urology

Surgery

Urology

J Urol

Laparoscopy and Robotics
Face, Content, and Construct Validation of the da Vinci Skills Simulator