Skip to main content
SpringerLink
Log in

Construct and face validity of MIST-VR, Endotower, and CELTS

Are we ready for skills assessment using simulators?

  • Published:
Surgical Endoscopy And Other Interventional Techniques Aims and scope Submit manuscript

Abstract

Background

Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators.

Methods

Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training).

Results

Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two parameters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of “expert” participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively.

Conclusions

All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. McCormick PH, Tanner WA, Keane FB, Tierney S (2003) Minimally invasive techniques in common surgical procedures: implications for training. Ir J Med Sci 172(1): 27–29

    Article  PubMed  CAS  Google Scholar 

  2. Powers TW, Murayama KM, Toyama M, Murphy S, Denham EW 3rd, Derossis AM, Joehl RJ (2002) Housestaff performance is improved by participation in a laparoscopic skills curriculum. Am J Surg 184(6): 626–629

    Article  PubMed  Google Scholar 

  3. Ali MR, Mowery Y, Kaplan B, DeMaria EJ (2002) Training the novice in laparoscopy. more challenge is better. Surg Endosc 16(12): 1732–1736

    Article  PubMed  CAS  Google Scholar 

  4. Gonzalez R, Bowers SP, Smith CD Ramshaw BJ (2004) Does setting specific goals and providing feedback during training result in better acquisition of laparoscopic skills? Am J Surg 70(1): 35–39

    Google Scholar 

  5. Scott DJ, Valentine RJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Jones DB (2000) Evaluating surgical competency with the Amerian Board of Surgery In-Training Examination, skill testing, and intraoperative assessment. Surgery 128(4): 613–622

    Article  PubMed  CAS  Google Scholar 

  6. Hyltander A, Liljegren E, Rhodin PH, Lonroth H (2002) The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 16(9): 1324–1328

    Article  PubMed  CAS  Google Scholar 

  7. Scott DJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus DM, Jeyarajah DR, Thompson WM, Jones DB (2000) Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 191(3): 272–283

    Article  PubMed  CAS  Google Scholar 

  8. Scott DJ, Young WN, Tesfay ST, Frawley WH, Rege RV, Jones DB (2001) Laparoscopic skills training. Am J Surg 182(2): 137–142

    Article  PubMed  CAS  Google Scholar 

  9. Hamilton EC, Scott DJ, Kapoor A, Nwariaku F, Bergen PC, Rege RV, Tesfay ST, Jones DB (2001) Improving operative performance using a laparoscopic hernia simulator. Am J Surg 182(6): 725–728

    Article  PubMed  CAS  Google Scholar 

  10. Ota D, Loftin B, Saito T, Lea R, Keller J (1995) Virtual reality in surgical education. Comput Bio Med 25(2): 127–137

    Article  CAS  Google Scholar 

  11. Sutton C, McCloy R, Middlebrook A, Chater P, Wilson M, Stone R (1997) MIST VR: a laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform 39: 598–607

    PubMed  CAS  Google Scholar 

  12. Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P (2001) Virtual reality computer simulation. Surg Endosc 15(3): 242–244

    Article  PubMed  CAS  Google Scholar 

  13. Ahlberg G, Heikkinen T, Iselius L, Leijonmarck CE, Rutqvist J, Arvidsson D (2002) Does training in a virtual reality simulator improve surgical perfomance? Surg Endosc 16(1): 126–129

    Article  PubMed  CAS  Google Scholar 

  14. Wilhelm DM, Ogan K, Roehrborn CG, Cadeddu JA, Pearle MS (2002) Assessment of basic endoscopic performance using a virtual reality simulator. J Am Coll Surg 195(5): 675–681

    Article  PubMed  Google Scholar 

  15. Gallagher AG, McClure N, McGuigan J, Crothers I, Browning J (1999) Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST-VR). Endoscopy 31(4): 310–313

    Article  PubMed  CAS  Google Scholar 

  16. Torkington J, Smith SG, Rees BI, Darzi A (2004) Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc 15(10): 1076–1079

    Article  Google Scholar 

  17. Jordan JA, Gallagher AG, McGuigan J, McClure N (2001) Virtual reality training leads to faster adaptation to the novel psychomotor restrictions encountered by laparoscopic surgeons. Surg Endosc 15(10): 1080–1084

    Article  PubMed  CAS  Google Scholar 

  18. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4): 458–463

    Article  PubMed  Google Scholar 

  19. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91(2): 146–150

    Article  PubMed  CAS  Google Scholar 

  20. Jordan JA, Gallagher AG, McGuigan J, McGlade K, McClure N (2000) A comparison between randomly alternating imaging, normal laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition. Am J Surg 180(3): 208–211

    Article  PubMed  CAS  Google Scholar 

  21. Kothari SN, Kaplan BJ, DeMaria EJ, Broderick TJ, Merrell RC (2002) Training in laparoscopic suturing skills using a new computer-based virtual reality simulator (MIST-VR) provides results comparable to those with an established pelvic trainer system. J Laparoendosc Adv Surg Tech A 12(3): 167–173

    Article  PubMed  Google Scholar 

  22. Hamilton EC, Scott DJ, Fleming JB, Rege RV, Laycock R, Bergen PC, Tesfay ST, Jones DB (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 16(3): 406–411

    Article  PubMed  CAS  Google Scholar 

  23. Brunner WC, Korndorffer JR, Sierra R, Dunne JB, Yau CL, Corsetti RL, Slakey DP, Townsend MC, Scott DJ (2005) Determining standards for laparoscopic competency using virtual reality. Am Surg 71: 29–35

    Google Scholar 

  24. Haluck RS, Webster RW, Snyder AJ, Melkonian MG, Mohler BJ, Dise ML, Lefever A (2001) A virtual reality surgical trainer for navigation in laparoscopic surgery. Stud Health Technol Inform 81: 171–176

    PubMed  CAS  Google Scholar 

  25. Stylopoulos N, Cotin S, Maithel SK, Ottensmeyer M, Jackson PG, Bardsley RS, Neumann PF, Rattner DW, Dawson SL (2004) Computer-enhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc 18(5): 782–789

    Article  PubMed  CAS  Google Scholar 

  26. Torkington J, Smith SG, Rees BI, Darzi A (2001) The role of the basic surgical skills course in the acquisition and retention of laparoscopic skill. Surg Endosc 15(10): 1071–1075

    Article  PubMed  CAS  Google Scholar 

  27. Munz Y, Kumar BD, Moorthy K, Mann S, Darzi A (2004) Laparoscopic virtual reality and box trainers: is one superior to the other? Surg Endosc 18: 485–494

    Article  PubMed  CAS  Google Scholar 

  28. Gallagher AG, Richie K, McClure N, McGuigan J (2001) Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg 25(11): 1478–1483

    Article  PubMed  CAS  Google Scholar 

  29. Gallagher AG, Lederman AB, McGlade K, Satava RM, Smith CD (2004) Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc 18: 660–665

    Article  PubMed  CAS  Google Scholar 

  30. Taffinder N, Sutton C, Fishwick RJ, McManus IC, Darzi A (1998). Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Stud Health Technol Inform 50: 124–130

    PubMed  CAS  Google Scholar 

  31. McNatt SS, Smith CD (2001) A computer-based laparoscopic skills assessment device differentiates experienced from novice laparoscopic surgeons. Surg Endosc 15(10): 1085–1089

    Article  PubMed  CAS  Google Scholar 

  32. Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on MIST-VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 81(4): 281–286

    PubMed  CAS  Google Scholar 

  33. Gallagher AG, Satava RM (2002) Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc 16(12): 1746–1752

    Article  PubMed  CAS  Google Scholar 

  34. Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2003) Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. Am J Surg 185(2): 146–149

    Article  PubMed  Google Scholar 

  35. Maithel SK, Villegas L, Stylopoulos N, Dawson S, Jones DB (2005) Simulated laparoscopy using a head-mounted display vs traditional video monitor: an assessment of performance and muscle fatigue. Surg Endosc 19: 406–411

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

We thank Stephane Cotin and Ryan Bardsley from CIMIT for their help in preparing and transporting CELTS for the SAGES annual meeting. We also thank Karl Storz Endoscopy and Meti for equipment and support.

Author information

Authors and Affiliations

  1. Department of Minimally Invasive Surgery, Beth Israel Deaconess Medical Center, 110 Francis Street, Suite 9, Boston, MA, 02215, USA

    S. Maithel, M. Callery & D. Jones

  2. Center for Minimally Invasive Surgery, Tulane University Health Sciences Center, 1440 Canal Street, Suite 2400, New Orleans, LA, 70112, USA

    R. Sierra, J. Korndorffer & D. Scott

  3. Center for Integration of Medicine and Innovative Technology (CIMIT), 65 Landsdowne Street, Suite 200, Cambridge, MA, 02139, USA

    P. Neumann & S. Dawson

Authors
  1. S. Maithel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Sierra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Korndorffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. P. Neumann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. S. Dawson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. M. Callery
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. D. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. D. Scott
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Maithel.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Maithel, S., Sierra, R., Korndorffer, J. et al. Construct and face validity of MIST-VR, Endotower, and CELTS. Surg Endosc 20, 104–112 (2006). https://doi.org/10.1007/s00464-005-0054-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-005-0054-4

Keywords

Search

Navigation