Elsevier

Journal of Surgical Research

Volume 246, February 2020, Pages 342-378
Journal of Surgical Research

Education and Career Development
Current Status of Technical Skills Assessment Tools in Surgery: A Systematic Review

https://doi.org/10.1016/j.jss.2019.09.006Get rights and content

Abstract

Background

Tools for assessment of technical skills are a crucial part of surgical education. They provide trainees with quantitative feedback highlighting both proficiency and areas for improvement. For this to be relevant to day-to-day practice, the tools used have to be validated and relevant to each surgical situation. This study aims to evaluate the validity of assessment tools used within surgery.

Materials and methods

Following Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines, a systematic review was conducted searching the MEDLINE and Embase databases (International Prospective Register of Systematic Reviews: CRD42018104674). Studies utilizing any assessment tool in any surgical specialty were included. Messick's criteria were used for literature evaluation, and the Modified Educational Oxford Centre for Evidence-Based Medicine was used to assess levels of recommendation.

Results

A total of 303 studies and 76 tools were included. The most commonly used tool was Objective Structured Assessment Tool Skills (OSATS; n = 137, 45.2%). OSATS was used in conjunction with another tool or tools in an additional 55 studies (18.2%). Seven further tools were used in at least 3 studies. A total of five studies evaluated contained all five aspects of Messick's validity.

Conclusions

There are several widely validated tools for assessing technical skills, the most common of which is OSATS. There is an emerging trend for crowdsourcing as a quick, cheap method for assessment of technical skills. This technique has been validated using both GEARS and GOALS. Numerous tools were found to be used only once and demonstrate a tendency for units to create their own tools for a specific task or specialty.

Introduction

Surgical training has classically revolved around the Halstedian1 training model of “see one, do one, teach one.” This technique of assessment of technical skills relies on subjective feedback from supervisors in order to improve.2 While once perceived as an effective method, the changes to working times, the desire for more objective feedback for trainees, and the ethical dilemmas of operating before being deemed competent has brought about a change to training practices.3 Furthermore, surgical competency is now considered to be an assimilation of both technical and nontechnical skills.

Today, technological advances provide the ability to simulate and hone technical skills outside of the operating room (OR). There has been a move away from curriculums based on cumulative OR experience toward those based on assessment of competency.3 The cornerstone of these programs is the requirement to evaluate technical proficiency using objective assessment tools.4 There is a plethora of tools currently available that are either task/procedure specific or generic.

For tools to be used in assessment, they must be validated. Consequently, this systematic review aims to (a) identify the currently utilized technical assessment tools across surgical specialties; (b) evaluate for validity evidence, as per the Messick framework5; and (c) assess the levels of recommendation of each assessment tool.

Section snippets

Material and methods

This was a priori protocol (International Prospective Register of Systematic Reviews registration number: CRD42018104674) systematic review, conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines.6 Registration of the trial with International Prospective Register of Systematic Reviews, a database of ongoing systematic reviews with health-related outcomes, ensured there was no duplication of work and provided a platform for comparison

Results

The literature search identified 2207 potentially relevant titles. A further 24 titles were added through bibliography searching. After removing duplicates and non-English studies, 1268 studies remained. The full texts of 692 studies were assessed—303 met the inclusion criteria (Figure). Seventy-six tools were used; the most popular tool of which was the Objective Structured Assessment Tool Skills (OSATS). Surgical specialties using tools included breast (n = 2), cardiothoracic (n = 6), ENT (n

Discussion

This review found that a large number of tools are currently being used to assess surgical competency across specialties in different settings. The most commonly used tool was found to be OSATS, which has been validated across all major surgical subspecialties, in bench models, animal models, VR simulators, and in the OR. It was used in 191 studies (139 alone and 55 in conjunction with other tools), which make up the majority of studies examined in this review. OSATS is commonly used as a

Conclusion

There is a need in surgery to be able to reliably assess the technical skills of trainees, both for education and examination. This review found that numerous tools have been validated for this purpose. OSATS had the highest LoR across surgical specialties in both simulation and OR settings. Other validated tools include GOALS, GRS, ROSATS, and GEARS. The advent of crowdsourcing has shown potential as an alternative to traditional assessment by specialists, which promises quicker, cheaper,

Acknowledgment

Authors' contributions: A.V., J.R., A.A., and N.R. contributed to data extraction, manuscript drafting, critical analysis, and revisions. A.A., P.D., and K.A. contributed to idea conception, manuscript drafting, critical analysis, and revisions.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References (313)

  • C.E. Buckley et al.

    Does aptitude influence the rate at which proficiency is achieved for laparoscopic appendectomy?

    J Am Coll Surg

    (2013)
  • S.N. Steigerwald et al.

    Does laparoscopic simulation predict intraoperative performance? A comparison between the Fundamentals of Laparoscopic Surgery and LapVR evaluation metrics

    Am J Surg

    (2015)
  • Y. Ahmed et al.

    Does surgeon leadership during robot-assisted surgery correlate with teamwork-related outcomes?

    J Urol

    (2018)
  • S. Polterauer et al.

    Effect of preoperative warm-up exercise before laparoscopic gynecological surgery: a randomized trial

    J Surg Educ

    (2016)
  • L. Rosen et al.

    The effect of resident simulation training on operative time of the retropubic midurethral sling procedure for the treatment of stress urinary incontinence

    Am J Obstet Gynecol

    (2017)
  • H.Y. Chiu et al.

    The effectiveness of a simulation-based flipped classroom in the acquisition of laparoscopic suturing skills in medical students-A pilot study

    J Surg Educ

    (2018)
  • S. De Montbrun et al.

    Establishing passing scores for technical performance in surgery: lessons learned from a 10-year experience of resident assessment

    J Am Coll Surg

    (2014)
  • D.J. Anastakis et al.

    Evaluating the effectiveness of a 2-year curriculum in a surgical skills center

    Am J Surg

    (2003)
  • A. Vogell et al.

    An evaluation of the utility of robotic virtual reality simulation in gynecologic resident surgical education

    J Minim Invasive Gynecol

    (2014)
  • V. LeBlanc et al.

    Examination stress leads to improvements on fundamental technical skills for surgery

    Am J Surg

    (2008)
  • G. Ault et al.

    Exporting a technical skills evaluation technology to other sites

    Am J Surg

    (2001)
  • J. Wood et al.

    The fann objective structured assessment of technical skills score in cardiothoracic set trainees undertaking coronary anastomosis

    Heart Lung Circ

    (2017)
  • G.G. Gosman et al.

    Focused assessment of surgical performance: difficulty with faculty compliance

    Am J Obstet Gynecol

    (2005)
  • P. Romero et al.

    Halsted's "See one, do one, and teach one" versus peyton's four-step approach: a randomized trial for training of laparoscopic suturing and knot tying

    J Surg Educ

    (2018)
  • S.L. York et al.

    Implementation and evaluation of a dilation and evacuation simulation training curriculum

    Contraception

    (2016)
  • S.E. Swift et al.

    Institution and validation of an observed structured assessment of technical skills (OSATS) for obstetrics and gynecology residents and faculty

    Am J Obstet Gynecol

    (2006)
  • A. Ghazi et al.

    Medical student deliberate practice on a virtualreality curriculum can achieve equivalency to senior urology resident real-time training

    J Urol

    (2015)
  • S. Arora et al.

    Mental practice enhances technical skills and teamwork in crisis simulations-a double blind, randomised controlled study

    J Am Coll Surg

    (2011)
  • A.L. VanBlaricom et al.

    A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS)

    Am J Obstet Gynecol

    (2005)
  • R. Kumar et al.

    Objective measures for longitudinal assessment of robotic surgery training

    J Thorac Cardiovasc Surg

    (2012)
  • F. Alici et al.

    Objective structured assessment of technical skills (OSATS) evaluation of hysteroscopy training: a prospective study

    Eur J Obstet Gynecol Reprod Biol

    (2014)
  • N.Y. Siddiqui et al.

    Objective structured assessment of technical skills for repair of fourth-degree perineal lacerations

    Am J Obstet Gynecol

    (2008)
  • M.D. Putnam et al.

    On orthopedic surgical skill prediction--the limited value of traditional testing

    J Surg Educ

    (2015)
  • S.V. Kotsis et al.

    Application of the "see one, do one, teach one" concept in surgical training

    Plast Reconstr Surg

    (2013)
  • J.L. Cameron

    William Stewart Halsted. Our surgical heritage

    Ann Surg

    (1997)
  • H. Hurreiz

    The evolution of surgical training in the UK

    Adv Med Educ Pract

    (2019)
  • P. Szasz et al.

    Assessing technical competence in surgical trainees: a systematic review

    Ann Surg

    (2015)
  • S. Messick

    Foundations of validity: meaning and consequences in psychological assessment

    Eur J Psychol Assess

    (1994)
  • D. Moher et al.

    The PG preferred reporting Items for systematic reviews and meta-analyses: the PRISMA Statement

    PLoS Med

    (2009)
  • A. Vaidya et al.

    Current status of assessment tools in surgical training. PROSPERO 2018 CRD42018104674

  • R. Satava

    Scientific inquiry: why don't surgeons practice what they preach?

    Surg Endosc

    (2003)
  • Y.A. Noureldin et al.

    Competency-based training and simulation: making a "valid" argument

    J Endourol

    (2018)
  • R.M. Sweet et al.

    A unified approach to validation, reliability, and education study design for surgical technical skills TrainingDesign for surgical technical skills training

    JAMA Surg

    (2010)
  • S.M. Downing

    Validity: on the meaningful interpretation of assessment data

    Med Educ

    (2003)
  • S. Messick

    Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning

    Am Psychol

    (1995)
  • F. Carter et al.

    Consensus guidelines for validation of virtual reality surgical simulators

    Surg Endosc

    (2005)
  • F.M. Sanchez-Margallo et al.

    An analysis of skills acquisition during a training program for experienced laparoscopists in laparoendoscopic single-site surgery

    Surg Innov

    (2014)
  • M.B.C. Balci et al.

    Applicability and effectiveness of virtual reality simulator training in urologic surgery: a double-blind randomised study

    Nobel Med

    (2014)
  • N. Jabbour et al.

    Assessing instrument handling and operative consequences simultaneously: a simple method for creating synced multicamera videos for endosurgical or microsurgical skills assessments

    Simul Healthc

    (2011)
  • G. Aldave et al.

    Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery

    J Neurosurg Pediatr

    (2017)
  • Cited by (0)

    Declaration of interest: None.

    View full text