Education and Career DevelopmentCurrent Status of Technical Skills Assessment Tools in Surgery: A Systematic Review
Introduction
Surgical training has classically revolved around the Halstedian1 training model of “see one, do one, teach one.” This technique of assessment of technical skills relies on subjective feedback from supervisors in order to improve.2 While once perceived as an effective method, the changes to working times, the desire for more objective feedback for trainees, and the ethical dilemmas of operating before being deemed competent has brought about a change to training practices.3 Furthermore, surgical competency is now considered to be an assimilation of both technical and nontechnical skills.
Today, technological advances provide the ability to simulate and hone technical skills outside of the operating room (OR). There has been a move away from curriculums based on cumulative OR experience toward those based on assessment of competency.3 The cornerstone of these programs is the requirement to evaluate technical proficiency using objective assessment tools.4 There is a plethora of tools currently available that are either task/procedure specific or generic.
For tools to be used in assessment, they must be validated. Consequently, this systematic review aims to (a) identify the currently utilized technical assessment tools across surgical specialties; (b) evaluate for validity evidence, as per the Messick framework5; and (c) assess the levels of recommendation of each assessment tool.
Section snippets
Material and methods
This was a priori protocol (International Prospective Register of Systematic Reviews registration number: CRD42018104674) systematic review, conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines.6 Registration of the trial with International Prospective Register of Systematic Reviews, a database of ongoing systematic reviews with health-related outcomes, ensured there was no duplication of work and provided a platform for comparison
Results
The literature search identified 2207 potentially relevant titles. A further 24 titles were added through bibliography searching. After removing duplicates and non-English studies, 1268 studies remained. The full texts of 692 studies were assessed—303 met the inclusion criteria (Figure). Seventy-six tools were used; the most popular tool of which was the Objective Structured Assessment Tool Skills (OSATS). Surgical specialties using tools included breast (n = 2), cardiothoracic (n = 6), ENT (n
Discussion
This review found that a large number of tools are currently being used to assess surgical competency across specialties in different settings. The most commonly used tool was found to be OSATS, which has been validated across all major surgical subspecialties, in bench models, animal models, VR simulators, and in the OR. It was used in 191 studies (139 alone and 55 in conjunction with other tools), which make up the majority of studies examined in this review. OSATS is commonly used as a
Conclusion
There is a need in surgery to be able to reliably assess the technical skills of trainees, both for education and examination. This review found that numerous tools have been validated for this purpose. OSATS had the highest LoR across surgical specialties in both simulation and OR settings. Other validated tools include GOALS, GRS, ROSATS, and GEARS. The advent of crowdsourcing has shown potential as an alternative to traditional assessment by specialists, which promises quicker, cheaper,
Acknowledgment
Authors' contributions: A.V., J.R., A.A., and N.R. contributed to data extraction, manuscript drafting, critical analysis, and revisions. A.A., P.D., and K.A. contributed to idea conception, manuscript drafting, critical analysis, and revisions.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
References (313)
- et al.
A 4-task skills examination for residents for the assessment of technical ability in hand trauma surgery
J Surg Educ
(2015) - et al.
Application of high fidelity simulation in teaching mitral valve annuloplasty to cardiac surgery residents
Can J Cardiol
(2015) - et al.
Assessment of resident surgical skills: is testing feasible?
Am J Obstet Gynecol
(2005) - et al.
Assessment of surgery residents' operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study
Surgery
(2014) - et al.
Assessment of technical skills transfer from the bench training model to the human model
Am J Surg
(1999) - et al.
Ceiling effect in technical skills of surgical residents
Am J Surg
(2004) - et al.
Construct validation of a cost-effective vessel ligation benchtop simulator
J Surg Educ
(2015) - et al.
Construct validity of a simple laparoscopic ovarian cystectomy model using a validated objective structured assessment of technical skills
J Minim Invasive Gynecol
(2017) - et al.
A curriculum for Burch colposuspension and diagnostic cystoscopy evaluated by an objective structured assessment of technical skills
Am J Obstet Gynecol
(2007) - et al.
Developing an objective structured assessment of technical skills for laparoscopic suturing and intracorporeal knot tying
J Surg Educ
(2016)
Does aptitude influence the rate at which proficiency is achieved for laparoscopic appendectomy?
J Am Coll Surg
Does laparoscopic simulation predict intraoperative performance? A comparison between the Fundamentals of Laparoscopic Surgery and LapVR evaluation metrics
Am J Surg
Does surgeon leadership during robot-assisted surgery correlate with teamwork-related outcomes?
J Urol
Effect of preoperative warm-up exercise before laparoscopic gynecological surgery: a randomized trial
J Surg Educ
The effect of resident simulation training on operative time of the retropubic midurethral sling procedure for the treatment of stress urinary incontinence
Am J Obstet Gynecol
The effectiveness of a simulation-based flipped classroom in the acquisition of laparoscopic suturing skills in medical students-A pilot study
J Surg Educ
Establishing passing scores for technical performance in surgery: lessons learned from a 10-year experience of resident assessment
J Am Coll Surg
Evaluating the effectiveness of a 2-year curriculum in a surgical skills center
Am J Surg
An evaluation of the utility of robotic virtual reality simulation in gynecologic resident surgical education
J Minim Invasive Gynecol
Examination stress leads to improvements on fundamental technical skills for surgery
Am J Surg
Exporting a technical skills evaluation technology to other sites
Am J Surg
The fann objective structured assessment of technical skills score in cardiothoracic set trainees undertaking coronary anastomosis
Heart Lung Circ
Focused assessment of surgical performance: difficulty with faculty compliance
Am J Obstet Gynecol
Halsted's "See one, do one, and teach one" versus peyton's four-step approach: a randomized trial for training of laparoscopic suturing and knot tying
J Surg Educ
Implementation and evaluation of a dilation and evacuation simulation training curriculum
Contraception
Institution and validation of an observed structured assessment of technical skills (OSATS) for obstetrics and gynecology residents and faculty
Am J Obstet Gynecol
Medical student deliberate practice on a virtualreality curriculum can achieve equivalency to senior urology resident real-time training
J Urol
Mental practice enhances technical skills and teamwork in crisis simulations-a double blind, randomised controlled study
J Am Coll Surg
A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS)
Am J Obstet Gynecol
Objective measures for longitudinal assessment of robotic surgery training
J Thorac Cardiovasc Surg
Objective structured assessment of technical skills (OSATS) evaluation of hysteroscopy training: a prospective study
Eur J Obstet Gynecol Reprod Biol
Objective structured assessment of technical skills for repair of fourth-degree perineal lacerations
Am J Obstet Gynecol
On orthopedic surgical skill prediction--the limited value of traditional testing
J Surg Educ
Application of the "see one, do one, teach one" concept in surgical training
Plast Reconstr Surg
William Stewart Halsted. Our surgical heritage
Ann Surg
The evolution of surgical training in the UK
Adv Med Educ Pract
Assessing technical competence in surgical trainees: a systematic review
Ann Surg
Foundations of validity: meaning and consequences in psychological assessment
Eur J Psychol Assess
The PG preferred reporting Items for systematic reviews and meta-analyses: the PRISMA Statement
PLoS Med
Current status of assessment tools in surgical training. PROSPERO 2018 CRD42018104674
Scientific inquiry: why don't surgeons practice what they preach?
Surg Endosc
Competency-based training and simulation: making a "valid" argument
J Endourol
A unified approach to validation, reliability, and education study design for surgical technical skills TrainingDesign for surgical technical skills training
JAMA Surg
Validity: on the meaningful interpretation of assessment data
Med Educ
Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning
Am Psychol
Consensus guidelines for validation of virtual reality surgical simulators
Surg Endosc
An analysis of skills acquisition during a training program for experienced laparoscopists in laparoendoscopic single-site surgery
Surg Innov
Applicability and effectiveness of virtual reality simulator training in urologic surgery: a double-blind randomised study
Nobel Med
Assessing instrument handling and operative consequences simultaneously: a simple method for creating synced multicamera videos for endosurgical or microsurgical skills assessments
Simul Healthc
Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery
J Neurosurg Pediatr
Cited by (0)
Declaration of interest: None.