Original article
Adult cardiac
The Joint Council on Thoracic Surgery Education Coronary Artery Assessment Tool Has High Interrater Reliability

Presented at the Forty-ninth Annual Meeting of The Society of Thoracic Surgeons, Los Angeles, CA, Jan 26–30, 2013.
https://doi.org/10.1016/j.athoracsur.2012.10.090Get rights and content

Background

Barriers to incorporation of simulation in cardiothoracic surgery training include lack of standardized, validated objective assessment tools. Our aim was to measure interrater reliability and internal consistency reliability of a coronary anastomosis assessment tool created by the Joint Council on Thoracic Surgery Education.

Methods

Ten attending surgeons from different cardiothoracic residency programs evaluated nine video recordings of 5 individuals (1 medical student, 1 resident, 1 fellow, 2 attendings) performing coronary anastomoses on two simulation models, including synthetic graft task station (low fidelity) and porcine explant (high fidelity), as well as in the operative setting. All raters, blinded to operator identity, scored 13 assessment items on a 1 to 5 (low to high) scale. Each performance also received an overall pass/fail determination. Interrater reliability and internal consistency were assessed as intraclass correlation coefficients and Cronbach’s α, respectively.

Results

Both interrater reliability and internal consistency were high for all three models (intraclass correlation coefficients = 0.98, 0.99, and 0.94, and Cronbach’s α = 0.99, 0.98, and 0.97 for low fidelity, high fidelity, and operative setting, respectively). Interrater reliability for overall pass/fail determination using κ were 0.54, 0.86, 0.15 for low fidelity, high fidelity, and operative setting, respectively.

Conclusions

Even without instruction on the assessment tool, experienced surgeons achieved high interrater reliability. Future resident training and evaluation may benefit from utilization of this tool for formative feedback in the simulated and operative environments. However, summative assessment in the operative setting will require further standardization and anchoring.

Section snippets

Subjects

The research protocol was approved by Northwestern University's Institutional Review Board (No. STU00057217), and all subjects provided written consent for participation.

Results

Table 1 presents reliability data, including interrater reliability among raters for composite scores, individual assessment item scores, overall P/F scores, and internal consistency reliability among the 13 assessment items.

Comment

The recent rise of integrated 6-year CT training programs requires trainees to achieve competency in surgical tasks in less time. Thus, the incorporation of simulation into current training paradigms will become even more critical for patient safety and quality of care 2, 3, 14, 15. Given the variability associated with existing simulators and the financial constraints of residency programs, there is an opportunity to develop validated, cost-effective simulators and to create a national

References (22)

Cited by (15)

  • Commentary: The time has come to measure and examine technical skills

    2020, Journal of Thoracic and Cardiovascular Surgery
  • Integration of simulation components enhances team training in cardiac surgery

    2018, Journal of Thoracic and Cardiovascular Surgery
  • Development and Evaluation of a Three-Dimensional Multistation Cardiovascular Simulator

    2016, Annals of Thoracic Surgery
    Citation Excerpt :

    This concept of deliberate home practice is particularly relevant today, in light of trainees’ limited opportunity for repetitive learning because of work-hour restrictions and increasing operation complexity. Still, few fellowships have a formal simulation curriculum or mandate [11, 12] Our current 3D multistation cardiovascular simulator addresses some of the barriers to deliberate home practice.

View all citing articles on Scopus
View full text