In Reply to Wayne et al.

We thank Wayne et al. for their thorough review of our article. We believe they raise some excellent points that are important to consider for any program attempting both educational outcomes research and procedural training. We offer the following discussion in response.

We agree that skills must initially be acquired to a high standard, and this has implications for skill decay. Most residency programs do not have the resources (e.g., finances, equipment, time, faculty)¹ to use simulation to train residents to mastery in all procedures. We were not aiming to meet a standard of simulation training that is not feasible or sustainable for most programs. In this study, we aimed to create what would be a reasonable evidence-based simulation curriculum^{2, 3} in a large internal medicine residency with average resources to replace the traditional “see one, do one, teach one” paradigm and follow decay thereafter. Less important was the change in skill level that occurred as a result of the simulation, but rather to examine, using a novel assessment tool², what happens to skill over time in novice learners.

Although the cut point checklist score, particularly for the minimum passing standard (MPS), was less than prior studies of mastery learning,^4,5,6 both the MPS and the unsupervised practice standard (UPS) were established using acceptable standard-setting procedures using experts from multiple institutions.^{6, 7} In addition, achievement of the MPS and UPS required a composite score that was not just checklist-based but also incorporated global skill level and entrustment measures, since both types of measures alone have their challenges in assessing competence³ . We would also assert that although the magnitude of the individual decline in each PCAT component was relatively small, the composite scores suggested significant reduction in the number of learners meeting either the MPS or UPS over time.⁸ Using this type of criterion-referencing⁷ is clinically relevant for it predicts whether a trainee is safe to perform a procedure unsupervised, or they are not. This has implications when we think about extrapolation to the live patient environment and consider supervision requirements when residents perform procedures.

We agree that rater training and quality assurance of the ratings could have been more robustly assessed and this is a limitation of the study. We do feel that the training raters in the study received is on par (or less) with what raters would receive before using the Paracentesis Competency Assessment Tool (PCAT) to assess residents on live patient procedures.

Wayne et al. and others have done tremendous work in using simulation as a tool to improve procedural confidence and competence, and reduce cost and complications.^{4, 9,10,11} We do not view our findings as detracting from this important body of work. In fact, we show that re-training at 3 months may be protective in interrupting skill decay.⁸ Procedural education and determination of competence are a challenge, and we have recently asked the question: “What is the safest way for a patient to undergo an invasive bedside procedure in a teaching hospital?”¹² We believe simulation, repeated opportunities for training, and valid assessment decisions are a part of the solution, within a larger systems-based approach.¹² We thank Wayne el al. for their comments and contribution to this crucial conversation surrounding procedural safety.