Original ReportsChanging Objective Structured Clinical Examinations Stations at Lunchtime During All Day Postgraduate Surgery Examinations Improves Examiner Morale and Stress
Introduction
The importance of human error as a major cause of accidents in aviation is well known, with recognition and teaching significantly improving air safety in recent years.1, 2 Several factors, common to both pilots and health care professionals such as team working,3 communication,4, 5 leadership,6 stress, fatigue, and burnout are crucial in minimizing human error.7, 8, 9, 10 Human failure itself can be broadly categorized into the following 4 main levels or domains: organizational influences, unsafe supervision, preconditions to unsafe acts, and the unsafe act itself.11, 12 The Human Factors Analysis and Classification System (HFACS), widely used to assess these domains, has been adapted for use in medicine to include both active failures, namely decisions, attitudes, or actions of individuals, and latent failures resulting from errors occurring in an organization.11, 12 The well-known Swiss cheese model occurs when deficiencies occur across all 4 levels to cause an error or adverse incident.13 Surprisingly, there is little research information available on the extent that human failure could play in an objective structured clinical examination (OSCE) situation.
OSCEs are widely used to minimize variability, thereby providing assessment consistency for candidates. Little is known about the effect of human factors (HF) on examining and marking. OSCE examiners have to concentrate for long periods, and endure much repetition by examining the same station time and again, leading to possible fatigue and boredom. A recent publication demonstrated that the ability to concentrate decreased and fatigue increased over time during an OSCE.14 A study of undergraduate medical OSCE found no evidence that examining duration in a communication OSCE bay influenced the marks awarded,15 whereas another showed that differential rater function over time was a significant factor potentially compromising marking reliability and validity.16
Little has been published on the effect of other factors (broadly categorized under the HFACS domains), in influencing examiner performance during OSCE.
We recently evaluated the possible influence of HF in the high-stakes postgraduate Membership of the Royal College of Surgeons (MRCS) examination, a requirement to enter higher surgical training in the United Kingdom,17 with approximately 2000 candidates sitting this examination every year.
A questionnaire based on the 4 HFACS domains was designed and statistically validated, followed by factor analysis, which revealed 4 main factors or domains accounting for the variance in participants’ scores.17 One of these identified factors termed “stress and pressure” was found to be significantly higher in examiners from a surgical college compared with 2 other colleges in the study. This finding matched the situation known in the colleges at the time of data collection, with that college having far greater candidate numbers than the other 2 studied.17
As a result, changes were made to the OSCE by the Intercollegiate Committee for Basic Surgical Examinations, the regulatory body responsible for the development quality of MRCS in the United Kingdom and Ireland, giving all 4 colleges the option to change examiner stations at lunchtime and providing a longer lunch break (40 min rather the 20 min allowed previously) for rest and familiarization of the new station. The actual break to eat lunch and rest is probably closer to 30 minutes as examiners have to check over their new station, including examining patients and confirming physical signs or histories in the communication scenarios. Therefore, realistically the break has only been usefully increased for examiners by an additional 10 minutes. The Royal College of Surgeons of England adopted this policy in 2015, and the Edinburgh College use it depending on candidate numbers, whereas the other 2 colleges (Glasgow and Ireland) did not feel the need to do so owing to fewer candidate numbers. A breakdown of the MRCS OSCE structure is shown in Table 1. Each station lasts for 9 minutes followed by a 1-minute break to enable candidates to move to the next scenario, and examiners to record their marks.
We recently published the results of more than 18,000 candidate-examiner interactions finding that candidate outcome and overall pass marks were not significantly different in OSCEs, whether the examiner cohort had changed stations at lunchtime or not.18 The examiner:candidate ratio does not change during the MRCS OSCE circuit, as each candidate passes through all of the 18 stations and meets each examiner only once. Examiners will usually examine for a maximum of 2 days at a time at their respective College, after which time a new cohort of examiners take over should that college have more candidates. For example, although the English College sometimes has to run the examination over a 2-week period owing to the large number of candidate applications, this should not adversely affect individual examiner performance.
In this study, we resurveyed examiners to determine if the previously identified stress and pressure issues had improved after the changing station at lunchtime option, and whether any other HF issues were apparent in MRCS examiners.
Section snippets
Materials and Methods
180 Examiners involved in the February 2016 examination were invited at the examiner briefing before the OSCE to complete our previously designed and validated 38-item questionnaire.17 The questions were listed randomly covering the 4 broad HFACS domains associated with human error equally (Appendix 1). As there is no ethical committee for the MRCS, the internal quality assurance and Intercollegiate Committee for Basic Surgical Examinations committees managing the examination approved the
Results
The number and college affiliation of examiners who completed the questionnaires are shown in Table 3 and type of examiner in Table 4. The comparative distributions of z-scores of amalgamated factor scores by college and year are shown in Figure 1. A nonparametric analysis of the published 2014 data in which parametric analysis was used17 came to the same conclusion as the original study that there was no evidence to suggest any difference in scores among the 3 UK colleges in factors 1, 2, and
Discussion
Careful planning, knowledge, and experience are required for the successful organization and running of an OSCE, with collaboration required between the examination center and its examiners.19 In a time, where some are questioning the fairness of the postgraduate examination process,20 potential and preventable causes of human error need to be recognized and understood, ensuring that examinations are run as smoothly and robustly as possible. A key component in this process is to try and ensure
Conclusion
There is relatively little known about the role that HF play in examiners for surgical (and other medical) OSCEs. We have found that allowing examiners to change stations at lunchtime during a long day of examining has no effect on overall candidate outcome, but has improved morale and reduced stress in Royal College of Surgeons Examiners. It has also identified some other areas that might affect other OSCEs, including the importance of being able to challenge more senior examiners if there are
Acknowledgments
We are grateful to the examiners who took part in this study and to the college examination directors, Mr Stephen Hills, Mr Duncan MacArthur, and Ms Valerie Hogg. We are also grateful to Mr Tim Mitchell, Chairman of the English College Court of Examiners for his help.
References (30)
- et al.
Safe surgery, the human factors approach
Surgeon
(2010) - et al.
Communication failure in the operating room
Surgery
(2011) - et al.
The effects of stress on surgical performance
Am J Surg
(2006) - et al.
Combating stress and burnout in surgical practice: a review
Adv Surg
(2010) - et al.
Does changing examiner stations during UK postgraduate surgery objective structured clinical examinations influence examination reliability and candidates’ scores?
J Surg Educ
(2016) - et al.
Five years after To Err Is Human: what have we leaned?
J Am Med Assoc
(2005) - et al.
Use of briefings and debriefings as a tool in improving team work, efficiency, and communication in the operating theatre
Postgrad Med
(2011) - et al.
Evaluation of a pre-operative checklist and team briefing among surgeons, nurses and anaesthesiologists to reduce errors in communication
Arch Surg
(2008) Error reduction through team leadership: what surgeons can learn from the airline industry
Clin Neurosurg
(2007)Professional stress and the anesthesiologist: how evident is it?
J Clin Monit Comput
(2011)
Burnout and medical errors among American surgeons
Ann Surg
Human factors analysis of post accident data: applying theoretical taxonomies of human error
Int J Aviat Psychol
Applying reason: the human factors analysis and classification system
Human Fact Aero Safety
A system approach to organizational error
Ergonomics
Examiner fatigue and ability to concentrate in objective structured clinical examinations for physical therapist students
J Allied Health
Cited by (6)
Do surgeon interviewers have human factor-related issues during the long day UK National Trauma and Orthopaedic specialty recruitment process?
2018, SurgeonCitation Excerpt :We have previously found that task repetition and continually examining the same scenario throughout a day at the high stakes UK Membership of the Royal of College of Surgeons (MRCS) objective structure clinical examination (OSCE) had an adverse effect on examiner morale and increased their stress.14 Following changes in OSCE delivery across the UK, including increasing the duration of regular breaks and allowing examiners to change scenarios at lunchtime, morale was significantly improved.15 Morale itself depends on many factors, both within the workplace and outside it, ideally ensuring a good work-life balance.16
Factors predicting success in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination: a summary for OMFS
2018, British Journal of Oral and Maxillofacial SurgeryHuman factors awareness and recognition during multidisciplinary team meetings
2019, Journal of Oral Pathology and MedicineThe effects of human factor related issues on assessors during the recruitment process for general and vascular surgery in the UK
2019, Annals of the Royal College of Surgeons of EnglandPredictors of success in the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination
2018, Annals of the Royal College of Surgeons of England