Elsevier

Journal of Surgical Education

Volume 74, Issue 4, July–August 2017, Pages 736-747
Journal of Surgical Education

Original Reports
Changing Objective Structured Clinical Examinations Stations at Lunchtime During All Day Postgraduate Surgery Examinations Improves Examiner Morale and Stress

https://doi.org/10.1016/j.jsurg.2016.12.013Get rights and content

Background

Human factors are important causes of error, but little is known about their possible effect during objective structured clinical examinations (OSCE). We have previously identified stress and pressure in OSCE examiners in the postgraduate intercollegiate Membership of the Royal College of Surgeons (MRCS) examination. After modifying examination delivery by changing OSCE stations at lunchtime with no demonstrable effect on candidate outcome, we resurveyed examiners to ascertain whether examiner experience was improved.

Method

Examiners (n = 180) from all 4 surgical colleges in the United Kingdom and Ireland were invited to complete the previously validated human factors questionnaire used in 2014. Aggregated scores for each of 4 previously identified factors were compared with the previous data. Unit-weighted z-scores and nonparametric Kruskal-Wallis methods were used to test the hypothesis that there was no difference among the median factor z-scores for each college. Individual Mann-Whitney-Wilcoxon tests (with appropriate Bonferonni corrections) were used to determine any differences between factors and the respective colleges.

Results

141 Completed questionnaires were evaluated (78% response rate) and compared with 108 responses (90%) from the original study. Analysis was based on 26 items common to both studies. In 2014, the college with the highest candidate numbers (England) was significantly different in 1 factor (stress and pressure), compared with Edinburgh (Mann-Whitney-Wilcoxon: W = 1524, p < 0.001) and Glasgow colleges (Mann-Whitney-Wilcoxon: W = 104, p = 0.004). No differences were found among colleges in the same factor in 2016, Kruskall-Wallis: (χ2 (3) = 1.73, p = 0.63). Analysis of responses found inconsistency among examiners regarding mistakes or omissions made when candidates were performing well.

Conclusion

After making changes to OSCE delivery, factor scores relating to examiner stress and pressure are now improved and consistent across the surgical colleges. Stress and pressure can occur in OSCE examiners and examination delivery should ideally minimize these issues, thereby improving morale is also likely to benefit candidates.

Introduction

The importance of human error as a major cause of accidents in aviation is well known, with recognition and teaching significantly improving air safety in recent years.1, 2 Several factors, common to both pilots and health care professionals such as team working,3 communication,4, 5 leadership,6 stress, fatigue, and burnout are crucial in minimizing human error.7, 8, 9, 10 Human failure itself can be broadly categorized into the following 4 main levels or domains: organizational influences, unsafe supervision, preconditions to unsafe acts, and the unsafe act itself.11, 12 The Human Factors Analysis and Classification System (HFACS), widely used to assess these domains, has been adapted for use in medicine to include both active failures, namely decisions, attitudes, or actions of individuals, and latent failures resulting from errors occurring in an organization.11, 12 The well-known Swiss cheese model occurs when deficiencies occur across all 4 levels to cause an error or adverse incident.13 Surprisingly, there is little research information available on the extent that human failure could play in an objective structured clinical examination (OSCE) situation.

OSCEs are widely used to minimize variability, thereby providing assessment consistency for candidates. Little is known about the effect of human factors (HF) on examining and marking. OSCE examiners have to concentrate for long periods, and endure much repetition by examining the same station time and again, leading to possible fatigue and boredom. A recent publication demonstrated that the ability to concentrate decreased and fatigue increased over time during an OSCE.14 A study of undergraduate medical OSCE found no evidence that examining duration in a communication OSCE bay influenced the marks awarded,15 whereas another showed that differential rater function over time was a significant factor potentially compromising marking reliability and validity.16

Little has been published on the effect of other factors (broadly categorized under the HFACS domains), in influencing examiner performance during OSCE.

We recently evaluated the possible influence of HF in the high-stakes postgraduate Membership of the Royal College of Surgeons (MRCS) examination, a requirement to enter higher surgical training in the United Kingdom,17 with approximately 2000 candidates sitting this examination every year.

A questionnaire based on the 4 HFACS domains was designed and statistically validated, followed by factor analysis, which revealed 4 main factors or domains accounting for the variance in participants’ scores.17 One of these identified factors termed “stress and pressure” was found to be significantly higher in examiners from a surgical college compared with 2 other colleges in the study. This finding matched the situation known in the colleges at the time of data collection, with that college having far greater candidate numbers than the other 2 studied.17

As a result, changes were made to the OSCE by the Intercollegiate Committee for Basic Surgical Examinations, the regulatory body responsible for the development quality of MRCS in the United Kingdom and Ireland, giving all 4 colleges the option to change examiner stations at lunchtime and providing a longer lunch break (40 min rather the 20 min allowed previously) for rest and familiarization of the new station. The actual break to eat lunch and rest is probably closer to 30 minutes as examiners have to check over their new station, including examining patients and confirming physical signs or histories in the communication scenarios. Therefore, realistically the break has only been usefully increased for examiners by an additional 10 minutes. The Royal College of Surgeons of England adopted this policy in 2015, and the Edinburgh College use it depending on candidate numbers, whereas the other 2 colleges (Glasgow and Ireland) did not feel the need to do so owing to fewer candidate numbers. A breakdown of the MRCS OSCE structure is shown in Table 1. Each station lasts for 9 minutes followed by a 1-minute break to enable candidates to move to the next scenario, and examiners to record their marks.

We recently published the results of more than 18,000 candidate-examiner interactions finding that candidate outcome and overall pass marks were not significantly different in OSCEs, whether the examiner cohort had changed stations at lunchtime or not.18 The examiner:candidate ratio does not change during the MRCS OSCE circuit, as each candidate passes through all of the 18 stations and meets each examiner only once. Examiners will usually examine for a maximum of 2 days at a time at their respective College, after which time a new cohort of examiners take over should that college have more candidates. For example, although the English College sometimes has to run the examination over a 2-week period owing to the large number of candidate applications, this should not adversely affect individual examiner performance.

In this study, we resurveyed examiners to determine if the previously identified stress and pressure issues had improved after the changing station at lunchtime option, and whether any other HF issues were apparent in MRCS examiners.

Section snippets

Materials and Methods

180 Examiners involved in the February 2016 examination were invited at the examiner briefing before the OSCE to complete our previously designed and validated 38-item questionnaire.17 The questions were listed randomly covering the 4 broad HFACS domains associated with human error equally (Appendix 1). As there is no ethical committee for the MRCS, the internal quality assurance and Intercollegiate Committee for Basic Surgical Examinations committees managing the examination approved the

Results

The number and college affiliation of examiners who completed the questionnaires are shown in Table 3 and type of examiner in Table 4. The comparative distributions of z-scores of amalgamated factor scores by college and year are shown in Figure 1. A nonparametric analysis of the published 2014 data in which parametric analysis was used17 came to the same conclusion as the original study that there was no evidence to suggest any difference in scores among the 3 UK colleges in factors 1, 2, and

Discussion

Careful planning, knowledge, and experience are required for the successful organization and running of an OSCE, with collaboration required between the examination center and its examiners.19 In a time, where some are questioning the fairness of the postgraduate examination process,20 potential and preventable causes of human error need to be recognized and understood, ensuring that examinations are run as smoothly and robustly as possible. A key component in this process is to try and ensure

Conclusion

There is relatively little known about the role that HF play in examiners for surgical (and other medical) OSCEs. We have found that allowing examiners to change stations at lunchtime during a long day of examining has no effect on overall candidate outcome, but has improved morale and reduced stress in Royal College of Surgeons Examiners. It has also identified some other areas that might affect other OSCEs, including the importance of being able to challenge more senior examiners if there are

Acknowledgments

We are grateful to the examiners who took part in this study and to the college examination directors, Mr Stephen Hills, Mr Duncan MacArthur, and Ms Valerie Hogg. We are also grateful to Mr Tim Mitchell, Chairman of the English College Court of Examiners for his help.

References (30)

  • T.D. Shanafelt et al.

    Burnout and medical errors among American surgeons

    Ann Surg

    (2010)
  • D.A. Wiegmann et al.

    Human factors analysis of post accident data: applying theoretical taxonomies of human error

    Int J Aviat Psychol

    (1997)
  • S.A. Shappell et al.

    Applying reason: the human factors analysis and classification system

    Human Fact Aero Safety

    (2001)
  • J. Reason

    A system approach to organizational error

    Ergonomics

    (2005)
  • M. Swift et al.

    Examiner fatigue and ability to concentrate in objective structured clinical examinations for physical therapist students

    J Allied Health

    (2016)
  • Cited by (6)

    • Do surgeon interviewers have human factor-related issues during the long day UK National Trauma and Orthopaedic specialty recruitment process?

      2018, Surgeon
      Citation Excerpt :

      We have previously found that task repetition and continually examining the same scenario throughout a day at the high stakes UK Membership of the Royal of College of Surgeons (MRCS) objective structure clinical examination (OSCE) had an adverse effect on examiner morale and increased their stress.14 Following changes in OSCE delivery across the UK, including increasing the duration of regular breaks and allowing examiners to change scenarios at lunchtime, morale was significantly improved.15 Morale itself depends on many factors, both within the workplace and outside it, ideally ensuring a good work-life balance.16

    View full text