Abstract
The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.
Similar content being viewed by others
References
Bandiera, G., & Lendrum, D. (2008). Daily encounter cards facilitate competency-based feedback while leniency bias persists. Canadian Journal of Emergency Medicine, 10(1), 44–50.
Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors’ perspectives. Advances in Health Sciences Education, 18(4), 559–571.
Bredo, E. (1994). Reconstructing educational psychology: Situated cognition and Deweyian pragmatism. Educational Psychology, 29(1), 23–35.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.
Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of Mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74–79.
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: SAGE Publications Inc.
De Vaus, D. (2014). Surveys in social research (6th ed.). Abingdon-on-Thames: Routledge.
Driessen, E., & Scheele, F. (2013). What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research. Medical Teacher, 35(7), 569–574.
Durning, S. J., & Artino, A. R. (2011). Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical Teacher, 33(3), 188–199.
Durning, S. J., Artino, A. R., Boulet, J. R., Dorrance, K., van der Vleuten, C., & Schuwirth, L. (2012). The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in Health Sciences Education, 17(1), 65–79.
Durning, S. J., Artino, A. R., Jr., Pangaro, L. N., van der Vleuten, C., & Schuwirth, L. (2010). Perspective: Redefining context in the clinical encounter: Implications for research and training in medical education. Academic Medicine, 85(5), 894–901.
Durning, S., Artino, A. R., Pangaro, L., van der Vleuten, C., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45(9), 927–938.
Eva, K. W. (2018). Cognitive influences on complex performance assessment: Lessons from the interplay between medicine and psychology. Journal of Applied Research in Memory and Cognition, 7(2), 177–188.
Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522.
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014a). Seeing the “black box” differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.
Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10 Suppl), S1–S7.
Gingerich, A., van der Vleuten, C. P. M., Eva, K. W., & Regehr, G. (2014b). More consensus than idiosyncrasy: Categorizing social judgments to examine variability in Mini-CEX ratings. Academic Medicine, 89(11), 1–10.
Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.
Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.
Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396.
Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239–260.
Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72(9), 861–871.
Hall, K. H. (2002). Reviewing intuitive decision-making and uncertainty: The implications for medical education. Medical Education, 36(3), 216–224.
Hill, Faith, Kathleen, Kendall, Galbraith, Kevin, & Crossley, J. (2009). Implemanting the undergradate Mini-CEX: A tailored approach at Southmpatom University. Medical Education, 43(4), 326–334.
Hoffman, K. G., & Donaldson, J. F. (2004). Contextual tensions of the clinical environment and their influence on teaching and learning. Medical Education, 38(4), 448–454.
Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004). Effects of training in direct observation of medical residents’ clinical competence: A randomized trial. Annals of Internal Medicine, 140(11), 874–881.
Huber, V. L., Neale, M. A., & Nofthcraft, G. B. (1987). Judgment by heuristics: Effects of ratee and rater characteristics and performance standards on performance-related judgments. Organizational Behavior and Human Decision Processes, 40(2), 149–169.
Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Durning, S. J., Hauer, K. E., & Holmboe, E. S. (2012). Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Medical Education, 46(2), 201–215.
Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.
Kogan, J. R., Conforti, L. N., Bernabeo, E., Iobst, W., & Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: A qualitative study. Medical Education, 49(7), 692–708.
Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721–727.
Lavelle, E., Vuk, J., & Barber, C. (2013). Twelve tips for getting started using mixed methods in medical education research. Medical Teacher, 35, 272–276.
Lee, V., Brain, K., & Martin, J. (2017). Factors influencing Mini-CEX rater judgments and their practical implications. Academic Medicine, 92(6), 880–887.
Maudsley, G. (2011). Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review). Medical Teacher, 33(2), e92–e104.
Mertens, F., de Groot, E., Meijer, L., Wens, J., Gemma Cherry, M., Deveugele, M., et al. (2017). Workplace learning through collaboration in primary healthcare: A BEME realist review of what works, for whom and in what circumstances: BEME Guide No. 46. Medical Teacher, 40(2), 117–134.
Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The Mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.
Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The Mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.
Pangaro, L., & ten Cate, O. (2013). Frameworks for learner assessment in medicine: AMEE Guide No. 78. Medical Teacher, 35(6), e1197–e1210.
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., & van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604–612.
Ramani, S., Könings, K. D., Ginsburg, S., & van der Vleuten, C. P. M. (2018). Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships. Medical Teacher, 7, 1–7. https://doi.org/10.1080/0142159X.2018.1432850.
Rogausch, A., Beyeler, C., Montagne, S., Jucker-Kupper, P., Berendonk, C., Huwendiek, S., et al. (2015). The influence of students’ prior clinical skills and context characteristics on Mini-CEX scores in clerkships—a multilevel analysis. BMC Medical Education, 15(1), 208.
Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: How trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education, 22(4), 931–949.
Schifferdecker, K. E., & Reed, V. A. (2009). Using mixed methods research in medical education: Basic guidelines for researchers. Medical Education, 43(7), 637–644.
Schuh, L. A., London, Z., Neel, R., Brock, C., Kissela, B. M., Schultz, L., et al. (2009). Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology, 73(11), 904–908.
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011a). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485.
Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011b). General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher, 33(10), 783–797.
Sheehan, D., Jowsey, T., Parwaiz, M., Birch, M., Seaton, P., Shaw, S., et al. (2017). Clinical learning environments: Place, artefacts and rhythm. Medical Education, 51(10), 1049–1060.
Stalmeijer, R. E., Mcnaughton, N., & Van Mook, W. N. K. A. (2014). Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher, 36(11), 923–939.
Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education, 18(2), 291–303.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.
Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292.
Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37.
Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education, 18(3), 523–525.
Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.
Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: Contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909–919.
Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013a). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325–341.
Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013b). “You”re certainly relatively competent: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.
Acknowledgements
The authors would like to thank Marjan Govaerts and Andrea Gingerich for their time and expertise in providing feedback regarding our survey and focus group questions. We would also like to thank staff at Monash University Eastern Health Clinical School for their assistance, the Australasian College for Emergency Medicine for permitting us to conduct this research and the Fellows of the College who participated.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
K Brain and J. Martin declare they have no conflicts of interest. V. Lee was chair of the Workplace-based Assessment working group during the period of this research, but this was a voluntary and unpaid position for the Australasian College for Emergency Medicine. This research was conducted independently without funding or coercion as part of his Master of Health Professional Education thesis at Monash University.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Lee, V., Brain, K. & Martin, J. From opening the ‘black box’ to looking behind the curtain: cognition and context in assessor-based judgements. Adv in Health Sci Educ 24, 85–102 (2019). https://doi.org/10.1007/s10459-018-9851-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-018-9851-0