Abstract
Medical examining bodies now commonly assess candidates' communication skills. However, there are a number of within-case sources of error that can mean examinations have poor reliability and validity. The aims of this study were to determine the main within-case sources of error and to identify the best methods of maximising reliability in highly structured communication skills assessments. Subjects were fifth year medical students at the University of Aberdeen. Subjects were videotaped giving clinical information to standardised patients and relatives toward the end of their fifth year and toward the end of their pre-registration year. Sources of variation, reliability coefficients and the most effective methods of reducing measurement error were calculated using general is ability theory. Systematic differences among subjects' communication skill was the main source of variation in two of the three assessment cases. The implications of the results are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Boon, H. & Stewart, M. (1998). Patient-physician communication assessment instruments: 1986–1996 in review. Patient Education and Counselling 35: 161–176.
Brown, S., Race, P. & Smith, B. (1996). As Assessment Manifesto: 500 Tips on Assessment. London: Kogan Press.
Colliver, J.A., Morrison, L.J. et al. (1990). Three studies of the effect of multiple standardised patients on intercase reliability of five standardised-patient examinations. Teaching and Learning in Medicine 2: 237–245.
Colliver, J.A., Vu, N.V. et al. (1991). Reliability and efficiency of components of clinical competence assessed with five performance-based examinations using standardized patients. Medical Education 25: 303–310.
Fowell, S.L., Southgate, L.J. & Bligh, J.G. (1999). Evaluating assessment: the missing link? Medical Education 33: 276–281.
General Medical Council (1997). The New Doctor. London: General Medical Council.
Gilhooly, K.J. (1996). Thinking: Directed, Undirected and Creative. London: Academic Press.
Hargie, O., Dickson, D. et al. (1998). A survey of communication skills training in UK schools of medicine: present practices and prospective proposals. Medical Education 32: 25–34.
Humphris, G.M. & Kane, S. (2001). Examiner fatigue in communication skills objective structured clinical examinations. Medical Education 35: 444–449.
Huysamen, G.K. (1990). The application of generalizability theory to the reliability of ratings. South African Journal of Psychology 20: 200–205.
Norcini, J. (2000). What should we do about unreliable scores? Medical Education 34: 501–502.
Norman, G.R., Tugwell, P. et al. (1985). Knowledge and clinical problem solving. Medical Education 19: 344–356.
Quine, L. & Rutter, D.R. (1994). First diagnosis of severe mental and physical disability: a study of doctor-patient communication. Journal of Child Psychology and Psychiatry and AlliedDisciplines 35: 1273–1287.
Roberts, J. & Norman, G. (1990). Reliability and learning from the OSCE. In W. Bender, R.J.Hiemstra et al. (eds.), Teaching and Assessing Clinical Competence, pp. 184–189. Groningen: Boewerk Publications.
Shavelson, R.J. & Webb, N.M. (1991). Generalizability Theory: A Primer. London: Sage Publications.
Silverman, J., Kurtz, S. & Draper, J. (1998). Skills for Communicating with Patients. Abingdon: Radcliffe Medical Press Ltd.
Stewart, M. (1995). Effective physician-patient communication and health outcomes: a review. Canadian Medical Association Journal 152: 1423–1433.
Streiner, D.L. & Norman, G.R. (1989). Health Measurement Scales: A Practical Guide to their Development and Use. Oxford: Oxford University Press.
Tweed,M. & Miola, J. (2001). Legal vulnerability of assessment tools. Medical Teacher 23: 312–314.
van der Vleuten, C.P.M. & Swanson, D.B. (1990). Assessment of clinical skills with standardized patients: state of the art. Teaching and Learning in Medicine 2: 58–76.
van der Vleuten, C.P.M., Dolmans, D.H.J.M. & Scherpbier, A.J.J.A. (2000). The need for evidence in education. Medical Teacher 22: 246–250.
Vu, N.V. & Barrows, H.S. (1994). Use of standardized patients in clinical assessments: recent developments and measurement findings. Educational Researcher 23: 23–30
Wakeford, R. (1983). Communication skills training in United Kingdom medical schools. In D. Pendleton & J. Hasler (eds.), Doctor-Patient Communication, pp. 233–247. London: Academic Press.
Wass, V., Jones, R. & van der Vleuten, C. (2001). Standardized or real patients to test clinical competence? The long case revisited. Medical Education 35: 321–325.
Wass, V., McGibbon, D. & van der Vleuten, C. (2001). Composite undergraduate clinical examinations: How should the components be combined to maximise reliability? Medical Education 35:326–330.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Keen, A., Klein, S. & Alexander, D. Assessing the Communication Skills of Doctors in Training: Reliability and Sources of Error. Adv Health Sci Educ Theory Pract 8, 5–16 (2003). https://doi.org/10.1023/A:1022609920310
Issue Date:
DOI: https://doi.org/10.1023/A:1022609920310