Psychiatry Clinical Skills Evaluation: Interrater Reliability of the American Board of Psychiatry and Neurology Required Assessment

Abstract

Objective

The American Board of Psychiatry and Neurology (ABPN) requires applicants to successfully complete at least 3 Clinical Skills Evaluations (CSE) as part of the credentialing requirements to sit for the psychiatry certification examination. The authors examined a database of CSE assessments conducted at a single site to determine the interrater and test-retest reliability of the evaluation.

Methods

The authors examined 159 CSE assessments of 51 practicing psychiatrists who completed residency prior to the implementation of CSEs, but did not previously sit for the ABPN examination and later required CSEs to do so. Of these, 36 were simultaneously observed by 2 evaluators, at least 1 of whom had extensive experience with the CSE and 8 of whom were novice evaluators. At least 3 CSEs were done on the same day by each candidate.

Results

The intraclass correlations for CSEs observed by 2 evaluators for the 3 clinical skills assessed (patient–physician relationship, conduct of the interview, and case presentation), were in the low-moderate “good” range for interrater reliability. Test-retest reliability for all candidates was in the moderate–high “good” range. Significant learning took place between the 1st and 3rd CSEs, with a mean improvement of 0.42–0.49 points on an 8-point scale (p < 0.01).

Conclusions

These data demonstrate good interrater reliability for the CSE even when conducted by novice faculty evaluators and good test-retest reliability despite significant learning that occurred over the series of assessments. The study supports both the educational and assessment value of the CSE.