Evaluating the effectiveness of rating instruments for a communication skills assessment of medical residents

Abstract

The investigators used evidence based on response processes to evaluate and improve the validity of scores on the Patient-Centered Communication and Interpersonal Skills (CIS) Scale for the assessment of residents’ communication competence. The investigators retrospectively analyzed the communication skills ratings of 68 residents at the University of Illinois at Chicago (UIC). Each resident encountered six standardized patients (SPs) portraying six cases. SPs rated the performance of each resident using the CIS Scale—an 18-item rating instrument asking for level of agreement on a 5-category scale. A many-faceted Rasch measurement model was used to determine how effectively each item and scale on the rating instrument performed. The analyses revealed that items were too easy for the residents. The SPs underutilized the lowest rating category, making the scale function as a 4-category rating scale. Some SPs were inconsistent when assigning ratings in the middle categories. The investigators modified the rating instrument based on the findings, creating the Revised UIC Communication and Interpersonal Skills (RUCIS) Scale—a 13-item rating instrument that employs a 4-category behaviorally anchored rating scale for each item. The investigators implemented the RUCIS Scale in a subsequent communication skills OSCE for 85 residents. The analyses revealed that the RUCIS Scale functioned more effectively than the CIS Scale in several respects (e.g., a more uniform distribution of ratings across categories, and better fit of the items to the measurement model). However, SPs still rarely assigned ratings in the lowest rating category of each scale.

Appendices

Appendix A

Items on the Patient-Centered Communication and Interpersonal Skills (CIS) scale

• 1. I felt you greeted me warmly upon entering the room.

• 2. I felt you were friendly throughout the encounter. You were never crabby or rude to me.

• 3. I felt that you treated me like we were on the same level. You never “talked down” to me or treated me like a child.

• 4. I felt you let me tell my story and were careful to not interrupt me while I was speaking.

• 5. I felt you were telling me everything; being truthful, up front and frank; not keeping things from me.

• 6. I felt you showed interest in me as a “person.” You never acted bored or ignored what I had to say.

• 7. I felt that you discussed options with me.

• 8. I felt you made sure that I understood those options.

• 9. I felt you asked my opinion, allowing me to make my own decision.

• 10. I felt you encouraged me to ask questions.

• 11. I felt you displayed patience when I asked questions.

• 12. I felt you answered my questions, never avoiding them.

• 13. I felt you clearly explained what I needed to know about my problem; how and why it occurred.

• 14. I felt you clearly explained what I should expect next.

• 15. I felt you were careful to use plain language and not medical jargon when speaking to me.

• 16. I felt you approached sensitive/difficult subject matters, such as religion, sexual history, tobacco/drug/alcohol history, sexual orientation, giving bad news, etc., with sensitivity and without being judgmental.

• 17. I felt the resident displayed a positive attitude during the verbal feedback session.

• 18. If given the choice in the future, I would choose this resident as my personal physician.

Note: All items are rated on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).

Appendix B

Revised UIC Communication and Interpersonal Skills scale

Instruction

Please choose the option that best describes how you feel toward the resident’s communication skills. Some items also have a “not applicable” option. Select this option when the context of the case does not allow you to observe that aspect of the resident’s performance.

• (1) Friendly communication

  • You did not greet me, or greeted me perfunctorily, or communicated with me rudely during the encounter.

  • Your greeting and/or behavior during the encounter was generally polite but impersonal or distant.

  • You greeted me warmly and communicated with me in a friendly, personal manner throughout the encounter.

  • Your greeting and overall communication were friendly and compassionate. Your tone of voice was appropriate for the situation. Overall, you created an exceptionally warm and friendly environment that made me feel comfortable to tell you all of my problems.

• (2) Respectful treatment

  • You showed an obvious sign of disrespect during the encounter. You treated me as an inferior.

  • You did not show disrespect to me. However, I observed some signs of condescending behavior. Although I believe it was unintentional, it made me feel that I was not at the same level with you.

  • You gave several indications of respecting me. If there was a physical exam, this includes draping me appropriately.

  • You were exceptionally respectful throughout the encounter. Your verbal and non-verbal communication showed respect for my privacy, my opinions, my rights, and my socioeconomic status.

• (3) Listening to my story

  • You rarely gave me any opportunity to tell my story or frequently interrupted me while I was talking, not allowing me to finish what I said. Sometimes I felt you were not paying attention (for example, you asked for information that I already provided).

  • You let me tell my story without interruption, or only interrupted appropriately and respectfully. You seemed to pay attention to my story and responded to what I said appropriately.

  • You allowed me to tell my story without interruption, responded appropriately to what I said, and asked thoughtful questions to encourage me to tell more of my story.

  • You were an exceptional listener. You encouraged me to tell my story and checked your understanding by restating important points.

• (4) Honest communication

  • You did not seem truthful and frank. I felt that there might be something that you were trying to hide from me.

  • You did not seem to hide any critical information from me.

  • You explained the facts of the situation without trivializing negative information or possibilities (e.g., side effects, complications, failure rates).

  • You were exceptionally frank and honest. You fully explained the positive and negative aspects of my condition. You openly acknowledged your own lack of knowledge or uncertainty, and things you would have to consult with others. When appropriate, you also suggested I seek a second opinion.

  • Not applicable. There was no information for the resident to provide.

• (5) Interest in me as a person

  • You never showed interest in me as a person. You only focused on the disease or medical issue.

  • In addition to talking about my medical issue, you spent some time getting to know me as a person.

  • You spent some time exploring how my medical issue affects my personal or social life.

  • You were exceptionally interested in me as a person. You not only explored how my medical problem affects my personal and social life, but also showed your willingness to help me address those challenges.

• (6) Discussion of options/plans

  • You did not explain any options or plans; you just told me what you would do without asking for my opinion.

  • You explained options to me, but did not involve me in decision making. If you solicited my opinion, you just ignored it. You made all the decisions for me based on your medical opinion.

  • You discussed options with me, made recommendations, solicited my opinion regarding the options/plans, and incorporated my opinion into your medical planning.

  • You not only solicited my input, but also explored the reasons for my choice and showed your understanding and respect for my decisions by negotiating a mutually agreeable plan.

  • Not applicable. There were no decisions to be made in this case.

• (7) Encouraging my questions

  • You did not solicit questions, or frequently avoided my questions, or did not provide helpful answers.

  • You sometimes asked if I had questions, but seldom waited at least 5 seconds to allow me to formulate questions. You addressed my questions briefly without avoiding them.

  • You actively encouraged me to ask questions, paused to allow me to formulate them, and provided clear and sufficient answers to all of my questions.

  • You actively encouraged me to ask questions several times during the encounter, with sufficient wait time. You spent significant time and effort to answer my questions clearly and confirmed that I understood the answer and that my concerns were addressed.

• (8) Providing clear explanation

  • You rarely explained things to me; you did not help me better understand my situation.

  • You gave me only brief explanations of my situation; you did not help me understand what would happen next.

  • You gave me a full and understandable explanation of my situation, pertinent findings, and important next steps.

  • You gave me a full explanation of my situation, your thinking about it and your recommendation, and probed my understanding by letting me summarize pertinent information.

  • Not applicable. There was nothing to be explained in this case.

• (9) Physical examination

  • You never or rarely warned me about what you were going to do with my body. You also never or rarely explained what you found from the physical examination.

  • You did not warn me about what you were going to do with my body, OR did not explain to me pertinent findings (both negative and positive) from your physical examination.

  • You told me what you were going to do to my body AND described what you found.

  • You helped me understand clearly what you were going to do to my body. You also provided clear explanation of what you found from the physical examination and the implications of your findings for my situation.

  • Not applicable. There was no physical examination in this case.

• (10) Appropriate vocabulary

  • You used vocabulary that was too simple or too complex for me, or frequently used medical terms without explaining them to me. Sometimes I could not understand what you told me without asking for explanations of terms you used.

  • Your vocabulary was generally appropriate but you sometimes inadvertently used medical terms without explaining them to me.

  • Your vocabulary was appropriate and if needed you provided brief explanations of any medical terms you used without need for prompting.

  • Your vocabulary was appropriate and you always provide clear and full explanation of relevant medical terms you used. In addition, you helped me better my understanding of my condition with the medical terms you explained to me.

• (11) Sensitive subject matters (e.g., sexual history, tobacco/alcohol/drug use, religious/cultural issues, giving bad news, or difficult emotional states)

  • You never warned me before approaching sensitive subject matters. You seemed judgmental and clearly expressed your disapproval of my positions or feelings, making me feel uncomfortable about discussing these subjects or feelings with you.

  • You were careful and non-judgmental in discussing sensitive subject matters. However, you did not express understanding of my feelings and did not provide much emotional support.

  • You were sensitive about discussing difficult subjects and were respectful of my feelings. I never sensed that you were judgmental or disapproving of my positions or feelings on these subjects. You showed empathic understanding of my position or feelings and provided appropriate emotional support.

  • You were unusually empathic, sensitive and respectful of me and of my feelings and provided exceptional emotional support. In addition, you verbally reflected these back to me (e.g., “You sound sad”) to show your understanding.

  • Not applicable. There were no sensitive subject matters in this case.

• (12) Receptiveness to feedback

  • You did not seem open to my feedback about your performance. You responded defensively or dismissively too many of my comments.

  • You listened to my feedback agreeably but passively. You did not actively participate during the feedback session.

  • You were able to describe some of your own effective and ineffective behaviors, were attentive to my comments, and had an open discussion with me about alternative behaviors.

  • You actively solicited additional feedback and showed signs of integrating my feedback into your behavioral repertoire. For example, you tried to role-play the communication techniques I suggested.

  • Not applicable. I provided no feedback.

• (13) Do I want to see you again as my personal physician?

  • I did not feel comfortable in communicating with you at all. I would rather see a different physician.

  • I think you were okay in general and might come see you again.

  • I was impressed by the way you communicated with me. I would like to see you again.

  • I was very impressed with you. I think you are one of the best physicians I have ever seen. I would feel very comfortable discussing any medical problems with you, and would recommend you to my friends.