Featured ArticleUse of artificial intelligence for gender bias analysis in letters of recommendation for general surgery residency candidates
Introduction
Letters of recommendation (LoRs) are valuable assets in general surgery (GS) resident selection, ranked as second in importance by general surgery program directors, only behind United States Medical Licensing Examination (USMLE) scores as determinants for interviewing applicants.1 LoRs complement academic success in clerkship grades and USMLE scores by accentuating non-cognitive factors and holistically highlighting candidate communication skills, work ethic, teamwork, technical performance, and personal characteristics, among others.2 LoR authors discuss attributes elucidating applicant traits readers otherwise might not know, engaging in advocacy while remaining accountable to their audience.
Despite their importance, LoRs unfortunately may not advocate equitably for all applicants. Analysis of large textual corpuses by Mondorf and Argamon et al., in 2002 and 2003, respectively, showed numerous gender-specific differences in underlying lexical and syntactic properties that implicitly communicate gender bias.3,4 While these have been reported repeatedly, their tacit nature often obscures audience detection.5,6 In medical education, several studies implementing different analytic techniques have elucidated various implicit social biases in LoRs, including race- and gender-based biases.7, 8, 9 Implicit gender bias in GS LoRs have been described at different levels of training.10, 11, 12, 13
The first step in mitigating gender bias is through detection. Natural Language Processing (NLP) is a rapidly deployable, malleable tool that utilizes codable algorithms for bias detection.14 NLP can be deployed through computer-based algorithms (CBAs) or cloud-based networks to perform subtasks, such as analyzing sentiment, emotion, tone, or personality within text to detect implicit biases.15, 16, 17 Of the cloud-based systems, artificial intelligence (AI) has been adopted by several industries due to automation, speed, adaptability, and machine learning capabilities.18, 19, 20, 21 To date, no published study has examined gender bias analysis capabilities of AI for GS LoRs. We describe our implementation of AI and CBAs to analyze NLP variables and gender bias in GS LoRs across three decades at one academic institution, examining changes in each over time by LoR, applicant, and author traits. We hypothesized that over time, gender bias in GS LoRs would decrease, with a corresponding increase in positive sentiment toward females, signifying gradual awareness of gender bias and improved opinion of GS residency candidates.
Section snippets
Materials and methods
Following institutional review board approval, we performed a retrospective analysis of 611 LoRs written for 171 categorical GS residency applicants who successfully matched at one tertiary academic institution between 1980 and 2011. Study data were collected in 2019 and only went through 2011 so as to avoid inclusion of data for residents currently matriculated into the program. LoRs were scanned from print copies then converted to text files with headings, greetings, and signatures removed
Applicant and author traits
APs 1, 2, and 3 had 306, 154, and 151 LoRs, respectively, written for 89, 40, and 42 applicants, respectively. Mean number of LoRs per applicant was 3.57 (±0.95) and remained similar over time (p = 0.11). SCG were reported by 93% of applicants in the study, while 64.9% had either a three-digit USMLE score or NBME Part I examination percentile to comprise an SBQ. SCG distribution did not differ significantly across AP (p = 0.30) or by applicant gender (p = 0.45), nor by applicant gender within
Discussion
Our study is the first to examine surgical residency candidate LoRs with AI and a gender bias algorithm. Both sentiment algorithms detected improved author opinion of applicants over time and with increasing LoR length as indicated by higher sentiment. Furthermore, our data suggest that gender bias in LoRs for both genders contained more overall female bias in earlier APs, with a gradual shift to male bias. This finding has been seen in other realms of graduate-level education but has not been
Conclusions
General surgery residency LoRs have made significant strides towards reducing female bias over the past three decades. However, the underlying maladaptation to male bias as the “norm” for descriptive success is problematic and should be addressed. More research is required to determine how to best mitigate implicit biases in the resident selection process. With appropriate datasets, time, and devoted analytics, gender bias detection can improve and be minimized, while simultaneously enabling
References (39)
- et al.
The role of letters of recommendation in the selection process of surgical residents in Canada: a national survey of program directors
J Surg Educ
(2017) - et al.
Gender and letters of recommendation: a linguistic comparison of the impact of gender on general surgery residency applicants
J Surg Educ
(2019) - et al.
Exploring the gender gap: letters of recommendation to pediatric surgery fellowship
Am J Surg
(2020) - et al.
Textual analysis of general surgery residency personal statements: topics and gender differences
J Surg Educ
(2018) - et al.
A survey on opinion mining and sentiment analysis: tasks, approaches and applications
Knowl Base Syst
(2015) Results of the 2020 NRMP program director survey
Gender differences in English syntax
J Engl Ling
(2002)- et al.
Gender, genre, and writing style in formal written texts
Text-The Hague Then Amsterdam Then Berlin-
(2003) The cognitive monster: the case against the controllability of automatic stereotype effects
Dual-process theories in social psychology
(1999)The malleability of automatic stereotypes and prejudice
Pers Soc Psychol Rev
(2002)
Linguistic Biases in Letters of Recommendation for Radiation Oncology Residency Applicants from 2015 to 2019
Gender-based differences in letters of recommendation written for ophthalmology residency applicants
BMC Med Educ
Gender and racial bias in radiology residency letters of recommendation
J Am Coll Radiol
Influence of gender on surgical residency applicants' recommendation letters
J Am Coll Surg
Multi-dimensional gender bias classification
Sentiment Analysis and Opinion Mining
Who will retweet this?: automatically identifying and engaging strangers on twitter to spread information
Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence
Nat Med
Cited by (16)
Gender Differences in Letters of Recommendation for Gynecology Surgical Fellowship Applicants: Analysis of One Program's Recruitment Cycle
2023, Journal of Minimally Invasive GynecologyExamining Implicit Bias Differences in Pediatric Surgical Fellowship Letters of Recommendation Using Natural Language Processing
2023, Journal of Surgical EducationCitation Excerpt :Additionally, differences in LORs have been identified by gender of the letter writer regardless of applicant gender.8 Previous work has identified race and gender differences in various specialties, including general surgery, urology, transplant surgery, vascular surgery and others.2,5-12 Linguistic differences were found to be implicit but may influence recruitment of a diverse workforce into surgical training programs.
Recruitment of the Next Generation of Diverse Hand Surgeons
2023, Hand ClinicsCitation Excerpt :Even in Plastic Surgery where standardized letters of recommendation have become more common, studies show that gender and minority status tend to predict poorer letters.27 Similar data also exist for applicants to General Surgery residency.28 To combat this problem, it may be important for schools and programs to consider letters of recommendation as potential sources of bias and not to weight them as heavily as the more unbiased elements of the application, such as transcripts and publications.
From the editor – In – Chief: Top papers from the Association for Surgical Education
2021, American Journal of SurgeryArtificial intelligence and gender bias in hiring surgeons and beyond
2021, American Journal of SurgeryWriting recommendation letters: The discourse of evaluation in academic settings
2024, Writing Recommendation Letters: The Discourse of Evaluation in Academic Settings