Introduction

Residency program directors [1, 2], students [3, 4], and clerkship directors [5] are among the stakeholders who recognize the post-clerkship year as an opportunity for more robust learning experiences to support the transition to residency. The introduction of the Core Entrustable Professional Activities for Entering Residency (CEPAER) or EPAs [5] has further underscored the importance of the post-clerkship year curriculum in teaching and assessing the competency of students to initiate the independent practice of medicine [5,6,7]. The Alliance for Clinical Education (ACE) called for the integration of Accreditation Council for Graduate Medical Education (ACGME) Core Competencies and Association of American Medical Colleges (AAMC) EPAs to guide curricular efforts during the fourth year, including recommendations related to assessment of competency, completion of capstone experiences, specialty-specific objectives, and targeted remediation [5]. The fourth year has been further influenced by internal and external efforts to help students prepare for their specialty of choice. Perceived increased competitiveness of the National Residency Match Program (NRMP) has led to increased specialization in the fourth year where students in competitive specialties are encouraged to complete multiple “acting intern” rotations at various institutions to secure a desirable residency spot.

Existing metrics to assess resident knowledge, skills, and attitudes (KSA) can be applied to the undergraduate medical education (UME) setting. Doing so can bridge expectations of UME and graduate medical education (GME) to the benefit of students and medical educators. Specifically, the integration of EPAs [5,6,7] and ACGME milestones as learning outcomes into fourth-year curricula [8,9,10] helps medical educators assess students on their path to residency and demystifies expectations of residency for students. Such efforts align to recommendations by various entities to better prepare medical students for the roles and responsibilities they will have as residents. Included, ACE recommends specialty-specific experiences and the Association of Program Directors in Surgery recommends surgery-specific experiences and EPA assessment for medical students pursing surgical residency [11].

A competency-based fourth year geared toward students’ specialty of choice and tied to EPAs and ACGME milestones can help address discrepancies in KSA observed by residency programs [9, 12]. Broadly speaking, fourth-year experiences that reflect responsibilities and expectations of residency are important [13]. Clinically immersive experiences [14], bootcamps [15], mentorship [16], and post-match competency handover [15,16,17,18,19] during the fourth year are among the curricular components identified as useful to aid the transition to residency. If a goal is to reduce the challenges in the transition from medical school to residency, how can programs design a fourth year that enacts recommendations and evidence-based practices?

Khan et al. first outlined the structure of the fourth-year curriculum at our institution [8]. They highlighted the structure and function of specialty-specific tracks and elective courses aligned to ACGME core competencies. The aim of this monograph is twofold: to revisit the design of our fourth-year curriculum and present the initial evaluation efforts to determine student and residency director perceptions of student preparedness for residency. Central to our post-clerkship curricular reform has been the integration of specialty-specific tracks and targeted opportunities to equip students with the knowledge, skills, and dispositions to function as interns and, ultimately, as future physicians.

Backwards Design of Fourth-Year Curriculum

The reformed fourth-year medical school curriculum was first implemented academic year 2015–2016 as part of the internally developed Lead.Serve.Inspire (LSI) curriculum. LSI is a 4-year, 3-part competency-based curriculum where part 1 includes the first 2 years of medical school focused on basic sciences integrated with patient care experiences, part 2 includes the third year of required clinical rotations, and part 3 encompasses the fourth post-clerkship year. Our conceptual framework for a multifaceted fourth-year curriculum draws on Fink’s backwards design with significant learning [20], calling for explicit attention to those factors that may influence teaching and learning [21] as learner outcomes, assessments, and learning experiences are designed. Fink’s taxonomy of significant learning—foundational knowledge, application, integration, human dimensions, caring, learning how to learn—facilitated the planning process. Specifically, quality of clinical experiences, prevalence of high-stake events (e.g., interviews, matching), exposure to general and specialty-specific activities, and balance between independence and supervision were among the key factors to designing a fourth year. Engagement in authentic learning experiences that mirror the roles and responsibilities of residents and attendings in their intended specialty were essential in integrating situational factors to meet learning outcomes.

The design process has been an iterative one, ongoing monitoring and modifications to ensure learning outcomes continue to be relevant and learning experiences are meaningful. Learning outcomes were determined based on established standards and core competencies. We have employed a combination of standards, to evaluate competency in the context of graduating medical school and progression toward competency parallel to expectations of an intern. Included are the AAMC Physician Competency Reference Set (PCRS) [22], ACGME milestones, and EPAs. The PCRS consists of eight domains of competency: Medical Knowledge/Knowledge for Practice, Patient Care, Practice Based Learning and Improvement, Interpersonal and Communication Skills, Professionalism, Systems-Based Practice, Interprofessional Collaboration, and Personal and Professional Development. They make up the core competencies for learner expectations across all four years of medical school. The general duties of any physician addressed by the EPAs, coupled with the specialty-specific KSA of ACGME milestones, bridge expectations of a graduating medical student with those of an intern. To formulate learning outcomes, first a crosswalk between established standards was performed similar to that conducted by the AAMC Drafting Panel [23]. For example, indicators within the Patient Care competency overlap with EPA 2 (“prioritize a differential diagnosis”). Next, specific and measurable learning outcomes were formulated using active developmentally appropriate verbs and actions aligned to the standard. For example, for Patient Care indicator 2 and EPA 2, a resulting learning outcome was “perform a comprehensive and pertinent physical exam for the chief complaint.” A broadly stated standard was transformed to an action that can be assessed through direct observation or other assessment method.

Learning experiences (e.g., clinical rotations, specialty-specific experiences) are mapped to measurable learning outcomes and established standards (core competencies, ACGME milestones, EPAs) to help students achieve the intended outcomes for the fourth year (Fig. 1). With previous curricula, as indicated by the dotted arrows, connections between outcomes and course instruction were decentralized. The current curriculum coordinated efforts toward students’ intended specialty, namely specialty-specific tracks as the capstone, as indicated by the solid arrows. The specialty-specific track for a student is the thread that overlaps required coursework and targeted competency-based experiences. Curriculum planning included multiple internally and externally developed outcomes in the design of assessments and learning experiences. The combination of required and elective courses provides opportunities to assess learner competency and progression toward readiness for the transition to residency.

Fig. 1
figure 1

Multifaceted fourth-year curriculum where students map out a schedule of electives and required clinical rotations that form the basis of their clinical track. Clinical tracks are unique to the specialty, with many supplemented by a bootcamp experience near the end of the year. A combination of general and specialty-specific learning goals informs assessments within and across courses experiences

Full size image

Clinical Tracks as a Fourth-Year Capstone

To bridge the gap between traditional third-year core clerkships and the internship year, our institution introduced specialty-specific tracks as courses of study customized for our students’ intended specialties [8]. Referred to as clinical tracks, these tracks serve as individualized educational plans and form the basis of part 3 course scheduling selections. Students select and complete at least one clinical track prior to graduation. The 13 currently available tracks represent the most common specialties applied to by our students (Table 1). Clinical tracks were designed with flexibility and ongoing support including the creation of new clinical tracks to align with a student’s residency plans. They function like a “major” toward their future specialty, consisting of longitudinal experiences and recommended elective courses to prepare students for residency by working toward knowledge, skills, and critical thinking outlined within ACGME milestones. As an example, a student pursuing General Surgery would meet with a clinical track advisor familiar with surgical residency prior to the fourth-year scheduling process to help guide the student in meeting their own educational priorities. Once enrolled in the General Surgery clinical track, they are able to select required rotations in clinical sites suited to their interests (e.g., a required ambulatory rotation in a surgical oncology office, or a hospital-based site with a high trauma census) and enroll in surgery-specific (e.g., ultrasound, surgery bootcamp) and/or broader elective experiences for the general physician (e.g., bioethics, medical education). Students undecided at the start of fourth year receive ongoing and individualized faculty mentorship to ensure students have ample opportunities to explore interests. Clinical experiences in the intended specialty, mentorship, bootcamps, and the feedforward process are four of the key elements of clinical tracks as the capstone experience throughout fourth year. The specialty-specific ACGME milestones serve as a tool to familiarize students with expectations of residency and formatively assess their progression within and across these key elements. Some of the clinical tracks, including Surgery and Internal Medicine, have additionally developed bootcamps for students to complete. These bootcamps include targeted clinical and non-clinical activities to hone KSA relative to EPAs and relevant ACGME milestones. Additional learning that occurs within the clinical track component has been outlined by Khan et al. [8]. In alignment with recent trends to “feedforward” [17, 18], a Post-Match, Milestone-Anchored Medical Student Performance Evaluation (mMSPE) was piloted with clinical track faculty during the academic year 2017–2018 academic year. First, students receive a letter written by the clinical track director detailing student progression along the specialty-specific ACGME milestones. Second, the student reviews a learning plan with their clinical track director. Third, the feedforward letter is sent to residency program directors. Beginning academic year 2018–2019, all fourth-year students were involved in the mMSPE process.

Table 1 Clinical track enrollment and student perception, 2015–2018
Full size table

Required and Elective Coursework for the Fourth Year

Advanced Management in Hospital-Based Care

Dedicated courses during fourth year including those concentrating on sub-internships, the intensive care unit, surgical clerkships, and emergency medicine have been positively correlated with better clinical evaluations as an intern [17]. Advanced Management in Hospital-Based Care (AMHBC) was designed as an intensive, 8-week integrated patient care experience in an acute care setting. It consists of two, separately scheduled, 4-week clinical blocks supplemented by asynchronous and online didactics, immersive simulation, skills workshops, and case conferences. Included in these blocks are targeted opportunities to engage in EPA 4 (order entry), EPA 8 (patient handover), EPA 10 (recognize patient in need of urgent or emergent care), EPA 11 (informed consent), and EPA 12 (perform general procedures of a physician) [6, 7]. The targeted experiences to practice these EPAs serve a secondary purpose—to provide opportunities to assess student competency. Clinical performance assessments (CPAs), objective structured clinical evaluations (OSCEs), and 360 evaluations are among the methods used to assess EPA related skills during AMHBC (Table 2). Similar to intensive curricular components investigated by Richards et al. [14], the clinical requirements for AMHBC are dispersed across the two clinical blocks—Mini Internship and Emergency Medicine. During the Mini Internship, students write admission orders, calculate bills for the patient, and discuss order entry decisions with their peers and faculty as part of the EPA 4 OSCE. This block also targets skills in acute inpatient management, interprofessional care, professional teamwork, and communication. During the Emergency Medicine block, students develop the skills needed to perform focused accurate assessments, order appropriate diagnostic studies, and initiate therapeutic interventions on acutely ill undifferentiated patients. Summative assessments include high-fidelity simulation for EPA 10 as it provides a balance between the realistic portrayal of an unstable patient and standardization across cases and assessment sessions [24]. Medical knowledge is assessed with an National Board of Medical Examiners (NBME) Shelf Exam, one of the few designed for the fourth year.

Table 2 Alignment of curricular entities to EPAs
Full size table

Advanced Management in Relationship-Centered Care

As outpatient and physician offices are inundated with more than one billion visits annually [25], acute care training is an important consideration for medical school curricula. Advanced Management in Relationship-Centered Care (AMRCC) was designed on a sub-internship model to offer students experiences in the breadth, scope, and importance of outpatient medicine. AMRCC consists of one clinical block each of outpatient and interprofessional chronic care medicine, with all students individually matched to clinical sites that complement their specialty of interest. They are expected to execute the responsibilities of an intern in clinic. Sites vary widely, from subspecialty, primary care, and palliative care clinics, to rural, indigent, interprofessional, and pediatric outpatient care. Students interested in primary care may also choose to complete clinical requirements longitudinally over the course of the year, with the goal of enhancing continuity of care and building authentic relationships with patients, physician preceptors, and interprofessional teams [26]. AMRCC’s scope extends to additional important areas for successful twenty-first century physicians—ethics and professionalism; health care access, finance, and delivery; advanced communication skills; self-care; outpatient evidence-based medicine; and care of the vulnerable patient/social justice (opioid addiction, physical and sexual violence, terminal illness, and racism). AMRCC provides targeted teaching and assessment of EPA 9 (interprofessional collaboration) including multisource clinical feedback evaluations from non-physician team members, nurse practitioner-medical student dyads, and an interprofessional team-based learning exercise on family violence (Table 2).

Advanced Competencies and Electives

There are four elective requirements in part 3, three being traditional clinical specialty-specific electives. The fourth elective is the Advanced Competency, a unique elective experience offering enhanced content mapped to the PCRS [22]. The Advanced Competency provides students targeted opportunities to develop skills and gain confidence in an area to benefit them in their future career. Yet, most are generalizable and translate to multiple practice areas. For example, the Advanced Competency in Ultrasound is often taken by students interested in Emergency Medicine and OBGYN. Students interested in a variety of subspecialties have enrolled in the Advanced Competency in Medical Ethics, and Professionalism and Humanism. Most are also interprofessional and may encompass a broad range of both clinical and non-clinical activities. Khan et al. provided examples of Advanced Competency and the core competencies on which they focus [8].

Longitudinal Projects

Health Systems, Informatics, and Quality Project

The AAMC and the Institute for Healthcare Improvement have increased their focus on Health Systems Science [27]. As a part of our curricular reform, Health Systems, Informatics, and Quality (HSIQ) was incorporated as a longitudinal patient safety/quality improvement component. It culminates with a High Value Care project during the fourth year as a significant vehicle for assessing EPA 13. During the fourth year, students complete and present a student-led interprofessional team capstone high value care quality improvement project using Define Measure Analyze Improve Control (DMAIC) methodology [28].

Educational Portfolio

The Educational Portfolio project is a pivotal experience in the LSI curriculum that begins during the first year and continues through part 3 to improve self-assessment skills and performance, enhance career achievement and satisfaction, and establish a pattern of lifelong, self-directed learning. It culminates during part 3 with the presentation of a formally assessed learning portfolio to faculty, where the student showcases evidence from their longitudinal education of success in the domains of competencies. Faculty coaching is a key element of the educational portfolio as it provides longitudinal, structured opportunities to engage students in reflection [29].

Results

This curricular evaluation was conducted in two stages to examine student and residency director perceptions of the fourth-year curriculum at our large Midwestern medical school in the USA. First, data from medical students who graduated from May 2016–May 2018 and consented to research involving their program data as part of an approved institutional umbrella IRB are reported here. Annually, approximately 90% of students consent to participate. Second, data from the 2019 AAMC Medical School Graduation Questionnaire (GQ) [30] are reported to (1) triangulate learner reactions to topics also addressed on internally developed surveys (e.g., perception of preparedness for residency) and (2) ensure learner reaction to intended goals are evaluated (e.g., communication skills). Since responses on the GQ are anonymous, all responses are included.

Curricular Evaluation of Program Data

The end of program survey is deployed electronically each spring. Descriptive statistics for 5-point Likert-type item responses (strongly disagree to strongly agree) were calculated. Included are the mean and standard deviation (Table 3) as well as percent agreement. For the end of program survey administered 2016, 2017, and 2018, student ratings of their clinical education during Med 4 have been, on average, favorable (Table 3). Over the 3 years, over 90% of students agreed or strongly agreed they were prepared or able to perform EPA 2, EPA 3, EPA 6, EPA 7, EPA 8, EPA 10, and EPA 11. Over 75% of students agreed or strongly agreed they were prepared or able to perform EPA4 and EPA 13. Favorable attitudes toward EPA 4 and EPA 13 preparedness were highest during 2018 (> 80%). Trends in Likert-type item responses were supplemented by narrative responses of 2018 graduates related to effective aspects of the fourth year and those needing improvement. While evaluation data are examined yearly by the part 3 program, 2018 responses were the most recent available to us for reporting here. In response to which aspects of the fourth year were effective, numerous related themes emerged. The most prevalent themes were flexibility, clinical experience, personalization of fourth-year experience, and preparation for intern year. Of the areas students cited as needing improvement, the only theme that emerged in over 10% of cases involved the timing and organization of HSIQ.

Table 3 Student perception of preparedness related to EPAs
Full size table

The Student Preparation Evaluation (SPE) is deployed annually between May and August to medical school alumni and their respective residency program directors at the end of their first post-graduate year (PGY-1) year. Descriptive statistics for 5-point Likert-type item responses (very low to very high) were calculated. Intern preparedness with respect to 10 of the EPAs, overall satisfaction, and progression toward residency milestones are assessed (Table 4). Graduates of 2016 and 2017 and program directors reported, on average, high levels of satisfaction with their medical training (Table 4). Given the nature of the variables being reported, we cannot meaningfully distinguish between fractional differences in ratings. However, the average responses coupled with the frequency distribution of responses suggest general satisfaction among alumni and program directors in the areas assessed. Across the two years, 98% (210/214) of program directors were pleased they matched with their intern, and 95% (203/214) believed their intern was progressing well toward meeting all residency milestones for their respective program. For EPA 3, EPA 4, EPA 8, EPA 9, EPA 10, and EPA 11, approximately 99% of program directors reported medium, high, or very high levels of satisfaction. Of program directors, 1.1–3.3% reported low satisfaction with student’s preparation for EPA 2 (2.3%), EPA 6 (3.3%), EPA 7 (2.4%), and EPA 13 (1.1%). There were no instances where a program director reported a very low level of satisfaction. Over 2 years, the majority of alumni reported medium, high, or very high levels of satisfaction with their training with respect to EPA 2 (97.4%), EPA 3 (97.4%), EPA 6 (98.2%), EPA 7 (98.2%), EPA 9 (96.6%), EPA 10 (97.4%), EPA 11 (97.4%), and EPA 13 (97.4%). Of graduates, 12.2% and 6.1% reported very low or low levels of satisfaction with training for EPA 4 and EPA 8, respectively. Notably, the percent of students who reported high or very high levels of satisfaction related to EPA 4 and EPA 8 was higher among 2017 graduates than 2016 graduates.

Table 4 Program director and PGY-1 level of satisfaction with entrustable professional activities
Full size table

2019 AAMC Medical School Graduation Questionnaire Individual Report

The 2019 AAMC GQ [30] collected responses from graduating medical students from February 14, 2019 to June 7, 2019. Of our fourth-year medical students, 85.6% (149/174) participated. One item (item 12) on the GQ asks students to assess their acquisition of KSA related to PCRS Domains of Competency. While the items are not specific the fourth year, the KSA highlighted occur in the clinical setting and would be assumed to arise from third or fourth-year experiences.

On statements related to Medical Knowledge/Knowledge for Practice, 97.2% of respondents agreed or strongly agreed they “acquired the clinical skills required to begin a residency program”; 96.5% (33.3% agree, 63.2% strongly agree) agreed or strongly agreed they had a “fundamental understanding of common conditions and their management encountered in the major clinical disciplines.” Regarding Patient Care, 97.3% respondents felt they had the “basic skills in clinical decision making and the application of evidence-based information to medical practice,” and 95.8% felt prepared to “care for patients from different backgrounds.” The statement most directly related to Interpersonal Communication Skills, “I have the communication skills necessary to interact with patients and health professionals” yielded 98.6% agreement, of which 77.6% of students responded strongly agreed. Professionalism was also represented in item 12. 98.6% of students felt they understood “the ethical and professional values that are expected of the profession,” with 78.5% of those having strongly agreed. Of student, 95.9% agreed or strongly agreed they were prepared to begin residency with an understanding of issues in the social science of medicines such as ethics and the health care system. This item most appropriately aligns to our AMRCC and HSIQ courses, respectively.

The GQ items also offered responses highlighting the personalized nature of the fourth-year curriculum. Specifically, 81.6% (n = 141) of student agreed or strongly agreed they “received appropriate guidance in the selection of electives” (item 15). Of the 142 students who responded, 50.7% reported having at least one away rotation, defined as rotations not required for graduations and not affiliated with their medical school (item 17).

Discussion

There is an abundance of literature outlining standards, frameworks, institutional models, and best practices to prepare fourth-year medical students for a transition to residency [2, 5,6,7]. We had an opportunity to draw upon the experiences of those institutions and professional organizations to inform our efforts and conceptualize a multifaceted fourth-year curriculum (Fig. 1). The aim of this monograph was to layout the fourth-year medical school curriculum at our institution and present stakeholder perceptions of the fourth year and preparedness for residency. Integration of targeted competency-based, EPA assessments, and specialty-specific tracks as the capstone with corresponding milestones have driven learning to prepare students for residency. The specialty-specific focus was critical in operationalizing early visions for personalization in the fourth year. Students have favored the flexibility and opportunity to personalize their fourth-year experience. Student and resident director perception reported on internally developed evaluations coupled with 2019 GQ results suggest the curriculum has been well received by stakeholders and is meeting our goal to help students transition to residency. The personalized and competency-based aspects of the curriculum are supported descriptively but could be better understood narratively and by reporting performance-based measures. Caution should be exercised when interpreting categorical data (e.g., Likert-type responses) as continuous.

The iterative nature of the design process and ongoing monitoring of learner perceptions has helped the curriculum remain relevant and rooted in best practices for curriculum design. We have added more robust assessment to the clinical tracks by anchoring specialty-specific milestones to required course assessments and advocated for protected time of clinical track faculty to implement change. The addition of the mMSPE for clinical tracks expands opportunities for feedback, providing students feedback on their progress for the given set of milestones and creating a pathway for feedback and communication with residency directors. Learning experiences to develop and assess learner competency around EPAs continue to expand. As an example, EPA 4 and EPA 9 assessments have made use of a simulated EHR environment with the help of the Medical Center IT to engage students. Advanced Competencies within the fourth year make adapting to current needs possible, most recently evident in the Disaster Preparedness Advanced Competency implemented April 2020 in response to COVID-19.

The timing of the curriculum and external pressures surrounding the match will always be a challenge. Anecdotally, students have a singular focus on the perceived increasing competitiveness of the match, and, as a by-product, coursework interfering with interviews, rank list, and specialty-specific rotations can be seen as having lower value. We have adapted to balance the needs of the students with the vision of a successful fourth year. As an example, the HSIQ curriculum was moved so the majority of activities are completed prior to interview season, including completion of the first project during the third year of medical school.

While curricular evaluation is an ongoing process, we have learned several lessons from our curricular reform process that may help others pursuing similar curricular endeavors. First, a diverse group of specialty-specific faculty, curriculum specialists, and stakeholders familiar with GME and LCME accrediting needs has been important at all stages of design, implementation, and review. Second, essential questions about learners and the context for learning guided the way backward design as outlined by Fink [20] was applied to design process (Fig. 1). What KSA do graduating medical students need the first day of intern year? How will we know when students are ready? How do we develop a curriculum that serves the individual needs of students? We suggest posing essential questions at the beginning of the process to guide decision making on learner outcomes, assessments, and authentic learning experiences. Such questions informed our decision to integrate multiple sets of existing standards—AAMC-endorsed competency-based outcomes (e.g., PCRS), ACGME milestones, and EPAs—as the backbone of learner expectations. In doing so, general and specialty-specific curricular needs emerged. Third, we suggest a vehicle to balance the need for students to demonstrate competencies in areas expected of all physicians with those needs that are specialty specific. For us, clinical tracks have served as a way to organize the fourth year, to drive scheduling, and to monitor progression through a complicated year. We believe it has helped place learner needs in pursuing their intended specialty at the forefront while balancing required general clinical coursework and longitudinal projects. Integration of competency-based EPA assessments (and making this transparent to the learner) in the required courses, and ACGME milestones for clinical track evaluations, parallels expectations for the transition to residency. Students within clinical tracks were paired with faculty well-equipped to advise students in their intended specialty to offset coaching in those general areas applicable to all prospective physicians, specifically PCRS core competencies and EPAs. Last, four months of elective experiences were intended to provide flexibility and further personalization of the fourth year, an aspect students appear to recognize as a strength in the curriculum. We have been able to diversify elective course offerings to meet current demand and address important contemporary issues impacting the future of medicine such as racial disparities and health inequities. Further studies will only better determine the level of effectiveness and impact the fourth-year curriculum has on student preparedness.

Future Directions

Our post-clerkship curriculum was informed by evidence- and research-based practices to provide learners with intensive clinical course work taken during the fourth year. Our clinical tracks provide a framework that aligns UME and GME such that students are working toward entry-level specialty-specific milestones in advance of intern year. Moving forward, we hope the mMSPE can better inform residency program perception of intern readiness. With ongoing implementation of the mMSPE process, we will have opportunities to investigate areas of strength and improvement in the design and implementation of the fourth year. Learner expectations during the four years of medical schools based on core competencies and, as such, KSA are developed over time. Further studies to evaluate the effectiveness of individual course components and determine direct associations between the fourth year and readiness for residency are needed. With continued curricular monitoring, we will be able to adapt teaching and learning to afford fourth-year medical students opportunities to develop KSA in the area will eventually practice medicine.