Introduction

The COVID-19 pandemic drove academic medical institutions to reimagine their approaches to medical education. This led to a widespread shift from in-person to virtual learning, presenting a particular challenge for surgical specialties that emphasize hands-on education. Medical student urologic education has since faced challenges, with many electives being canceled and a high proportion of medical students reporting decreased exposure to urology as a result [1]. Furthermore, the apropos recommendation by the Coalition for Physician Accountability to discourage all visiting subinternship dissolved the traditional pathway whereby rising fourth-year medical students commonly solidify their interest in surgical specialties, advance their knowledge base, and begin to select a residency program [2].

This situation presented an opportunity for novel tools in virtual education to emerge. Foundational urologic learning in medical school is traditionally based in the operating room and supplemented by lectures and literature. While the technical aspects of surgical training are best taught in the operating room, surgical specialties also demand a cognitive skill set that includes clinical decision-making, problem-solving, critical thinking, physiologic and anatomic mastery, and literature appraisal – all of which can be taught in a virtual setting. Departments capitalized on this opportunity by offering an array of virtual educational initiatives, including online lecture series and virtual electives for medical students [3,4,5,6].

While these virtual initiatives have been effective, their reliance on synchronous interaction between students and teachers represent a barrier to widespread implementation and future reusability. In particular, the coordination of rigorous didactic schedules with physicians’ time constraints is significantly challenging. Therefore, the development and utilization of asynchronous educational tools can be particularly valuable, as they offer the opportunity to accommodate physicians’ varying schedules and limited availability.

Herein, we describe the development, implementation, and educational outcomes of a novel blended learning curriculum for senior medical students that utilize interactive online modules in combination with small-group discussion to enhance urologic learning. This curriculum was designed to fill the gap in urologic learning opportunities during the pandemic, and it was offered as a component of both virtual and in-person urology electives. This survey study demonstrates that incorporating synchronous and asynchronous online learning into urologic education can enhance the educational experience for medical students during subinternships.

Material and Methods

Assessment of Need

Due to the COVID-19 pandemic, our institution suspended all in-person medical student clinical clerkships in March 2020. In response, we sought to develop virtual educational resources that could be utilized as part of an entirely virtual rotation, as well as incorporated into future live rotations. In May 2020, we conducted a 13-person focus group to evaluate the educational needs of medical students interested in a career in urology. The group included seven medical students from our institution’s medical school, all of whom were considering applying to residency in urology, and six recent medical school graduates who are currently in urology residencies. This assessment helped inform the development of a comprehensive 4-week virtual subinternship which we have described and evaluated separately [7]. Two of the core objectives of the curriculum therein were:

  • To teach foundational urologic knowledge, pertaining to the 6 ACGMA core competencies and the ACGME Urology Milestones

  • To assess students’ individual capabilities and potential to succeed as residents

Based on this, we aimed to build an interactive online curriculum that could teach foundational knowledge and provide a platform for qualitative performance assessment, while affording scheduling flexibility for both participating physicians and medical students. We hoped this virtual format would keep students engaged and would be compatible with both virtual and in-person urology electives.

Structure

We created a curriculum of four interactive online modules on the Canvas platform (Instructure, Salt Lake City, UT). Each module consisted of a fictional clinical scenario, in which students were guided step-by-step through a complex urologic case in a self-paced format. Each case involved a multimodal array of resources that were strategically woven into various parts of the patient’s unfolding story. Cases included links to journal articles, textbook chapters, recorded lectures, surgical videos, and radiology studies. At different steps of the cases, students would be prompted to obtain a relevant history and physical exam, formulate a differential diagnosis, order laboratory tests, interpret imaging studies, read textbook chapters, and appraise relevant literature.

At each step of the case, students were asked open-ended questions and posted their answers on a discussion board (Fig. 1). Questions were largely focused on clinical reasoning, pathophysiology, and evidence-based medicine. Faculty and residents in our department were invited to view the discussion boards and post comments in response. Students were also invited to comment on each other’s posts. This enabled dialogue and individualized feedback in a timeframe that was flexible for both the teachers and the students. Students were encouraged to collaborate with one another and to use all resources at their disposal when answering questions. Questions were intentionally open-ended, in that there were not always “right” answers and thus no standard grading. The intent of the questions, therefore, was to stimulate further discussion. Every question intended to deepen the students’ understanding of the topic while advancing the patient’s narrative. Faculty and residents were instructed to incorporate an assessment of the students’ clinical reasoning, medical knowledge, and literature appraisal from the discussion boards into their overall performance assessment for the subinternship.

Fig. 1
figure 1

Screenshot of a discussion board regarding the evaluation of a man with elevated PSA (student and resident names removed)

Full size image

Once a week, the students would also review the cases in a live didactic session with the course directors. Each module was designed to require approximately 1 h of work per day over the course of 1 week.

Content

Topics for the modules were selected by the course directors in order to provide students with a breadth of exposure to common urologic problems. Topics were drawn primarily from the American Urological Association (AUA) curriculum for medical students [8,9,10]. We fine-tuned module content through extensive discussions with academic urologists at our institution, ensuring that we balanced benign and malignant conditions, adult and pediatric conditions, inpatient and outpatient care, and clinical and surgical management.

We selected the following topics for the modules:

  1. 1.

    Hematuria and bladder cancer

  2. 2.

    Kidney stones and ureteral stricture

  3. 3.

    Vesicoureteral reflux and cryptorchidism

  4. 4.

    Prostate cancer, erectile dysfunction, and urinary incontinence

An excerpt from one of the modules is included in Online Resource 1. In total, these modules incorporated eight of the core content topics from the AUA curriculum (hematuria, bladder drainage, kidney stones, urologic emergencies, pediatric urinary tract infection, prostate cancer, erectile dysfunction, and incontinence). Topics were grouped based on similarities in clinical features, surgical complications, and patient population. All content in the modules was reviewed and critiqued by fellowship-trained urologists with expertise in the relevant subject matter.

We also incorporated several of the core competencies provided by the Accreditation Council for Graduate Medical Education (ACGME) into each module. We focused on patient care (PC), medical knowledge (MK), practice-based learning and improvement (PBLI), and interpersonal and communication skills (ICS). PC was taught using scenarios involving clinical evaluation and surgical steps, MK with questions about pathophysiology and diagnosis, PBLI using evidence-based medicine and individualized feedback, and ICS through collaboration with fellow students, as well as scenarios involving patient counseling.

Curriculum Integration

All fourth-year medical students enrolled in a urology elective at our institution from July through October 2020 completed these modules over the course of their elective, in addition to their standard curriculum. These included students enrolled in both virtual and in-person electives.

We offered two types of virtual electives: a research elective and a clinical elective. For the virtual research elective, students were paired with a faculty mentor and a resident mentor to design and implement a research project, as well as participate in ongoing research projects. For the virtual clinical elective, the standard curriculum consisted of department conferences, small group didactic sessions, virtual clinic participation, virtual surgical participation, virtual social events, mentorship meetings, and feedback sessions. Each student spent approximately 15 h per week in clinical/surgical encounters and an additional 13 h in conferences and didactics. Students were assessed on the quality of a number of deliverables including a Grand Rounds presentation, small group presentations, and a quality improvement group project. Students were also assessed based on participation in all didactic and clinical sessions, attendance, and interactions with other members of the department. A full description and evaluation of the virtual clinical elective has been previously published [7].

For students enrolled in in-person electives, the standard curriculum consisted of daily participation in urologic surgery and inpatient care, as well department conferences, outpatient clinic, and feedback sessions. These students were assessed based on their participation in clinical and surgical care, quality of a Grand Rounds presentation, attendance, and interactions with other members of the department. Both the virtual and in-person electives were 4 weeks long, were formally accredited by our institution’s medical school, and were graded in accordance with our institution’s guidelines.

Students in the virtual electives were given 1 week to complete each of the 4 modules. Due to the heavy clinical and surgical workload required during the in-person electives, students completing these electives were given 2 weeks to complete each module, thus only completing 2 of the 4 modules during their rotations. Students in the in-person electives completed the “hematuria and bladder cancer” module and the “kidney stones and ureteral stricture” module, which were chosen to help reinforce scenarios that the students encounter in the inpatient setting. The students would gain the knowledge from the other two modules, as well as a range of other urologic topics, through their clinical and surgical encounters, supplemented by independent reading. Student performance for the modules was assessed based on the accuracy and completeness of their responses, their engagement in the discussion boards, and their participation in the live review sessions with the course directors; this was incorporated as part of the students’ overall assessment for the course. Because the modules did not involve direct patient interaction and were based on fictional cases, time spent on the modules was not considered to be clinical time.

Survey Design

At the conclusion of the elective, we sent anonymous surveys to the students (Online Resource 2) and participating physicians (Online Resource 3). The surveys were designed to evaluate (i) perceived success in achieving the curriculum’s primary objectives, (ii) perceived learning outcomes, (iii) efficacy of the different components of the curriculum, and (iv) future reusability. Surveys also included space for free text feedback.

Students’ learning outcomes were measured using the framework of the ACGME core competencies, focusing on PC, MK, PBLI, and ICS. We included 2 questions specific to each core competency to evaluate the modules’ effect on medical student learning. Questions pertaining to the core competencies were drawn from the ACGME Urology Milestones [11]. Outcomes were measured using a five-point Likert scale (5 = strongly agree/extremely effective, 1 = strongly disagree/not effective at all). The scores from the questions related to each core competency were averaged to determine learning outcomes. Mean Likert scale scores of 4 or greater were considered a positive outcome. Statistical analysis was done using Stata 12.1 (StataCorp, College Station, TX). This study obtained an exemption determination by the Columbia University Institutional Review Board.

Results

Over the course of 4 months, 14 students completed the online modules as a component of their urology electives, including 4 students during an in-person elective, 8 students during a virtual clinical elective, and 2 students during a virtual research elective. Sixteen physicians (10 residents and 6 faculty) participated in reviewing student module responses during these rotations. Thirteen students (93% response rate) and 12 physicians (75%), including 7 residents and 5 faculty, completed the survey.

Overall, 12/13 students and 11/12 physicians “strongly agree” or “somewhat agree” that the modules improved the rotation, with average ratings 4.5 and 4.4, respectively (Fig. 2). This held true both for students in in-person rotations (average rating 4.5) and in virtual rotations (4.6). Both students and physicians reported the modules were able to achieve the primary objectives of the modules regarding learning/teaching foundational knowledge (average ratings 4.8 and 4.5) and facilitating performance assessment (4.4 and 4.0).

Fig. 2
figure 2

Curriculum objectives. Physician and student perspectives of whether the online modules’ primary objectives were achieved: (i) overall improvement of the elective, (ii) teaching and learning foundational knowledge, (iii) providing a fair assessment of students. Objectives were scored on a five-point Likert scale

Full size image

The majority of students (12/13) and physicians (10/12) strongly agree or somewhat agree that the online modules should be incorporated into future urology electives (average ratings 4.5 and 4.0). Additionally, 12/13 students strongly agree or “somewhat agree that the online modules were fun/engaging (average rating 4.5).

Student learning outcomes for each ACGME core competency are shown in Fig. 3. Students perceived excellent learning opportunities in PC (average rating 4.5), MK (4.6), and PBLI (4.5), with slightly lower scores for ICS (3.9). Student ratings of the various components of the online modules are shown in Fig. 4. The most effective components were rounds/discussion boards (average rating 4.5), the live review of the modules (4.5), and radiology studies (4.3) — each of which was rated “extremely effective” or “very effective” by 11/13 students.

Fig. 3
figure 3

Learning outcomes. Student learning outcomes, scored on a five-point Likert scale, with 5 being the best/highest score. Scores for each ACGME core competency are derived from the average of the two survey questions pertaining to that competency

Full size image
Fig. 4
figure 4

Module components. Student ratings of the various components of the online modules. Ratings were scored on a five-point Likert scale

Full size image

The median amount of time that students reported spending on the modules was 1–2 h per day (range: less than 1 h – 2–3 h). All students reported that the difficulty level of the questions was “appropriate,” though 4 students (3 virtual, 1 in-person) reported that the modules were “too long.”

Feedback was very positive, overall. One student commented, “These modules did an incredible job delivering information and promoting curiosity and learning.” One resident commented that the literature review and discussions were “very impressive and something that I may have benefited from during my medical school education.” The most common constructive feedback related to the time commitment of the modules which some students, both from virtual and in-person electives, found to be burdensome on top of their other responsibilities during the rotation.

Discussion

We determined that a blended curriculum of interactive online modules can be effectively integrated into senior medical student rotations to fill the gaps in clinical and surgical education in the context of the COVID-19 pandemic. The blended curriculum has the capacity to augment learning and facilitate performance assessment for students engaging in both in-person and virtual urology electives, with high potential for continued use after the pandemic. This platform offers students an opportunity to work through complex cases while interpreting studies, synthesizing the results from evidence-based investigations, collaborating with one another, and receiving individualized feedback. Most importantly, both students and physicians felt that the curriculum improved the elective overall, regardless of whether the elective was in-person or virtual.

Our curriculum was intended to supplement, rather than replace, the knowledge that students gain from their electives. We demonstrate that both students and physicians perceived the online curriculum to be effective in enhancing knowledge acquisition and qualitative performance assessment. We were pleased to find that the modules helped students learn a variety of skills spanning multiple ACGME core competencies in a fun and engaging way.

Our students found the live review and online discussion boards to be the most effective components of our modules. This finding underscores the importance of discussion and feedback in the learning process, as opposed to the lower-rated reviews of articles, textbook chapters, record lectures, and surgical videos. The discussion-based learning tools, both on the online discussion boards and the weekly synchronous review, allowed students not only to understand the concepts but also to apply their knowledge to clinical scenarios. Notably, the other tool that students found most useful was the interpretation of radiology studies. This is consistent with prior literature showing that radiologic image interpretation is a widely unmet need among medical students [12].

Our interactive online curriculum builds upon prior studies on virtual education and blended learning. Interactive online modules have previously shown to be effective in augmenting knowledge acquisition in a variety of medical settings, with a range of topics including preparation for residency [13], value-based care [14], and microbiology [15]. Blended curricula that augment virtual self-paced learning with live discussion may offer additional benefits in terms of satisfaction and knowledge acquisition [16, 17]. A meta-analysis evaluating virtual education found that qualities associated with improved learning outcomes include interactivity, practice exercises, and feedback — qualities that are featured in our curriculum’s format [18].

While virtual learning is frequently incorporated into surgical education in the form of procedural simulation, the implementation of interactive case-based curricula is uncommon [19,20,21]. Web-based modules for surgical trainees have previously demonstrated a benefit in knowledge acquisition and surgical proficiency in general surgery [22, 23], vascular surgery [24], urology [25, 26], and plastic surgery [27] programs. Our curriculum builds on these initiatives by blending online learning with synchronous and asynchronous discussion forums. To our knowledge, this is the first surgical curriculum to involve clinical cases with embedded open-ended questions and a forum for individualized feedback and discussion, which we believe to be the major strength of our modules.

The discussion-based format, in addition to facilitating feedback and learning, opens the door for improved student performance assessment. While prior virtual education programs have demonstrated a benefit in terms of knowledge acquisition, they did not examine how their curricula affected performance assessment. The existing structure of student assessment in clinical electives is highly dependent on subjective clinical skills evaluations, which are susceptible to structural and interpersonal biases [28]. Additionally, with a trend towards deemphasizing medical student standardized test scores, it is perhaps more essential than ever to ensure consistent and fair assessment. Through written responses and case-based discussions, our curriculum provides a platform for students to demonstrate their knowledge, clinical reasoning, and literature appraisal in a standardized format outside the pressures of the hospital setting. While the assessment of these responses may also be susceptible to bias, students and educators found our curriculum to be effective in facilitating performance assessment. Further study is needed to evaluate the impact of virtual discussion boards on bias in performance assessment.

A major strength of online curricula is accessibility. The virtual format allows the curriculum to reach students at other institutions, without being dependent on geographic proximity. This allows for extended access to high-quality education to students who might otherwise lack that opportunity. Particularly for subspecialties that do not have a presence at every institution, this online format can level the playing field and offer students exposure to a field that they might otherwise never experience, while minimizing the geographic and financial barriers that are associated with visiting electives. With visiting electives estimated to cost up to $1000, online curricula may offer a more cost-effective alternative [29].

Another strength of this format is the ability to easily make adjustments and improvements with each iteration based on student and faculty feedback. We have already implemented changes to the curriculum, including shortening some of the modules and having students lead the live review sessions with physician oversight. Potential opportunities for further improvement include adding more radiology studies, decreasing the number of articles and lectures, and expanding the breadth of material. While the four modules cannot capture the entirety of medical student urologic education, we may consider including additional topics that students, urologists, and other physicians have considered to be important components of medical student education, including acute scrotum, benign prostatic hyperplasia, and adult urinary tract infection [8].

While our results are promising, we recognize that the time commitment required to effectively implement our curriculum is significant for physicians and students alike. The curriculum relies on buy-in and commitment from physicians to read the cases and reply to student responses. Despite the curriculum’s flexible nature, multiple students felt that the modules were time-intensive in the context of their other clinical demands. A potential strategy to address this could involve more thoughtful pacing of the modules throughout the clerkship and removing some components that may be less relevant to the students’ learning. This could decrease the burden on students while also encouraging more physician engagement. Certainly, it is important to allow time for thoughtful responses without detracting from the students’ clinical and surgical experience.

Limitations of our study include small sample size, lack of a comparison group, and potential bias in survey responses due to the small number of students. Additionally, our metrics for measuring learning and assessment were subjective and based on student and physician perceptions, rather than test scores or demonstrated proficiency, as the National Board of Medical Examiners does not offer subject examinations in urology, and there are no validated instruments for assessing medical student urologic knowledge in this setting. Despite these limitations, our experience supports the addition of blended online curricula into surgical education, with potential to enhance foundational learning and facilitate qualitative performance assessment for medical students. By broadening access, removing barriers, and minimizing costs, online curricula may open the door for reimagining traditional visiting electives in the future. We believe the framework of this curriculum is adaptable by other institutions and other specialties, and we hope this can serve as a springboard for further advances in surgical education.

Conclusions

A blended learning curriculum utilizing online modules is an effective tool for urologic education, improving perceived learning outcomes and facilitating performance assessment. This curriculum can be utilized as a component of in-person and virtual courses, and it can enhance the standard of foundational learning for medical students.