In existence since 1937, the American Board of Surgery (ABS) provides board certification to individuals meeting defined standards of education, training, and knowledge in surgery. These generally include completion of training programs with associated requirements, and successful completion of the ABS examination process, which includes both a Qualifying (QE) and Certifying Examination (CE). ABS certification in Complex General Surgical Oncology (CGSO) is in a relatively nascent period of development. The first CGSO QE was administered in 2014, with the first certificates conferred in 2015. As with other surgery specialty boards, obtaining CGSO certification is a rigorous process. Candidates must have met the requirements for primary ABS certification in General Surgery, including the QE and CE in this specialty. Additionally, candidates must have completed an Accreditation Council for Graduate Medical Education (ACGME)-accredited CGSO training program in the United States, or a general surgical oncology training program accredited by the Royal College of Physicians and Surgeons in Canada. Recognizing the rigor required to obtain certification, there is a comprehensive and rigorous psychometric process applied to ensuring that the examination is fair and reflective of a safe standard. At present, there are nearly 500 ABS CGSO diplomates, and this number continues to increase.1

The QE assesses if an examinee has achieved defined standards of knowledge in CGSO. The passing score on the QE is expected to reflect the minimum acceptable standard of knowledge that an examinee is required to demonstrate, therefore reflecting the necessary knowledge to be a board-certified surgical oncologist. Thus, defining these minimum requirements is critical.

Why is the “minimum acceptable standard” of knowledge used as the bar for passing the QE?

 

The ABS has the responsibility, as its mission, to ensure the safety of the public. One of the tools it implements in pursuing this goal is identifying standards for board certification. The ABS does not try to stratify clinicians or identify exceptional individuals, but rather seeks to ensure that anyone who is certified meets a safe minimum standard to provide care to patients. In consideration of this goal, the ABS tasks a Standard Setting Panel with identifying what would be the minimum expected level of surgical knowledge to ensure that a surgeon is safe in providing care to a patient. In the case of the CGSO QE, the focus of the Standard Setting Panel is on identifying the minimum expected level of surgical oncology knowledge

 

Standard setting is the comprehensive and systemic process through which this passing score is set. It is conducted for examinations in all ABS specialties at scheduled intervals, generally every 5–7 years. For CGSO, a standard setting was conducted in September 2023 to determine the passing score for the 2023 CGSO QE. To provide the CGSO community with insight into the details of this process, we herein describe the methodology utilized.

Methodology and Results

It is important to make the distinction between criterion-referenced and norm-referenced passing standards. In a norm-referenced approach, a predetermined failure rate is prescribed such that a certain portion of examinees fail each year. In this model, for example, the bottom 15% of examinees may fail, irrespective of overall performance. Candidates are evaluated relative to one another, rather than to an absolute standard based on the content of the examination. One disadvantage of this approach is that it does not consider the variability seen in examinee preparedness across years. Theoretically, one examinee group may perform significantly better than previously, but 15% would fail regardless. The converse is also true; an examinee group in one year might perform significantly worse, but only 15% would fail. Norm-referenced approaches can result in some candidates passing who should not, and vice versa.

In contrast, the ABS utilizes a criterion-referenced approach wherein standards are established based on a minimum level of performance in relation to the content of the examination. Criterion-referenced approaches result in a fairer passing standard that is applied equally to all examinees each year. Several strategies for the criterion-referenced standard setting are available. For the 2023 CGSO QE, the modified Angoff method was utilized.2 Panelists first come to a shared understanding of a minimum acceptable standard of knowledge an examinee would possess, and second, evaluate each question on the examination relative to that understanding. Panelists review each question on the examination and individually assign the probability that an examinee with the minimum acceptable standard of knowledge would answer correctly. Probability ratings are averaged across all questions and panelists to generate a percentage correct passing score. Notably, in criterion-reference approaches, the ‘fail’ rate will vary, including some years in which all examinees may pass.

The CGSO Board convened a highly selected group of surgical oncologists with varied backgrounds and practice contexts to set the standard for the 2023 CGSO QE. The panel consisted of 15 practicing surgeons who were recruited to provide diverse representation from the field of surgical oncology with regard to demographics, training setting, practice type, and location. The panelists included CGSO Board members, CGSO Consultant Committee members, members of the certified surgical community, and Society of Surgical Oncology (SSO) members. The process was facilitated by ABS psychometric and examination development staff utilizing a validated and objective methodology. The meeting began with an explanation of the task at hand (to define the passing score for the 2023 CGSO QE) along with an explanation of the examination development process and basic psychometric concepts (e.g., question difficulty). This ensured that all participants had a reasonable understanding of the psychometric concepts. Next, the panelists defined the knowledge, skills, and abilities (KSAs) that would define the minimum acceptable standard of knowledge to provide safe and effective care of CGSO patients. The purpose of this exercise was to create a shared framework to establish this level of performance expected from a qualified examinee with a minimum acceptable standard of knowledge. To achieve this goal, content categories from the examination were reviewed (as outlined by the examination blueprint, which was revised in 2022 and implemented for the 2023 QE3), and panelists defined the KSAs that were collectively viewed to ensure the minimum standard expectations.

Once the minimum standard was defined, all panel members reviewed each question on the 2023 QE independently and assigned their rating of the probability that an examinee meeting the minimum standard would answer the question correctly. Probability ratings were averaged across all questions for each rater, thus representing the percentage of questions each panelist believed the qualified examinee with a minimum acceptable standard of knowledge would answer correctly. Values were then averaged across panelists to generate the minimum passing score for the examination, represented as the percentage correct. To provide additional detail, the panelists first practiced this process on a sample set of 20 questions, and were given feedback on (1) how their item-level ratings compared with the rest of the panel; and (2) what the passing score and resulting failure rate would be based on those ratings. After a brief discussion, the panelists repeated this process for the questions on the QE from 2023. The panelists provided ratings for the first half of the examination questions, again reviewed the feedback described above, and then completed the second half of the examination. Once all ratings were completed, panelists were again presented with information regarding their ratings as compared with the rest of the panel, the final passing score, and resulting failure rate.

Despite the rigor associated with this process, standard setting by nature includes some subjectivity that could lead to a degree of variability across multiple panels. In recognition of the variability associated with standard setting panels, the ABS provided the CGSO Board with the opportunity to adjust the standard by one standard error of measurement. After reviewing methods used to set the standard and the resulting passing score, the consensus by the CGSO Board was to endorse the score set by the panel, opining that any adjustment by the Board would be arbitrary. This substantive passing score will be carried forward and applied to subsequent years’ examinations, with statistical adjustments depending on their relative easiness/difficulty compared with the 2023 CGSO QE, until the next standard setting.

The panel members felt this information should be communicated not only to the SSO Executive Council but also to the CGSO Program Directors. As such, presentations were delivered to both the Executive Council and the SSO Program Directors’ Committee at their regularly scheduled meetings in October 2023. It was noted that current fellows would also be interested in this information, adding motivation for this editorial.

Conclusion

The central missions of the ABS include promoting the highest standards for professionalism, building inclusive excellence, and providing rigorous evaluation and assessment. The spirit of the standard setting process was in accordance with these missions to not only determine a passing score based on expected performance but to also convey the methodology to the public. Informing CGSO fellowship program leadership, faculty, and trainees to these efforts in parallel with ongoing multi-organizational initiatives to improve the educational foundations of CGSO trainees will maintain the benchmarks for potential diplomates going forward. We hope better understanding of the activity involved in setting standards for the CGSO QE will bolster confidence in the process and demonstrate the importance of ABS certification.