Introduction

High quality training in interventional radiology (IR) with appropriate methods of assessment of competency is vital to ensure the quality of outcomes, performance and ultimately patient safety [1, 2]. Since its introduction just over a decade ago, the European Board of Interventional Radiology (EBIR) has evolved to become a highly regarded, internationally recognised and sought-after examination, which certifies the expertise of interventional radiologists (IRs) throughout the world. The examination is organised by the Cardiovascular and Interventional Radiology Society of Europe (CIRSE) under the leadership of the Chair and Deputy Chair of the CIRSE EBIR Council. In 2022, the examination is delivered in the English, German and Spanish languages with the potential in future to expand the choice to additional languages. Under the careful guidance of the CIRSE Executive Board and the EBIR Chairs, the EBIR has gone from strength to strength with more than 1,000 EBIR holders now worldwide. As we look to the future development of the EBIR, it is timely to reflect on how the examination was developed and what lessons might be learned for the future.

In the Beginning/First Steps

The concept of an examination for European Interventional Radiologists was first introduced by Prof Jim Reekers during his CIRSE presidency, at the spring meeting of the CIRSE Executive Committee in Lisbon in 2008. The aim of such an examination was to provide successful candidates with a recognised qualification in interventional radiology, which could be used for employment applications, in situations involving turf issues, and when IRs move from one country to another in Europe as a certificate of IR knowledge.

Subsequent to that meeting, Jim Reekers invited Prof. Robert Morgan to design and implement an examination with a target date for the initial examination to be held at the CIRSE congress in Valencia in 2010. Some years previously, the Society of Interventional Radiology had introduced its own IR examination. Prof Robert Morgan held discussions with Prof. John Kaufman of the SIR regarding the SIR experience in setting up an examination and used the basis of these discussions to decide on the format of the proposed examination.

The examination would consist of written and oral components to be taken during the same day. Dr. Raman Uberoi was invited to appoint a small team to generate oral examination questions, and Prof. Patrick Haage was asked to select a team to produce multiple-choice questions to be used for the first examination. Prof Robert Morgan was appointed as the first EBIR Council Chair with Dr Raman Uberoi and Patrick Haage as deputy chairs. Everybody worked hard to finalise the oral and written examination questions to be ready for the scheduled time of the examination in September 2010.

At the same time, discussions held with both the European Union of Medical Specialists (UEMS) and the European Society of Radiology (ESR) resulted in both bodies granting their endorsement for the new EBIR examination. A requirement of this endorsement was that the examination would only be taken at completion of specialist training in diagnostic radiology.

Twenty senior IRs in CIRSE were invited to be examiners for the first examination and underwent examiner training on the morning of the examination. Twenty candidates sat the first EBIR examination, and sixteen candidates passed the examination. The examination was deemed a success and plans were implemented to hold the examination twice a year—in March on the European Congress of Radiology and in September on the day before the CIRSE congress.

At the same time as the development of the EBIR examination, in 2009 under the direction of Prof Anna-Maria Belli, CIRSE produced a Curriculum in Interventional Radiology to be used by IR trainees in Europe. This was the first curriculum in IR to be developed and the Syllabus that was derived from this document formed the basis of the creation of appropriate questions for the EBIR [3]. The second iteration of the CIRSE IR Curriculum was produced under the direction of Raman Uberoi in 2017.

The new EBIR examination was popular and applications to take the examination filled available capacity for each scheduled occasion. IRs throughout Europe could see the benefits of accreditation of their knowledge and skills to both help their career progression and to improve mobility around Europe and were keen to obtain this examination [4]. To address the increasing demand for questions, the small EBIR Council was modified to a larger EBIR Examination Board.

Although initially a European examination for European IRs, CIRSE expanded the eligibility criteria to IRs outside Europe. There was early interest to adopt the EBIR examination by the Australasian IR society IRSA. In 2014, CIRSE signed a memorandum of understanding (MOU) with IRSA and the Royal Australian and New Zealand College of Radiologists (RANZCR) to hold the EBIR examination once a year in Australasia. This significantly increased the workload and pressure on the EBIR Examination Board to produce an ever-larger bank of questions every year for the three separate sittings of the EBIR without re-using the same material. This required further expansion of question writers and committees to not only satisfy this demand but also expand the bank of examination questions for future use.

Evolution of the EBIR–UEMS–CESMA Accreditation

Although CIRSE and the EBIR Examination Board were confident that they had created a high-quality examination, it was evident from the start that the examination would need to evolve to achieve international recognition. Moreover, there are internationally accepted standards for the accreditation of examinations. The EBIR would need further develop to satisfy the necessary standards to achieve this accreditation and recognition both within Europe and worldwide. Over the subsequent years, the EBIR examination underwent constant improvement under the Direction of the successive EBIR Chairs (Prof Robert Morgan, Prof Klaus Hausegger, Prof Otto van Delden and Dr Raman Uberoi) to reach the high standard that it is today.

During his CIRSE Presidency, Prof Michael Lee recommended to the Executive Board that CIRSE appoint a professional educationalist with academic experience in the provision of scientific examinations. This was a major step forward, and the successful applicant, Dr Muirne Spooner, highlighted the challenges involved in progressing the EBIR to become an accredited quality examination. Dr Spooner also stated the need for change if CIRSE wanted the EBIR to develop to become the premier examination in the world for IRs.

The Examination Board embarked on a program of training all of the examiners and question (item) writers to improve the quality of the questions and examination techniques. This included learning the basis of exam theory for higher-level examinations in testing higher cognitive skill rather than just recall of facts, creating blueprints for the exam, understanding the concept of minimally competent candidates, and weighting examination questions using the Angoff method.

CIRSE now had a high-quality team of trained examiners; some of the best in the world. This led to a significant uplift in the overall quality for the examination, which resulted in a high level of satisfaction among EBIR candidates [5]. Moreover, CIRSE was now able to demonstrate that the society had created a world-class examination in interventional radiology.

The next step in the maturity of the EBIR examination was to seek approval of the examination from the accreditation body UEMS-CESMA (The European Union of Medical Specialists—Council for European Specialists Medical Assessment). CIRSE submitted the application to UEMS-CESMA for formal international recognition in April 2016. UEMS-CESMA accreditation involves a rigorous process of assessment of the examination processes. The EBIR Chair and Deputy Chair Otto van Delden and Raman Uberoi had to demonstrate the excellence of the EBIR examination process and show the team of assessors every aspect of the examination. These included the complex production cycle of creating questions by item writers, the review by senior editors, by the EBIR deputy chair and finally by the EBIR chair of the Examination Board before the final acceptance. The application process also required a description of the weighting process of the examination questions, an assessment of the examination process on the day itself, the process of setting the pass mark level, the communication of the results and the appeals process. This overall process took several months but thanks to the hard work of the entire team UEMS-CESMA approval was granted in March 2017 on the first application attempt. This was a major milestone for the EBIR examination [5].

Maturation—COVID, Digitisation and Increasing Capacity

The quality of the EBIR examination had significantly improved over the years since its inception. Examiners underwent comprehensive training through several multiple dedicated training sessions and the creation of comprehensive online training materials. However, it became evident that there remained a problem with the Viva (oral) component of the examination. Despite the use of standardised questions and answers, there was still significant variability between examiners. The examination had also become very popular with a gradual increase in the waiting list to sit the examination. At one time, the waiting list extended to more than two years, which was unacceptable.

Further research into examination science revealed evidence that the higher cognitive abilities that are normally assessed by oral examinations could also be tested adequately by using sophisticated and well-written multiple-choice questions. Moreover, these types of questions performed better than the traditional Viva examination in terms of reliability, validity and defensibility [6,7,8].

In 2019, the Executive Board took the decision to create a completely digitised examination and to replace the Viva component of the EBIR with new so-called F type questions [9]. This differed significantly from standard A type single-best-answer MCQs in being more complex containing a clinical scenario and sequentially linked four or five multiple-response questions. Each question would follow on from the previous question in a uniform manner, testing the same higher level of knowledge and clinical skills as the Viva examination.

Although there was initial scepticism among some of the more traditionalist examiners, all examiners were soon persuaded of the validity of this new question format. New training courses were developed to teach question writers to produce these more complex questions. The first sitting of the EBIR in this new examination format in Barcelona 2019 showed that the validity and reproducibility of the examination had improved substantially.

Dr Uberoi as chair of the EBIR board was tasked with the implementation of this new digitised examination which also removed the need to hold oral examinations with examiners and examinees physically in an examination room. This had the added benefit that the examination could be changed to a digital online examination. By removing the need for people to attend in person, it was now possible to break the linkage of the EBIR with CIRSE/ESR/IRSA congresses so that candidates could sit the examination anywhere in the world. The digital format also removed the limitation on the numbers of candidates, which would address and deal with the waiting list to sit the examination.

A large number of companies were sourced that might be able to provide the digital services required to provide the EBIR in a secure online format. This search for a provider coincided with the lockdowns produced by the COVID pandemic. The pandemic produced significant challenges in developing and delivering this new format because of the inability of personnel to meet face to face. However, the CIRSE office and EBIR committees responded well to the challenges and using the new ways of meeting online to make decisions produced the new electronic examination. The new examination was tested in mock examination format by members of the EBIR committee. To minimise risks of any issues occurring on the day of the first digital EBIR, all candidates were offered the opportunity to also sit a mock examination in the week before the formal examination to see how the new examination format works and to test their local IT environment.

The first completely virtual EBIR examination was delivered in January 2021. Dr Raman Uberoi (EBIR Committee Chair) and Dr Colin Nice (EBIR Committee Deputy Chair) observed anxiously online in case there were any major issues; CIRSE office staff were also available to deal with IT issues. Apart from some minor logging in issues, the examination proceeded without major incident and a new dawn of a virtual EBIR had commenced. Reassuringly, the EBIR statistics improved significantly and the waiting list disappeared soon after.

Soon after the implementation of the first electronic EBIR, work commenced to provide the EBIR in other languages apart from English. This work was assisted by our regional experts and translators, which started with the German language examination in 2021 and was closely followed by the Spanish sitting of the examination during 2022.

The Future

Having now reached the landmark of over a thousand EBIR holders worldwide, what are the next goals for the EBIR? While the examination format has matured beyond recognition from its first iteration back in 2010, there is a need to avoid complacency and to look at constantly improving the quality of the examination. The amount of questions in the question bank always requires increasing. It would be good to offer the examination in progressively more languages as countries mature in their development of IR service provision.

Many countries are already looking to create a specialty of IR to help improve the training, visibility, reach and ultimately the provision of IR services. The EBIR can be used by these countries to further their goal to achieving specialty status, in combination with the already established well regarded CIRSE IR curriculum and syllabus, which are being constantly updated, most recently in 2022 by the editorial team led by Dr Uberoi and supported by the EBIR exam team led by Maria Weren [10]].

Subject to development of MOUs with CIRSE to adopt the EBIR, it will be possible for nations to use the examination for their own purposes to accredit IR trainees without having to recreate the entire examination process themselves. This may be done by incurring minimal or no cost to their organisation, while at the same time having the credibility of a well-established internationally recognised examination.

Already countries such as Germany, Switzerland, Austria and the Netherlands have made the EBIR examination part of their assessment or revalidation process and Spain is looking to do the same. The EBIR has a bright future and has become a model for other professional examinations around the world. With 13 years since the delivery of the first EBIR examination, it will be interesting indeed to review the progress of the EBIR over the next 13 years.