Abstract
Aims
The ongoing COVID-19 pandemic has disrupted and delayed medical and surgical examinations where attendance is required in person. Our article aims to outline the validity of online assessment, the range of benefits to both candidate and assessor, and the challenges to its implementation. In addition, we propose pragmatic suggestions for its introduction into medical assessment.
Methods
We reviewed the literature concerning the present status of online medical and surgical assessment to establish the perceived benefits, limitations, and potential problems with this method of assessment.
Results
Global experience with online, remote virtual examination has been largely successful with many benefits conferred to the trainee, and both an economic and logistical advantage conferred to the assessor or organization. Advances in online examination software and remote proctoring are overcoming practical caveats including candidate authentication, cheating prevention, cybersecurity, and IT failure.
Conclusion
Virtual assessment provides benefits to both trainee and assessor in medical and surgical examinations and may also result in cost savings. Virtual assessment is likely to be increasingly used in the post-COVID world and we present recommendations for the continued adoption of virtual examination. It is, however, currently unable to completely replace clinical assessment of trainees.
Cite this article: Bone Jt Open 2021;2(2):111–118.
Take home message
The ongoing COVID-19 pandemic stalled medical and surgical examinations where attendance is required in person, and may have delayed career development for some young surgeons. The introduction of online assessment would circumvent the delay in assessment associated with the pandemic and similar events, and additionally provide many benefits both locally and globally if adopted as the ongoing standard.
Introduction
Medicine is a career of lifelong learning and assessment. Whether in core training, specialist training, or at a consultant level, continual assessment remains central to quality improvement in medicine. A typical Trauma and Orthopaedic (T&O) surgical speciality training programme in the UK exemplifies this. Core surgical training must be completed for two years, and an intercollegiate membership exam, i.e. MRCS, must be passed succesfully prior to entry into any surgical speciality training. Success in the exit exam following speciality training, i.e. FRCS (Tr & Orth) for Trauma and Orthopaedics, is mandatory for completion of specialist training.1 These exams consist of written, clinical, and viva voce assessment. During speciality training the trainee’s progress is continuously monitored in the annual reviews of competency progression (ARCPs), which include reviews of operative performance consisting of procedure-based assessments, and direct observation of procedure skills, along with nonoperative performance, including case-based discussions and clinical evaluation exercises.1 The Fellowship exam is a summative assessment of the clinical skills developed during training and assesses the trainee against the skills necessary to be at the level of a ‘Day 1 Consultant’ in the relevant surgical speciality.
While assessment has evolved over the past century, exams have continued to be administered in a traditional ‘exam hall’ manner.2 The ongoing COVID-19 pandemic has exposed flaws in this system, which poses rigid parameters of time, date, and location for candidates and assessors. All parts of both the MRCS and FRCS exams were postponed due to the COVID-19 pandemic until an alternative COVID-safe format could be devised. The Fellowship section 1 exams, which follow a single best answer (SBA) and extended matching item questions (EMQ) format, were postponed until examination centres were able to put in place COVID-safe measures. The section 2 exams, the clinical component which consists of clinical examination, and scenario- and patient-based viva voce examinations (vivas), were also postponed for the development of COVID-safe examinations and remote assessment options. Going forward, it is time for the surgical profession to adopt online remote assessment as an intrinsic part of both undergraduate and postgraduate medical training, both during and after the COVID-19 pandemic. In this article, we aim to assess the validity of summative online assessments for examinations outside of the hospital clinical environment, the advantages to both trainee and assessor, and the challenges to implementation through a review of the available literature. This will provide a background of evidence and validation for the implementation of online examinations, and may act as a checklist for future and continual implementation.
Methods
Literature searches were performed by three independent reviewers (MP, SS, JZ) to identify relevant articles in both medical education and computational technology. Medline and IEEE Xplore were searched using the following terms: ‘computer-based’, ‘online’, ‘virtual’, ‘remote’, ‘computer-based’, ‘medic*’, ‘surg*’, ‘Exam*’, ‘assessment’, ‘valid*’, ‘outcomes’, ‘advantages’, and ‘challenges’. Articles were included if they were related to the validity of summative assessment, the benefits of virtual assessments or the challenges to the implementation of virtual assessment.
This literature search identified 10,836 articles after removal of duplicates, while a further 74 articles were identified from other sources. After title and abstract screening 96 articles remained, and 49 full-text articles were included in this narrative review.
Results
Following the literature search, articles were analyzed and broadly grouped into different categories including 1) assessment methods, 2) trainee and trainer perspective, and 3) challenges to implementation of virtual examination, which are discussed below.
Computer-based assessment and validity
Prior to the COVID-19 pandemic, the majority of computer-based assessments (CBAs) had been trialled in medical schools rather than in postgraduate training, and had been formative in nature.3 The Ottawa consensus statement for good medical assessment identifies construct validity, reproducibility, equivalence, acceptability, feasibility, educational benefit, and timely feedback as key elements for successful medical assessment.4 Kane’s framework provides a construct validity argument in a step-by-step framework and represents the most recent evolution in validity theory (Figure 1).5
Fig. 1
Kane’s framework for interpreting the validity argument of an assessment, which it achieves by emphasizing scoring, generalization, extrapolation, and implication arguments.
MCQ and EMQ assessment
The UK and Ireland Royal Colleges of Surgeons (RCS) use MCQ, SBA, and EMQ in the written components of their examinations. Computer-based SBA and EMQ are valid by proxy, as they are currently conducted at CBA centres in the written section 1 of FRCS examinations.6 Studies in higher education confirm that similar exam scores are awarded with both paper- and computer-based MCQ assessment, displaying valid scoring and generalization.7 MCQs also display valid extrapolation and implication, as they correlate well with other measures of clinical performance.8
Viva voce examinations
Viva voce examinations are considered a valid form of assessment, however scoring and generalization may follow a subjective, examiner-dependent process.9 To ensure that the marking in these examinations remains objective, the Intercollegiate Fellowship examiners (FRCS) are trained and provided with an objective scoring system. For computer-based vivas to be valid, interpretation of subjective features should not change with the use of telecommunication. Factors influencing subjective scoring include personal qualities of fluency and creativity, nervousness, incoherence, cultural differences, values, and language factors, including discourse style, which may be perceived differently over video.9 The system should therefore provide sufficient audiovisual quality for the accurate interpretation of language and non-verbal communication, and the examination should be carried out in a controlled environment such that personal qualities are not perceived differently to a traditional interview setting. Sufficient technical quality is available in medical telecommunication. Current technology including smartphones and tablets have been shown to provide accurate image-based teleconsultations between physicians.10,11 Virtual patient consultations additionally report high efficacy across multiple specialties, equalling or improving outcomes compared to face-to-face consultation.12
Clinical ability
The validity of virtual assessments for clinical ability, in both technical and non-technical skills, must be viewed with more caution. Technical skills can be broadly grouped into practical skills and patient assessment. Virtual patient assessments intuitively cannot be valid if physical examination is required, as they cannot be extrapolated to a clinical environment. While a recent literature review highlighted the paucity of evidence for the validity of online case-based simulations in assessment of clinical ability, evidence is emerging for the validity and efficacy of online assessment in clinical reasoning and case presentation given standardized patient encounters, and the use of virtual patients for history taking.13-15 The use of such virtual patient encounters may precede the online adaptation of patient-based objective structured clinical examinations (OSCEs) assessing clinical ability, especially in a context where examination-based findings can be supplemented. This is supported by preliminary evidence validating the use of ‘teleOSCEs’ assessing clinical skills.16
Non-technical skills are wide-ranging and include a core set of communication skills. In this area current computer-based assessments utilize recorded consultations which are later analyzed (Objective Structured Video Examination format), and virtual assessment of active communication skills is now reported.17 It can be hypothesized that standardized online simulated patient encounters may be validated for postgraduate exams. Evidence demonstrates that online teaching of communication skills is as effective as traditional learning in an undergraduate setting, suggesting valid scoring and generalization.18 Virtual patient encounters also appear to validly extrapolate, provoking similar emotional responses to real standardized encounters in communication skills assessment.19 Such simulation has high efficacy for undergraduate communication skills assessment, and evidence exists to suggest that screen-to-screen communication has no difference in patient perceived information exchange, interpersonal relationship building, and shared decision-making versus face-to-face communication.20-22
Trainee and trainer perspective
Virtual assessment will only be adopted if accessible and advantageous to both candidates and examiners.
Accessibility
Online learning and assessment are associated with the ‘any time anywhere’ mantra, and can be taken from anywhere around the world. On a global scale, this could promote professional accreditation and quality improvement, reducing ‘brain drain’ in countries without formal postgraduate training or standardized entry and exit exams.23 In Malawi, online surgical sciences MSc scholarships were awarded to 24 surgical trainees through the RCS Edinburgh.24 Not only did the scholarship candidates score similarly to the cohort average, but they also reported improved patient safety and decision-making skills following completion of the course.
Cost saving
Virtual exams may enable savings for both the assessment body and trainee. One may estimate a significant reduction to cost without the need for exam hall hire, invigilator and examiner employment, travel expenses, exam paper printing, and digitization and marking of exam papers.25 Through a top-down market sizing approach applied to the RCS of England annual report and accounts for the year ending 30 June 2019, we provide an estimated saving for assessment of T&O surgeons by the RCS of England of between GBP £52,000 and £157,000 (Figure 2).26 Trainees will also incur savings related to travel and accommodation.
Fig. 2
Cost estimate of savings provided by changing to an online examination system. The most conservative estimate splits to the left, while the best-case estimate splits to the right. Savings were estimated through market sizing strategies and the Royal College of Surgeons of England Annual report and accounts for the year ending 30 June 2019. T&O, trauma and orthopaedics.
Efficient administration
A reduction in administration time, including the use of digital marking sheets for vivas, along with automated marking for MCQ, SBA, EMQ, and some SAQ formats, may enable trainees to receive results sooner, if not the same day.2 Using a virtual assessment, the time taken to mark over 14,000 very short answer questions (VSAQs) was reduced from 70 hours to five hours in an undergraduate pathology examination.27 Virtual assessment would also remove the handwriting variability and the halo effect associated with these questions, where examiners associate poor handwriting with lower performance.28 Virtual assessment may therefore increase the feasibility of traditionally labour intensive examination methods such as SAQs.
Challenges to implementation
Practical caveats
Although remote assessments are attractive, variation between candidates’ home environments remains a concern. Noisy, busy households are not conducive to examination. Mitigating such factors may be possible by continuing to provide students with the opportunity to undertake exams at local CBA centres rather than remotely at home, and by requiring a standardized virtual background for viva examination.
Variations in household internet access should also be considered, as unreliable bandwidth may disadvantage candidates. It is possible to deliver online exams ‘offline’ or with adjusted time constraints to counter unreliable or insufficient bandwidth. Services offer downloadable exams and the capability for seamless offline examination with connectivity loss. For stratification of candidates to pre-downloaded remote examinations, bandwidth and connectivity should be registered during the enrolment process. Alternatively, an unexpected drop in bandwidth during a fully online examination can be compensated with a Time Adaptive Mobile E-Exam (TAMEx) format.29 This monitors the examinee’s device, and if the device loses connectivity, the total time of this occurrence is added to the remaining examination time.
Assessment of students across multiple time zones is disadvantageous for some candidates if a uniform examination start time is used, as international candidates take the examination at irregular hours. The optimal examination start time for cognitive functioning, which disadvantages the least number of chronotypes, is around noon.30 A standardized start time in each time zone would therefore be optimal.
In order for time zone varied start times to be enabled without an increased risk of cheating and plagiarism, a different set of questions is required for each exam. This is possible given a large question bank from which an examination software randomly selects a generalisable question set.31 Question bank development may also enable annual integrated continual formative assessment and feedback, improving training quality.32,33 Question bank design is labour-intensive. Importing or generating a question bank requires checking, editing, and tagging for all questions and exam blueprinting.31,34 Additionally, it has been reported that 90.8% of questions in online medical assessments have an associated image.35 If this is representative, administrative costs may increase in relation to data protection, copyright, and server storage.25 It is therefore essential for costings to be performed during procurement.
Ongoing IT support must also be available during exams, along with fail-safe measures to continuously back up progress in case of a systems failure.25 This agrees with university students’ attitudes, whose primary concerns regarding online examinations were data security and data loss.36
The overall wellbeing and performance of candidates is important. Transition of written exams to CBA has received favourable student responses, with no difference in results versus written exams.7 A proportion of candidates do, however, believe that IT skills enable faster and easier exam completion. Additionally, a survey of CBA using individuals’ own devices found IT compatibility to be an issue.37 This would warrant a priori instruction and troubleshooting to reduce anxiety and test day errors. It is also known that viva voce examination elicits higher anxiety than the written equivalent, and may not give an accurate indication of future performance.38 To reduce anxiety, trainees may have an initial group brief in an online conference setting, and structured viva voce (SVV) may be used rather than traditional viva voce (TVV). SVV is a valid assessment method, and is potentially a better discriminator than TTV.39 Additionally, the majority of medical students trialled with SVV reported a preference for this over TVV.
Exam security
Cheating is a primary concern, as it may invalidate assessments, and may occur in many forms. Broadly, an individual may impersonate the candidate, or the candidate may access prohibited materials during the assessment. Such behaviours are overcome or minimized in an examination centre where proctors monitor behaviour. New mitigating strategies are essential in order to reduce cheating during virtual assessments.
Authentication
Static authentication involves confirmation of the candidate’s identity at the beginning of the examination, while in continuous authentication the candidate’s identity is repeatedly confirmed throughout the examination. Authentication can be through either a knowledge factor (for example username and password combinations), an ownership factor such as an access card, or a unique biometric factor.40 Knowledge and ownership factors are not highly secure and biometric factors tend to be favoured.
Virtual exams may employ a live proctoring system with ID comparison made by an invigilator who monitors candidates remotely. Automated proctoring software is a viable alternative, where biometrics are utilized for live authentication, including: keystroke recognition, facial recognition, fingerprint recognition, and voiceprint or iris scanning.40 These measures require candidate registration prior to examination.
Fingerprint recognition is typically used for static authentication. It requires a universal serial bus fingerprint scanner and is commonly used in conjunction with other biometric measures.41 Keystroke analysis can provide static authentication through the typing of a standardised text, and is able to continuously authenticate if typing is required during the exam.42,43 Keystroke dynamics act as an individual signature, and current analysis using Artificial Neural Networks can extract user patterns with 80% accuracy.43 Static authentication using facial recognition has been possible for almost forty years but requires sufficient camera quality.44 Continuous facial recognition poses the additional problems of movement, angling, and lighting changes. Recently, a continuous authentication algorithm has been developed with 100% accuracy based on a three-minute training video.45 Current models for integrated biometric authentication utilize these three methods.
Other biometric technologies may be integrated in the future. Theoretical models for voice recognition have been proposed and validated, but not widely studied, and carry practical implications for continuous authentication in a silent examination atmosphere.46 Similarly, iris scanning has been proposed, but requires an infrared camera which is not readily accessible.47
Cheating detection
Cheating detection is a continuous process. In a live proctored online exam, the individual will be directly observed at their workspace by a proctor. This carries limitations, as the proctor receives a narrow field of view with typical camera hardware. Additionally, it is costly to acquire staffing. Therefore, an automated proctor system may be used where images and video are flagged for proctor review if suspicious activity is detected by the system.
Eye tracking is an automated proctoring technique which flags viewing of offscreen content, or inappropriate areas of the screen. Initial calibration is required through recording candidate pupil position while they focus on particular onscreen coordinates. In a sample of 30 adults an eye tracking system was able to detect cheating with accuracy over 97%.48 Further theoretical advances to continuous cheating detection include thermal imaging, whereby the intention to cheat may manifest by an increase in the candidate's body temperature.49 Such technology requires validation and benefit to be shown over current biometrics, and provides an insight into innovative future cheating detection solutions.
Furthermore, test and software design can reduce cheating behaviour. Multiple steps have been suggsted by Cluskey et al50 to prevent cheating in un-proctored online examination, with the aim of increasing practical difficulties associated with cheating behaviour (Table I). Finally, a data science approach may be considered, requiring analysis of answer concordance in order to flag excessively similar examination scripts for proctor review.51
Table I.
Question design strategies to mitigate cheating behaviour. Adapted from Cluskey et al.50
| All examinees should sit an exam at one particular time slot |
| The questions should be accessible for a short period of time |
| The sequence of questions should be randomised |
| Questions should be presented one at a time, with no going back |
| Question difficulty should be such that the questions can just be completed without much excess time |
| The exam should only be accessible once, with no resetting unless proof of technical failure exists |
| The exam questions should be frequently changed |
| Students should be required to use a restrictive ‘locked-in’ browser |
Cybersecurity
Cyberattacks may be orchestrated by candidates, or a third party hoping to disrupt the exam for malicious reasons. In the context of virtual medical examination, cyber security measures are increasingly important if patient cases are discussed or presented, to ensure the maintenance of patient anonymity. Software attacks can be mapped using attack-tree frameworks. Rosmansyah et al52 describe such a framework for cyberattacks on virtual exams. Each attack can be carried out in multiple ways for which there are specific cybersecurity countermeasures. For instance, a Denial of Service attack is a web application attack which aims to overload a server by sending large amounts of data. This can be mitigated by incorporation of dynamic path identifiers, fuzzy reasoning-based windows firewall, or channel hopping.53-55
Discussion
Despite a paucity of literature describing summative online assessment, we have outlined the validity of online written examinations, and of online vivas. There is also a developing argument for the validity of virtual patient encounters to assess clinical skills, which are now being piloted at several institutions. Furthermore, we outlined the benefits of the continued adoption of a virtual examination system, along with the practical issues its adoption faces. Given the potential issues, we have outlined our recommendations for online assessment implementation in Table II.
Table II.
Challenges to the implementation of online assessment, and recommendations for the introduction of online assessment.
| Challenges | Recommendations |
|---|---|
| Maintaining exam integrity across multiple time zones | Use a standardized examination start time on the same date to disadvantage as few candidates as possible. Utilise question banks, automated exam generation from blueprinting, order randomization, and restrictive browsers to prevent cheating behaviour. |
| Mitigating the impact of home environments and subjective influences | Require candidates to have a standard virtual background or plain wall for viva voce examinations, and maintain a dress code. Use SVV formats to further ensure objectivity. Enable candidates to book education facilities for examinations where a home environment is not suitable. |
| Poor candidate internet access | Provide a system which provides seamless offline-online capability. This may be through pre-downloading and synchronisation or other means. Prior to viva voce examinations, ensure candidates provide evidence of compatible bandwidth. |
| Question bank development | Seek a system which allows automated importation of existing questions from a word format or similar, and allocate administration time to blueprinting and tagging of questions. Perform a costing, and ensure a long-term plan or contract is in place if an external system is used, such that data is not lost at the end of a contract. |
| Systems failure and compatibility | Utilize a system which has continuous back-up of progress, or one which utilizes hard-disk space to enable back-up in the event of internet loss. Ensure candidates have had prior instruction and sample questions to prevent compatibility issues, or user-related failures. |
| Cybersecurity threats | Ensure systems employ a reputable internet security system. We recommend consulting an expert or external company due to the complexities involved. |
| Questions concerning the validity of online clinical assessment using virtual patients and virtual patient encounters | Continue to record and present data to explicitly establish the validity of summative online examinations in the surgical field. Furthermore, in the near future, record data regarding outlier cases who are stratified to remote examination and use this to investigate the validity of virtual patient encounters in surgical examination. |
| Impersonation | Ensure static authentication at the beginning of the exam. This should be at least bimodal, requiring a knowledge factor and/or ownership factor, as well as a biometric factor. |
| Cheating detection | Employ artificial intelligence-driven systems which utilize multiple cheating detection methods to assess the risk of cheating and use statistical analysis to flag excessively similar scripts. Prevent cheating behaviour through exam design, and application lock-in. Produce and keep a webcam recording of the candidate for manual review. |
-
SVV, structured viva voce
Prior to the COVID-19 outbreak, only a small number of medical institutions utilized online platforms. Universities had begun to implement online assessment, including Imperial College London and the University of Edinburgh, which had implemented summative MCQ, EMQ and VSAQ into their undergraduate courses with good outcomes.27,31 The implementation of such systems required question bank importation, and blueprinting of exams. This system enabled a fourteen-fold increase in the speed of marking, along with the integration of previously unused VSAQs. Similarly, a number of Royal Colleges, including the governing bodies for paediatrics, general practice and radiology in England, had implemented online examination systems. These systems offered the ability to blueprint the curriculum, import existing questions from microsoft word, and provide fully online or offline capability provided that Wi-Fi synchronization had occurred. These systems concurrently enabled automated set up of the exam from question banks, and instant marking and feedback for those taking the exam. They were, however, still conducting exams at a centralized examination centre.
Since the introduction of COVID-19 restrictions, we have seen a movement from the Royal Colleges to provide remote examination. This includes the Royal College of Surgeons, who have provided a fully remote online MRCS part A examination. This utilized static authentication in a two-fold process, using a username and password combination, and subsequently biometric facial recognition.56 Remote proctoring was additionally employed, with an artificial intelligence (AI) system tracking on-screen movements, gaze, and audio recording, with second tier human review guided by the AI flagging of behavioural anomalies. This type of online proctoring and authentication is offered by numerous companies. Additional features currently available include locking of the screen, and blacklisting of applications to reduce cheating behaviour. Systems currently available provide a high level of accuracy for remote proctoring, of approximately 99.5% in flagging behavioural abnormalities, and provide a high level of automation, with approximately 7% of candidates being manually reviewed. Given the current capability in delivering online written examination and the success of MRCS part A virtual examinations, the authors would suggest provision of remote virtual assessment for Intercollegiate Fellowship examinations including FRCS (Tr&Orth) section 1 in the future. Additionally, the data from such examinations may be used to add to the small body of literature which exists surrounding summative post-graduate virtual examination within medical professions.
While there has been good uptake of virtual online written examination, remote patient- and case-based vivas were not the default scenario-based structured interviews in the FRCS (Tr & Orth) section 2 unless absolutely required, and remote OSCEs were not used in the Oct 2020 MRCS part B. These assessment methods have some evidence promoting their validity in virtual examination, with the exception of patient physical examination. However, for the November 2020 sitting of FRCS (Tr & Orth) clinical examination, measures were taken to eliminate direct interaction with patients. The authors would promote the uptake of virtual case-based and patient-based vivas in the FRCS (Tr & Orth) section 2, and the subsequent utilization of these data to further validate virtual vivas and OSCEs. Furthermore, it may be speculated that the utilization of virtual and augmented reality may enable remote virtual clinical examination in a merged reality space, which has been trialled for postoperative follow-up, or the examination of a completely virtual patient, similar to anatomical assessments trialled in dentistry examinations.57,58
In conclusion, online assessment provides numerous benefits to both trainee and assessor in the field of written and viva voce examinations, while retaining validity, but there are challenges underlying its implementation. Online assessment is not well suited to clinical assessment involving patient examination. However, the paradigm shift in the current working environment as a result of the COVID-19 pandemic has indeed provided us the opportunity to adopt and validate innovative ways of assessment in medical training. Virtual assessment is providing a path forward to overcome the obstacle posed by the current pandemic, which may in fact be the impetus for long-term change.
References
1. No authors listed . Specialist training in trauma and orthopaedics . 2020 . https://www.iscp.ac.uk/static/public/syllabus/syllabus_to_2017.pdf (date last accessed 12 October 2020 ). Google Scholar
2. Marston A . The final surgical examination: then and now . Bulletin . 2011 ; 93 ( 3 ): 86 – 88 . Google Scholar
3. Al-Amri S , Ali Z . Systematic review of computer based assessments in medical education . Saudi J Med Med Sci . 2016 ; 4 ( 2 ): 79 – 88 . Crossref PubMed Google Scholar
4. Norcini J , Anderson B , Bollela V , et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 conference . Med Teach . 2011 ; 33 ( 3 ): 206 – 214 . Crossref PubMed Google Scholar
5. Cook DA , Brydges R , Ginsburg S , Hatala R . A contemporary approach to validity arguments: a practical guide to Kane's framework . Med Educ . 2015 ; 49 ( 6 ): 560 – 575 . Crossref PubMed Google Scholar
6. No authors listed . England Royal College of Surgeons of international fellowship examination (JSCFE) . 2020 . https://www.rcseng.ac.uk/education-and-exams/exams/search/international-fellowship-examination-jscfe/ (date last accessed 23 May 2020 ). Google Scholar
7. Boevé AJ , Meijer RR , Albers CJ , Beetsma Y , Bosker RJ . Introducing computer-based testing in high-stakes exams in higher education: results of a field experiment . PLoS One . 2015 ; 10 ( 12 ): e0143616 – 13 . Crossref PubMed Google Scholar
8. Fallatah HI , Tekian A , Park YS , Al Shawa L , Al SL . The validity and reliability of the sixth-year internal medical examination administered at the King Abdulaziz University medical college . BMC Med Educ . 2015 ; 15 ( 1 ): 1 – 6 . Crossref PubMed Google Scholar
9. Iqbal IZ , Naqvi S , Abeysundara L , Narula AA . The value of oral assessments: a review . Bulletin . 2010 ; 92 ( 7 ): 1 – 6 . Google Scholar
10. Boissin C , Blom L , Wallis L , Laflamme L . Image-Based teleconsultation using smartphones or tablets: qualitative assessment of medical experts . Emerg Med J . 2017 ; 34 ( 2 ): 95 – 99 . Crossref PubMed Google Scholar
11. Deldar K , Bahaadinbeigy K , Tara SM . Teleconsultation and clinical decision making: a systematic review . Acta Inform Med . 2016 ; 24 ( 4 ): 286 – 292 . Crossref PubMed Google Scholar
12. Greenhalgh T , Vijayaraghavan S , Wherton J , et al. Virtual online consultations: advantages and limitations (vocal) study . BMJ Open . 2016 ; 6 ( 1 ): e009388 – 13 . Crossref PubMed Google Scholar
13. Mooney CJ , Peyre SE , Clark NS , Nofziger AC . Rapid transition to online assessment: practical steps and unanticipated advantages . Med Educ . 2020 ; 54 ( 9 ): 857 – 858 . Crossref PubMed Google Scholar
14. Kononowicz AA , Woodham LA , Edelbring S , et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration . J Med Internet Res . 2019 ; 21 ( 7 ): e14676 – 20 . Crossref PubMed Google Scholar
15. Ward RC , Muckle TJ , Kremer MJ , Krogh MA . Computer-Based case simulations for assessment in health care: a literature review of validity evidence . Eval Health Prof . 2019 ; 42 ( 1 ): 82 – 102 . Crossref PubMed Google Scholar
16. Lara S , Foster CW , Hawks M , Montgomery M . Remote assessment of clinical skills during COVID-19: a virtual, High-Stakes, Summative pediatric objective structured clinical examination . Acad Pediatr . 2020 ; 20 ( 6 ): 760 – 761 . Crossref PubMed Google Scholar
17. Hulsman RL , Mollema ED , Hoos AM , de Haes JCJM , Donnison-Speijer JD . Assessment of medical communication skills by computer: assessment method and student experiences . Med Educ . 2004 ; 38 ( 8 ): 813 – 824 . Crossref PubMed Google Scholar
18. Kyaw BM , Posadzki P , Paddock S , et al. Effectiveness of digital education on communication skills among medical students: systematic review and meta-analysis by the digital health education collaboration . J Med Internet Res . 2019 ; 21 ( 8 ): e12967 . Crossref PubMed Google Scholar
19. O'Rourke SR , Branford KR , Brooks TL , et al. The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study . Teach Learn Med . 2020 ; 32 ( 2 ): 139 – 149 . Crossref PubMed Google Scholar
20. Guiton G , Hodgson CS , Delandshere G , Wilkerson L . Communication skills in standardized-patient assessment of final-year medical students: a psychometric study . Adv Health Sci Educ Theory Pract . 2004 ; 9 ( 3 ): 179 – 187 . Crossref PubMed Google Scholar
21. Jabeen D . Use of simulated patients for assessment of communication skills in undergraduate medical education in obstetrics and gynaecology . J Coll Physicians Surg Pak . 2013 ; 23 ( 1 ): 16 – 19 . PubMed Google Scholar
22. Tates K , Antheunis ML , Kanters S , Nieboer TE , Gerritse MB . The effect of Screen-to-Screen versus face-to-face consultation on doctor-patient communication: an experimental study with simulated patients . J Med Internet Res . 2017 ; 19 ( 12 ): e421 – e421 . Crossref PubMed Google Scholar
23. Muula AS . Successes and challenges of Malawi's only medical school . Croat Med J . 2009 ; 50 ( 2 ): 189 – 191 . Crossref PubMed Google Scholar
24. Smith PJW , Garden OJ , Wigmore SJ , Borgstein E , Dewhurst D , et al. The effectiveness of an online, distance-learning Master’s in Surgical Sciences programme in Malawi . Afr J Health Prof Educ . 2018 ; 10 ( 3 ): 159 – 165 . Google Scholar
25. Kuikka M , Kitola M , Laakso M-J . Challenges when introducing electronic exam . Research in Learning Technology . 2014 ; 22 : 22817 . Google Scholar
26. No authors listed . Annual report and accounts for the year ended 30 June . 2019 . https://register-of-charities.charitycommission.gov.uk/charity-search?p_p_id=uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet&p_p_lifecycle=2&p_p_state=maximized&p_p_mode=view&p_p_resource_id=%2Faccountsresource&p_p_cacheability=cacheLevelPage&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_fileName=0000212808_AC_20190630_E_C.pdf&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_objectiveId=A10229642&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_priv_r_p_mvcRenderCommandName=%2Faccounts-and-annual-returns&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_priv_r_p_organisationNumber=212808 (date last accessed 16 October 2020 ). Google Scholar
27. Sam AH , Peleva E , Fung CY , et al. Very short answer questions: a novel approach to Summative assessments in pathology . Adv Med Educ Pract . 2019 ; 10 : 943 – 948 . Crossref PubMed Google Scholar
28. Nisbett RE , Wilson TD , DeCamp Wilson T . The halo effect: evidence for unconscious alteration of judgments . J Pers Soc Psychol . 1977 ; 35 ( 4 ): 250 – 256 . Google Scholar
29. Sukadarmika G , Hartati RS , Linawati L , et al. Proposed model for e-exam availability in WLAN environment . 2016 International Conference on smart green technology in electrical and information systems (ICSGTEIS . Denpasar : Udayana University , 2016 : 89 – 92 . Google Scholar
30. Evans MDR , Kelley P , Kelley J . Identifying the best times for cognitive functioning using new methods: matching university times to undergraduate chronotypes . Front Hum Neurosci . 2017 ; 11 : 188 . Crossref PubMed Google Scholar
31. Kluth D , Hill J , De'Ath P , Jaap A . Introduction of a new online platform for formative and summative assessments in the MBChB programme . 2020 . http://www.docs.hss.ed.ac.uk/iad/Learning_teaching/Academic_teaching/Events/LT_Conference/2018/s1/1C2_KluthD_LTConf2018.pdf (date last accessed 15 June 2020 ). Google Scholar
32. Sharma S , Sharma V , Sharma M , Awasthi B , Chaudhary S . Formative assessment in postgraduate medical education - Perceptions of students and teachers . Int J Appl Basic Med Res . 2015 ; 5 ( Suppl 1 ): 66 – 70 . Crossref PubMed Google Scholar
33. Arja SB , Arja SB , Ramey J , et al. Logistics behind the OSCE in undergraduate medical education and an experience at a Caribbean medical school . MedEdPublish . 2018 ; 7 ( 4 ). Google Scholar
34. Whitelock DM , Brasher A . Developing a Roadmap for e-Assessment: Which Way Now? Danson M . ed . Proceedings of the 10th CAA international computer assisted assessment conference . Loughborough : Loughborough University , 2006 : 487 – 501 . Google Scholar
35. Egarter S , Mutschler A , Tekian A , Norcini J , Brass K . Medical assessment in the age of digitalisation . BMC Med Educ . 2020 ; 20 ( 1 ): 101 . Crossref PubMed Google Scholar
36. Pagram J , Cooper M , Jin H , Campbell A . Tales from the exam room: Trialing an e-exam system for computer education and design and technology students . Educ Sci . 2018 ; 8 ( 4 ): 188 . Google Scholar
37. Hillier M . To type or handwrite: student’s experience across six e-Exam trials . In : Reiners T , von Konsky BR , Gibson D . eds . Globally connected, digitally enabled. Proceedings ascilite 2015 . Perth : Perth University , 2016 : 143 – 154 . Google Scholar
38. Arndt CB , Guly UM , McManus IC . Preclinical anxiety: the stress associated with a VIVA voce examination . Med Educ . 1986 ; 20 ( 4 ): 274 – 280 . Crossref PubMed Google Scholar
39. Imran M , Doshi C , Kharadi D . Structured and unstructured VIVA voce assessment: a double-blind, randomized, comparative evaluation of medical students . Int J Health Sci . 2019 ; 13 ( 2 ): 3 – 9 . PubMed Google Scholar
40. Sabbah Y , Saroit I , Kotb A . Synchronous authentication with bimodal biometrics for e-assessment: A theoretical model . 2012 6th International Conference on sciences of electronics, technologies of information and telecommunications (SETIT . New York : IEEE , 2012 : 139 – 145 . Google Scholar
41. Alotaibi SJ . Using biometrics authentication via fingerprint recognition in e-exams in e-learning environment . The 4th Saudi International Conference, Friday 30 and Saturday 31 July 2010 . Manchester : The University of Manchester , 2010 . Google Scholar
42. Flior E , Kowalski K . Continuous biometric user authentication in online examinations . 2010 seventh International Conference on information technology: new generations . New York : IEEE , 2010:4 : 88 – 492 . Google Scholar
43. Abisado MB , Gerardo BD , Fajardo AC . Towards Keystroke Analysis using Neural Network for Multi-Factor Authentication of Learner Recognition in On-Line Examination . Manila International Conference on “Trends in Engineering and Technology” . Manila : IAETR , 2017 : 71 – 74 . Google Scholar
44. Sirovich L , Kirby M . Low-Dimensional procedure for the characterization of human faces . J Opt Soc Am A . 1987 ; 4 ( 3 ): 519 – 524 . Crossref PubMed Google Scholar
45. Sukmandhani AA , Sutedja I . Face Recognition Method for Online Exams . Proceedings of 2019 International Conference on information management and technology . New : IEEE , 2019 : 175 – 179 . Google Scholar
46. Rudrapal D , Das S , Debbarma S , Kar N , Debbarma N , et al. Voice recognition and authentication as a proficient biometric tool and its application in online exam for p.h people . Int J Comput Appl . 2012 ; 39 ( 12 ): 6 – 12 . Google Scholar
47. Curran J , Curran K . Biometric Authentication Techniques in Online Learning Environments Kumar AV . ed . Biometric authentication in online learning environments . Hershey : IGI Global , 2019 : 266 – 278 . Google Scholar
48. Bawarith R , Basuhail A , Fattouh A . Gamalel-Din S. E-exam Cheating Detection System . Int J Adv Comput Sci Appl . 2017 ; 8 : 176 – 181 . Google Scholar
49. Alfaries A , Mengash H , Yasar A , Shakshuki E . Advances in data science, cyber security and it applications . Cham : Springer Nature , 2019 . Google Scholar
50. Cluskey GR , Ehlen CR , Raiborn MH . Thwarting online exam cheating without proctor supervision . J Acad Bus Ethics . 2011 ; 4 : 1 – 8 . Google Scholar
51. Haney WM , Clarke MJ , Tests Con . Prevalence, Detection, and Implications for Online Testing . In : Anderman EM , Murdock TB . Psychology of academic cheating . London : Elsevier , 2007 : 255 – 287 . Google Scholar
52. Rosmansyah Y , Ritonga MH , Hardi AB . An Attack-Defence Tree of e-Exam System. . iJET . 2019 ; 14 : 251 – 259 . Google Scholar
53. Luo H , Chen Z , Li J , Vasilakos A V . Preventing distributed Denial-of-Service flooding attacks with dynamic path identifiers . IEEE Trans Inf Forensics Secur . 2017 . Google Scholar
54. Naik N , Jenkins P , Cooke R , et al. Augmented windows fuzzy firewall for preventing denial of service attack . Proceedings of the 2017 IEEE International Conference on fuzzy systems (FUZZ-IEEE . Red Hook : Curran Associates , 2017 : 1 – 6 . Google Scholar
55. Djuraev S , Choi J-G , Sohn K-S , Nam SY . Channel hopping scheme to mitigate jamming attacks in wireless LANs . EURASIP J Wirel Commun Netw . 2017 ; 2017 ( 1 ). Google Scholar
56. No authors listed . The Examplify testing application from ExamSoft . 2020 . https://examsoft.com/resources/examplify-testing-application-examsoft (date last accessed 20 September 2020 ). Google Scholar
57. Ponce BA , Brabston EW , Shin Zu , et al. Telemedicine with mobile devices and augmented reality for early postoperative care . Annu Int Conf IEEE Eng Med Biol Soc . 2016 ; 2016 : 4411 – 4414 . Crossref PubMed Google Scholar
58. Kim-Berman H , Karl E , Sherbel J , Sytek L , Ramaswamy V . Validity and user experience in an augmented reality virtual tooth identification test . J Dent Educ . 2019 ; 83 ( 11 ): 1345 – 1352 . Crossref PubMed Google Scholar
Author contributions
M. Pettit: Conducted the literature search, Wrote the manuscript.
S. Shukla: Designed the study, Conducted the literature search, Wrote the manuscript.
J. Zhang: Conducted the literature search, Wrote the manuscript.
K. H. Sunil Kumar: Conceptualized and designed the study, Conducted the literature search, Wrote the manuscript.
V. Khanduja: Conceptualized and designed the study, Revised the manuscript.
Funding statement
No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.
Follow K. H. Sunil Kumar @kumarkhs
Follow Vikas Khanduja @CambridgeHipDoc
© 2021 Author(s) et al. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/
