The results of this qualitative study will be presented into three main sections as following:
Step 1: The thematic analysis of the focus group contributions
A number of themes emerged from analyzing participant contributions in the focus group as follows:
1.1 The role and attributes of the leaders for successful online learning
Most of the participants agreed that leaders should set rules and be decisive especially in time of crisis. Encouraging teamwork and including everyone are the cornerstones for achieving goals. Another important attribute is flexibility and how to cope with different personalities and work requirements. Visionary leaders who predict the future and act proactively to suggest solutions, crisis management, support and sustainability were added as important roles of the leaders. One of the participants added that ‘effective leadership requires receiving feedback from significant stakeholders including the students and they must be seen and heard for highly relevant feedback’. Proper communication can help in overcoming several obstacles as one participant mentioned ‘Leaders should provide early and continuous student and faculty orientation about milestone in the learning experience.’
1.2 Resources needed to conduct a successful online learning.
There was a collective agreement among participants on the importance of resource allocation for online learning. These resources include user friendly learning management systems (LMS), internet services, ready-made or self-generated digital tools and equipment to support the online learning. Moreover, an important resource are the personnel involved in the online learning. They also added the importance of having a supportive IT team and conducting a well-organized faculty development program.
1.3 Institutional bylaws
The participants' responses varied regarding the modifications of the bylaws. Some of the participants highlighted the importance of revising the institutional mission and vision. Others confirmed that the mission and vision will not be changed for the use of a new mode of learning. Important points should be added to the bylaws, as per participants, including the ratio of online to face-to-face learning, the assessment plan and attendance ratio in addition to a clear description of online learning competencies and required staff and faculty qualifications.
1.4 Key points to consider while shifting face to face programs into online format.
Participants agreed that institutions have to start with development of the skills and knowledge of their faculty. Faculty should understand the difference between face to face and online learning. The role of faculty, the nature of the content and instructional methods will change with this shift. Also, faculty and students needed to develop some essential skills to cope with this transformation. Materials should be simplified, interactive and motivating. Proper platforms or LMSs are important to conduct successful online learning since it is the ‘vehicle for all the activities. Use of all the available tools to engage students and create interactive activities such as whiteboard, share-screen, assignments, e-portfolio, online quizzes, online discussion forums. Still there are difficulties with conducting practical and clinical sessions. Virtual reality and simulation may help in this point, but funding will remain the main obstacle.
Guidance is required in a comprehensive way in online learning to guide the students and avoid isolation. It is important to provide the students with different alternatives that facilitate and ensure their engagement and participation even with poor internet connections particularly in rural areas. In online learning, mentorship and coaching are needed even more than in face-to-face learning. Finally, finding alternatives for clinical and practical skills teaching is a big challenge. Formative assessment and feedback are also critical points to consider.
1.5 Creating a motivating/engaging environment in online learning.
Student engagement was a major problem that faced most of the universities last year as mentioned by most of the participants. Therefore, participants highlighted some important practices that should be considered while implementing online learning. One participant added the need for the use of more formative assessment to keep the students engaged: “Students become more engaged when they are about to have exams…”. However, there is a need for redesigning and adapting teaching and learning materials to fit the new learning environment as one participant reported that “We need to redesign our lectures to adapt to the new era of online learning…”
Most of the participants linked the interesting content with student engagement. Therefore, they recommended the use of gamifications, quizzes, and the use of Multimedia learning principles “If we followed multimedia learning principles, that would help in both instruction & assessment…”. Though it is still of high importance to select the suitable platform.
According to the participants, selecting the best model for learning may help the students’ development of clinical reasoning skills with the help of scenarios, interactive diagnostic reasoning softwares and virtual simulation.
Student centered approaches and methods can be of great benefit especially in online learning. According to one of the participants “It will allow the students to lead and this may help them to feel secure”. Another participant added “Engaging the students with a student-centered activity will get them out of isolation and will help the faculty to detect any student that was left behind”. Additionally, the use of group work learning/teaching methods as online TBL may foster the development of a collaborative environment.
1.6 Criteria of effective online assessment
Assessment becomes one of the most important dilemmas when shifting to online learning. Ensuring validity and reliability of the exam is a challenge when examinations are done at a distance. “Student Assessment in online learning should be innovative, secure, out of the box, creative and aligned with the teaching methodology.”
There is also the concept of accessibility and how the exam is made available to students. This requires availability of alternatives and ensuring flexibility of format. “Assessment methods used in online learning should be open-book exam, case-based scenarios, single best answer, assignments, virtual OSCE, pattern recognition sessions e.g., histopathological slides, X-rays identification, clinical signs. Choosing the suitable online assessment method depends on the nature of the course, the available resources, student number and student staff ratio”.
1.7 Criteria of online learning evaluation
Types of program evaluation that seemed popular among participants were process and outcome evaluation. However, participants emphasized the need for a comprehensive model of evaluation as the CIPP model because of the complexity of the online learning. “Merging more than one model of evaluation is indicated and highly important in online learning evaluation” was added by one of the participants.
When the participants were asked about the differences between face-to-face course evaluation and online course evaluation, one of the participants mentioned “Modifying and updating the traditional course evaluation surveys to include evaluation of learning management systems (LMS), connectivity and technical support”.
“Collecting the contact details of the registered students is an important step to facilitate online courses evaluation” was added by one of the participants.
Using different data collection tools in online formats was recommended by participants, surveys and students quizzes are preferred.
Participants agreed that student engagement in online learning should be evaluated in terms of student interaction, performance and assessment. “Learning management systems (LMS) analytics such as submission of assignments, synchronous sessions attendance and dropout rate are indicators for students engagement”. Finally, finding the suitable benchmark program and logic model for evaluation is of high importance as well as the need for external peer review to validate the evaluation process.
1.8 Important Evaluation Questions
Participants agreed on some areas that should be covered in evaluation questions e.g. evidence to prove learning (prove of learning), student satisfaction, management system analytics, faculty performance, student interaction, and student performance including the analysis of quiz grades, drop out rate, delay in assignment submission, discussion participation.
Examples of evaluations questions as suggested by most of the participants included:
- If this course/program is optional, would students apply for it?
- Are the staff satisfied with the online learning experience? Do they prefer face to face learning?
- Does the program/course help students to learn and grow?
- Does online learning help tutors/faculty to be better teachers?
1.9 Faculty, Students, Administration attributes in online learning.
Most of the participants discussed the competencies faculty members should have in online learning. They agreed that all faculty members should be skilled in using technology and online platforms and show creativity and innovation. Faculty also should show proficiency in online communication, course design, online assessment, time management and in engaging students in an online learning environment. The participants also stressed on the importance of student-centered strategies to be adopted by faculty members.
Beside the proper use of technology in learning, engagement, critical thinking, collaboration, teamwork and communication, students should also know how to manage learning in an online context. To master such a skill, they should be familiar and trained on self-regulated learning, time management, setting goals and how and when to seek help.
The role of administration in assuring quality of online learning is an integral one. Participants nominated different attributes that will help administration maintain quality online learning including management skills, technical skills, strategic planning attributes and risk management, decision making, ethics and professionalism, communication, monitoring and evaluation.
Step 2: Formulation of descriptors of best practice
The focus group contributions were analysed and reformulated by the authors into quality standards and indicators. Three main quality areas were identified: Organizational capacity, Learning and assessment and Human resources. The standards were designed as follows:
- ORGANIZATIONAL CAPACITY
- Governance
School Leadership is accountable and committed to support and lead the institution to delivering quality online education.
Indicators:
- Leadership encourages a collaborative environment to plan, implement and monitor the quality of online learning activities.
- Leadership shares and cements the values, beliefs, and the operational expectations for a quality online learning.
- Leadership holds themselves accountable to disclose accurate information about the recruitment process, policy, fees, courses/programs, and reports.
- Leadership demonstrates proactive understanding and analyzing organizational needs to deliver effective online education.
- Leadership creates a culture of acceptance and encouragement for online learning.
- Leadership delegates responsibility to multidisciplinary teams and facilitates their work to implement and monitor online learning activities.
- Resources:
Resources for the online learning are allocated in a fair, reasonable manner that responds to the identified needs
Indicators:
- Presence of a learning management system (LMS) that ensures user-friendly and secure online environment.
- Presence of accessible Internet services
- Presence of digital tools that are aligned with the educational needs of learners.
- Presence of the equipment that support successful online learning.
- Presence of trained technical support team.
- Financial resources allocated to online learning.
- Provisional needs documents are available designed annually and approved by proper authorities.
- Budget is well-managed in a transparent and documented way.
- Organizational bylaws
Bylaws clearly define the administrative issues, credit points calculation and the roles and responsibilities of team members
Indicators:
- Presence of written policies & procedures or all online courses
- There is a defined and documented process related to the online programs
- There is a documented clear policy governing the ongoing training and support to the working staff.
- All students have equitable access to the online learning resources.
- EFFECTIVE LEARNING AND ASSESSMENT
- Educational Program
The program has a clear robust design that respects the school vision, mission, and values and that demonstrates a clear understanding of the nature of the required graduate attributes.
Indicators:
- There is an approved, updated and well-constructed longitudinal online education plan that includes sufficient data to support decisions and is aligned with the educational program.
- There are aligned and cascaded goals: strategic, long term, intermediate and short-term goals.
- There is clear identification of the required resources to ensure sustainability of the online programs and courses.
- Course Design
Courses have a clear robust design that respects the school vision, mission and values with a clear distinction of the allocation of online teaching/ learning practices.
Indicators:
- Courses have clearly stated learning objectives/ competencies that are aligned with the organization goals
- The course learning objectives or competencies describe outcomes that are measurable.
- The selected contents are UpToDate, related to learning goals and follow the legal requirements (ownership, intellectual property, copyrights)
- The instructional materials contribute to the achievement of the stated learning objectives or competencies.
- Online instructional methods and tools support active learning, student involvement, support interaction amongst students and between instructors and students and are based on recent best practices.
- Online instructional methods are variable and support development of higher order thinking.
- The relationship between learning objectives or competencies and course activities is clearly stated.
- There is use of digital tools that best support students’ involvement and better understanding of the learning material.
- Learning and assessment schedules are clear, applicable, and fair for all students.
- Online student Assessment methods planned are clear, fair for all students and include frequent formative assessment with feedback and summative assessment with clear and transparent reporting.
- Course delivery
Courses should be delivered in the safest most accessible way providing standardized learning opportunities.
Indicators:
- There is a plan for frequent evaluation that is approved and implemented with identified data collection methods eg. observation, questionnaires, focus group.
- There is a well-organized plan for delivery with a backup.
- A troubleshooting and complaint policy and procedure exists and is announced and used by learners.
- Designed learning activities are implemented with minimal deviation from plans.
- Technologies required in the course are readily obtainable.
- Student assessment
Student assessment to measure student achievement using multiple assessment methods that align with the learning objectives and the instructional methods. Data from assessment is evaluated and feeds into educational decision making.
Indicators
- Digital tools are used to ensure secure, fair, valid, and applicable assessment.
- There is use of multiple assessment methods to measure cognition, skills, and attitude of the students.
- There is use of frequent formative assessment with feedback for better learning.
- There are clear reports after the summative assessment.
- There is a plan for academic counseling that is clear, manageable and is executed.
- Evaluation
Educational monitoring and evaluation plans are available with clearly assigned evaluation questions, key performance indicators and assigned personnel. The plan is implemented and the information it generates feeds into the educational replanning.
Indicators
- There is documented continuous monitoring and evaluation for the online learning materials/ process by internal reviewers to collect and analyze data for continuous improvement. (about LMS, Faculty performance and satisfaction, and Students’ Engagement, Satisfaction, and Achievement)
- There is documented periodic evaluation by external reviewers to validate the internal evaluation process and assess the goal achievement.
- There is disclosure of the evaluation results with the stakeholders.
- Data is used to drive decisions for continuous improvement.
III. Human Resources
The organization has personnel who can manage the educational process effectively and who are under continuous monitoring and development.
Indicators:
1- Faculty
- There is a wide variety of professional development activities for the faculty pertaining to skills needed for online education.
- There is timely and effective technical support to the faculty.
- There is timely, frequent, and constructive feedback about instructor performance.
- Faculty have an opportunity to add to their professional portfolio within online learning in the school.
- The number of assigned faculty is reasonable, sufficient, and aligned with the student number and educational activities.
- There is a clear definition of faculty roles and responsibilities.
2- Students
- There is briefing and orienting the students about the accessibility and availability of the online learning resources and digital tools.
- Equity and accessibility to technology to all students is ensured.
- There is timely and effective technical support to students to overcome limitations of technology & computer literacy.
- There is timely and constructive feedback to students.
- There are guidelines for student-teacher and student-student communication.
3- Administration
- There exists a supporting administration team that is reasonable and aligned with the educational processes, number of students, number of faculty etc.
- There is a solid development plan for administration of the online learning program.
- There is a clear role definition for administration.
- There is a definite pathway for troubleshooting and for complaints for administrators in the program.
Step 3: Expert Consensus session response and Delphi Technique
The above-mentioned descriptors of online learning were handled by experts the following results were achieved:
A- All suggested standard areas were agreed upon with no further additions or amendments by 100% of experts except:
B- The following standards were amended:
- Digital tools are used to ensure secure, fair, valid, and applicable assessment.
Suggested amendment:
Digital tools need to be more specified into KPI e.g., number of trained faculty on the digital tools of assessment, blueprinting to ensure content validity.
- There is a plan for academic counseling that is clear, manageable and is executed.
Suggestion addition of:
Supported by the administration.
C- The following standards were proposed to be added (Table 1).
Based on the above findings and recommendations a set of checklists were developed (table 2-4).