This study has an exploratory sequential mixed-method design, including a qualitative systematic review followed by a quantitative Delphi study [15]. The systematic review aimed to identify requirements for an e-platform for collaborative developments in nursing science and nursing research education. The findings of the systematic review generated the input for the Delphi study, which aimed to achieve consensus among experts on the requirements to be prioritized.

A systematic literature search was conducted according to the PRISMA guidelines by researchers collaborating internationally from Kazakhstan, the Netherlands, and Finland. Consensus on search strategy and screening, as well as the selection of results, was reached through online meetings. A series of search terms were identified in relation to e-platforms or online environment, nursing, research or evidence, and education or learning. From these search terms, different search strings were developed to browse the electronic databases IEEE, TRIP, EBSCO, ABI, PubMed, and CINAHL (Table 1). The searches were conducted between February 10 and April 12, 2018. In addition to this systematic approach, literature introduced in the included studies was also examined (also known as the snowball technique) to minimize the possibility of overlooking important research. The studies identified were required to involve humans, be published in peer-reviewed academic journals not older than six years, and be written in the English or Russian language. Further inclusion criteria were that studies should concern nursing students or clinical nurses who had participated in an online platform or learning program concerning research or evidence-based nursing. The studies should have evaluated the efficacy or impact of the online platform or program using an evaluation instrument or should have evaluated the platform’s development process. The students’ perceptions, experiences, or opinions should have been an element of the evaluation.

Initially, two assessors (ATO, AA) completed the first round of assessments on titles and abstracts in accordance with the PRISMA guideline. Subsequently, one reviewer (ATO) screened and reviewed the full text of articles as needed to determine eligibility. Eligible articles were discussed with two other reviewers (WP, AA) and only included if the researchers reached an agreement. Two researchers (ATO, WP) graded the evidence level of included studies according to the Oxford Centre for Evidence-based Medicine [16] and conducted the methodological quality appraisal according to the Joanna Briggs Institute checklists [17]. Data extraction and data synthesis from the articles took place by assessing the objectives, types of platforms or programs, evaluation tools, key findings, and major recommendations. Three researchers (ATO, WP, AA) analyzed the included studies to identify essential requirements for an e-platform. These independent analyses were compared in two consensus meetings to arrive at a definitive overview of the results and key findings of the studies.

A Delphi study was conducted to prioritize e-platform requirements since this consensus-building tool has been applied in a variety of fields, including innovations in health care [18]. It is an appropriate method when respondents are located at considerable geographic distances, as was the case in this study.

The Delphi panel in this study is quantitative in nature. The respondents were asked to score items on a three-point scale to facilitate the calculation of content validity ratios as well as percentages. A panel of experts was established, including 59 respondents originating from Kazakhstan, Finland, and the Netherlands. Since the e-platform aims to match the context of Kazakh nursing developments, most experts (75%) were from Kazakhstan. Experts in education, research, clinical practice, and information technology as well as professional leadership and the government, were included. The respondents were either participant in ProInCa activities or connected to the project as associated partners or international project members. They were also professionally related to the field of nursing and had adequate understanding or experience in at least one of the following four domains: nursing education at the vocational, bachelor, or master level; nursing research; ICT expertise in nursing; or ICT expertise in online collaboration. Project members engaged in the e-platform development were excluded because they would be too closely involved.

The Delphi study included two rounds of questionnaires containing statements concerning e-platform requirements. For the preparation of the questionnaire, the key findings of the systematic review were discussed at a ProInCa project meeting and translated to requirement statements for the e-platform for nursing in Kazakhstan. This resulted in a list of 62 requirement statements to be included in the Delphi questionnaire. These statements were clustered into four groups: “content requirements,” “functional requirements,” “usability requirements,” and “technical requirements.” The group “content requirements” also included nursing content. For prioritizing nursing content issues, the seven domains of the Nursing Intervention Classification (NIC) were used, which are physiological basic, physiological complex, behavior, safety, family, community, and health system [19].

The first questionnaire was transferred to the SurveyMonkey tool in order to electronically aggregate the survey data [20]. Respondents received a link to the questionnaire by e-mail and were requested to provide informed consent. They were invited to prioritize all requirements on a three-point scale: 1) essential, 2) useful but not essential, or 3) not necessary [21]. Rating was based on the respondents’ own opinions, and anonymity was guaranteed. In addition, two open-ended questions were included through which respondents could contribute additional requirements. A reminder was sent after two weeks and a second reminder after four weeks.

As described by Linstone and Turoff (1975), the Delphi procedure enables individual experts to provide feedback on the results of the first round by offering the opportunity to modify their previous ratings. Therefore, in the second-round questionnaire, all requirements were presented again either as 1) essential or 2) useful but not essential. In addition, based on responses to the open-ended questions in the first round, some minor linguistic changes were made and five additional statements were included. Experts were invited to agree or to disagree with the lists of “essential” and “useful but not essential” requirements. Both questionnaires were provided in English and Russian. Anonymity for individual responses was guaranteed [22].

For the analysis of the ratings in the first round, a content validity ratio (cvr) was applied, which is the appropriate technique for determining consensus among a panel of experts [21]. The cvr should be determined as (n_e - N/2)/(N/2) in which n_e is the number of panelists indicating “essential,” and N is the total number of panelists. In a panel of 40 or more experts, a cvr of 0.29 is considered the minimum [21]. The outcomes of the second round in which requirements could be rated as “essential” or “useful but not essential” were analyzed by allocating percentages. The consensus among experts was determined by applying a threshold of 75%, which is the appropriate analysis technique at this stage in a Delphi study [23].

After removing duplicates and studies that did not meet the inclusion criteria, a further selection was accomplished. This selection included the availability of the full text and screening based on title and abstract. Additional consideration of these criteria resulted in six relevant studies. After reading the full text of the six studies, the researchers determined that references of two studies were relevant to include, although not included by the first queries [snowball searching]. As a result, the literature search strategy generated eight studies. The selection and screening process are presented in the PRISMA flowchart in Fig. (1).

Fig. (1). PRISMA flowchart.

The studies identified are presented in Table 2, representing a variety in country and study participants, study design, and evidence grading. The methodological quality of all studies was good to excellent. In one cross-sectional study, excellence was achieved since confounding factors were addressed [24]. Excellence was also achieved in one qualitative study by specifically addressing the researchers’ roles [25].

In all studies, the e-platform users’ perceptions and experiences were assessed. Objectives were to evaluate, assess, or discuss either the efficacy and impact or the development of an online platform or program. The objectives, types of platforms or programs, and evaluation tools are presented in Table 3, including the key findings and major recommendations. The key findings and recommendations generated an overview of e-platform requirements that were classified into nine categories [1]: aim and users [2], educational content and materials [3], nursing skills [4], platform structure [5], communication, interaction, and feedback [6], knowledge acquisition and achievement of learning outcomes [7], management [8], infrastructural and technological support and [9] usability.

In the first round of the Delphi study, 59 experts completed the online questionnaire, including 48 respondents from Kazakhstan (81%) and 11 from European countries (19%), as illustrated in Table 4. The table also shows the respondents’ employment and expertise, with the highest percentage of respondents (46%) being employed in a health care organization and the highest percentage of respondents’ expertise (46%) being in nursing practice. In the second round, 17 experts completed the questionnaire: 65% from Kazakhstan and 35% from European countries. In contrast to the first round, most respondents (76%) in the second round were employed at a university, and most respondents’ expertise (47%) was in nursing education.

Most requirements identified from the systematic review were included as statements in the Delphi study for prioritizing. These requirements, which were divided into nine categories, were covered by four groups of statements for the Delphi study, as shown in Table 4.

Requirements identified from the systematic review and prioritized in the Delphi study are presented below by group and by category, with ratings shown in Table 6 and Table 7. Table 6 presents a list of 52 requirements that were rated as “essential,” with a cvr of 0.29 or higher. Table 7 presents a list of 15 requirements that were rated as “useful but not essential,” with a cvr below 0.29.

The aim of the platforms studied in the systematic review is to enhance the ability to put acquired nursing knowledge into practice [26]. In the systematic review of Du, clinical nursing skills performance was one of the main outcomes [27]. All authors stated that the acquisition of research skills should enhance the transfer of knowledge to clinical practice and should also increase the access to and use of evidence in the clinical nursing profession [28]. A further requirement is that knowledge should be disseminated among colleagues [26]. It was suggested that it is important to allocate additional time for staff to realize knowledge transfer at work [28].

The Delphi study showed that all statements concerning aim and users were rated as essential (Table 6, Statements No. 1–4), with the highest scores given to enhancing the ability to put nursing knowledge into practice and enhancing nursing skills performance. Enhancing nursing management and leadership (Table 6, No. 5) was added by respondents in the first round as an essential requirement and was confirmed in the second round.

The systematic review indicated that the choice of language should be considered thoroughly before developing the e-platform [26]. In the Delphi study, it was considered essential that information be available in Russian and English as well as in Kazakh (No. 6).

Respondents in the Delphi study also considered it essential that the e-platform include materials for different target groups (No. 7). The platforms in the literature identified addressed a variety of users and levels varying from undergraduates and graduates to clinical nurses or combinations of these. McIntyre et al. (2013) addressed undergraduate as well as graduate nurses, including student levels of master’s degree, bachelor’s or baccalaureate’s degree, or associate or diploma degree [25]. Some platforms explicitly targeted clinical nurses [26, 28, 29], or employed students [27].

The primary requirement related to educational content and materials identified by the systematic review was access to evidence-based guidelines [28]. The literature also indicated the importance of basic nursing research skills. Three studies explicitly explained what educational content concerning nursing research skills should be included [28, 30, 31], such as expertise in basic research principles, expertise in research paradigms like quantitative and qualitative methods [30], and understanding of the major characteristics, strengths, and limitations of quantitative and qualitative research designs [31]. Research skills also involve implementing quest-based search strategies and formulating clinical research questions [31] and using the Patient, Intervention, Comparison, Outcome (PICO) structure [28, 31]. Subsequent skills include performing literature search strategies using a variety of library databases [26, 31]; all studies mentioned the importance of the ability to use electronic databases to access scientific nursing publications. Most statements in the Delphi concerning educational content and materials were rated as essential, as demonstrated in Table 6 (No. 8–18).

The systematic review indicated that additional nursing research skills concern statistical data analysis [26], critical appraisal of scientific reports [31], and academic writing [25]. These research skills were rated as useful but not essential (Table 7, No. 8–10).

Statements concerning nursing skills were obtained from the seven NIC domains, and six of these were rated as essential, as shown in Table 6 (Statements No. 19–24). The domain physiological-basic was rated as useful but not essential, as shown in Table 6 (No. 8), although no agreement was achieved in the second round.

Nursing skills concerning documentation, quality management, standard operating procedures, and laws and regulations (No. 25–28) were added by respondents as essential requirements after the first round, which was confirmed by high levels of agreement in the second round.

A variety of platform structures was reported in the studies identified by the systematic review. All platforms or websites were structured in a way to contain distance learning courses or series of nursing learning modules [27]. Some of the platforms included additional structures that enabled content and tools such as forum discussions, webinars about nursing topics, blogs or podcasts [25], or teleconferencing [27]. The content can be provided in text as well as in audio and video formats [27]. These formats may also include audio-visual presentations as well as graphics and animations, usually with multimedia and interactive elements [29]. The platform structure should allow links to electronic databases, as is mentioned by all the studies.

Although identified as crucial in the systematic review, all statements in the Delphi study concerning platform structure were rated useful but not essential. Only one statement was prioritized as essential – an agenda or calendar with relevant activities (Table 6, No. 29).

The review revealed that interactive functionalities for online collaboration and online discussions are the basic characteristics of a successful educational e-platform. This includes interaction between students, teachers, and content, which must always be maintained [25]. This type of e-platform design is explicitly referred to as an interactive design [27], and all modules or courses that are included should be interactive [30]. Interaction is aimed at achieving learning outcomes using feedback from tutors and peers. In addition, feedback should be individual-based and prompt or timely [31].

In the Delphi study, the statements concerning online collaboration and feedback were rated as essential (Table 6, No. 30–31). However, the statements concerning online discussions and communication and interaction with tutors and peers were rated as useful but not essential (Table 7, No. 12–13).

The review demonstrated that requirements for a successful e-platform include clear learning goals and learning outcomes to support knowledge acquisition through self-directed learning. Consequently, courses should provide explicit instructions, examples, and assignments. Davidson et al. (2016) demonstrated the need for guaranteeing progression by increasing levels of learning activities [31]. The specific characteristics of knowledge acquisition through e-learning were considered thoroughly in all studies. Specific consideration should be given to assessments for measuring progression in learning. Specifically, in self-directed learning, it is important to distinguish between formative and summative assessments [27]. In testing the achievement of learning outcomes, priority was on formative assessment, which corresponds with the typical features of self-directed learning [26]. It is recommended that the e-platform includes materials to download for use in classroom settings as well.

In the Delphi study, all statements concerning knowledge acquisition were rated as essential (Table 6, No. 32–39), apart from the statement concerning downloadable educational materials for classroom use, which was rated as useful but not essential (Table 7, No.14).

A platform should include management tools for developing and maintaining structures and materials and for secure administration of courses. It should be simple to further develop the platform and to add and change information. Management requirements involve monitoring tools to track students [27], check logins, and ensure secure personal data and user accounts. It may also be necessary to monitor the number of users and the popularity of different items.

The experts in the Delphi panel rated most management requirements as essential (Table 6, No. 40–42), although monitoring the number of users and popularity of different items was rated as useful but not essential (Table 7, No. 15).

All studies in the review demonstrate that the e-platform must be compatible for use with different devices, such as a laptop, tablet, or smartphone. Requirements concerning infrastructural and technological support also stipulate that access to the Internet should not be restricted in terms of time, place, or costs [27]. The unlimited availability of technological support by a support center or helpdesk is a basic requirement because hardware and software problems commonly occur [27]. The effectiveness and quality of the technological support provided are essential [29]. Testing by information technology personnel is required to check for viruses in the system [30]. All studies reported that the technological infrastructure should enable unrestricted access to other websites or electronic databases.

All statements in the Delphi study concerning technical requirements (Table 6, No. 43–45) were rated as essential.

In all studies reviewed, usability was an important feature that involves user-friendliness, student motivation, and engagement. Usability requirements include easy site navigation and a rapid load time [26] as well as an attractive design with an adequate amount of text that is easy to read. In addition, ergonomic usability is required [24]. Another feature of usability is accessibility, which may be open, closed, or a mixture. In some of the studies, access was limited to registered members of a specific learning community [28]. In all nine studies of Du’s systematic review (2013), users had open access [27]. Usability of the platform should enhance and support flexibility, which is the key to distance learning and includes flexibility in time, place, and learning activities [31]. Flexibility in accessing education encourages students to work across time zones [25]. Usability also means information should be easy to locate, including clear course information by curriculum mapping [27] and useful instructions [26].

In the Delphi study, all usability requirements were rated as essential (Table 6, No. 46–52).

The exploratory sequential mixed-method design, which has also been applied in other health-related studies [18], appeared to be an appropriate method for identifying and prioritizing requirements for an e-platform in nursing. The requirements ascertained from the literature are translated and validated for the specific context using the Delphi method.

A strength of the Delphi study was the panel composition with most experts from Kazakhstan and others from European countries. Consequently, the results are well embedded in the national context. At the same time, a limitation is that the panel composition varied from experienced e-platform users to non-users, whereas the results from the systematic review were derived from samples of study participants who were all experienced platform users.

Another limitation is the low response rate in the second round of the Delphi, with only 17 respondents compared to 59 respondents in the first round, and a higher percentage (76%) of respondents employed at universities in the second round compared to the first round (44%).

In this study, the cvr was used to analyze ratings in the first round, and frequencies were utilized in the second round, applying a consensus threshold of 75%. One strength of this combined analysis technique is that it enables panel members to provide feedback and adjust scores during the study. Limitations of the technique were that no explicit criteria for removing items from the list of requirements were determined and the number of Delphi rounds were established beforehand. Although these limitations appear in most Delphi studies [23], it is recommended to prevent this when conducting a Delphi study.

Beyond the scope of this study is the organizational structure of monitoring the e-platform. In general, to guarantee the long-term success of an e-platform, it is important to delineate responsibilities related to monitoring the content, functionality, usability, and technology.

To strengthen and substantiate the nursing profession in Kazakhstan, it was recommended to develop an e-platform following the requirements established in the study. The e-platform, launched in 2020, is an innovative resource on a national level in Kazakhstan. It provides a permanent platform for nursing stakeholders from clinical practice, education, and research to strengthen cooperation and collaboration to enhance nursing and to enforce the quality of nursing care [32]. The current e-platform will be further enriched and will contribute to substantive innovations in the near future. For example, recently developed national clinical nursing guidelines are now disseminated through the e-platform. The platform will also be used to support Masters and PhD programs in nursing [33].

Regarding follow-up research, we suggest measuring the actual medium- and long-term use and effects of the e-platform. It is recommended to investigate different effects, such as the influence of the e-platform on the use of evidence-based interventions in clinical practice. Effects on quality of care and patient satisfaction must ultimately also be included in the research.

Our research offers new insights into the requirements for an e-platform to successfully strengthen nursing on a national level in the trinity of nursing education, nursing research, and clinical practice. Based on this new understanding, the current platform can be applied even more widely and can grow further in terms of content. In addition, the nine categories and four groups of essential and useful requirements may provide a starting point for developing other e-platforms of national scope in other countries or for other health care professionals. It is recommended to consult a panel of experts who represent future users’ perceptions and are well embedded in the specific context. Since the application of ICT is increasing rapidly, it is crucial to continue studying how these developments affect knowledge sharing and collaboration in the nursing profession.