1 Introduction

During the COVID-19 pandemic, MOOCs gained much more attention in addressing the limitations of remote learning (Hu et al., 2022). However, the development of MOOCs faces considerable challenges, including high dropout rate, low completion rate, failure to provide the same quality certification as on-campus education, and weak teaching and learning process. Recently, Wang et al., (2022) reveled that MOOCs component was not enough for maintaining students’ curiosity and engagement. These challenges MOOCs faces are caused by instructional design and pedagogy (Kovanović et al., 2018), which disconnect the practice and research in online and distance education. Online and distance education research has achieved impressively systematic and fruitful results in the past decades. Among these accomplishments was the CoI framework, which provides guidance for the design, organization, and implementation of MOOCs. At the beginning, online teaching is an extremely complex and challenging work (Anderson et al., 2001). Practice and research in this particular context require appropriate theoretical perspective. The CoI framework developed by Garrison et al., (2000) provides a new perspective to understand the nature of teaching and learning in an online environment (Rourke et al., 2001). It also helps acknowledge that deep and meaningful learning occurs with sufficient teaching, social, and cognitive presence, and it serves as a theoretical framework for practitioners to structure online learning process (Stefan, 2018). The CoI instrument developed by Arbaugh et al., (2008) is a research tool that explores, reveals, and assesses students’ online learning experience from the perspective of three critical presences, and thus to evaluate the quality of online course.

As online learning becomes popular, the CoI framework informs practitioners and researchers of online teaching and learning (Swan et al., 2008; Garrison et al., 2010; Halverson et al., 2014). Numerous empirical researches have applied and expanded the CoI framework and instrument in different educational settings (Yu & Li, 2022). These studies tell us that the CoI framework and instrument has great versatility and may be suitable for different teaching settings. For example, Shea et al., (2010) modified the teaching presence coding framework of original CoI to make it suitable for the entire asynchronous online course rather than just the threaded discussion. Szeto (2015) applied the CoI framework to blended synchronization teaching context. Recently, Stefan (2018) systematically reviewed the research on CoI instrument. After rigorous screening, 103 scholarly papers continue using this instrument to investigate critical issues in online courses. Flock (2020) made a literature review focusing on CoI instructional strategies. Form their study we can see the fruitful research outcomes on how to improve online teaching practice from the perspective of CoI. Caskurlu et al., (2021) made a thematic synthesis on the existing empirical studies to investigate the factors influencing students’ online learning experiences, and findings strongly suggested that factors from the CoI are important variables can predict student online learning experiences. In addition, some studies focus on the influence of CoI on students’ learning related variables, such as the relationship between CoI indicators and student retention (Boston et al., 2019),the mediating effects of the three presences of CoI on the influence of student enrollment and motivation on their learning performance (Lawa et al., 2019), and the differences in students’ perceived CoI and its predictive effects on students’ affective learning outcomes according to their academic disciplines (Lim & Richardson, 2021). Researchers’ interest in CoI framework (Shea et al., 2022; Garrison, 2022) and the CoI instrument continues (Parulla et al., 2022).

On the other hand, some studies focus on the construct validity and reliability of the CoI instruments in online course (Heilporn & Lakhal, 2020; Dempsey & Jang, 2019; Velázquez et al., 2019; Kovanović et al., 2018; Caskurlu, 2018; Olpak & Kiliç Çakmak, 2018). For instance, Heilporn & Lakhal (2020) investigating the reliability and validity of the CoI framework by analysis of the ten categories within the CoI framework. On the basis of study, Heilporn & Lakhal (2020) highlight the need for revising the CoI instrument and better refine some of its categories. With the continuous development of online education, the original CoI survey instrument should “be revisited and adjusted over time” (Lowenthal & Dunlap, 2014; Wertz, 2022) evaluated alternative structures of the CoI instrument, a series of confirmatory factor analyses (CFA) to evaluate the measurement models of the four CoI constructs individually, followed by a model including all four constructs simultaneously, this study produced evidence on the multidimensionality of the CoI constructs. Moreover, Wang et al., (2022) have qualitatively explored how students experience all four types of presences outlined in the revised CoI framework.

However, only a few studies verify the applicability of the CoI instrument in MOOC settings. Damm (2016) measured students’ online learning experience in MOOC setting using the CoI instrument and verified the survey results through in-depth interviews, but he did not verify its factor structure. Recently, Parulla et al., (2022) validated the CoI instrument in Brazil in MOOC setting by only employing the component analysis. Kovanović et al., (2018) gathered MOOC learners’ online learning experience using the CoI instrument. Moreover, Exploratory Factor Analysis (EFA) was used to analyze the data that offered a six-factor structure CoI model with teaching presence (TP), social presence (SP), and cognitive presence (CP) from the original CoI framework, and additional design and organization (Org), affective expression (Aff), and resolution (Res). The three new factors are all subcategories of TP, SP, and CP, which separate from the original CoI and formed a new one themselves. Unfortunately, the researchers failed to verify the new six-factor model using CFA and its construct validity, especially the discriminant validity among the three new factors and the three original factors they separated from. According to Hair et al., (2010), the factors obtained by CFA require further discriminant validity test, and good discriminant validity must be established among the factors to separate them effectively. Kovanović et al., (2018) did not compare the six-factor model with the original three-factor model. Therefore, we do not know whether the six-factor CoI model is significantly better than the three-factor CoI model.

Therefore, this study aims to determine the optimal model of CoI instrument through model comparison and verify its construct validity. It first explores the presence of other potential models of the CoI instrument in MOOC settings that conduct EFA and CFA on it. In addition, the study will verify the existing original three-factor (Arbaugh et al., 2008) and the new six-factor model (Kovanović et al., 2018) via CFA. It will then use the chi-square test to compare these three models with different factor structures to select the optimal one and test the structural validity of the optimal model. Accordingly, this study has three research questions as follows:

RQ1. Is there another potential model structure of the CoI instrument in MOOC setting?

RQ2. Among the original three-factor model, the new six-factor model, and another potential n- factor model of the CoI instrument, which is the optimal one?

RQ3. Does the optimal model have good structural validity?

The current study aims to explore the applicability of the CoI instrument in MOOC setting to provide new perspectives, methods, and tools for future MOOC research, which is a starting point for studying student learning experience in MOOC and analyzing the effectiveness of course based on MOOCs from the perspective of CoI framework. Doing so will extend the research in MOOCs, and increase the cultural adaptability of MOOC practice and research in the field of online and distance education. In addition, this will help to achieve a comprehensive understanding of students’ learning experience in MOOC setting, and helps us to better design and implement high quality MOOC to improve student learning engagement and learning experience that ultimately promote students’ online learning outcomes.

2 CoI framework and survey instrument

2.1 CoI framework

The CoI framework has three factors, namely, social, teaching, and cognitive presence (see Fig. 1) (Garrison et al., 2001). CP is defined as “the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry” (Garrison et al., 2001, P. 11). It is directly related to meaningful learning, which has four stages: triggering events, exploration, integration, and resolution. Triggering events refer to presenting a problem or task that will trigger students’ attention to learning. Exploration refer to student exploring relevant information. Integration refer to student connecting and integrating different ideas and understandings of the problems to be solved. Res refers to the fourth phases of CP, where learners apply the newly gained knowledge to new educational contexts or workplace settings (Garrison et al., 2001).

Fig. 1
figure 1

Community of Inquiry framework (Garrison et al., 2000)

Full size image

TP is described as “the design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes” (Anderson et al., 2001, P. 5). It has three subcategories, namely, design and organization, facilitating discourse, and direct instruction. Design and organization is the first subcategory of TP, which refers to the planning and design of the structure, process, interactions, and evaluation of an online course, includes preparing for presentations, developing audio/video mini-lectures and lecture notes for the online course, providing personal insights into the course content, and creating a desirable mix of individual and group activities and an accompanying schedule (Anderson et al., 2001). In addition, the tasks that reflect TP include creating curriculum content, designing learning activities, setting time parameters, monitoring and managing purposeful collaborative reflection activities, and ensuring expected learning outcomes by diagnosing needs and assessing and evaluating student learning with timely feedback for learning improvement.

SP is “the ability of participants in a community of inquiry to project themselves socially and emotionally, as real people” (Garrison et al., 2001, P. 94). SP includes affective expression, open communication, and group cohesion. Affective expression is one of the subcategories of SP, which refers to “where learners share personal expressions of emotion, feelings, beliefs, and values” (Lowenthal, 2010, P. 125). Its importance is due to the limited uses of face-to-face emotional communications in the online learning environment. SP requires participants the ability to connect themselves with the learning community, communicate in an environment they trust, and develop relationships in the learning community. Students must create personal and purposeful relationships through SP.

2.2 CoI survey instrument

The CoI framework became increasingly popular in explaining the effectiveness of learning in the online environment. Hence, Arbaugh et al., (2008) developed the CoI instrument (five-point Likert-type) with 34 items to measure three presences of CoI framework. The CoI instrument attempts to operationalize the CoI framework. Thirteen (1–13) items for TP, nine (14–22) items for SP, twelve items (23–34) for CP, to determine an efficient quantitative orientation measure of the CoI theoretical framework. This instrument has been shown to have high internal consistency for the three presences, it has high Cronbach’s α of 0.94 for TP, 0.91 for SP, and 0.95 for CP (Arbaugh et al., 2008).

The reliability of the English CoI instrument has been verified by other studies (Swan et al., 2008; Díaz et al., 2010; Shea & Bidjerano, 2009; Kozan & Richardson, 2014). Moreover, a few researchers validated its reliability and validity in other cultural contexts. For their research sample, Moreira et al., (2013) selected 510 students from different universities who major in different professions and enrolled in blended online courses to verify the reliability and validity of the CoI instrument in Portugal. Yu & Richardson (2015) explored the reliability and validity of the CoI instrument with 995 Korean undergraduate students in online universities. All these studies have verified the good reliability of this instrument in terms of high Cronbach value. As for validity, the previous validation studies have provided empirical support that all the items are loaded significantly on their corresponding factors and are aligned with the CoI framework as operationalized by the CoI instrument. However, Arbaugh et al., (2008), Díaz et al. (2010), and Kozan & Richardson (2014) suggested the existence of a potential fourth factor apart from CP, TP, and SP factors. Both Arbaugh et al., (2008) and Díaz et al. (2010) found that the construct of TP consisted of two factors. On the other hand, Kozan & Richardson (2014) found a fourth factor might exist as a subcategory of the CP. In short, the previous validation studies indicated that the CoI instrument has good reliability. However, there is some controversy regarding the validity of the CoI instrument.

In addition, the previous studies have verified the reliability and validity of the CoI instrument in traditional online courses. Damm (2016) and Kovanović et al., (2018) applied the CoI framework and instrument in MOOC context. However, these studies have some limitations as described in the introduction section, this study will further study the applicability of CoI instrument in MOOC setting.

3 Methods

3.1 Research context

The course Modern Educational Technology was selected as our research context in this study, which requires students to learn through interactions with others rather than individual learning. The leading instructor of this course is a nationally renowned professor, with a team of six professors, three associate professors, six doctoral students, and ten master students. It has won the honorary title of National Excellent Course in China, and has been offering on the “MOOC platform of Chinese university” since 2016. This MOOC is open to everyone, thousands of learners register and study in this course every year.

Students’ online learning activities in this MOOC include watching videos, participating in forum discussions, completing unit online tests, finishing two design assignments, and participating in the final online examination. Students are also required to participate in two peer review activities. In the final course evaluation, students’ participation in all learning activities and their learning outcomes are recorded. Two peer-review activities, online discussion, unit online tests, and final examination account for 30%, 10%, 24%, and 36% of students’ rating. Students who complete all the online learning activities and tasks can obtain an ordinary or an excellent certificate according to their final scores.

3.1.1 Online discussion

This MOOC has three fora, namely, Content Discussion, Q&A, and General Discussion. The instructor must actively assist or guide the online discussions to be effective. One technique to achieve this goal is by posing meaningful and relevant questions (Kanuka & Garrison, 2004). In this MOOC, teachers post two questions in the Content Discussion Forum each week for student discussions. Students can only comment on or reply to other students’ posts, and they are not allowed to publish new posts in this forum. Different from this forum, students can post any questions during the course in the Q&A Forum, where their teachers or peer learners can help them. The TP is high due to the teachers’ frequent participation in the discussions and prompt responses to students’ questions. The third forum is the General Discussion Forum and is prepared for students to publish topics about this course and their study, internship, life, and all their other interests and things they want to share. Teachers rarely participate in this open forum. Students can freely speak and communicate with their peers, socially develop themselves, connect themselves with others through open communications to establish personal relationships, and finally form their personal learning community. Notably, the SP of CoI framework advocates these activities.

3.1.2 Peer review

The two design assignments are evaluated by peers anonymously on the basis of the assessment criteria set by the teaching team. All students are randomly divided into groups by the online course system. Each group has 10 students. For the group with less than 10 members, teachers will participate in peer review as team members, and each peer review lasts for one week. Every student should review 6 to 10 assignments. Students who fail to participate in the peer reviews can only achieve 30% of the full score for the given assignment. Those who complete 6 assignments will gain 50% of the full assignment scores, and those who review 10 assignments for their peers will obtain full scores.

3.2 Data collection

An online survey was conducted to collect data using the CoI instrument (Arbaugh et al., 2008). The course is open, anyone can participate in learning, but we only collected the data from students from one university. All participating students were informed of the purpose, procedures, and contents of this survey. The data was collected in the first two weeks of January 2016. The study involved 1,186 Chinese undergraduate students from three different departments with 11 majors. A total of 491 (41.4%) were males, and 695 (58.6%) were females. Among these participants, 691 (58.3%) had previous online learning experience, and 495 (41.7%) had no online learning experience. The age of the participants ranges from 19 to 20.

3.3 Data Analysis

All the data collected were preprocessed to ensure that no missing or abnormal values appear in the overall sample data. SPSS21. was used to divide the data into two samples randomly. The first sample group (n = 582) was used for EFA, and the second sample group (n = 604) was used for CFA. EFA is a statistical method to increase the reliability of the instrument by removing inappropriate items and to identify the dimensionality of constructs by examining relations between items and factors (Netemeyer et al., 2003). CFA is a statistical method to verify the structure of the instrument, specifically, to examine the relationships among the latent and manifest variables supported by logic or theory.

Chi-square difference test was utilized to compare the CFA models with different factor structures of the CoI instruments. Structural validity was tested via convergent and discriminant validity analysis. Convergent validity means that the measurement variables of the same factor should be at the same construct level with a high factor loading value and a high correlation between these measurement variables. The convergent validity of the factors obtained from CFA can be tested from factor loading value and verified via composite reliability (Hair et al., 2010). High factor loading value represents good convergent validity of the measurement variable; the general factor loading value should be greater than 0.5 (Chai et al., 2013). If the factor loading value is greater than 0.7, then the measurement variable has ideal quality (Hair et al., 2010). Combinatorial reliability analysis was used to verify the convergent validity of the CFA factors. The value of the combinatorial reliability of factors should be greater than 0.6 (Fornell & Larcker, 1981). A high combination reliability implies a high correlation between the measured variables. Thus, the measurement variables have good isomorphism.

Chi-square difference test was used to verify if these factors in the six-factor model of the CoI instrument (Kovanović et al., 2018) have good discriminant validity. If the square value difference between the unconstrained and constrained (the covariance between two factors was fixed as 1) model is greater than the threshold of \({\chi }_{0.05}^{2}=\)3.841 and reached the significant level (\(p<.05\)), then the relations between the two factors are only partially related, namely, clearly indicating good discriminant validity (Anderson & Gerbing, 1988) of the two factors.

4 Results

4.1 Descriptive analysis

Descriptive statistical analysis was conducted on the 34 measurement variables, and the results showed that the kurtosis of all the measurement variables is less than the threshold of 5. Thus, the data is normally (or close to normally) distributed, which is sufficient for our analysis (Bentler, 2006).

The Cronbach’s α of TP, SP, and CP are 0.945, 0.924, and 0.925, respectively, whereas that of the overall instrument is 0.970. Thus, the internal consistency of this instrument and its subcategories has good internal consistency (Hair et al., 2010).

4.2 Results of EFA analysis

An EFA using principal component analysis was conducted on the first sample group (n = 582) via SPSS 21. Maximum variance factor rotation method was used to extract factors. The Kaiser–Meyer–Olkin (KMO) is 0.973, which is higher than the accepted threshold of 0.5 (Kaiser, 1974). The results of Bartlett’s test are significant [χ2 [561] = 14263.419, P < 001]. Thus, the sampling adequacy is verified, and the correlations between the items are sufficiently large for EFA. Therefore, the data are suitable for factor analysis. Table 1; Fig. 2 show the results of eigenvalues and the scree plot. Principal components analysis showed there were four factors: TP, SP, CP, and an additional fourth factor with an eigenvalue > 1. The fourth factor is a subcategory (resolution) of CP, as shown in Table 2. The eigenvalues of TP, SP, CP, and resolution are 17.179, 1.173, 2.144, and 1.053, respectively. The results of EFA showed that the four-factor model accounted for 63.38% of the variance, with TP, SP, CP, and resolution accounting for 50.53%, 3.45%, 6.31%, and 3.10%, respectively. In addition, the correlation analyses results show that there is a strong correlation between each of the four factors (See Table 3).

Table 1 Eigenvalues from principal component analysis
Full size table
Fig. 2
figure 2

Scree plot

Full size image
Table 2 Factor loading in the factor pattern matrix
Full size table
Table 3 Correlation between factors in the four-factor model
Full size table

4.3 Results of CFA analysis

CFA was conducted on the four-factor structure obtained from EFA using the second sample group data (n = 604) via LISREL8.7. Several fit indices were used to evaluate the model, including the chi-square value, ratio between χ2 and degrees of freedom, the Incremental Fit Index (IFI), Comparative Fit Index (CFI), Goodness-of-Fit Index (GFI), Normed Fit Index (NFI), and Root Mean Square Error of Approximation (RMSEA). Although the insignificant χ2 value indicates good fit model, χ2 value is sensitive to sample size (Kline, 2005). Alternatively, using a ratio χ2/df less than 3 is recommended as a reliable indicator of good model adjustment. CFI, IFI, and NFI above 0.95 and GFI above 0.90 are indicative of model fit (Hu & Bentler, 1999). RMSEA below 0.05 is considered excellent in terms of fit, whereas RMSEA below 0.10 is considered adequate (Browne & Cudeck, 1992).

In this study, CFA results show that the four-factor structure model fit the data very well (χ2 = 1499.18, p < .001, df = 521, χ2/df = 2.878), with acceptable NFI = 0.98, CFI = 0.99, GFI = 0.87, IFI = 0.99, and RMSEA = 0.058 (Table 4). The completely standardized factor loading ranges from 0.61 to 0.85 (Fig. 3). Finally, the results of the CFA confirmed that the model fit is excellent between the four-factor model and the data.

Table 4 Fit indices of different factor structure models
Full size table
Fig. 3
figure 3

Latent four-factor solution for the CoI instrument with completely standardized factor loadings

Full size image

In addition, CFA on the original three- and six-factor structures recently obtained by Kovanović et al., (2018) using EFA was conducted. The results of CFA revealed that the fitting indices of the original three- and six-factor models are all significant (Table 4). First, the original three-factor structure model fits the data very well (χ2 = 1527.26, p < .001, df = 524, χ2/df = 2.915), with acceptable NFI = 0.98, CFI = 0.99, GFI = 0.86, IFI = 0.99, and RMSEA = 0.059 (Table 4). The completely standardized factor loading ranges from 0.61 to 0.85 (Fig. 4). The results of the CFA confirmed that the model fit is excellent between the three-factor model and the data. Second, the six-factor structure model also fits the data very well (χ2 = 1216.47, df = 512, χ2/df = 2.376), with acceptable NFI = 0.99, CFI = 0.99, GFI = 0.89, IFI = 0.99, and RMSEA = 0.049 (Table 4). The completely standardized factor loading ranges from 0.63 to 0.86 (Fig. 5). The results of the CFA confirmed that the model fit is excellent between the six-factor model and the data. According to the traditional evaluation criteria (fit indices) for model fit, the model fit of the three-, four-, and six-factor structure models are all ideal. Therefore, these three models are all acceptable.

Fig. 4
figure 4

Latent three-factor solution for the CoI instrument with completely standardized factor loading

Full size image
Fig. 5
figure 5

Latent six-factor solution for the CoI instrument with completely standardized factor loading

Full size image

4.4 Model comparison

Chi-square difference test was used to compare the three different models of the CoI instrument (Table 5). The chi-square difference between model 3 F (three-factor: TP, SP, and CP) and model 4 F (four-factor: TP, SP, CP, and Res) is significant (Δ χ2 [3] = 28.08 > 11.35, p < .01). Thus, model 4 F is better than model 3 F. The chi-square difference between model 3 F and model 6 F (six-factor: TP, SP, CP, Org, Aff, and Res) is significant (Δ χ2 [12] = 310.79 > 26.22, p < .01). Thus, model 6Fis better than model 3 F. The chi-square difference between models 4 and 6 F is significant (Δ χ2 [9] = 282.71 > 21.67, p < .01). Thus, model 6 F is better than model 4 F. Therefore, model 6 F is the optimal model.

Table 5 Result of chi-square difference test for the three different factor structure models
Full size table

4.5 Construct validity

4.5.1 Convergent validity

The standardized factor loading value of all measurement variables of model 6 F is between 0.63 and 0.86 (Fig. 5). Most of them are greater than 0.7, which indicates that most measurement variables have ideal quality (Hair et al., 2010). According to the standardized factor loading value of the measurement variables of each factor in model 6 F, the following calculation formula was used (Fornell & Larcker, 1981):

$${\rho }_{c=}\frac{{\left(\sum \text{S}\text{t}\text{a}\text{n}\text{d}\text{a}\text{r}\text{d}\text{i}\text{z}\text{e}\text{d} \text{f}\text{a}\text{c}\text{t}\text{o}\text{r} \text{l}\text{o}\text{a}\text{d} \text{v}\text{a}\text{l}\text{u}\text{e}\text{s}\right)}^{2}}{\left[{\left(\sum \text{S}\text{t}\text{a}\text{n}\text{d}\text{a}\text{r}\text{d}\text{i}\text{z}\text{e}\text{d} \text{f}\text{a}\text{c}\text{t}\text{o}\text{r} \text{l}\text{o}\text{a}\text{d} \text{v}\text{a}\text{l}\text{u}\text{e}\text{s}\right)}^{2}+\sum \left(\text{E}\text{r}\text{r}\text{o}\text{r} \text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e} \text{o}\text{f} \text{m}\text{e}\text{a}\text{s}\text{u}\text{r}\text{e}\text{m}\text{e}\text{n}\text{t} \text{v}\text{a}\text{r}\text{i}\text{a}\text{b}\text{l}\text{e}\text{s}\right)\right]}$$

.

The factor combination reliability value \({\rho }_{c}\) on each factor was obtained as follows: 0.93 for TP, 0.92 for SP, 0.92 for CP, 0.81 for Res, 0.77 for Aff, and 0.86 for Org. The combined reliability values of the six factors are all greater than 0.7. Therefore, the measurement variables of each factor have good isomorphism (Chai et al., 2013). The combined reliability value indicates that all measurement variables of the CoI instrument have good convergent validity. They effectively reflect the corresponding factors (potential constructs), and the internal consistency of the measurement variables of the same factor is high.

4.5.2 Discriminant validity

Chi-square difference test was used to verify the discriminative validity of the six-factor model of CoI instrument by calculating the chi-square differences between the unconstrained and the constrained models. Any two of Org, TP, Aff, SP, Res and CP were combined in pairs. Fifteen CFA hypothesis models (Model 1 to Modle15 in Table 6) were obtained. For each CFA hypothesis model, the covariance between each two factors was set as 1 obtaining the constrained model, whereas the covariance between each two factors was set as free estimation parameters, obtaining the unconstrained model (M0). CFA was carried out on the unconstrained model 0 and also the fifteen constrained models by the ML method, one pair of constructs at a time. Finally, χ2, df, and chi-square difference (Δχ2) of the unconstrained model and the fifteen constrained model models showed in Table 6.

Chi-square difference test results show that the chi-square differences between the fifteen constrained and the unconstrained models are all significantly higher than the accepted threshold of factor discriminant validity \({\chi }_{0.05}^{2}=\) 3.841. Thus, the square differences between the constrained and unconstrained models all achieve significance at the 0.05 level, and significant differences exist in the latent trait expressed by any two of the six factors. Take model 1 as an example: the discriminant validity was tested by calculating the chi-square difference between the constrained model1 and the unconstrained Model0.The result shows the chi-square difference is significantly higher than 3.841, indicating Org and TP has good discriminant validity, the same as other pairs of constructs. In summary, the results of chi-square difference tests indicated that six-factor model shows good discriminant validity (Anderson & Gerbing, 1988).

Table 6 Unconstrained and constrained model differences for discriminant validity
Full size table

5 Discussion

A four-factor CoI model, including TP, SP, CP, and the resolution of CP, was obtained using EFA. This echoes with the previous researches claiming that there is a potential fourth factor (Arbaugh et al., 2008; Díaz et al., 2010; Kozan & Richardson, 2014). For example, Arbaugh et al., (2008) Díaz et al. (2010) pointed to the potential fourth factor serving as a subcategory of TP. This present study does not confirm this possibility. The result of EFA however shows that the fourth factor is the one assessing the resolution of CP. In this sense, the present study is consistent with Kozan & Richardson (2014), which noted the potential fourth factor serving as a subcategory of CP in their initial EFA, and Kovanović et al. (2018), which indicated that the fourth stage of CP separated from CP into a subcategory of its own.

The results of CFA show that the data fit the original three-factor structure model very well, and all the fit indices meet the requirements(Table 4; Fig. 3). Similarly, the four- and six-factor structure models are also good, for all the fit indices meet the requirements (Table 4; Figs. 4 and 5). After model comparison using the chi-square difference test, we found that the four-factor structure model is better than the original three-factor structure model, and the six-factor structure model is better than both three- and four-factor models. Therefore, the six-factor structure model, which includes TP, SP, CP, Res, Org, and Aff, is the optimal model (Table 5). The result of Chi-square difference test shows that the six-factor model has good discriminant validity (See Table 6). Especially, the chi-square difference results indicate that Org can be significantly distinguished from TP (Model M1), whereas Aff can be significantly distinguished from SP (Model M11), and Res can be significantly distinguished from CP (Model M15). In addition, the combined reliability values of the six factors are all greater than 0.8, thereby indicating that the six factors all have good convergent validity.

As for the fourth factor, the items assessing the Res of CP unexpectedly loaded on a separate factor. Res require student apply the new concept or idea to the original task or challenge (Parrish et al., 2021). It is well known that students face challenges in transitioning into later stages of CP, and especially Res phase (Vaughan & Garrison, 2005). TP plays an important role for the development of CP in traditional online courses. Meyer (2004) pointed out that the problems or tasks in the trigger event stage directly affect students’ following cognitive activity. When the problems or tasks in this stage explicitly require students to participate in learning, students’ discussion may rise to the Res stage. Archibald (2010) emphasized that students’ inquiry activities have better chances of reaching the Res stage in case-based discussion than with an open-ended discussion. However, TP, especially facilitation and direct instruction, is limited in MOOCs. Students who actively participate in the online discussion forum are more likely to reach higher levels of CP than those who focus on individual learning in MOOC settings (Kovanović et al., 2018). In our study, the study found 48,938 posts in Content Discussion Forum, which comprised 32 theme posts and 48,906 reply posts. A total of 5,930 posts were found in the Q&A Forum, including 1,293 first posts and 4,637 reply posts. In addition, 4,813 posts were recorded in the General Discussion Forum, including 1,612 first posts and 3,201 reply posts. These numbers indicated active participation in discussion forums in MOOC selected for this research. Although the number of participants (who continue to participate in the forum activities) is important, the quality of their participation is even more important. If students simply post comments without engaging in knowledge construction (Kanuka & Garrison, 2004), then their learning can scarcely reach the higher levels of CP. Higher quality cognitive activities (participation) require strong and high TP (Garrison et al., 2001; Meyer, 2003), which is what MOOCs lack. Without such presence, students’ cognitive activities cannot reach the resolution stage in MOOC setting.

As for the fifth factor, the items related to Org loaded on a separate factor. This finding is consistent with the previous research (Caskurlu, 2018; Arbaugh et al., 2008; Laves, 2010; Shea et al., 2006) conducted principal component analysis on the TP scale developed on the basis of the CoI framework that closely resembles the items for TP in the CoI instrument. The items of facilitation and direct instruction loaded on one factor, whereas the items of Org loaded on another factor. Shea et al., (2006) also obtained a similar result. Caskurlu (2018) noted that Org are distinct from direct instruction and facilitation, whereas direct instruction and facilitation may not be distinct from each other in the traditional online course. It is related to the discriminant of the scale itself (Laves, 2010), and it is also possible because students may think that the facilitation is part of direct instruction (Garrison & Arbaugh, 2007). In addition, the separate factors reflected the different times at which these teaching activities take place; Org take place before, whereas facilitation and direct instruction take place during the course (Arbaugh et al., 2008). According to Kovanović et al., (2018), this scenario is further emphasized in MOOC settings. Different researchers have proposed different explanations. However, as suggested by Kovanović et al., (2018), the separation between Org and facilitation and direct instruction is more obvious in MOOCs than in the traditional online courses. MOOC is likely designed well prior to students’ learning. Moreover, little facilitation and direct instruction, which manifest as two separate constructs, are allocated during the entire process of students’ learning. In addition, Wang et al., (2022) reveled that students’ perception level of Org was low in MOOC-based flipped learning. This study believes that this may also be the reason for the separation of org from the other two factors of TP. Additional research is necessary in this direction to confirm the explanation.

As for the sixth factor Aff, the first two items related to Aff were separate from the other items of SP and thus form a new factor of its own. This result is consistent with previous research (Poquet et al., 2018; Kovanović et al., 2018; Damm, 2016; Akyol et al., 2011). Large student cohorts and a short course duration render the establishment of an Aff difficult in MOOC setting (Kovanović et al., 2018; Poquet et al., 2018). One recent study (Poquet et al., 2018) compared the perceived SP among learners in three different MOOCs in edX. In MOOC with a large number of students, only learners who continue to participate in the forum activities can perceive high levels of SP. By contrast, in MOOCs with a small number of students, all learners can perceive high levels of SP. However, in the same group of learners, learners who perceive high levels of SP have low scores in Aff but high scores in group cohesion and open communication (Poquet et al., 2018). Therefore, student cohorts prevent the establishment of SP and Aff, which are the challenges faced in MOOCs. Akyol et al., (2011) discovered that long-term (thirteen weeks) traditional small online courses have more frequent affective communications than short-term (six weeks) small online course. Thus, short course duration influences the establishment of SP. However, in this study, the course duration of MOOC that lasted for 17 weeks had no effect on the results. An alternative explanation might be that the large student cohorts influenced the results instead of course duration. However, this speculation requires further verification.

It is worth noting that only the two items related to Aff were separate from the other one and form a new factor. The reason may rest on the distinct large student cohorts of MOOC and also the discriminant of the items themselves. First, although getting to know other course participants gave a sense of belonging in the course (#SP1: Getting to know other course participants gave me a sense of belonging in the course), it is a difficult job because MOOC had large student cohorts. Similarly, because of the large student cohorts, they may interact with different course participants every day, this may make it difficult for a course participant to form distinct impressions of others (#SP2: I was able to form distinct impressions of some course participants). However, large student cohorts likely does nothing to do with students’ perceptions on #SP3 (Online or web-based communication is an excellent medium for social interaction). Thus #SP3 does not loaded together with #SP1 and #SP2 to form a distinct factor. Second, according to previous studies, we understand that Aff focuses on student’s interaction with others (Carlon et al., 2012). However, item # SP3 does not align directly with any indicators of students discourse, such as using humor and self-disclosure (Lowenthal & Dunlap, 2014), while the #SP1 and #SP2 focus on students discourse. These differences may have a strong influence on students assessment on these items. Thus, students may think that item # SP1 and # SP2 were about the interaction among classmates, while # SP3 is about the medium rather than the interaction (discourse) with using the medium. Hence, they may give them difference score, finally results in # SP1 and #SP2 were separated from #SP3.

In brief, there is a lack of TP in MOOCs, which has an impact on students’ cognitive activities and causes their CP to fail to reach the Res stage. In addition, the large number of students in MOOCs may affects SP, making Aff a separate factor itself. Namely, the three additional factors are not simply extensions of the original subcomponents of CoI model, which is related to the important essential characteristics of MOOC itself. From a theoretical perspective, as we have discussed above, the six-factor model is more in line with the characteristics of MOOCs. This can help us have a better understanding of MOOCs, especially their particularity. From a practical perspective, the six-factor model can enlighten us more about designing and implementing MOOCs: The separation between Org and TP indicates that designers and practitioners have to focus on the design of MOOCs prior to students’ learning. More importantly, during the entire process of students’ learning, the learning design should be adjusted as needed. The separation between Aff and SP, indicates that designers and practitioners have to focus on the interaction between participants. The separation between Res and CP indicates that designers and practitioners have to focus on establishing enough TP to push student’s learning to achieve the resolution stage, so that they finally achieve deep and meaningful learning.

It is worth mentioning that Lowenthal & Dunlap (2014) who argued that the original CoI survey instrument should “be revisited and adjusted over time”. Kozan & Caskurlu (2018) recommended to clarify or even to enlarge the three presences of CoI instrument. Heilporn & Lakhal (2020) advised that the items should be “refined to avoid content overlaps and better define distinct categories”. However, based on the finding that the original three-factor structure model have good fit indices, we suggested that there is no need to make major changes to the original CoI instrument. Based on the finding that the new six-factor structure model is better than the three- or four- factor structure model, the present study suggested that the CoI survey instrument should consider to readjust the items under its three factors into six factors in MOOC settings.

6 Conclusions, limitations and future research

6.1 Conclusion

The current study has also verified the generalizability and persistence of the CoI instrument used in the Chinese context, which contributes to the CoI framework and instrument literature. In this study, the three-, four-, and six-factor structure models of the CoI instrument were verified in a MOOC setting using CFA. These three models all have good fit indices of the hypothetical model, and the data reached a significant level. However, the six-factor structure model with TP, SP, and CP, Res, Org, and Aff is the optimal one. The change in chi-square between the models suggested that the six-factor model with a lower chi-square value is significantly better than the four- or three-factor model. In addition, the GFI value of the six-factor model is higher, and its RMSEA value is lower than the four- or three-factor model. Moreover, the six-factor model of the CoI instrument has good convergent validity and discriminant validity. The six-factor model is better than the three-factor model, which is related to the particularity of MOOCs. MOOC is designed well prior to the course, whereas facilitation and direct instruction happen during students’ learning, which manifest as two separate constructs of TP. Limited TP prevents students’ cognitive activities from reaching the Res stage. Large student cohorts impose challenges to the establishment of affective expression.

Firstly, the six-factor structure model provide deeper conceptual and theoretical insights and expand our understanding of student learning in MOOC context. Secondly, from the perspective of practice, in MOOC settings, the designers and practitioners should combine their teaching objectives, curriculum content, platform functions, and the number and the characteristics of learners, using the items of the CoI survey instrument as a guide to design high quality MOOC. Moreover, they should pay special attention to the pedagogy to provide meaningful learning experience for students. In particular, it is necessary to pay attention to establish and maintain enough teaching presence, to promote the development of students’ cognitive presence to research the resolution (Res) stage, and also need to pay attention to affective expression (Aff) of social presence. From this perspective, this research has significance for MOOC design and implementation from the pedagogical perspective. In addition, the CoI survey instrument with six-factor structure can be used to investigate MOOC learners’ online learning experience to evaluate their learning outcomes in MOOC context.

6.2 Future research

This study has taken a critical first step in focusing on CoI in MOOCs. Whereas each MOOC is extremely different from each other in the number of learners, platform functions, and pedagogy, future research can investigate the applicability of CoI in other MOOCs settings. However, based on the findings of the present study, we suggest that future research can use the new six-factor structure CoI instrument to investigate critical issues that MOOCs face, such as learners’ perception of the three presences of the CoI to evaluate the quality of MOOCs. Moreover, how to establish and maintain the three presences in MOOCs? In designing and implementing MOOCs, what support should be provided for students with different academic backgrounds from the perspective of the three presences of the CoI framework? What effect do three presences have on students’ learning in MOOCs settings? What relationship between three presences and other important variables(such as student learning performance) related to student learning, are all worthy of further study.

6.3 Limitations

This study also has several limitations. We derived our sample data from only one single university in China. Therefore, our sample profiles may not be representative of all student learning in MOOCs and affect the generalizability of this research. More empirical studies to examine the construct validity of the CoI instrument are needed to collect data from students with different cultural background, institutions, disciplines and MOOCs learning experience. Another limitation is that although the back-translated method was used to translate the original English version CoI instrument into a Chinese version. Specifically, the original English CoI instrument was first translated into Chinese by educational technology professionals, and then translated back into English by English professionals. Nevertheless, we think this may have an impact on the results of the study.