Internet use at and outside of school in relation to low- and high-stakes mathematics test scores across 3 years

Abstract

Background

The excessive use of Internet-based technologies has received a considerable attention over the past years. Despite this, there is relatively little research on how general Internet usage patterns at and outside of school as well as on weekends may be associated with mathematics achievement. Moreover, only a handful of studies have implemented a longitudinal or repeated-measures approach on this research question. The aim of the current study was to fill that gap. Specifically, we investigated the potential associations of Internet use at and outside of school as well as on weekends with mathematics test performance in both high- and low-stakes testing conditions over a period of 3 years in a representative sample of Estonian teenagers.

Methods

PISA 2015 survey data in conjunction with national educational registry data were used for the current study. Specifically, Internet use at and outside of school as well as on weekends were queried during the PISA 2015 survey. In addition, the data set included PISA mathematics test results from 4113 Estonian 9th-grade students. Furthermore, 3758 of these students also had a 9th-grade national mathematics exam score from a couple of months after the PISA survey. Finally, of these students, the results of 12th-grade mathematics national exam scores were available for 1612 and 1174 students for “wide” (comprehensive) and “narrow” (less comprehensive) mathematics exams, respectively.

Results

The results showed that the rather low-stakes PISA mathematics test scores correlated well with the high-stakes national mathematics exam scores obtained from the 9th (completed a couple of months after the PISA survey) and 12th grade (completed approximately 3 years after the PISA survey), with correlation values ranging from r = .438 to .557. Furthermore, socioeconomic status index was positively correlated with all mathematics scores (ranging from r = .162 to .305). Controlled for age and gender, the results also showed that students who reported using Internet the longest tended to have, on average, the lowest mathematics scores in all tests across 3 years. Although effect sizes were generally small, they seemed to be more pronounced in Internet use at school.

Conclusions

Based on these results, one may notice that significantly longer time spent on Internet use at and outside of school as well as on weekends may be associated with poorer mathematics performance. These results are somewhat in line with research outlining the potentially negative associations between longer time spent on digital technology use and daily life outcomes.

Introduction

The relationship and possible impact of information and communication technology (ICT) on academic outcomes has received considerable attention over the past years. Diffusion of various devices, such as personal computers and laptops, smartphones, and tablets have the potential of enhancing productivity and efficiency of learning. For instance, it is possible to have a library of materials in one’s personal device, replacing a heavy backpack filled with textbooks. Information-seeking is easier than ever—popular search engines, such as Google, and online encyclopedias (e.g., Wikipedia) allow to browse for up-to-date knowledge, supplementing or even replacing textbooks. Various applications may be useful in visualizing mathematical concepts (e.g., the nature of a regression equation, geometry, etc), and simulations could help with understanding scientific concepts (Atit et al., 2020). Paper-and-pencil testing could be supplemented or substituted by computer-based assessment (Nissen, Jariwala, Close, & Dusen, 2018). There is also evidence that educational technology applications may have a positive effect on mathematics achievement (Cheung & Slavin, 2013; Miller, 2018).

On the other hand, it has also been shown that ICT use may not necessarily enhance academic experience as expected (Star et al., 2014). Furthermore, several studies have reported that excessive use of Internet-based digital technologies¹ is associated with impairments in daily life. Not only has it been demonstrated to inversely correlate with psychological well-being (Twenge & Campbell, 2018), it has also been shown that educational factors have negative associations with excessive Internet-based digital device use (Kates, Wu, & Coryn, 2018; Lepp, Barkley, & Karpinski, 2015; Rozgonjuk, Saal, and Täht, 2018). From an instructor’s point of view, it may be relevant to understand the association between online- and offline-teaching (Yang, 2017) as well as study habits that could be influenced by ICTs (Hora & Oleson, 2017; Rozgonjuk, Kattago, and Täht, 2018). In addition, it has been found that the potential benefits of ICTs in mathematics education context may not be fully harnessed if the educator lacks ICT-related knowledge, has minimal training as well as learning opportunities regarding ICT, and if the technical support of ICT is limited (Zakaria & Khalid, 2016). Therefore, it would be relevant to take a closer look at Internet use in relation to mathematics achievement.

In the current study, we focus on students’ general self-reported duration of Internet use at and outside of school, as well as on weekends, in relation to PISA 2015 mathematics test results and 9th- and 12th-grade national mathematics exam scores. The results could be helpful, as they may provide insights into the association between ICT and mathematics outcomes; this knowledge, in turn, could be potentially useful in designing cyber-hygiene practices that may be helpful in improved academic achievement.

Literature overview

Mathematics has been demonstrated to be a key factor in future academic success (Konvalina, Wileman, & Stephens, 1983). The results of mathematics exams could play an important role in students’ future educational path: for instance, in Estonia, a students’ admission to university may heavily depend on how well one performs on a mandatory national mathematics exam. Although there are many studies that have investigated the interplay between academic outcomes and Internet-based digital technology use, studies have not generally focused on mathematics or have focused on very specific educational technology applications or platforms (Cheung & Slavin, 2013; Fabian & Topping, 2019). Nevertheless, it has been suggested that these digital technologies may shape the future of mathematics education (Engelbrecht, Llinares, & Borba, 2020).

Even though the usage of ICTs has the potential to improve the efficiency of a classroom experience (e.g., by substituting paper-based books with digital resources, as well as by allowing the implementation of distance learning), previous studies have demonstrated that the effects of ICT use are not necessarily learning-enhancing. In fact, some studies have found that ICT use may even impair learning—this also in the context of mathematics (Bulut & Cutumisu, 2017; Zhang & Liu, 2016). Ravizza, Hambrick, & Fenn (2014) demonstrated that non-academic Internet use was negatively related to learning even when the students’ ability was controlled for.

Several factors may explain this association (Hu, Gong, Lai, & Leung, 2018). First, school-level indicators, such as the availability of Internet-based technologies as well as the size of school could play a large role (Eickelmann, Gerick, & Koop, 2017; Luu & Freeman, 2011). Second, several factors regarding individual differences in students’ perception and attitudes towards ICT use may be relevant (Hu, Gong, Lai, & Leung, 2018). Third, where (e.g., at school vs outside of school) and for what purposes (e.g., purely for learning vs entertainment) the ICTs are used may also be helpful in explaining the association between ICT use and academic outcomes (Petko, Cantieni, & Prasse, 2017; Skryabin, Zhang, Liu, & Zhang, 2015). Finally, individual differences in predisposing factors, such as personality traits, emotion regulation, and tendency to procrastinate have been shown to be relevant in developing problematic ICT use patterns (Brand, Young, Laier, Wölfling, & Potenza, 2016, Brand et al., 2019; Rozgonjuk, 2019) .

While there are studies that have investigated the relationship between Internet and other ICT use and academic achievement, research on general ICT usage patterns in relation to mathematics outcomes is rather scarce. Moreover, most of these studies tend to rely on cross-sectional data. Of note, however, it should be mentioned that Zhang and Liu (2016), for instance, have also provided evidence over a longer period of time, finding that ICT use at school is negatively correlated to academic achievement.

In the current study, we aim to present a more comprehensive set of empirical evidence on the association between Internet use and mathematics achievement. Specifically, we take into account the differences in Internet use at and outside of school and on weekends, and we include both low- and high-stakes mathematics achievement test scores. The former includes the PISA 2015 (see the Sample and procedure section) mathematics test scores, while the latter data are from two additional time points: from 9th- and 12th-grade national mathematics exams. Importantly, while PISA 2015 mathematics test could be considered as a low-stakes test—since the outcome of this test does not affect a student’s future significantly (Mägi, Adov, Täht, & Must, 2013; Silm, Must, & Täht, 2013)—the national mathematics exams can be considered as high-stakes tests within the Estonian educational system, because the results of these tests could play a proportional role in a student’s admission scores for the next stage of education.

Additionally, students’ gender and socioeconomic status (SES) may play a role in academic achievement and mathematics results. According to a meta-analysis by Voyer and Voyer (2014), female students tend to achieve better grades in school—this is a common finding, regardless of culture, school subject, or time period (based on literature from 1914 to 2011). But in addition to better academic achievement, female students also achieve better rates in higher education graduation and post-secondary school enrollments, and demonstrate better overall retention when compared with male students (Clark, Sang Min, Goodman, & Yacco, 2008). On the other hand, meta-analysis that included standardized test performance results indicates that male students tend to achieve better results in mathematics (Lindberg, Hyde, Petersen, & Linn, 2010) and natural sciences (Hedges & Nowell, 1995). Men also tend to have a higher propensity towards choosing a science, technology, engineering, and mathematics (STEM)-related career (Ketenci, Leroux, & Renken, 2020). On the other hand, female students have been found to have more positive mathematics attitudes compared with male students (Zuo, Ferris, & LaForce, 2020).

The positive relationship between students’ SES and academic achievement has been shown in many studies (e.g., Jimerson, Egeland, Sroufe, & Carlson, 2000; Liu & Schunn, 2020; Sirin, 2005; White, 1982). For instance, a meta-analysis by White (1982) demonstrated that the relationship yielded an effect size of r = .343, while a meta-analysis by Sirin (2005) found the effect size of r = .299. Therefore, one may conclude that SES generally has a medium-sized positive effect on academic achievement, including in mathematics and natural sciences. Therefore, it would be a necessary covariate in investigating links between mathematics performance and other variables.

The data set of the current study is based on an Estonian student sample. This may be of interest for mathematics as well as educational scientists in general for two reasons. Firstly, according to PISA 2015 results, the achievements of Estonian students in mathematics (as well as functional reading and science) were among the highest within the countries participating in PISA 2015 survey (OECD, 2016). Second, Estonia is also well-known for its high levels of digitalization and diffusion of e-governance (Solvak et al., 2019) that may also promote higher digital technology implementation in education. The results of this study could shed light on whether these aspects could be relevant when compared with previous findings in the field of ICT use and academic achievement (in mathematics).

Conceptual framework

One of the explanations to these findings is the so-called displacement hypothesis, according to which the negative effects of ICT use are directly proportional to time spent on one’s device. This is because time spent on a digital device decreases the potential time one could spend on reading books, exercising, and/or socializing in non-digital settings (Neuman, 1988). According to another explanation, the digital Goldilocks hypothesis, engaging too little or too much in digital technology use could result in poorer outcomes (Przybylski & Weinstein, 2017). On one hand, not using the Internet may result in a student’s inability to seek for additional information and to discuss homeworks via social networking sites. On the other hand, too much Internet use could result in a student not paying attention to learning relevant materials—especially when it is being communicated orally by a teacher. In fact, some ICT use has been shown to be related to more favorable outcomes than no or too much use in children’s subjective well-being (Przybylski & Weinstein, 2017; Twenge & Campbell, 2018) and cognitive test results (Rozgonjuk & Täht, 2017). In general, the lowest test scores were associated with individuals reporting highest time spent online.

The current study could add to this theoretical framework by providing empirical evidence that could further help the discussion on the association between ICT use and academic outcomes. Specifically, we show how groups based on self-reported Internet use at and outside of school as well as on weekends differ from each other in mathematics performance. Furthermore, the associations are controlled for students’ SES and gender, variables that are typically associated with academic performance, including STEM subjects (Sirin, 2005; Voyer & Voyer, 2014). Therefore, these results could potentially show if any of these conceptual frameworks could be used as a potential explanation for the results.

Aim of this work

The general aim is to investigate if self-reported Internet use at and outside of school and on weekends that was queried in 9th grade is associated with mathematics test performance across both low- and high-stakes mathematics testing conditions over a period of approximately 3 years. It should be noted, however, that in the current study, Internet use at school means not only using it for schoolwork, but also for leisurely activities.

Based on previous findings, there is reason to believe that the lowest mathematics achievement scores are associated with the highest self-reported Internet use. However, in addition to providing insights into Internet use at school in association with mathematics performance, we also provide evidence for how mathematics performance is correlated with Internet use outside of school as well as on weekends.

The findings of this study may be informative regarding how general Internet use could be a predictive factor for mathematics achievement. For instance, large-effect associations would mean that Internet use at school could play a pivotal role either in improving or hindering students’ achievement in mathematics. On the other hand, small or non-significant associations between Internet use and mathematics test scores may be helpful for recalibrating people’s expectations towards technology’s effects on mathematics achievement. As has been shown recently, the potential effects of digital technology use may be smaller (yet negative) than could be expected from sensationalized headlines in mass media (Orben & Przybylski, 2019) which may unnecessarily fuel technology use-related panics (Orben, 2020; Segool, Goforth, Bowman, & Pham, 2016). In other words, due to sensationalized headlines in the media, teachers, parents, and other stakeholders may be led to believe that digital technology use is more harmful than it actually may be. In the context of the current study, for instance, mathematics teachers may choose not to use digital technology-based teaching methods because of the fear that digital technology use may be detrimental for students’ achievements. Similarly, small or non-significant associations could also be helpful in curbing one’s enthusiasm regarding the hype typically associated with new technological advances. Accordingly, this study may have the potential to provide input into policymaking regarding ICT use in education (e.g., whether the general public should be concerned or not).

In addition, if the results demonstrate the associations between Internet use (e.g., the more Internet is used, the lower the mathematics results) outside of school or on weekends, there could be a reason to believe that by targeting the Internet use on one’s time outside of school could have potential to improve a student’s mathematics performance.

Method

Sample and procedure

The data that we used was combined from two sources: (1) the Economic Co-operation and Development (OECD) public PISA 2015 data repository and (2) the National Exam Data Repository managed by the Innove Foundation in Estonia. Variables regarding Internet use at school as well as sociodemographics were retrieved from the former, and national mathematics exam scores from the latter. Since the Innove Foundation was responsible for administering both PISA survey as well as national mathematics exams, the data were merged in-house based on students’ personal identification code. The data set shared with the reviewers did not include personal identifiers and was therefore fully anonymous. Merging these datasets allows to use self-report measures in combination with high-stakes exam scores from nationally standardized settings in repeated-measures study design.

The PISA is a triennial international survey which attempts to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. In the current study, we focused on the PISA 2015 Estonian sample. The PISA 2015 used a stratified systematic sample where sampling probabilities had to be proportional to the estimated number of 15-year-old students (OECD, 2016). Sampling consisted of two stages: first, schools were sampled from all schools of Estonia (which included 15-year-old students); second, students were sampled from these schools. This method should provide a high-standard sample that is representative of 15-year-old students in Estonia. The PISA 2015 survey was carried out in Estonia in April 2015. Data for all of the participating countries is available on the OECD website (OECD, 2017a). The Estonian sample used in the current study included 6147 15-year-old students participating, 49.3% of whom were boys. Students were assessed in science, mathematics, and reading comprehension. These data were merged with national mathematics exam scores administered at the end of secondary school.

As national exams are different across each year, and students who took the PISA test while in 8th grade took their exams a year later than students who were 9th graders at the time of the PISA survey, we excluded 8th graders from the effective sample so that we could consistently compare the results (remaining n = 4709). Then, we included only students who had responded to the Internet use, socioeconomic status, and gender variables (remaining n = 4113). The 4113 students (48.23% male, 51.76% female) had the PISA mathematics test scores. Out of them, 3758 students (48.32% male, 51.68% female) had a 9th-grade mathematics exam results from approximately 2 months after the PISA survey. Finally, 1612 students (51.24% male, 48.76% female) had a “wide” mathematics exam score, while 1174 students (32.45% male, 67.55% female) had a “narrow” mathematics exam score from 12th grade. Sample breakdown by Internet response variables and gender, along with each group’s average SES, are in Supplementary Table 1.

It is necessary to comment that the reason for sample attrition over time is not completely clear; it may be that some students decided not to pursue their educational path beyond 9th grade. It could also be that some of the students decided not to take the national mathematics exam in 2019 for other reasons (e.g., because of studying abroad, choosing to continue their education in a vocational school, etc.).

Another important issue necessary to be mentioned is the somewhat nested nature of the data. While the PISA data set does include some school-level information, and it may be accounted for in analyses involving the PISA mathematics test, students in Estonia may change their school when they pursue their educational path in a secondary school. In addition, many Estonian schools only provide education until 9th grade, meaning that students who would like to pursue studying need to change their school for secondary education.

Measures

In the current work, we focused on Internet use and mathematics achievement variables. Nevertheless, the PISA survey data set also includes some sociodemographic variables. We used participants’ gender (coded as 1 = female, 2 = male), and socioeconomic status (SES) index which consituted standardized z-scores computed from different relevant SES-related items queried in the PISA 2015 survey. SES index variable is standardized across all OECD countries that participated in the PISA study.

Internet use items

The study participants provided self-reported estimation for the duration of their Internet use (a) at school, (b) outside of school, and (c) on weekends. The scale for all variables was the same: 1 = No time; 2 = 1–30 min per day; 3 = 31–60 min per day; 4 = Between 1 and 2 h per day; 5 = Between 2 and 4 h per day; 6 = Between 4 and 6 h per day; 7 = More than 6 h per day.

PISA mathematics test

The PISA 2015 dataset (OECD, 2017a) includes mathematics achievement test results along with other educational test results. As each student responded only to a fraction of the entire assessment, other answers were imputed. The PISA student achievement dataset provides so-called “plausible values” for secondary analysis. Plausible values are multiple imputations of unobservable latent achievement for each student; details about the imputation procedure could be found in Wu (2005). Ten plausible values were given to all subjects within the framework of PISA 2015 (OECD, 2017b).

Wu (2005) states that it is possible to use one of the plausible values in order to recover population parameters. However, as the current study is not modeling population parameters, it is more appropriate to use all plausible values in order to retrieve more accurate parameter estimates. Therefore, we followed the guidelines by OECD (2018) and computed the results for PISA mathematics test scores across all ten plausible values in all relevant analyses.

The mathematics achievement test is meant to measure mathematics literacy (defined as students’ capacity to formulate, employ, and interpret mathematics in a variety of contexts). It includes reasoning mathematically and using mathematical concepts, procedures, facts, and tools to describe, explain, and predict phenomena (OECD, 2016).

Mathematics national exam scores for 9th graders (2018)

The mandatory 9th-grade mathematics national exams are administered to students in Estonia in spring during the last period of the 9-grade education program. In the current case, the students took the exam in spring 2015. In Estonia, a mathematics exam is compulsory for graduating from primary school (9-grade education). This exam covers all the topics taught in primary school. The 9th grade mathematics exams are scored on the scale of 0 to 100 points.

Mathematics national exam scores for 12th graders (2018)

The mandatory mathematics national exams at the end of 12th grade are administered to students in Estonia in spring during the last period of the secondary education program. In the context of the current study, the students took the exam in spring 2018. In Estonia, a mathematics exam is compulsory for graduating from secondary school. However, students can choose whether to take the simplified (“narrow”) or more comprehensive (“wide”) mathematics exam. The latter may have more relevance in continuing one’s education on the next, higher education stage. Both exams cover all topics taught in secondary school (e.g., geometry and trigonometry, among others), while tasks in the narrow mathematics exam are less comprehensive. Both exams are scored on the scale of 0 to 100 points.

Analysis

The data were analyzed in RStudio version 4.0.3 (R Core Team, 2020). In order to compute the results that include PISA 2015 mathematics test scores, we followed the guidelines by OECD (2018) and estimated all statistics across ten plausible values of those mathematics scores.

We conducted Pearson correlation analysis (p-values adjusted with the Holm’s method) to investigate the relationship between 9th- and 12th-grade national mathematics exam scores and SES using the rcorr.adjust() function from the RcmdrMisc package v 2.7.1 (Fox, 2020).

Welch’s two sample t tests were used to compare average 9th- and 12th-grade national mathematics exam scores across gender from the R’s base package. Cohen’s d–s as group difference effect size measures were computed using the cohensD() function from the lsr package v 0.5 (Navarro, 2015).

In order to compare groups of students based on their self-reported Internet use at and outside of school and on weekends, we computed a series of analyses of covariances (ANCOVAs) using the ancova() function from the jmv package v 1.2.23 (Selker, Love, & Dropmann 2020), controlling for gender and SES effects. These ANCOVAs were computed with 9th- and 12th-grade national mathematics exam scores as dependent variables in each model. In addition, Holm’s post hoc tests were used for computing the differences between each pair of groups, complemented with effect size estimates (Cohen’s d–s) for group differences.

Commonly used effect size benchmarks according to Cohen (1988) and Sawilowsky (2009) are reported for post hoc comparisons, where d = 0.01, 0.20, 0.50, 0.80, 1.20, and 2.00 denote “very small”, “small”, “medium”, “large”, “very large”, and “huge” effect sizes, respectively. This said, we also report the recently proposed benchmarks by Kraft (2020) who argues that, in educational context, it may be more justified to interpret d < 0.05, d < 0.20, and d > 0.50 as “small”, “medium”, and “large” effect sizes, respectively. Partial eta-squared (η²_p) are presented as effect size measures in ANCOVA models. Commonly used benchmarks are η²_p = .010, .059, and .138 for small, medium, and large effects, respectively (Cohen, 1988; Richardson, 2011).

Results

Descriptive statistics for and correlations between SES and mathematics scores

In Table 1 below, we present the descriptive statistics and correlations for SES and mathematics scores.

Table 1 Descriptive statistics and correlations between SES and mathematics test scores

As can be seen from Table 1, SES is positively correlated with all mathematics variables, yielding small-to-medium effect sizes. PISA mathematics test scores have rather medium-to-large positive associations with 9th- and 12th-grade exam scores. Ninth-grade national mathematics exam scores have strong positive correlations with 12th-grade exam scores.

The results from Welch’s two sample t tests showed that male students (M = 534.72, SD = 80.55) had higher scores in PISA 2015 mathematics test (estimated across all ten plausible values) than female students (M = 525.56, SD = 76.31), t(4047.95) = − 3.737, p = .002, d = .117. Girls (M = 66.36, SD = 22.62) scored higher in the 9th-grade mathematics exam than boys (M = 63.66, SD = 24.03), t(3696.2) = 3.54, p > .001, d = 0.12. While there were no gender differences in the “wide” mathematics exam scores, t(1608.5) = 0.77, p = .444, d = 0.04, girls (M = 43.02, SD = 24.54) had higher scores than boys (M = 37.92, SD = 23.78) in the “narrow” mathematics exam, t(771.51) = 3.403, p < .001, d = 0.21.

Group differences in mathematics scores based on internet use

Below, the results for ANCOVAs are presented. Figure 1 depicts group differences based on self-reported Internet use at and outside of school and on weekends across the PISA mathematics test (estimated across ten plausible values), 9th-grade national mathematics exam, and 12th-grade (both “wide” and “narrow”) mathematics exam scores, controlled for potential SES and gender effects (estimated marginal means are depicted on y-axis). Holm’s post hoc comparisons for 9th- and 12th-grade exam scores are presented in Supplementary Table 2.

Fig. 1

Estimated marginal means of mathematics scores based on Internet use at and outside of school and on weekends, controlled for SES and gender. Notes: On x-axis (Self-reported Internet Use in 2015): 1 = “No time”; 2 = “1–30 min per day”; 3 = “31–60 min per day”; 4 = “Between 1 and 2 h per day”; 5 = “Between 2 and 4 h per day”; 6 = “Between 4 and 6 h per day”; 7 = “More than 6 h per day”. G12 (Narrow) = Grade 12 national mathematics exam (narrow) score; G12 (Wide) = Grade 12 national mathematics exam (wide) score; G9 = Grade 9 national mathematics exam score; PISA PV = PISA 2015 mathematics test scores (estimated across ten plausible values as suggested in OECD (2018). The 95% CIs are depicted.

Overall, the results indicate to either curvilinear or linearly decreasing scores with the increase of self-reported Internet usage. As can be observed below in Tables 2, 3, 4, and 5, SES was a significant covariate in all models predicting mathematics results. In addition, significant group differences due to Internet use yielded small (based on Cohen, 1988, and Sawilowsky, 2009) or medium (based on Kraft, 2020) effects. Below, the results for each model are presented.

PISA mathematics test scores and internet use

Table 2 shows that Internet use variables as well as SES and gender were all significant predictors of PISA mathematics test scores in all models. It could be observed that the effects of SES are consistent throughout all models, yielding medium size. While gender has consistently small effects, Internet use at school has medium, while Internet use outside of school and on weekends have small-sized associations with PISA mathematics test scores.

Table 2 Results of ANCOVA analyses for PISA 2015 mathematics test scores (estimated across ten plausible values) and Internet use variables, controlled for SES and gender

Post hoc test results (in Supplementary Table 2) show that, in general, some Internet use at school has positive associations with PISA mathematics test scores, whereas the no or 6+ h per day of Internet use at school tend to be negatively associated with PISA mathematics test scores. In general, one may also notice that there may be a sweet spot of Internet use at school after which there seems to be a linear decline in PISA mathematics scores.

Within Internet use outside of school, somewhat similar patterns could be observed: students who reported no or 6+ h per day of Internet use outside of school had lower PISA mathematics test scores than students who reported some Internet use outside of school.

Finally, similar patterns described above tended to be also present in relation to Internet use on weekends. Here, too, the significant group differences tended to emerge either between students who reported not using Internet on weekends and students who reported using Internet on weekends between 2 and 4 h a day, or between students who reported using the Internet for 6+ h per day on weekends and those who tended to use the Internet on weekends for a couple of hours per day.

In general, therefore, students who either reported that they did not use Internet or used it for 6+ h per day tended to score lower than other students.

Ninth-grade mathematics national exam scores and internet use

The results in Table 3 show that, as with PISA mathematics scores, 9th-grade mathematics national exam scores were predicted by Internet use variables as well as SES and gender in all models. While potential gender effects were small, SES yielded small-to-medium effects. Internet use variables, too, had small associations with 9th-grade mathematics national exam scores. Highest partial eta-squared for Internet use was in the model regarding Internet use at school.

Table 3 Results of ANCOVA analyses for 9th-grade mathematics national exam scores and Internet use variables, controlled for SES and gender

Holm’s post hoc test results (in Supplementary Table 2) indicated to similar patterns in 9th-grade national exam scores as observed in PISA mathematics test: in general, moderate Internet use was associated with better outcomes than no or 6+ h per day of Internet use at school. Somewhat similar patterns could be observed for Internet use outside of school. Interestingly, when looking into Internet use on weekends, only some of the differences that were significant included 6+ h per day of Internet use—indicating to poorer outcomes than when Internet was used for a couple of hours per day on weekends.

Twelfth-grade “wide” mathematics national exam scores and internet use

The results in Table 4 are somewhat different than in the case of PISA and 9th-grade mathematics national exam scores. While SES remains to be a significant covariate predicting 12th-grade “wide” mathematics exam scores (with small effect sizes), gender is not a significant covariate in any models. While Internet use variables were significant predictors, the effect sizes were small, with highest partial eta-squared in the model for Internet use at school as predictor.

Table 4 Results of ANCOVA analyses for 12th-grade “wide” mathematics national exam scores and Internet use variables, controlled for SES and gender

Results of Holm’s post hoc tests (in Supplementary Table 2) show that across Internet use at school, there seems to be a relatively linear decline in mathematics scores with the increase of Internet use. Students who reported using the Internet at school for 6+ h per day scored lower on the “wide” mathematics national exam than students who reported not using the Internet at school. In general, students reporting using the Internet at school for 6+ h per day at school tended to have the lowest scores in mathematics exams.

While there were no group differences in “wide” mathematics scores between students who reported not or moderately using the Internet outside of school, 6+ h per day of Internet use outside of school was associated with poorer mathematics exam scores.

Finally, there were no group differences in the “wide” mathematics exam scores across Internet use on weekends.

Twelfth-grade “narrow” mathematics national exam scores and internet use

According to Table 5, SES and gender were significant covariates predicting 12th-grade national “narrow” mathematics exam scores (with small-sized effects). While Internet use at and outside of school had small yet significant associations, Internet use on weekends was not a significant predictor of these mathematics exam scores.

Table 5 Results of ANCOVA analyses for 12th-grade “narrow” mathematics national exam scores and Internet use variables, controlled for SES and gender

Across these exam scores, the Holm’s post hoc test results (in Supplementary Table 2) showed that using the Internet at school for up to 30 min tended to be associated with better “narrow” mathematics exam outcomes than using the Internet for more than 2 h per day at school. Students reporting using the Internet for 6+ h per day outside of school scored lower than students who reported using the Internet for between 12 h per day. There were no group differences in “narrow” mathematics scores based on Internet use on weekends.

Discussion

The main aim of the present study was to investigate how students’ Internet use at and outside of school and on weekends predicts mathematics performance across several years in low- and high-stakes testing conditions. Below are some of the insights into the findings as well as the contribution and limitations of this study.

We expected to find that students who report using Internet the most (at and outside of school, on weekends) to have the lowest average scores in mathematics tests. This was generally the case in all exams across all Internet use conditions—with the exception of Internet use on weekends as a predictor in 12th-grade mathematics exam scores. The results of this study are mostly in line with some previous, more general findings that have demonstrated the small negative relationships between Internet-based technology use and academic achievement (Kates, Wu, & Coryn, 2018; Lepp, Barkley, & Karpinski, 2015; Rozgonjuk, Saal, et al., 2018), as well as in the domain of mathematics education (Bulut & Cutumisu, 2017; Eickelmann, Gerick, & Koop, 2017; Hu,Gong, Lai, & Leung, 2018; Skryabin, Zhang, Liu, & Zhang, 2015). Yet—and especially in the case of Internet use at school—it also seemed that both the students who reported using the Internet for 6+ h per day at school as well as students who reported not using the Internet at all had lower mathematics test scores than students who reported some Internet use. This finding is in line with the digital Goldilocks hypothesis which states that there may be an optimal time of ICT use which may be associated with favorable outcomes (Przybylski & Weinstein, 2017). Moreover, these patterns seemed to be highly similar across different mathematics tests over a 3-year period.

What do these findings tell us? From one hand, these results support the notion that perhaps general Internet use, especially for a very long duration, may not be unidirectionally beneficial to mathematics performance. Perhaps too much Internet use may lead to activities which are irrelevant to learning objectives and outcomes, therefore resulting in poorer performance on mathematics tests. Using Internet during classes could potentially contribute to multitasking behavior while studying, which in turn could lead to distraction and be detrimental for understanding class context (Sana, Weston, & Cepeda, 2013). In fact, a recent study demonstrated that interruptions due to pop-up notifications are associated with more surface approach to learning (Rozgonjuk, Elhai, Ryan, and Scott, 2019).

On the other hand, it should also be stressed that these associations, although negative, are relatively small. Of course, it also depends on what exactly constitute small effects. Recently, Kraft (2020) argued that perhaps research in the field of education should not rely on common benchmarks proposed by Cohen (1988), since effect sizes in educational research may be small yet still pragmatically relevant. By benchmarks proposed in Kraft (2020), several group differences yielded medium-sized effects. This said, while the negative associations of Internet use seem to be present in some conditions, there are probably other more important aspects of education that drive the mathematics performance of a student. Mapping these variables should be in focus of subsequent studies.

Although it may be hardly plausible to assume that a teenager’s Internet use may affect their mathematics performance after 3 years, the results seem to suggest that there may be some substance to it that deserves further attention. On one hand, it could be that Internet use patterns do not change dramatically over time, potentially explaining these findings. Of course, although the current study has a repeated-measures approach, this causal hypothesis cannot be answered with a high degree of validity based on the data used in this study. On the other hand, there may be an alternative explanation for these findings. Importantly, it should be stressed that a student’s socioeconomic status (SES) was positively associated with all mathematics scores both in bi- and multivariate analyses. Furthermore, with the exception of “wide” mathematics exam scores, gender was also a significant covariate in models predicting mathematics performance. While boys scored higher in PISA mathematics test, girls had higher scores in 9th- and 12th-grade “narrow” mathematics exam scores. These findings are in line with common results in educational research, including mathematics, where SES and gender are associated with academic performance (Lindberg, Hyde, Petersen, & Linn, 2010; Sirin, 2005; Voyer & Voyer, 2014; White, 1982). It is nevertheless curious that even when mathematics results were controlled for these covariates in the current study, some of the relationships between Internet use variables and mathematics test scores remained significant, albeit small. In general, no Internet use or using the Internet for a very long time (e.g., 6+ h per day) was associated with poorer outcomes.

The main contributions of the current study regard the nature of the data, and some novel findings. First, the PISA 2015 sampling standards are very high and resemble the whole population of 15-year-old students in Estonia. Therefore, this sampling method provides more generalizability with regards to results, because one of the more common limitations in survey-based studies—convenience sampling and self-selection bias—are mitigated. Second, in addition to self-reported Internet use, the data included actual (that is, not self-reported) results for mathematics performance. Furthermore, these results were obtained for both low-stakes (PISA 2015 mathematics test) as well as high-stakes tests (9th- and 12th-grade national mathematics exams). This should support the validity of findings. While PISA 2015 mathematics test scores correlated strongly with national mathematics exam scores, the patterns in the relationships between Internet use variables and mathematics scores were similar, regardless of the low- or high-stakes testing conditions, and even when the possible effects of students’ SES and gender were taken into account. Third, mathematics results from three time points were used, further increasing validity and reliability of findings. While repeated-measures design provides a stronger case for causality in the reported relationships (Cole & Maxwell, 2003; Gollob & Reichardt, 1987), in order to establish a causal link, chronological order of measurements may not be sufficient, and a (quasi-)experimental study should be conducted to replicate these findings with more robust confidence. Therefore, we also want to explicitly state that our results do not support strong causal interpretation.

Of more theoretical contributions, the results show that the association between Internet use and mathematics achievement could be predicted over a period of three years, both across low- and high-stakes testing conditions. Furthermore, the results also seem to indicate that it may be necessary to distinguish Internet use at and outside of school (also Internet use on weekends) in this line of research, since the relationship patterns with mathematics achievement may vary due to that. This is also in line with some previous findings (Petko, Cantieni, & Prasse, 2017; Skryabin et al., 2015).

The limitations of the study ought to be addressed as well. Firstly, although the mathematics scores were not self-reported, Internet use variables did rely on self-reports. Studies have shown that people are not very accurate in estimating their digital technology use duration and frequency in relation to objectively measured device use (Boase & Ling, 2013; Kobayashi & Boase, 2012; Loid, Täht, & Rozgonjuk, 2020; Rozgonjuk, Levine, Hall, & Elhai, 2018). Therefore, future studies should aim towards documenting, or tracking, the actual digital device use in classroom, e.g., as done in learning analytics (Schneider, Reilly, & Radu, 2020) and digital phenotyping in other fields (Baumeister & Montag, 2019; Rozgonjuk, Elhai, & Hall, 2019). This could provide a more valid picture regarding the potential effects of digital technology use in relation to mathematics achievement.

Secondly, as mentioned earlier, the causal interpretation regarding the effects of Internet use at school is largely based on chronology of measurements—this, of course, does not necessarily mean that there is a direct causal effect. It could also be that other factors influence both Internet use (or how the student recalls their typical Internet use) and mathematics performance. For instance, individual differences could influence both (self-reported) technology use and academic outcomes. For example, more conscientious students could engage in less non-purposeful digital technology use and are also more disciplined to attain to learning tasks. In addition, one may also hypothesize that mathematics anxiety hinders learning mathematics and may motivate using ICTs instead of learning. It has recently been demonstrated that mathematics anxiety is associated with more surface approach to learning (Rozgonjuk et al., 2020), a factor also associated with problematic ICT use (Alt & Boniel-Nissim, 2018).

A third limitation is using rather general Internet use measures and correlating these self-reported estimates with mathematics scores. In order to gain insights into the potential effects of digital technology on mathematics, it would be highly informative to include data about digital technology use in mathematics learning, specifically. It could be questionable whether students really spend 6+ h per school day online. The question asking about the duration of Internet use at school does not specify whether the activities spent on the Internet are school-related or not. Therefore, it could be that students who reported using 6+ h of Internet per school day also included Internet use outside of coursework (e.g., during recess, etc) in their estimation. Clearly, this is a limitation of the study, and should be taken into account in future investigations.

Finally, it should be noted that it was not possible to account for the nested nature of the data for the national exam scores, and the reason for sample attrition over time was not completely clear. Further research should address these limitations.

In conclusion, this study demonstrates that the relationship between Internet use and mathematics achievement is rather curvilinear, but students who reported using the Internet for 6+ h per day tended to have, on average, lower mathematics test scores than students reporting less Internet use. This said, it is also interesting that these patterns may have variations depending on where and when Internet is used—either at or outside of school, or on weekends.