Changing Students Minds and Achievement in Mathematics: The Impact of a Free Online Student Course

Introduction

There are a number of damaging and pervasive myths about mathematics learning in the US that are believed by millions of school children, their parents and their teachers. These different myths hold students back on a daily basis and reduce their learning and achievement significantly (Boaler, 2016). One of the most damaging is the idea that some people are born with a “math brain” and some are not, and that high achievement is only available to some students. Two areas of research are important in challenging this myth, and improving student learning. First, recent neuroscience showing the plasticity of the brain, revealing that brains can grow and change (Maguire et al., 2000). Second, research on mindset showing that when people change their ideas about the malleability of their potential, from “fixed” (my ability is not changeable) to “growth” (my ability changes as I learn) their learning and achievement improves (Dweck, 2006). Different studies, pioneered by Carol Dweck, have shown that students with a growth mindset achieve at higher levels than those with a fixed mindset (Blackwell et al., 2007; Claro et al., 2016) and that when students change their mindset their achievement changes (Aronson et al., 2002; Good et al., 2003). A second damaging myth is the idea that mathematics learning is all about procedures and memorization, rather than ideas, concepts, and creativity. Research shows that students who approach mathematics as a subject of memorization are lower achieving than those who approach it as a subject of ideas that they can think deeply about (Boaler and Zoido, 2016). A third myth that students believe is that good mathematics students have to be fast when some of the world's leading mathematicians are slow thinkers (Boaler, 2016). This study examines the impact of a “massive open online course” (MOOC) for students centered on changing these ideas and teaching students how to learn mathematics well.

The MOOC includes six modules, each of which takes 15–20 min to complete. The teacher of the course is the lead author, Jo Boaler, professor of mathematics education at Stanford, accompanied by some of her undergraduate students. Some of the key ideas in the course are:

• Everyone can learn mathematics to high levels

• Mistakes, challenge and struggle are the best times for brain growth

• Depth of thinking is more important than speed

• Mathematics is a creative and beautiful subject

• Good strategies for learning mathematics including talking and drawing

• Mathematics is all around us in life and is important—this was shown by different undergraduates showing mathematics in soccer, nature, juggling, and dance.

The course includes a series of short videos interspersed with opportunities for students to reflect on the ideas, connect with other students in the course, and work on open-ended mathematics tasks designed to shape students' perceptions related to these core ideas. (see Supplemental Note for more information about the online course.)

This paper describes the results of a randomized controlled trial (RCT) which examined the impact of the course on middle school students' engagement in mathematics class, their beliefs and mindset, and their academic achievement on state tests—the Smarter Balanced Assessment Consortium (SBAC) Summative Assessment. The SBAC assessments determine students' progress toward college and career readiness in English language arts/literacy and mathematics. These are given at the end of the school year and consist of two parts: a computer adaptive test and a performance task.

Research into the impact of free online classes—or MOOCs—has shown disappointing results with the early promise of equitable access to education being replaced with a harsh reality of low finishing rates and a predominance of privileged learners (Hansen and Reich, 2015). This study gives a very different result, showing that a strategically designed course, with careful considerations to access, significantly impacted students' mathematics learning pathways and subsequent achievement, regardless of students' gender, ethnicity, language learning level, or wealth.

Design

California school districts were recruited through a variety of announcements at conferences and workshops. School districts that were willing to provide data on the impact of the course were admitted. This study was carried out in accordance with the recommendations of Stanford University Research Compliance Office. The protocol was approved by the Stanford Graduate School of Education Institutional Review Board. All subjects gave written informed consent in accordance with the Declaration of Helsinki. Our analysis shows results from four school districts in California, with 1,090 students enrolled in 10 different middle schools across four districts. There were 439 students who took the online class, and 651 students who were control students. There were 14 teachers in this sample. Tables 1, 2 provide additional descriptive statistics about the sample.

Table 1. Observations used in the study.

Table 2. Baseline traits description.

Using a delayed-treatment research design that enabled randomization of students without the constraint of certain students getting access to a helpful course while their classmates did not, we recruited middle school teachers who taught at least 2 classes of 6th, 7th or 8th grade mathematics. For each teacher half of their classes were randomly assigned to the treatment group and half to the control group. Students in the treatment and control conditions were taught by the same teachers, thus controlling for teacher characteristics. Classes assigned to the treatment group took the online class in the first few months of the school year. Students who completed at least 4 of the modules were considered as having received the treatment.

Table 3 provides a project timeline of key activities. The students in the control group were given access to the course at the conclusion of the study.

Table 3. Project timeline.

Results

Using Ordinary Least Squares (OLS) regression controlling for baseline differences (gender, ethnicity, free and reduced lunch status, English language learner (ELL) status, and special education status) we tested for the effects of the intervention. Summary descriptive statistics for SBAC measures are provided in Table S1.

We found a treatment effect indicating that MOOC participants obtained higher scores in their SBAC math overall scale score, overall proficiency levels, and concepts and procedures (Smarter Balanced Assessment Consortium, 2013). In fact, students who receive the treatment obtained 0.33 standard deviation gains in SBAC math overall scale score; i.e., the average student in the treatment group would score higher than 63% of the control group that was initially equivalent (see Tables 4, 5). The subscales of the SBAC test were also significantly higher for the treatment group. More details on the model specifications is given in Table S2. In addition, further analyses show a positive and significant treatment effect for student subgroups defined by ethnicity, gender, economic disadvantage status, ELL status, special education status, and school grade (see Table S3).

Table 4. Regression estimates, MOOC effect on SBAC math scores.

Table 5. Standard deviations gains on SBAC math scores.

The strong design offered by the RCT performed in this study was partially offset by data access constraints. At different stages of the data collection, some schools that were part of the original study design provided incomplete data on their students. The first call for participants yielded 193 teachers who expressed interest and participated in the initial orientation to the project. On the first survey measure in August 2014, 73 teachers and 6,727 students responded in 27 school districts. By December 2014, the sample size had reduced to 31 teachers and 1,645 students in 10 school districts. The attrition in the study was largely due to technical difficulties at the school level–students needed an email address and password to take the online class, which many districts could not provide. Some schools also reported being unable to access the online course from their classrooms because of district firewall security settings that could not be resolved in the timeframe of the study. Further attrition occurred when some districts did not provide full data from state tests, usually because of staff capacity.

Importantly, the attrition was not systematic and was not linked to the outcome variables. In fact, attrition can introduce selection bias in randomized trials so this was investigated fully, as explained below. The most crucial internal validity concern when estimating causal effects is the assumption that students' assignment to treatment and control condition is random. Under this assumption, the estimates are valid if students' baseline traits are statistically similar for treatment and control students. Table 6 validates this assumption by examining whether students' traits vary with treatment/control condition (Table S4 includes complete regression information). Each point estimate is from a separate regression where each baseline student's covariate (i.e., gender, ethnicity, economic disadvantage status, limited English proficiency, and special education) is the dependent variable. The estimated effect of treatment status on these covariates is small and statistically insignificant, suggesting that students' baseline traits are statistically similar for both treated and control students. This validity check shows that the treatment and control groups are comparable and equivalent at baseline. In other words, treatment and comparison groups are statistically equivalent in every observable aspect except for the intervention, ruling out the threat of selection bias.

Table 6. Auxiliary regressions of baseline covariate balance.

Changes in Student Classroom Engagement

To understand possible mechanisms for the improved academic achievement treatment effect, the study also examined student engagement and beliefs related to mathematics teaching and learning. Participating teachers were asked to evaluate students' engagement, before and after students took the online class, in both their treatment and control classrooms. Teachers observed students along four dimensions of engagement: (a) student participates in class discussions, (b) student works as hard s/he can, (c) student appears to be involved in classwork, and (d) student gives up quickly. Table 7 indicates the two practices which showed significant differences in how students participated in class between control and treatment classes.

Table 7. Outcomes and baseline traits of students in the randomized controlled trial MOOC.

The study of student engagement demanded a lot of teacher time, and only 4 of 14 teachers returned full data on the students' engagement in math class. The data from this subset show significant effects for students who took the online course. The effect size of the treatment on student engagement was 0.47 SD (see last row of Table 5), meaning that the average student in the treatment group would score higher on the engagement scale than 68% of the control group accounting for baseline differences (see Pre/post-gains in student engagement, the last column of Table 8 for the regression estimates). Students in the treatment group participated more in class discussions and did not give up on work as quickly as their counterparts in the control classes. These findings provide insight into the reasons that students in the treatment group achieved at significantly higher levels on state mathematics tests. One compromise in our design is that because we used a “within-teacher” design, teachers were aware of which of their classes were designated as control and treatment, thus posing a potential threat to the validity of this measure. We include the engagement gain measure as a mediator variable for the positive treatment effect on standardized test introduced earlier. Our aim in using this measure was to explore possible factors through which students' beliefs about math may have resulted in deeper forms of classroom engagement, helping to explain increases in student achievement. Future studies will include student engagement surveys that do not rely on teacher reports, which will strengthen our design.

Table 8. Regression estimates, MOOC effect on student mindset surveys.

Changes in Student Mindset

A pre- and post-survey, measuring shifts in students' beliefs, was completed by 156 students and provides further insight into students' increased academic achievement. (These numbers are low as although 1,090 students took surveys, only 156 students took both the pre and the post-survey). Despite the response rate of 14%, Table S5 shows that the subset of students who completed the pre and post-survey were, in fact, representative of the larger sample group of 1,090 students.

There was a significant treatment effect on three student beliefs (see Table 8 for regression estimates and Figure 1 which compares treatment and control group survey responses). Students in the treatment had significantly higher reports of growth mindset (Mindset) and their perceptions of mathematics being an interesting and creative subject (Math Creative). They also reported feeling less fearful or easily deterred in math (Fear of Math). The specific survey items for each cluster and alpha levels are given in Table 9.

Figure 1. Student mindset survey scores and effect size by group.

Table 9. Measures of student mindset survey (cluster items with alpha levels) 1 = Strongly disagree, 2 = Disagree, 3 = Somewhat disagree, 4 = Somewhat agree, 5 = Agree, 6 = Strongly agree.

The significant changes in engagement and beliefs that students showed is likely to explain, at least partly, the increase in students' mathematics achievement. This finding supports a growing body of work that shows a link between students' mindsets about their potential and their ideas about mathematics. It is difficult to maintain a growth mindset and the idea you can learn any mathematics when the subject is presented as a series of short, closed questions—with no space for growth or learning within them. Sun (2015) showed that students developed more growth mindsets when teachers presented mathematics as subject with more opportunities for growth and learning, as opposed to performing and answering questions. The finding that students in this study shifted in seeing mathematics as a more creative subject, as well as developing more growth mindsets, supports this important link (see also Boaler, 2016).

Discussion

Consistent with previous research, this study finds a significant connection between students' mindset and their learning outcomes (Mueller and Dweck, 1998). Students in the treatment group reported more growth mindset beliefs and more challenge-seeking behaviors than those in the control group. What is distinctive about this study is the impact of an online class in changing students' mindsets toward mathematics, with subsequent changes in student achievement. Much of the research on mindset has focused on changing students' mindsets outside of any content teaching and learning; by contrast this study examines an intervention that combines mindset with changed views of mathematics and mathematical engagement. This study shows that an intervention addressing the intersection of mindset and mathematics can improve students' academic achievement, as well as students' behavior and beliefs about mathematics.

These findings are particularly important in light of continued concerns with US mathematics achievement. In the most recent international comparisons students in the US ranked 40th out of 72 countries (OECD, 2016). This is an issue that has prevailed for decades despite a vast body of research that has shown the ways to teach mathematics well (Schoenfeld, 2002; Boaler, 2015, 2016). Low mathematics achievement is not the only problem that faces the US—math anxiety is widespread among school children and the general population (Ashcraft and Krause, 2007; Foley et al., 2017). Most of the attention that is given to this issue considers the curriculum standards and textbooks used in classrooms. While these issues are important they may not be more important than a completely neglected issue—the fact that most students sit in mathematics classrooms, from kindergarten to University, thinking “I am not a math person.” In addition to this damaging belief, few students have learned to approach mathematics as a conceptual domain, rather than a set of procedures. The evidence from this randomized control trial shows the academic impact of changing these beliefs and approaches for students.

We acknowledge the limitations of our study. Our sample was drawn from middle school students in one state, and so additional studies with wider grade spans and in more varied geographical areas would be needed to generalize more broadly. In addition, for the student engagement measures, we relied on teachers' classroom observations of students. Our study would have been strengthened if we had also included student engagement surveys.

Most of the research on MOOCS has portrayed disappointing results—with online classes having low retention and perpetuating the inequities of open access that MOOCs were originally aimed to challenge (9). The online class that was the subject of this study had a different outcome of students continuing the course and significantly improving their beliefs and achievement, regardless of students' gender, ethnicity, language learning level, or wealth. The fact that this MOOC was used as part of an educational intervention and administered by teachers is part of the reason for the students continued participation. Another, we contend is the pedagogy of active engagement inside the course. Most MOOC's are lecture based, which would likely have been ineffective, even inside a classroom setting. In the “How to Learn Math” course students were invited to engage every few minutes, through answering questions, commenting on videos, and interacting with others. As MOOCs are developed and refined over the next few decades, it seems that an important advancement will be the inclusion of opportunities for more active engagement.

Many school students in the US and world are held back by damaging ideas about learning and their potential—particularly in mathematics. There is a widespread myth that students are either born with a math brain or they are not, and when students struggle they often decide they are just not a math person. The RCT that is the focus of this study has shown that students can be liberated from these damaging ideas and when they are it improves their participation and achievement. Online courses for teachers that also focus on mindset messages, and ideas for teaching mathematics actively, have also been shown to change students' achievement and beliefs (Anderson et al., under review). Together these studies reveal the importance of changing the mindsets of teachers and students, in order that students can learn mathematics without being held back by damaging beliefs. They also show the potential of online courses - which have great scalability and wide-scale access—as effective teaching opportunities, bringing some of the best teachers and the most cutting-edge research to the students who most need it.

Author Contributions

JB designed and directed the study, CW led recruitment and implementation, GP-N and KS ran analyses, the full team interpreted results, JD and GP-N contributed to and managed the writing and revising process among all team members.

Funding

This study was funded by National Science Foundation (NSF), Research on Education and Learning (REAL), Award Number 1443790. JB, Principal Investigator, Stanford University.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2018.00026/full#supplementary-material

Supplementary Note on Online Student Course

The online class that was the focus of this study has now been taken by over 160,000 participants—students of mathematics of all levels from elementary school to college. It has also been taken by tens of thousands of teachers and parents as both sets of adults are helped by knowledge of the latest research on ways to learn mathematics. In addition to individuals taking the course teachers of students as young as 5 have shared the videos with their students. The class is free and can be taken at any time and at any pace. Students can take the class in their school class, as students in the study did, or at home. The modular nature of the course has enabled teachers to use the course in a variety of ways: using the course in summer school, as a way to launch the school year, or infused throughout the year.

The class, which is also available with Spanish sub-titles, is open to anyone with an internet connection. The ideas from the class are also disseminated in different forms including papers, videos and mathematics curriculum materials on youcubed.org, a Stanford center and accompanying website of almost entirely free resources. Accompanying teacher courses on ways to teach mathematics well are also available.

References