Describing undergraduate students’ reasoning and use of evidence during argumentation about socioscientific issues systems

Introduction

A goal of science education is to facilitate students’ ability to make sense of scientific information and make science-informed decisions in the context of complex issues they will confront as citizens (NRC, 2012; OECD, 2013; NASEM, 2016; European Commission, 2019). Students’ appropriate interpretation and application of scientific information to explain complex problems within science-related social contexts is a means to support personal and societal-level decision making (Roberts, 2007; Feinstein, 2011; Yacoubian, 2018). Enactment of this complex practice in science classrooms remains a challenge for both students and educators.

Asking students to reason about evidence is a primary way instructors support students’ ability to appropriately interpret, synthesize, and apply scientific information. Many of the studies and interventions that support student reasoning about evidence are focused on argumentation (Osborne et al., 2016), particularly of discipline-specific science concepts, such as explaining biology or chemistry phenomena. A smaller subset of the evaluation of evidence and argumentation literature is situated within SSI contexts (Cavagnetto, 2010; Henderson et al., 2018), which are complex ill-structured social issues with conceptual or technological ties to science (Zeidler, 2014). SSI contexts differ from disciplinary contexts in that they offer students the opportunity to reason about multidisciplinary (scientific, economic and socio-cultural) evidence, and how evidence intersects with values, ethics, norms and beliefs (Evagorou and Osborne, 2013). The social context of SSI can intersect with students’ identities and world view, sets the potential for more complicated reasoning in comparison to reasoning about strictly natural phenomena. Additionally, in comparison to evidence evaluation in the context of less familiar biology or chemistry phenomena (e.g., cell division or redox reactions), many students are often familiar with SSI contexts and have already formed ideas and opinions about phenomenon and mechanisms that arise from their experiences. This can be an advantage, allowing students to draw on existing resources and knowledge in their reasoning (Osborne et al., 2004), but can also be problematic in terms of open-minded evidence evaluation, particularly if students have a stake in the issue or vested interest arising from a personal worldview. Prior knowledge or attitudes about the issue may allow students to engage in motivated reasoning when constructing arguments in an SSI context (Kunda, 1990; Sinatra et al., 2014), considering only evidence that supports rather than challenges ideas they may have already formed about the issue (Wu and Tsai, 2011; Epley and Gilovich, 2016). Other SSI researchers have observed the dominant role that students’ prior knowledge, value, and experience play in SSI decision making, naming the notion “intellectual baggage” (Rundgren et al., 2016). Because of the increased complexity emerging from the interdisciplinarity and interacting social components in the context of SSI, there is a need to clearly define aspects of SSI argumentation, and carefully consider how best to investigate and characterize students’ practice of applying evidence to support their reasoning.

In this study, we engaged in qualitative research to achieve our primary aim: describing students’ proficiency in reasoning about evidence when constructing arguments in the broader context of SSI decision making. We used science argumentation literature to help separate and define aspects of SSI argumentation. We also lean on philosophy and psychology literature to consider the role of assumptions in science, which were a prominent feature in our resulting framework of students’ SSI argumentation.

Needs for distinguishing argument quality in the context of socioscientific issues

There are multiple types of arguments with different epistemic functions that exist within SSI argumentation. One type of argumentation embedded in SSI argumentation is “what we should do,” which is often the culminating decision-making focus of classroom activities and requires students to reason about multidisciplinary systems plus bring to bear values, morals, ethics, and preferences for desired outcomes in considering SSI solutions. One part of the “what we should do” argument may be considered another type of SSI argumentation that involves weighing of priorities, for example, “are economic advantages more important than species conservation?” that require students to give ethical or moral arguments. There are no definitive answers to the question of how to weigh priorities, so there are no definitive solutions to the “what we should do” question, and proposed solutions cannot be determined by empirical or theoretical evidence alone (Sadler, 2011; Bächtold et al., 2022).

Within an “what should we do” argument about SSI, students must consider “how something occurs” in SSI systems (which is the focus of this study). Arguments about SSI systems span natural phenomenon and disciplinary systems as well as large-scale human systems that include society, economics, culture, environment, health, policy, etc., and requires multidisciplinary reasoning. Arguments of “how something occurs,” more closely resembles disciplinary science argumentation in education that is generally described as the justification of claims with reasoning and evidence, whether empirical or theoretical (Erduran and Jiménez-Aleixandre, 2007), and centers on explanations of natural phenomenon or systems, or valid answers to research questions. There are challenges in determining the quality of student arguments about “how something occurs” in SSI systems because they are multi-faceted, with a variety of possible lines of evidence. For example, Jensen (2002) classified knowledge about environmental SSIs into four dimensions: (1) what kind of problem is it? (knowledge about effects); (2) why do we have the problems we have? (knowledge about root causes, including the socioeconomic sphere); (3) how do we change things? (knowledge about strategies for direct and indirect action); and (4) where do we want to go? (knowledge about alternatives and visions). Yet much of the literature describing student arguments and explanations in SSI-contexts is limited to science concepts (e.g., Sadler and Zeidler, 2005; Lombardi et al., 2017; Zangori et al., 2017). Our study expands the scope of the work on SSI to incorporate multidisciplinary arguments about causes and effects as well as efficacy of strategies for societal change, by investigating student claims about how a particular SSI solution may impact the SSI system.

Generally, frameworks to analyze differences in the quality of students’ SSI argumentation are limited. Many of the assessments of SSI argumentation in prior literature have emphasized structure of arguments (e.g., Venville and Dawson, 2010; Capkinoglu et al., 2020) or a count of justifications and degree of elaboration (e.g., Sadler and Fowler, 2006; Wu and Tsai, 2011; Christenson et al., 2014; Baytelman et al., 2020). While these approaches are useful, particularly to understand the mechanics of student argumentation, they fall short in evaluating the complicated content of student SSI argumentation. Patterns of informal reasoning have been described and commonly used in SSI argumentation (Sadler and Zeidler, 2005; Grooms et al., 2014) specifically – rational (i.e., grounded in logical reasoning), emotive (i.e., driven by care, concern or sympathy/empathy toward others), and intuitive (i.e., arguments predicated on gut-level reaction to an SSI) reasoning. Though the informal reasoning framework provides a useful descriptive taxonomy, it is not meant to evaluate the quality or content of the arguments themselves. Additionally, this framework is used when students give an opinion on an SSI with a rationale, rather than to compare possible solutions to an SSI problem or engage in decision making, which does not separate epistemic aims of arguments about how the system works, (which may elicit rational reasoning), compared to “should we” and “weighing” arguments (which necessitate emotional or intuitive reasoning tied to values, morals, or ethics). These examples underscore the need for conceptual ideas for understanding differences in students’ SSI argumentation quality that also clearly delineate the epistemic function of the argument.

Conceptualization of assumptions in SSI argumentation about SSI systems

Evaluating and applying evidence to explain SSI systems is a complicated task that does not occur devoid of social context. Our work is informed by social constructivism (Bingle and Gaskell, 1994; Zeidler et al., 2005), which is a theoretical perspective stating that knowing and learning is situated in social practice, and that understanding of the world is co-constructed through interactions with others inside and outside the classroom (Vygotsky and Cole, 1978). Importantly, students are influenced by their own prior and current social experiences, identities, motivations, and discourses. In particular, cultural experiences and personal narratives have been shown to influence students’ reasoning and decision making about SSI (Levinson, 2008; Simonneaux and Simonneaux, 2009; Evagorou et al., 2012; Gupta et al., 2022). This study focused on the socially constructed assumptions that students bring to the classroom when arguing about SSI systems.

When people interact with the world around them, their actions are influenced by a set of beliefs, or “rules of inference,” that have shaped their reality that influences how they function in the world. Once established, people often do not push beyond the paradigm’s rules but rather mold everything they see to fit within it (Kuhn, 2012). These rules influence assumptions, or natural thoughts about the way the world should be. Our study uses the definition that assumptions are “less-than-fully established propositions” (Reitman, 1964; Fortus, 2009) held to be true without need of further evidence. Assumptions are latent and often unobserved dispositions that help people explain a complex world (Quine and Ulian, 1978). However, assumptions, particularly the explicating of assumptions, play an important role in scientific practice. Scientists often develop theory by making explicitly informed assumptions based on prior knowledge and scientific evidence (Kuhn, 2012). Explicit assumptions help set parameters for problem-solving and are an important part of the scientific process (Seino, 2005; Fortus, 2009; Milbourne and Wiebe, 2018). Scientists rely on arguments from evidence (Walton, 1996), or use evidence to question assumptions that explain phenomena in the world.

The complication in relying on assumptions is that they may not always be accurate in every context, and unless they are explicitly noticed and examined in the light of evidence, making decisions based on assumptions could be problematic. When confronted with new evidence that contradicts assumptions, people generally reconcile the new information by modifying their assumptions or disregarding new information. How, and if, they acknowledge the logical inconsistency between their assumptions and new information can depend on multiple factors. The ability to notice, verify, or refute assumptions can vary depending on the extent of scientific training and knowledge of the problem-solver (Schraw and Moshman, 1995; Fortus, 2009; Milbourne and Wiebe, 2018). Additionally, the potential for the information to be more plausible than the assumption, perceived source and author credibility, along with motivation, cognitive and emotional dispositions toward the information can all play a role in a person’s ability to examine assumptions (Sinatra and Chinn, 2012; Lombardi et al., 2016). Assumptions can inform cognitive biases or may take the form of simple heuristics that are applied to the problem (Kahneman, 2011; Sinatra and Chinn, 2012). Therefore, given these multiple factors, the possibility exists of individual differences among students in their ability to notice their assumptions and verify or refute the assumption based on evidence.

Argumentation creates an environment where students practice negotiating uncertainty, generated by differing claims and evidence created by conflicting or incomplete knowledge (Chen and Qiao, 2020). Ideally, in science classrooms students are trained to construct arguments from evidence by noticing uncertainty and questioning their assumptions, as well as seek evidence to validate assumptions, or be open to evidence that contradicts their assumptions (Walton, 1996). This skill may be especially important as students provide arguments to explain social, economic, and natural systems embedded in a controversial SSI-context. Students enter science classrooms with their own assumptions of how the world works based on their lived experiences that may have various degrees of accuracy. However, implicit assumptions can potentially cause students to forgo seeking evidence, not be open to contradictions, and quickly discard or misinterpret evidence. For example, students could engage in confirmation bias by showing an affinity toward evidence that aligns rather than contradicts their personal beliefs and assumptions (e.g., Sadler et al., 2004). How students create and support SSI claims with evidence may indicate how they monitor their existing assumptions and may provide a useful framework to reveal student proficiencies in SSI argumentation.

Current study: argumentation embedded in a decision-making context

To describe how students construct arguments about multidisciplinary scientific evidence in an SSI-context, we collected data within a course that centered on SSI decision making. Decision making is a multi-step process in which students are presented with a problem, set specific criteria that a solution should meet, identify several potential solutions, employ scientific data, knowledge and practices to analyze evidence regarding the potential solutions, and consider tradeoffs in ultimately deciding on one solution to the problem (Acar et al., 2010; Garrecht et al., 2018; Kirby and Dauer, 2021; Dauer et al., in review). We consider decision making to be a complete argument that addresses “what we should do” by proposing a desirable direction to ameliorate an SSI as a culmination of the entire decision-making process (Sparks et al., 2022). Within the multi-step decision-making process, there are opportunities for learners to engage in different SSI argument types, including constructing and evaluating evidence-based claims about the SSI system (Jiménez-Aleixandre et al., 2014; Jiménez-Aleixandre and Brocos, 2017). We focused on one step of the decision-making process that asks students to present an argument about the performance of each alternative solution based on evidence, embedded in decision making, as opposed to the entire decision-making process. Several researchers have studied SSI decision making in the classroom in recent years (e.g., Fang et al., 2019; Alred and Dauer, 2020; Dauer et al., in review), though we do not extensively review this literature here as students’ decision making as a whole was not the focus of our research.

Our research aimed to characterize students’ ability to reason about evidence by asking the following research question: what are the levels of proficiency that describe how students use evidence to support their claim about SSI systems? We also compared student performance across the two semesters to support the validity of the levels of proficiency in the evidence evaluation construct, expecting increased proficiency after research-based course revisions.

Methods

Research design

This study employed qualitative methods (Creswell and Poth, 2016), to engage in foundational research that described students’ levels of proficiency in using evidence and reasoning to support their claims when constructing arguments in an SSI-context. We collected students’ responses in fall 2016 and inductively coded to create and refine a coding scheme. We then collected students’ responses in a revised course in fall 2018 and qualitatively coded a subset of student responses from two groups of students fall 2016 and fall 2018, blinded to semester. Our research was done with the approval of an institutional IRB (# blinded).

Instructional setting

A science literacy course, [blinded] in a large Midwestern university, incorporated a structured decision making (SDM) process, which served as an external representation of students’ reasoning about SSI, and was designed to improve students’ decision making and information literacy practices (Ratcliffe, 1997; Kolstø, 2001; Grace, 2009; Acar et al., 2010). The SDM process (Gregory et al., 2012; Hammond et al., 2015) used was a stepwise process that is published elsewhere (Dauer et al., 2017, 2022; Alred and Dauer, 2020; Jimenez et al., 2023). Several SSI-contexts were used in the SDM context throughout the class semester. We collected data during the final iteration of the SDM in an SSI framed as “how do we conserve water in the state?” At the start of the SSI module, instructors defined the issue through a general background of the SSI. Then students proposed desired goals that were collated by the instructor into four discrete Objectives (Figure 1). Students assigned weights to the four student-generated Objectives based on how personally important each Objective is; the weights’ sum must equal one. Third, instructors presented.

Figure 1. An illustration of the SDM matrix that students use to analyze the water SSI, with each OBJ x ALT combination present in the module assessment. Also illustrated are hypothetical student weights for Objectives, and hypothetical Objective 1 performance scores for each Alternative. In the fall 2018 semester, performance scores for Objectives 3 and 4 were given to students so they could focus on evidence for Objectives 1 and 2.

four Alternatives (Figure 1), or solutions, to the SSI. Fourth, students were asked to find and evaluate information to predict how each of Alternatives might satisfy each of the Objectives. Instructors directed students to seek information from multiple sources including class content, online sources, peer-reviewed studies, and experts, including those in their own personal networks. In this step, students used their research to claim how well the Alternative might satisfy an Objective based on the evidence they found. The 16 resulting arguments (hereafter, ALT × OBJ) completed a matrix (Figure 1) with “performance scores” that ranked which Alternative performed the best in achieving each Objective. Ultimately students use total weighted performance scores based on this matrix to determine the best performing Alternative based on the individual students’ Objective priorities. To complete the matrix, student groups were researched one ALT × OBJ argument, wrote an argument, got feedback from instructors on an initial draft, and then presented their research and argument in a recitation section. Then students individually completed 16 ALT × OBJ arguments (in fall 2016) or 8 ALT × OBJ arguments (in fall 2018) as part of their module summative assessment. The module assessment required students to work through each step of the SDM process, with grading emphasis associated in their application of evidence to support their decision-making process.

Data collection and analysis

We collected data from de-identified module assessment arguments of consenting undergraduates in the science literacy course. Students in this required course self-reported a major we classified as STEM (82%), as freshmen (57%), and from rural areas (41%). At the end of the semester in 2016, we collected arguments from students’ (n = 160) water SSI module assessments. Since each student provided more than one argument, throughout we will use notation to indicate the number of students (n_s) and the number of arguments (n_a).

Design and development of coding schemes

We engaged in inductive developmental coding to describe the levels of proficiency that we observed in how students used evidence when arguing about complex phenomena in an SSI-context. We first randomly selected a subset of students (n_s = 30 out of 160 students from two lecture sections in the course) from the fall 2016 semester and coded arguments pertaining only to four Alternatives (status quo, less restrictions on irrigation, more restrictions on irrigation, encourage the use of new technology) and four Objectives (farmers remain economically profitable, water is conserved for future use, sufficient household water, and sufficient water for wildlife and their habitat) (total ALT × OBJ arguments n_a = 64). The research team used constant comparison (Creswell and Poth, 2016) in multiple coding iterations to consider meaningful differences in proficiency in students’ claim, evidence, and reasoning in evaluating the quality of their argument. Based on the central patterns observed we transformed related codes into broader categories (Creswell and Poth, 2016) that represented proficiency levels.

One consistent pattern we noted was students basing claims on assumptions about how Alternatives would satisfy Objectives, or simplistic lines of reasoning commonly provided by students without any evidentiary support. Assumptions were common in students’ ALT × OBJ arguments despite directions and a grading rubric that asked for specific information to support their ideas, including evidence discussed in class or researched. Throughout developmental coding, we attempted to separate students’ reasoning that addressed assumptions using evidence, from other qualities of the argument. We noted two distinct practices, variability in students’ ability to (1) provide reasoning about evidence to address assumptions (RAAVE), and (2) link evidence to their reasoning (LER). Additionally, we coded the source-type of evidence (SOE) students provided in their arguments. We reached 71% interrater agreement on the RAAVE coding scheme and 83% on the LER coding scheme for 3 raters before consensus discussions where all discrepancies were resolved.

Course revisions and evaluation

During developmental coding of fall 2016 arguments, we noted that most of the students were in the lowest proficiency levels in our coding scheme. We substantially revised the course to address this concern in multiple ways including decreasing the number of ALT × OBJ arguments students were required to write in their module assessments from 16 to 8 (students focused only on OBJ 1 and 2, while arguments were provided from the instructors for OBJ 3 and 4 so students could still complete the full tradeoff analysis), and creating grading rubrics that more clearly indicated proficiencies in providing reasoning about evidence. We randomly selected students from each semester (fall 2016 n_s = 44; fall 2018 n_s = 30) and coded four arguments per student (fall 2016 n_a = 176; fall 2018 n_a = 120) based on the RAAVE and LER coding schemes. Student arguments were blinded by semester, and we coded these arguments independently for inter-rater reliability (RAAVE IRR: 66% and LER IRR = 83% for 3 raters) and then reached consensus for discrepancies in our codes.

Findings

We present our findings by first describing our three coding schemes (RAAVE, LER, and SOE) and giving student exemplars, noting key characteristics for each (Jimenez, 2021). Then, we compare the application of all coding schemes across two semesters, fall 2016 and fall 2018, where instructional changes were made for the latter semester. For the sake of continuity, as we describe the of the coding schemes in this section, we provided student arguments for the following ALT × OBJ scenario: if we further restrict farmers’ use of water for irrigation (Alternative), would this lead to farmers remaining economically profitable (Objective)?

Reasoning to address assumptions via evidence

The RAAVE coding scheme described levels of proficiency of student reasoning that addressed assumptions with evidence (Table 1). This coding scheme focused on the quality of reasoning in students’ ALT × OBJ arguments rather than how students supported their reasoning with evidence (which was independently coded for in the LER coding scheme). We noted that the common assumption students made was that more restrictions on irrigation would lead to farmers using less water and obtaining lower profits.

Table 1. The RAAVE coding scheme describes students’ levels of proficiency in providing reasoning that addresses assumptions when constructing arguments about the consequences of an Alternative (e.g., more restrictions) satisfying an Objective (e.g., farmers remain profitable) during the SDM process of a water conservation SSI.

Students’ reasoning at the highest proficiency, level 4, must use evidentiary support to state how (a mechanism) or by how much the consequences of an Alternative may satisfy the Objective (Table 1). One student illustrated this in their argument by first stating that “restricting water flow could potentially severely reduce farmer’s yield and profit,” then justified this statement by using evidence to indicate how much profit would be lost: “…according to [extension.psu.edu source] each inch of water produces 9 more bushels of corn grain on average. This means that if a farmer is restricted to using 20 inches of groundwater instead of 22, he could lose 18 bushels per acre. If the farmer owned 1,000 acres and the price of corn was $3.50 this would result in a $63,000 loss.” Similarly, another student stated the general assumption that “increasing restrictions on water use [does not allow] farmers to remain economically profitable” and reasoned that “a majority of farms across the United States rely on irrigation in order to have a successful operation. Irrigation is shown to double crop production in parts of the United States (Wagner, 2012). Increasing restrictions on water use would limit the availability of water and decrease production.” Both students provide explicit evidence about how much money is at stake or the economic value of irrigation to support the idea that restrictions would reduce farmers’ economic profitability.

For a student to achieve a level three, their reasoning must attempted to use evidence to address assumptions in their reasoning (Table 1). In these responses, students provided a new idea related to the ALT × OBJ scenario that were not one of the commonly made assumptions we had noted, and helped explain how or by how much an Alternative might satisfy an Objective. For example, one student stated “if water use is more strict, it might not allow some farmers to irrigate their crops, which in return could allow them to not remain economically profitable… but stricter water allocations could also mean that the farmers that truly need water for irrigation (like smaller farmers in low precipitation areas), might actually receive the water they need if the water is more wisely distributed.” Although this student reiterated the most common assumption for this ALT × OBJ argument (more restrictions would lead farmers to be less economically profitable), they introduced a new, related idea that attempted to explain how stricter water allocations could allow some farmers to receive more water depending on the level of competition with surrounding farms, however, the student did not support their ideas using evidence. We considered new ideas to be important factors that might influence the general assumption of restrictions reducing farmer’s economic status, such as long and short-term economics, farm size, crop type, competition for groundwater, annual rainfall variability, the type of restriction, the cost of irrigation infrastructure, or energy use and fees. Another student stated that “On slide 26 from 10/20, I learned that corn needs 19–20 inches of water in order thrive. Given that corn makes up a large portion of the agriculture in our state, having stricter allocations of water would not result in more corn production… it would ultimately lower the amount of economic profitability.” They provided a new idea concerning the amount of water the crop needs to thrive that was provided as evidence in class, however the student does not relate this evidence to how or why restrictions on irrigation would result in the crop receiving less than the optimal amount of water. Students’ reasoning in this proficiency level provided new ideas beyond common assumptions, but unfortunately either lacked evidentiary support, gave evidence unrelated to their reasoning, or gave evidence that they then misinterpreted.

In level 2 responses, students’ reasoning did not address how or by how much the general assumption was supported by evidence (Table 1). For example, one student reasoned that “stricter allocations would not allow the farmers to be more profitable because they can water less, and they will get worse yields without… more water.” Another student provided a more detailed response: “…if regulations are put on a farmer’s water supply, then crops will have lower yield. Especially in drought years, farmers need as much water as possible but cannot afford to pay for over-appropriating… Less water because of more restrictions, and depending on the weather, will directly affect economic profit.” Here, they even provided useful descriptions of the issue regarding the ALT × OBJ scenario using informal understanding of fees associated with over-appropriating and farmer profit margins, but these descriptions still did not give evidence about how or by how much their assumption was accurate or supported.

For the lowest level of proficiency, students provided non-justificatory arguments (Kuhn et al., 1997) that were not justified by evidence, may have had an emotive appeal, and may not have connected the evidence to the Alternative and Objective scenarios (Table 1). For example, one student stated: “I think this is the worst alternative for farmers. We cannot continue to put restriction on water because if we do this, farmers will have no water to grow crops and raise livestock. If we have no water to feed livestock and crops, how will we feed the world?” In this argument, the student’s main reasoning was an emotional appeal in that they viewed increased restrictions as linked to a drastic outcome of having no food. Another student stated that “more [restrictions] got a no because that would reduce the amount of water [farmers] get.” In both arguments, students did not relate the Alternative to the Objective, specifically addressing how the Alternative would allow farmers to remain economically profitable.

Linking evidence to reasoning

This coding scheme describes three proficiency levels in students’ ability to link evidence to their reasoning about the claim (LER, Table 2). The LER coding scheme focused solely on how well students linked evidence in their arguments regardless of whether the reasoning itself expanded understanding of common assumptions, and were assigned independently from the RAAVE coding scheme.

Table 2. The LER coding scheme describes students’ levels of proficiency in linking evidence to their reasoning as they constructed arguments about the consequences of an Alternative (e.g., more restrictions) satisfying an Objective (e.g., farmers remain profitable) during the SDM process of a water conservation SSI.

Students’ ALT × OBJ arguments as a level three linked traceable evidence that was clearly related to the reasoning in their argument. One student demonstrated this by stating that “…strict water restrictions have been shown to influence farmers to choose crops that increase returns based off of water and profitability (Wichelns, 1999). This means that farmers, when under pressure of strict restrictions, do not lose out money because they choose crops that allow them to work with the amount of water they have. However, a different study found that in some situations, farmers’ income was reduced by 7–13% due to high taxes on water allocations (Majias, Varela-Ortega, & Flichman).” This student provided explicitly cited and traceable evidence that supported their reasoning of stricter water restrictions (Alternative) somewhat allowing farmers to remain economically profitable (Objective).

For level two, students’ ALT × OBJ arguments gave problematic evidence, defined as instances where (a) the evidence was not clearly connected or was unrelated to students’ reasoning, (b) no actual information from the stated source was given or it was unclear what the exact evidence was to support their reasoning, and (c) the evidence was apparently misinterpreted or used in a way that did not make sense. One student illustrated an instance of (b), “If you are a farmer that has a low amount of groundwater it will also lead to less income due to the fact that the entire population will have to split the total water amount. American Water Resources Association showed that people with the strictest restrictions achieved the greatest savings, which is not quite what we predicted.” Although they referenced a source it is unclear what the specific evidence was or how it tied to their reasoning about how farmers’ profit would be impacted by restrictions. Students with problematic evidence may not be fully aware of how to link evidence to support their reasoning or possibly overlook the need to provide specific information. Students’ ALT × OBJ arguments were assigned a level one code if no evidence was used to support their reasoning (LER, Table 2).

Relationship between RAAVE and LER coding schemes

Separating these two coding schemes allowed us to distinctly document students’ reasoning about the Alternative and Objective, and not get distracted by the presence or absence of evidence. The RAAVE coding scheme described how well students’ argument addressed an assumption about the Alternative, though the highest level could only be achieved if a student supported their reasoning with evidence. The RAAVE and LER coding schemes were considered separate but were not completely mutually exclusive, for example, students could not receive a RAAVE proficiency level of four (relevant reasoning with evidentiary support) with an LER proficiency level of one (no evidence).

Sources of evidence

We also developed a coding scheme that described the sources of evidence (SOE) students provided in their ALT × OBJ arguments (Table 3). The SOE coding was thematic rather than suggesting levels of proficiency. The sources of evidence came from in-class sources, websites, and scientific articles (peer-reviewed and technical). Students often provided multiple instances of evidence to support their reasoning throughout their arguments, and each source was counted as an instance of evidence (Table 3).

Table 3. The sources of evidence (SOE) coding scheme describes the types of evidence students provided as they constructed arguments about the consequences of an Alternative (e.g., more restrictions) satisfying an Objective (e.g., farmers remain profitable) during the SDM process of a water conservation SSI.

Student proficiencies after course revisions

We explored proficiency levels of students’ ALT × OBJ arguments after course modifications were made in fall 2018. We coded and compared a subset of student ALT × OBJ arguments from fall 2016 (n_a = 176) and the revised course in fall 2018 (n_a = 120). We found an overall increase in students’ proficiency, with more students giving arguments that used evidence in their reasoning in fall 2018 (RAAVE coding scheme: 36% at Level 4; Table 1) than in fall 2016 (2% at Level 4, Table 1). Likewise, more students provided ALT × OBJ arguments that linked evidence to their reasoning in fall 2018 (LER coding scheme: 49% in Level 3; Table 2) than in fall 2016 (3% at Level 3, Table 2). There were also more instances of students providing evidence within their ALT × OBJ arguments, with a mean of 2 references per student in 2016 and a mean of 22 references per students in 2018, and with more peer-reviewed articles and technical papers referenced in fall 2018 (29%) compared to fall 2016 (10%) (Table 3).

Discussion

As the primary aim of our study, we developed coding schemes that describe students’ levels of proficiency in reasoning about multidisciplinary scientific evidence to address assumptions regarding ecological, agricultural, and social systems in an SSI-context. This work demonstrated how the framing of “assumptions” might be a useful lens for determining how well students provide mechanistic reasoning to support their claims with evidence in the context of an SSI. Additionally, we compared students’ responses across two semesters of instruction, before and after course revisions based on our initial findings. We interpreted this is an indication of increased validity—that our coding schemes were useful in describing students’ level of proficiency in reasoning about evidence, and when informed grading rubrics, supported students’ ability to use and apply multidisciplinary scientific evidence to better understand different systems in an SSI-context.

Using assumptions in scientific arguments within an SSI-context

In the course, students were required to find and use multidisciplinary scientific evidence that helped them predict the potential consequences of a phenomenon as they worked through an SDM process to examine an SSI. Using a lens of “assumptions” allowed us as researchers and instructors to go beyond a simplistic characterization of whether evidence was present or absent to support students’ claims, and instead evaluate the quality of the reasoning to support a students’ claim about SSI systems. We found that students address assumptions when they provide reasoning about a mechanism (how or why) or an amount (by how much). Using assumptions as a lens for students’ reasoning leans on a social constructivist perspective and acknowledges that students arrive in our classrooms with ideas about SSI systems that have been formed through their experiences in the world. Foregrounding students’ prior socially constructed knowledge is an important way to determine how student reasoning might evolve as they learn about an SSI and apply evidence to their arguments of SSI systems.

Instructional focus on assumptions may have utility because without awareness of existing assumptions and careful consideration of information, students may not be able to recognize the complexity of SSI and may resist adjusting their prior knowledge when making SSI decisions (Grace, 2009; Sinatra et al., 2014; Alred and Dauer, 2020). Encouraging the practice of making assumptions explicit and vetting them based on available evidence within science classrooms may help students adopt reasoning strategies that mirror scientific practices (Sinatra and Chinn, 2012). By applying an “assumptions” lens to science teaching and research, students can be trained to recognize and question their assumptions, then use relevant evidence and deeper reflection on experiences to construct richer and more accurate understandings of the nature of SSI systems. We found that asking students to explicitly think about assumptions in the revised semester may have made the role of scientific evidence clearer for students and may have ultimately encouraged students to develop arguments that were more complex, nuanced, and interdisciplinary.

Our work focused on how students reason about evidence in their scientific arguments of SSI systems as one step in the larger decision-making process, as opposed to the approach of some previous researchers (e.g., Bell and Lederman, 2003; Sadler and Zeidler, 2005; Kolstø, 2006; Bravo-Torija and Jiménez-Aleixandre, 2018) that have examined how well students used evidence to make a decision about SSI without providing a conceptual structure to lead students through the complex process of decision making. For example, Bravo-Torija and Jiménez-Aleixandre (2018), describe an initial learning progression for students’ application of evidence in decision-making contexts. In their study, students were given ecology evidence about marine resource management (though not economic or other multidisciplinary evidence) and one objective for biological efficiency (therefore tradeoffs around decision making were not part of the process). The resulting proposed learning progression of Bravo-Torija and Jiménez-Aleixandre (2018) had 5 levels that integrated a description how well students both applied evidence and compared outcomes across alternative solutions. In comparison, we asked students to systematically compare each Alternative and Objective. Bravo-Torija and Jiménez-Aleixandre’s (2018) highest sophisitication level was student reasoning that synthesized and found patterns in the evidence from multiple sources, coordinated with theoretical models, to compare consequences of different alternatives based on available evidence. In contrast, our highest proficiency level in the RAAVE coding scheme was characterized by the nature of student reasoning in using evidence that students selected to address an assumption about an Alternative SSI solution. Each approach advances understanding of students’ SSI argumentation in different ways, though the differences in decision-making-component contexts between studies make it difficult for science educational researchers to compare and build upon one another’s work in a coherent way.

It is complicated to characterize student reasoning about decision making without isolating different types of arguments. Previous studies have struggled to characterize the interplay between multiple components of SSI decision making such as considering values, analyzing tradeoffs, and applying science information in student reasoning. Some earlier studies indicated that students tend to emphasize values at the expense of seeking additional scientific information that would clarify different choices (Bell and Lederman, 2003; Sadler and Zeidler, 2005; Kolstø, 2006). However, decisions about what we should do about SSIs ought to be based on our values (Keeney, 2009; Gregory et al., 2012). In our study, the SDM process made the separate and important roles of evidence and values explicit, which provided a useful lens for being able to examine students’ evidence evaluation more fully in an SSI-context. The separation of students’ arguments about the SSI system from their evaluation of the ultimate decision based on their Objectives, allowed us to focus our research on students’ reasoning toward how something is apart from their reasoning of what should we do. In this way, despite the overall framing of a value-laden SSI, students’ evaluation of evidence was meant to be a more impartial exploration of the nature of ecological, agricultural, and social systems. Jensen’s (2002) knowledge dimensions (e.g., causes, effects, actions, and visions) of environmental issues also reminded us that students may pursue different types of arguments as they worked toward an overall argument for an SSI solution. Our work might be a useful model for creating classroom tasks that deliberately support individual steps in reasoning about complex SSI and untangle the components of students’ arguments, as well as help instructors and researchers consider how to evaluate the multiple proficiencies involved in SSI argumentation and decision making.

Limitations to the use of assumptions in classrooms to evaluate student reasoning

We found limitations to understanding what assumptions students hold, and how their assumptions impact their reasoning. We characterized common statements that students made without evidentiary support in student responses in the first semester as “assumptions” made by students, though we recognize there may be many other assumptions that students bring to bear on these issues that we did not capture. Additionally, asking students to be explicit about assumptions when giving arguments may be helpful, but may not completely avoid the potential role assumptions play in how students engage with scientific evidence they encounter (West et al., 2012). Importantly, we were not able to distinguish between “assumptions” that were made because an issue has not been carefully thought through by a student, and a carefully considered understanding of the nature of SSI systems that was either unsubstantiated with evidence, or that comes from ways of knowing that are often unacknowledged by Western science, such as traditional knowledge from indigenous cultures.

Generally, more work needs to be done to consider how multiple ways of knowing can be integrated into SSI-context. The course we studied was an introductory science course and emphasized science as the primary source of evidence in explaining SSI systems. However, we emphasize that our definition of “assumptions” is not meant to be a contrast between a positivistic Western-science frame versus other ways of knowing, rather we consider “assumptions” to be born out of a lack of deep thought or consideration about the nature of an SSI system. Science is one way of knowing that helps clarify the nature of social and natural systems, though limited in its pursuit of ultimate truths (Waddington and Feinstein, 2016). Asking students to consider scientific evidence alongside their experiences and other ways of knowing should enhance the accuracy of their ideas about the nature of these systems. The intention in having students explore scientific evidence is not necessarily to replace students’ inherent understanding of the world with scientific ways of knowing, as it is possible to have parallel knowledges that mutually enrich understandings and give a more holistic picture of SSI systems (Michie et al., 2018).

Importantly, students’ assumptions are only one of the potential psychological, cognitive, or social elements that impact reasoning about SSI evidence. Lombardi et al. (2016) found that emotional (personal knowledge and experience) and psychological factors (emotional and cognitive dispositions) influence a students’ judgment of the plausibility of scientific arguments based on the provided evidence. Moreover, the SSI-context may remain challenging for students due their inherent uncertainty, and the uncertainty of the ‘frontier’ science around these ill-structured issues (Kolstø, 2006), making it difficult for students to construct comprehensible arguments about SSI systems based on evidence. All these factors may induce students to construct arguments about complex phenomena based on their assumptions devoid of any self-reflection or evidence (Gregory et al., 2012). More work needs to be done to investigate all the cognitive, epistemological, psychological, and social aspects that are brought to bear in student reasoning about the causal mechanisms of SSI systems.

Conclusion

These findings are an important contribution to the science education community because the ability to construct evidence-based SSI arguments is a key practice in fostering science literate individuals. This critical component of science literacy requires students to identify and make sense of multidisciplinary scientific evidence alongside their own knowledge and experience, then synthesize and use evidence to explain complex phenomena that help them reason about SSI that they will confront as professionals or citizens. Assumptions might be a useful lens for researchers and instructors to foster arguments from evidence about SSIs in classrooms to enhance students’ skills in making decisions about complex societal issues.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by the Institutional Review Board for the Protection of Human Subjects University of Nebraska-Lincoln (20140813907EP). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

PJ: Formal analysis, Investigation, Writing – original draft. AA: Formal analysis, Investigation, Writing – review & editing. JD: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Funding was provided by the National Science Foundation IUSE:EDU 2216214 and EHR:BCSER 1937657, The College of Agricultural Science and Natural Resources and the School of Natural Resources at the University of Nebraska-Lincoln.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References