Using Competencies to Explain Mathematical Item Demand: A Work in Progress

Abstract

This chapter describes theoretical and practical issues associated with the development and use of a rating scheme for the purpose of analysing mathematical problems—specifically, to assess the extent to which solving those problems calls for the activation of a particular set of mathematical competencies. The competencies targeted through the scheme are based on the mathematical competencies that have underpinned each of the PISA Mathematics Frameworks. The scheme consists of operational definitions of the six competencies (labelled as communication; devising strategies; mathematisation; representation; using symbols, operations and formal language; and reasoning and argument), descriptions of four levels of activation of each competency, and examples of the ratings given to particular items together with commentary that explains how each proposed rating is justified in relation to the competency definition and level descriptions. The mathematical problems used so far to investigate the action of those competencies are questions developed for use in the PISA survey instruments from 2000 through to 2012. Ratings according to the scheme predict a large proportion of the variation in difficulty across items, providing evidence that these competencies are important elements of students’ problem solving capabilities. The appendix gives definitions of each competence and the specification of each of four levels for each.

Appendices

Appendix 1: Initial Competency Definitions and Level Descriptions (April 2005)

Reasoning and Argumentation: Logically rooted thought processes that explore and connect problem elements to work towards a conclusion, and activities related to justifying, and explaining conclusions; can be part of problem solving process

0: Understand direct instructions and take the actions implied 1: Employ a brief mental dialogue to process information, for example to link separate components present in the problem, or to use straightforward reasoning within one aspect of the problem 2: Employ an extended mental dialogue (for example to connect several variables) to follow or create sequential arguments; interpret and reason from different information sources 3: Evaluate, use or create chains of reasoning to support conclusions or to make generalisations, drawing on and combining multiple elements of information in a sustained and directed way

Communication: Decoding and interpreting stimulus, question, task; expressing conclusions

0: Understand short sentences or phrases containing single familiar ideas that give immediate access to the context, where it is clear what information is relevant, and where the order of information matches the required steps of thought 1: Identify and extract relevant information, and use links or connections within the text, that are needed to understand the context, or cycle between the text and other related representation/s; some reordering of ideas may be required 2: Use repeated cycling to understand instructions and decode the elements of the context; interpret conditional statements or instructions containing diverse elements; actively communicate a constructed explanation 3: Create an economical, clear, coherent and complete presentation of words selected to explain or describe a solution, process or argument; interpret complex logical relations involving multiple ideas and connections

Modelling: Mathematising, interpreting, validating

0: Either the situation is purely intra-mathematical, or the relationship between the real situation and the model is not needed in solving the problem 1: Interpret and infer directly from a given model; translate directly from a situation into mathematics (for example, structure and conceptualise the situation in a relevant way, identify and select relevant variables) 2: Modify or use a given model to satisfy changed conditions; or choose a familiar model within limited and clearly articulated constraints; or create a model where the required variables, relationships and constraints are explicit and clear 3: Create a model in a situation where the assumptions, variables, relationships and constraints are to be identified or defined, and check that the model satisfies the requirements of the task; evaluate or compare models

Problem solving: The planning, or strategic controlling, and implementation of mathematical solution processes, largely within the mathematical world

0: Direct and obvious actions are required, with no strategic planning needed (that is, the strategy needed is stated or obvious) 1: Identify or select an appropriate strategy by selecting and combining the given relevant information to reach a conclusion 2: Construct or invent a strategy to transform given information to reach a conclusion; identify relevant information and transform it appropriately 3: Create an elaborated strategy to find an exhaustive solution or a generalised conclusion

Representation: Concrete expression of an abstract idea, object or action; a transformation or mapping from one form to another; can be part of modelling or PS

0: Handle direct information, for example translating directly from text to numbers, where minimal interpretation is required 1: Make direct use of one standard or familiar representation (equation, graph, table, diagram) linking the situation and its representation 2: Understand and interpret or manipulate a representation; or switch between and use two different representations 3: Understand and use an unfamiliar representation that requires substantial decoding and interpretation, or where the mental imagery required goes substantially beyond what is stated

Symbols and Formalism: Activating and using particular forms of representation governed by special rules (e.g. mathematical conventions)

0: No mathematical rules or symbolic expressions need to be activated beyond fundamental arithmetic calculations, operating with small or easily tractable numbers 1: Make direct use of a simple functional relationship (implicit or explicit); use formal mathematical symbols (for example, by direct substitution) or activate and directly use a formal mathematical definition, convention or symbolic concept 2: Explicit use and manipulation of symbols (for example, by rearranging a formula); activate and use mathematical rules, definitions, conventions, procedures or formulae using a combination of multiple relationships or symbolic concepts 3: Multi-step application of formal mathematical procedures; working flexibly with functional relationships; using both mathematical technique and knowledge to produce results

Appendix 2: Competency Definitions and Level Descriptions (May 2013)

Communication: The communication competency has both ‘receptive’ and ‘constructive’ components. The receptive component includes understanding what is being stated and shown related to the mathematical objectives of the task, including the mathematical language used, what information is relevant, and what is the nature of the response requested. The constructive component consists of presenting the response that may include solution steps, description of the reasoning used and justification of the answer provided. In written and computer-based items, receptive communication relates to understanding text and images, still and moving. Text includes verbally presented mathematical expressions and may also be found in mathematical representations (for example titles, labels and legends in graphs and diagrams). Communication does not include knowing how to approach or solve the problem, how to make use of particular information provided, or how to reason about or justify the answer obtained; rather it is the understanding or presenting of relevant information. It also does not apply to extracting or processing mathematical information from representations. In computer-based items, the instructions about navigation and other issues related to the computer environment may add to the general task demand, but is not part of the communication competency. Demand for the receptive aspect of this competency increases according to the complexity of material to be interpreted in understanding the task; the need to link multiple information sources or to move backwards and forwards (to cycle) between information elements. The constructive aspect increases with the need to provide a detailed written solution or explanation.

Definition: Reading and interpreting statements, questions, instructions, tasks, images and objects; imagining and understanding the situation presented and making sense of the information provided including the mathematical terms referred to; presenting and explaining one’s mathematical work or reasoning.

0: Understand short sentences or phrases relating to concepts that give immediate access to the context, where all information is directly relevant to the task, and where the order of information matches the steps of thought required to understand what the task requests. Constructive communication involves only presentation of a single word or numeric result 1: Identify and link relevant elements of the information provided in the text and other related representation/s, where the material presented is more complex or extensive than short sentences and phrases or where some extraneous information may be present. Any constructive communication required is simple, for example it may involve writing a short statement or calculation, or expressing an interval or a range of values 2: Identify and select elements to be linked, where repeated cycling within the material presented is needed to understand the task; or understand multiple elements of the context or task or their links. Any constructive communication involves providing a brief description or explanation, or presenting a sequence of calculation steps 3: Identify, select and understand multiple context or task elements and links between them, involving logically complex relations (such as conditional or nested statements). Any constructive communication would involve presenting argumentation that links multiple elements of the problem or solution

Devising strategies: The focus of this competency is on the strategic aspects of mathematical problem solving: selecting, constructing or activating a solution strategy and monitoring and controlling the implementation of the processes involved. ‘Strategy’ is used to mean a set of stages that together form the overall plan needed to solve the problem. Each stage comprises a sub-goal and related steps. For example a plan to gather data, to transform them and to represent them in a different way would normally constitute three separate stages. The knowledge, technical procedures, mathematising and reasoning needed to actually carry out the solution process are taken to belong to those other competencies. Demand for this competency increases with the degree of creativity and invention involved in identifying a suitable strategy, with increased complexity of the solution process (for example the number, range and complexity of the stages needed in a strategy), and with the consequential need for greater metacognitive control in the implementation of the strategy towards a solution.

Definition: Selecting or devising a mathematical strategy to solve a problem as well as monitoring and controlling implementation of the strategy.

0: Take direct actions, where the solution process needed is explicitly stated or obvious 1: Find a straight-forward strategy (usually of a single stage) to combine or use the given information 2: Devise a straight-forward multi-stage strategy, for example involving a linear sequence of stages, or repeatedly use an identified strategy that requires targeted and controlled processing 3: Devise a complex multi-stage strategy, for example that involves bringing together multiple sub-goals or where using the strategy involves substantial monitoring and control of the solution process; or evaluate or compare strategies

Mathematising: The focus of this competency is on those aspects of the modelling cycle that link an extra-mathematical context with some mathematical domain. Accordingly, the mathematising competency has two components. A situation outside mathematics may require translation into a form amenable to mathematical treatment. This includes making simplifying assumptions, identifying variables present in the context and relationships between them, and expressing those variables in a mathematical form. This translation is sometimes referred to as mathematising. Conversely, a mathematical entity or outcome may need to be interpreted in relation to an extra-mathematical situation or context. This includes translating mathematical results in relation to specific elements of the context and validating the adequacy of the solution found with respect to the context. This process is sometimes referred to as de-mathematising.

The intra-mathematical treatment of ensuing issues and problems within the mathematical domain is dealt with under other competencies. Hence, while the mathematising competency deals with representing extra-mathematical contexts by means of mathematical entities, the representation of mathematical entities is dealt with under the representation competency. Demand for activation of this competency increases with the degree of creativity, insight and knowledge needed to translate between the context elements and the mathematical structures of the problem.

Definition: Translating an extra-mathematical situation into a mathematical model, interpreting outcomes from using a model in relation to the problem situation, or validating the adequacy of the model in relation to the problem situation.

0: Either the situation is purely intra-mathematical, or the relationship between the extra-mathematical situation and the model is not relevant to solving the problem 1: Construct a model where the required assumptions, variables, relationships and constraints are given; or draw conclusions about the situation directly from a given model or from the mathematical results 2: Construct a model where the required assumptions, variables, relationships and constraints can be readily identified; or modify a given model to satisfy changed conditions; or interpret a model or mathematical results where consideration of the problem situation is essential 3: Construct a model in a situation where the assumptions, variables, relationships and constraints need to be defined; or validate or evaluate models in relation to the problem situation; or link or compare different models

Representation: The focus of this competency is on decoding, devising, and manipulating representations of mathematical entities or linking different representations in order to pursue a solution. By ‘representation of a mathematical entity’ we understand a concrete expression (mapping) of a mathematical concept, object, relationship, process or action. It can be physical, verbal, symbolic, graphical, tabular, diagrammatic or figurative. Mathematical tasks are often presented in text form, sometimes with graphic material that only helps set the context. Understanding verbal or text instructions and information, photographs and graphics does not generally belong to representation competency—that is part of the communication competency. Similarly, working exclusively with symbolic representations lies within the using symbols, operations and formal language competency. On the other hand, translation between different representations is always part of the representation competency. For example, the act of transforming mathematical information derived from relevant text elements into a non-verbal representation is where representation commences to apply. While the representation competency deals with representing mathematical entities by means of other entities (mathematical or extra-mathematical), the representation of extra-mathematical contexts by mathematical entities is dealt with under the mathematising competency. Demand for this competency increases with the amount of information to be extracted, with the need to integrate information from multiple representations, and with the need to devise representations rather than to use given representations. Demand also increases with added complexity of the representation or of its decoding, from simple and standard representations requiring minimal decoding (such as a bar chart or Cartesian graph), to complex and less standard representations comprising multiple components and requiring substantial decoding perhaps devised for specialised purposes (such as a population pyramid, or side elevations of a building).

Definition: Decoding, translating between, and making use of given mathematical representations in pursuit of a solution; selecting or devising representations to capture the situation or to present one’s work.

0: Either no representation is involved; or read isolated values from a simple representation, for example from a coordinate system, table or bar chart; or plot such values; or read isolated numeric values directly from text 1: Use a given simple and standard representation to interpret relationships or trends, for example extract data from a table to compare values, or interpret changes over time shown in a graph; or read or plot isolated values within a complex representation; or construct a simple representation 2: Understand and use a complex representation, or construct such a representation where some of the required structure is provided; or translate between and use different simple representations of a mathematical entity, including modifying a representation 3: Understand, use, link or translate between multiple complex representations of mathematical entities; or compare or evaluate representations; or devise a representation that captures a complex mathematical entity

Using symbols, operations and formal language: This competency reflects skill with activating and using mathematical content knowledge, such as mathematical definitions, results (facts), rules, algorithms and procedures, recalling and using symbolic expressions, understanding and manipulating formulae or functional relationships or other algebraic expressions and using the formal rules of operations (e.g. arithmetic calculations or solving equations). This competency also includes working with measurement units and derived quantities such as ‘speed’ and ‘density’. Developing symbolic formulations of extra-mathematical situations is part of mathematisation. For example, setting up an equation to reflect the key elements of an extra-mathematical situation belongs to mathematisation, whereas solving it is part of the using symbols, operations and formal language competency. Manipulating symbolic expressions belongs to the using symbols, operations and formal language competency even though they are mathematical representations. However, translating between symbolic and other representations belongs to the representation competency. The term ‘variable’ is used here to refer to a symbol that stands for an unspecified number or a changing quantity, for example C and r in the formula C = 2πr. Demand for this competency increases with the increased complexity and sophistication of the mathematical content and procedural knowledge required.

Definition: Understanding and implementing mathematical procedures and language (including symbolic expressions, arithmetic and algebraic operations), using the mathematical conventions and rules that govern them; activating and using knowledge of definitions, results, rules and formal systems.

0: State and use elementary mathematical facts and definitions; or carry out short arithmetic calculations involving only easily tractable numbers. For example, find the area of a rectangle given the side lengths, or write down the formula for the area of a rectangle 1: Make direct use of a simple mathematical relationship involving variables (for example, substitute into a linear relationship); use arithmetic calculations involving fractions and decimals; use repeated or sustained calculations from level 0; make use of a mathematical definition, fact, or convention, for example use knowledge of the angle sum of a triangle to find a missing angle 2: Use and manipulate expressions involving variables and having multiple components (for example, by algebraically rearranging a formula); employ multiple rules, definitions, results, conventions, procedures or formulae together; use repeated or sustained calculations from level 1 3: Apply multi-step formal mathematical procedures combining a variety of rules, facts, definitions and techniques; work flexibly with complex relationships involving variables, for example use insight to decide which form of algebraic expression would be better for a particular purpose

Reasoning and argument: This competency relates to drawing valid inferences based on the internal mental processing of mathematical information needed to obtain well-founded results, and to assembling those inferences to justify or, more rigorously, prove a result. Other forms of mental processing and reflection involved in undertaking tasks underpin each of the other competencies. For example the thinking needed to choose or devise an approach to solving a problem is dealt with under the devising strategies competency, and the thinking involved in transforming contextual elements into a mathematical form is accounted for in the mathematising competency. The nature, number or complexity of elements that need to be brought to bear in making inferences, and the length and complexity of the chain of inferences needed would be important contributors to increased demand for this competency.

Definition: Drawing inferences by using logically rooted thought processes that explore and connect problem elements to form, scrutinise or justify arguments and conclusions

0: Draw direct inferences from the information and instructions given 1: Draw inferences from reasoning steps within one aspect of the problem that involves simple mathematical entities 2: Draw inferences by joining pieces of information from separate aspects of the problem or concerning complex entities within the problem; or make a chain of inferences to follow or create a multi-step argument 3: Use or create linked chains of inferences; or check or justify complex inferences; or synthesise and evaluate conclusions and inferences, drawing on and combining multiple elements of complex information, in a sustained and directed way