Skip to main content
SpringerLink
Log in

Developing fair tests for mathematics curriculum comparison studies: the role of content analyses

  • Original Article
  • Published:
Mathematics Education Research Journal Aims and scope Submit manuscript

Abstract

This article describes the process of development of assessment instruments for a three-year longitudinal comparative study that focused on evaluating American high school students’ mathematics learning from two distinct approaches to content organization: curriculum built around a sequence of three full-year courses (Algebra 1, Geometry, and Algebra 2) and a sequence of integrated mathematics courses (algebra and geometry content, together with functions, data analysis, and discrete mathematics is integrated each year). The study was conducted in six school districts in five states involving over 4,000 students from schools that were using both curricular approaches but with different groups of students. In order to develop assessment instruments that were not biased towards either of the two curriculum programs (Fair Tests), an iterative process of content analyses, identification of common topics, internal and external reviews, pilot tests, and revisions was followed, resulting in five tests that were used in the three years of the study. Results indicate that these tests have solid discrimination properties and address adequately mathematics content common to both secondary curriculum programs. The corresponding scoring rubrics are highly reliable, with interrater reliability above 94% for all tests. Mathematics education researchers involved in curriculum comparison studies need to conduct content analyses of the curriculum materials under study in order to identify salient relationships between curriculum programs and student outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Baker, E. L., & Herman, J. L. (1983). Task structure design: beyond linkage. Journal of Educational Measurement, 20(2), 149–164.

    Article  Google Scholar 

  • Center for Mathematics Education (CME), P. (2009). CME project: Algebra 1: Student Edition. Boston: Pearson.

    Google Scholar 

  • Coxford, A. F., Fey, J. T., Hirsch, C. R., Schoen, H. L., Burrill, G., Hart, E. W., et al. (1998). Contemporary mathematics in context: a unified approach (course 1). New York: Glencoe/McGraw-Hill.

    Google Scholar 

  • Dahl, T., Johnson, J., Morton, M., & Whalen, S. (2005). Mathematics assessment sampler, grades 9–12: Items aligned with NCTM’s principles and standards for school mathematics. Reston: National Council of Teachers of Mathematics.

    Google Scholar 

  • De Lange, J. (2007). Large-scale assessment and mathematics education. In F. K. Lester Jr. (Ed.), Second handbook of research on mathematics teaching and learning (pp. 1111–1142). Charlotte: Information Age Publishing.

    Google Scholar 

  • Domino, G., & Domino, M. L. (2006). Psychological testing: an introduction. Cambridge: Cambridge University Press.

    Google Scholar 

  • Dossey, J., Halvorsen, K., & McCrone, S. (2008). Mathematics education in the United States 2008: a capsule summary book written for the eleventh International Congress on Mathematical Education (ICME-11). Reston: National Council of Teachers of Mathematics.

    Google Scholar 

  • Gorin, J. S. (2007). Test construction and diagnostic testing. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: theory and applications (pp. 173–201). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Hirsch, C. R. (Ed.). (2007). Perspectives on the design and development of school mathematics curricula. Reston, VA: National Council of Teachers of Mathematics.

    Google Scholar 

  • Holliday, B., Cuevas, G. J., Moore-Harris, B., Carter, J. A., Marks, D., Casey, R. M., Day, R., & Hayek, L. (2005). Algebra 1. New York: Glencoe/McGraw-Hill.

    Google Scholar 

  • Keitel, C., & Kilpatrick, J. (1999). The rationality and irrationality of international comparative studies. In G. Kaiser, E. Luna, & I. Huntley (Eds.), International comparisons in mathematics education (pp. 241–256). London: Falmer.

    Google Scholar 

  • Kilpatrick, J. (2003). What works? In S. L. Senk & D. R. Thompson (Eds.), Standards’ based school mathematics curricula. What are they? What do students learn? (pp. 471–493). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Kline, P. (1986). A handbook of test construction. London: Methuen.

    Google Scholar 

  • Kline, P. (2000). The handbook of psychological testing (2nd ed.). New York: Routledge.

    Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

    Article  Google Scholar 

  • Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based assessment: expectations and validation criteria. Educational Researcher, 20(8), 15–21.

    Google Scholar 

  • McNaught, M., Tarr, J., & Grouws. D. (2008). Assessing curriculum implementation: insights from the comparing options in Secondary Mathematics Project. Paper presented at the Annual meeting the American Educational Research Association, New York.

  • Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.

    Google Scholar 

  • Murphy, K. R., & Davidshofer, C. O. (2005). Psychological testing: principles and applications (6th ed.). Upper Saddle River: Pearson Prentice Hall.

    Google Scholar 

  • National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standardsfor school mathematics. Reston: Author.

    Google Scholar 

  • National Council of Teachers of Mathematics. (1995). Assessment standards for school mathematics. Reston: Author.

    Google Scholar 

  • National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematicss. Reston: Author.

    Google Scholar 

  • National Council of Teachers of Mathematics. (2009). Focus in high school mathematics: reasoning and sense making. Reston: Author.

    Google Scholar 

  • National Research Council. (2000). How people learn: brain, mind, experience, and school: Expanded Edition. Washington, DC: National Academies Press.

    Google Scholar 

  • National Research Council. (2001). Knowing what students know: the science and design of educational assessment. Washington, DC: National Academies Press.

    Google Scholar 

  • National Research Council. (2004). On evaluating curricular effectiveness: judging the quality of K-12 NSF-supported and commercially generated mathematics curriculum materials. Washington, DC: National Academies Press.

    Google Scholar 

  • Papick, I. (2006). Algebra connections. Upper Saddle River: Prentice Hall.

    Google Scholar 

  • Pearson, D. P., & Garavaglia, D. R. (2003). Improving the information value of performance items in large-scale assessments: NAEP validity studies. Washington, D.C.: National Center for Education Statistics.

    Google Scholar 

  • Reynolds, C. B., Livingston, R. B., & Willson, V. (2006). Measurement and assessment in education. New York: Pearson.

    Google Scholar 

  • Ross, D., Reys, R., Chávez, O., McNaught, M., & Grouws, D. (2011). Lessons learned from student strategies on an algebra problem. School Science and Mathematics.

  • Royer, J. M., Cisero, C. A., & Carlo, M. S. (1993). Techniques and procedures for assessing cognitive skills. Review of Educational Research, 63(2), 201–243.

    Google Scholar 

  • Schmidt, W. H., McKnight, C. C., & Raizen, S. A. (1997). A splintered vision: an investigation of U.S. science and mathematics education. Dordrecht: Kluwer.

    Google Scholar 

  • Senk, S., Thompson, D., Viktora, S., Usiskin, Z., Ahbel, N., Rubenstein, R., Levin, S., et al. (2001). UCSMP Advanced Algebra. Prentice-Hall.

  • Silver, E. A., & Lane, S. (1993). Assessment in the context of mathematics instruction reform: the design of assessment in the QUASAR project. In M. Niss (Ed.), Cases of assessment in mathematics education (pp. 59–69). London: Kluwer Academic Publishers.

    Google Scholar 

  • Silver, E. A., Alacaci, C., & Stylianou, D. A. (2000). Students’ performance on extended constructed-response tasks. In E. A. Silver & P. A. Kenney (Eds.), Results from the seventh mathematics assessment of the National Assessment of Educational Progress. Reston: NCTM.

    Google Scholar 

  • Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeley, G. (2001). An evaluation of the second edition of UCSMP Advanced Algebra. Unpublished manuscript.

  • Thompson, D. R., Witonsky, D., Senk, S. L., Usiskin, Z., & Kaeley, G. (2003). An evaluation of the second edition of UCSMP Geometry. Unpublished manuscript.

  • Webb, N. L. (1992). Assessment of students’ knowledge of mathematics: steps toward a theory. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning. Reston: National Council of Teachers of Mathematics.

    Google Scholar 

  • Yoong, W. H. (Ed.). (1999). New elementary mathematics: Syllabus D, volume 1. Singapore: Pan Pacific Publications.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Missouri, 303 Townsend Hall, Columbia, MO, 65211-2400, USA

    Óscar Chávez & Douglas A. Grouws

  2. 10696 West Brome Road, Rocheport, MO, 65279, USA

    Ira Papick

  3. 206 Sutton Science Center, Maryville College, 502 E. Lamar Alexander Pkwy, Maryville, TN, 37804, USA

    Daniel J. Ross

  4. University of Nebraska, Lincoln, NE, USA

    Ira Papick

Authors
  1. Óscar Chávez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ira Papick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel J. Ross
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Douglas A. Grouws
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Óscar Chávez.

Additional information

This paper is based on research conducted as part of the Comparing Options in Secondary Mathematics: Investigating Curriculum (COSMIC) project, a research study supported by the National Science Foundation under grant number REC-0532214. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the National Science Foundation. Portions of this paper were presented at the Annual Meeting of the American Educational Research Association, Denver, May, 2010.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chávez, Ó., Papick, I., Ross, D.J. et al. Developing fair tests for mathematics curriculum comparison studies: the role of content analyses. Math Ed Res J 23, 397–416 (2011). https://doi.org/10.1007/s13394-011-0023-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13394-011-0023-2

Keywords

Search

Navigation