Skip to main content

Advertisement

SpringerLink
Log in

Validation of a research-based student survey of instruction in a college of education

  • Published:
Educational Assessment, Evaluation and Accountability Aims and scope Submit manuscript

Abstract

As a preparer of students for the profession of teaching, a college, school, or department of education in higher education has an expectation that instructors teach competently. In higher education, the student survey of instruction is one of the most important among other indicators of teaching effectiveness. This article presents a validation of a new student survey of instruction in a college of education. In this validation, we found the new survey to be as troublesome as the previous survey. Four major findings emerged from this study that offer hope for a new and improved survey of instructional effectiveness. These findings involve estimating reliability, subscore validity, course relevance and rigor of the instructor, and construct-irrelevant variables that threaten validity. Given the failed validation, many cogent observations provide guidance for the design and development of improved forms that may overcome these threats to validity and guide others in validating teaching effectiveness surveys.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abrami, P. C., D’Apollonia, S., & Cohen, P. (1990). Validity of student ratings of instruction: what we know and what we no not. Journal of Educational Psychology, 82, 219–231.

    Article  Google Scholar 

  • Aleamoni, L. M. (1981). Student ratings of instruction. In J., & Millman (Eds.), Handbook of teacher evaluation (pp. 110–145). Beverly Hills, CA: Sage.

    Google Scholar 

  • Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., & Bell, D. (2006). Electronic course evaluations: does an online delivery system influence student evaluations? Journal of Economic Education, 37(1), 21–38.

    Article  Google Scholar 

  • Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64(8), 723–733.

    Google Scholar 

  • Cohen, P. A. (1981). Student ratings of instruction and student achievement: a meta-analysis of multisection validity studies. Review of Educational Research, 51(3), 281–309.

    Google Scholar 

  • Cronbach, L. J. (1988). Five perspectives on validity argument. In H., Wainer, H., & Braun (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • D’Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198–1208.

    Article  Google Scholar 

  • Darby, J. A. (2006a). The effects of the elective or required status of courses on student evaluations. Journal of Vocational Education and Training, 58(1), 19–29.

    Article  Google Scholar 

  • Darby, J. A. (2006b). Evaluating courses: an examination of the impact of student gender. Educational Studies, 32(2), 187–199.

    Article  Google Scholar 

  • Dommeyer, C. J., Baum, P., Hanna, R., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623.

    Article  Google Scholar 

  • Feldman, K. A. (1976). The superior college teacher from the students’ view. Research in Higher Education, 5, 243–288.

    Article  Google Scholar 

  • Feldman, K. A. (1998). Identifying exemplary teachers and teaching: Evidence from student ratings. In K. A., Feldman, M. B., & Paulsen (Eds.), Teaching and learning in the college classroom (pp. 391–414). Needham Heights, MA: Association for the Study of Higher Education (ASHE) Reader Series.

    Google Scholar 

  • Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52(11), 1182–86.

    Article  Google Scholar 

  • Greenwald, A. G., & Gillmore, G. M. (1997). No pain, no gain? The importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology, 89(4), 743–51.

    Article  Google Scholar 

  • Haladyna, T. M. (2007). Roles and importance of validity studies in test development. In S. M., Downing, T. M., & Haladyna (Eds.), Handbook of test development (pp. 739–758). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Haladyna, T. M., & Hess, R. K. (1994). The detection and correction of bias in student ratings of instruction. Research in Higher Education, 35(6), 669–687.

    Article  Google Scholar 

  • Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27.

    Article  Google Scholar 

  • Hellman, C. M. (1998). Faculty evaluation by students: a comparison between full-time and adjunct faculty. Journal of Applied Research in the Community College, 6(1), 45–50.

    Google Scholar 

  • Hendry, G. D., Cumming, R. G., Lyon, P. M., & Gordon, J. (2001). Student-centred course evaluation in a four-year, problem based medical programme: issues in collection and management of feedback. Assessment & Evaluation in Higher Education, 26(4), 327–339.

    Article  Google Scholar 

  • Hobson, S. M., & Talbot, D. M. (2001). Understanding student evaluations: what all faculty should know. College Teaching, 49(1), 26–31.

    Article  Google Scholar 

  • Husbands, C. T. (1997). Variations in student evaluations of teachers’ lecturing in different courses on which they lecture: a study at the London school of economics and political science. Higher Education, 33(1), 51–70.

    Article  Google Scholar 

  • Johnson, R. (2000). The authority of the student evaluation questionnaire. Teaching in Higher Education, 5(4), 419–434.

    Article  Google Scholar 

  • Kane, M. T. (2006a). Content-related validity evidence. In S. M., Downing, T. M., & Haladyna (Eds.), Handbook of test development (pp. 131–154). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Kane, M. T. (2006b). In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger.

  • Kogan, J. R., & Shea, J. A. (2007). Course evaluation in medical education. Teaching and Teacher Education: An International Journal of Research and Studies, 23(3), 251–264.

    Article  Google Scholar 

  • Kolitch, E., & Dean, A. V. (1999). Student ratings of instruction in the USA: hidden assumptions and missing conceptions about “good” teaching. Studies in Higher Education, 24(1), 27–42.

    Article  Google Scholar 

  • Li-Ping Tang, T. (1997). Teaching evaluation at a public institution of higher education: factors related to the overall teaching effectiveness. Public Personal Management, 26, 1997.

    Google Scholar 

  • Marsh, H. W. (1982). Validity of students’ evaluation of college teaching: a multi-trait, multi-method analysis. Journal of Educational Psychology, 74(2), 264–279.

    Article  Google Scholar 

  • Marsh, H. W., & Dunkin, M. J. (1997). Students’ evaluation of university teaching: A multidimensional perspective. In R. P., Perry, J. C., & Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 241–320). New York, NY: Agathon.

    Google Scholar 

  • McKeachie, W. J. (1997). Student ratings: the validity of use. American Psychologist, 52(11), 1218–1225.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement, 3rd ed. (pp. 13–103). New York: American Council on Education and Macmillan.

    Google Scholar 

  • Mohanty, G., Gretes, J., Flowers, C., Algozzine, B., & Spooner, F. (2005). Multi-method evaluation of instruction in engineering classes. Journal of Personnel Evaluation in Education, 18(2), 139–151.

    Article  Google Scholar 

  • Moss, J., & Hendry, G. (2002). Use of electronic surveys in course evaluation. British Journal of Educational Technology, 33(5), 583–592.

    Article  Google Scholar 

  • Oliver, R. L., & Sautter, E. P. (2005). Using course management systems to enhance the value of student evaluations of teaching. Journal of Education for Business, 80(4), 231–234.

    Article  Google Scholar 

  • Ramsden, P. (1991). A performance indicator of teaching quality in higher education: the course experience questionnaire. Studies in Higher Education, 16(2), 129–150.

    Article  Google Scholar 

  • Roberts, T. G., Irani, T. A., Telg, R. W., & Lundy, L. K. (2005). The development of an instrument to evaluate distance education courses using student attitudes. American Journal of Distance Education, 19(1), 51–64.

    Article  Google Scholar 

  • Ryan, J. M., & Harrison, P. D. (1995). The relationship between individual instructional characteristics and the overall assessment of teaching effectiveness across different instructional contexts. Research in Higher Education, 36(5), 577–594.

    Article  Google Scholar 

  • Schmelkin, L. P., Spencer, K. J., & Gellman, E. S. (1997). Faculty perspectives on course and teacher evaluations. Research in Higher Education, 38(5), 575–592.

    Article  Google Scholar 

  • Shevlin, M. (2000). The validity of student evaluation of teaching in higher education: love me, love my lectures? Assessment & Evaluation in Higher Education, 25(4), 397–405.

    Article  Google Scholar 

  • Timpson, W. W., & Andrew, D. (1997). Rethinking student evaluations and the improvement of teaching: instruments for change at the University of Queensland. Studies in Higher Education, 22(1), 55–65.

    Article  Google Scholar 

  • Tricker, T., Rangecroft, M., & Long, P. (2005). Bridging the gap: An alternative tool for course evaluation. Case study. Open Learning, 20(2), 185–192.

    Article  Google Scholar 

  • Trout, P. A. (1997). What the numbers mean: providing a context for numerical student evaluations of courses. Change, 29(5), 24–30.

    Google Scholar 

  • Williams, W. M., & Ceci, S. J. (1997). “How’m I doing?” Problems with student ratings of instructors and courses. Change, 29(5), 12–23.

    Google Scholar 

  • Wilson, K. L., & Lizzio, A. (1997). The development, validation and application of the course experience questionnaire. Studies in Higher Education, 22(1), 33–53.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Arizona State University, PO Box 37100, Phoenix, 85069-7100, Arizona, USA

    Thomas M. Haladyna & Audrey Amrein-Beardsley

Authors
  1. Thomas M. Haladyna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Audrey Amrein-Beardsley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Audrey Amrein-Beardsley.

Appendix

Appendix

Rights and permissions

Reprints and permissions

About this article

Cite this article

Haladyna, T.M., Amrein-Beardsley, A. Validation of a research-based student survey of instruction in a college of education. Educ Asse Eval Acc 21, 255–276 (2009). https://doi.org/10.1007/s11092-008-9065-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11092-008-9065-8

Keywords

Search

Navigation