Who is the expert? Construct and criteria validity of student and teacher ratings of instruction

Abstract

In this article, we examine the construct and criteria validity of student and teacher ratings as indicators of instructional features. As parsimonious measures, student and teacher reports are frequently used to assess aspects of the learning environment. From a methodological perspective, however, both approaches have been questioned. Whereas student ratings are occasionally criticised as being undifferentiated and easily influenced by personal preferences, teacher reports are sometimes considered to be biased by self-serving strategies or teaching ideals. Instead of pitting one method against the other, our study aimed at establishing the specific value of each approach. The analyses draw on data from a German extension to the 2003 PISA assessment, in which 288 mathematics teachers and their students completed questionnaires using identical items to tap various aspects of instruction. Mean student ratings were computed for each class. Exploratory and confirmatory factor analyses revealed specific conceptual structures for student and teacher ratings, with teachers elaborating on the use of tasks and methods, and students focusing on their teacher’s support in personal and learning matters. Three shared dimensions were also established: occurrence of classroom management problems, degree of cognitive autonomy during lessons, and tempo of interaction. Teacher/student agreement on these dimensions varied across constructs, with considerable agreement on classroom management, low agreement on cognitive autonomy, and no significant agreement on whether or not the interaction tempo was appropriate. Accordingly, correlations with external criteria (student motivation, teacher motivation, mathematical achievement scores, and characteristics of the tasks set in class) also varied across constructs. We conclude that student and teacher ratings are best suited to tapping different aspects of the learning environment, and recommend that care be taken in choosing a data source appropriate for the construct to be measured.

Appendices

Appendix 1

Student questionnaire items with loadings on the latent factors and interfactor correlations

Questionnaire item	Factor loadings
	Inefficient classroom management	Inadequate interaction tempo	Tasks	Experience of cognitive autonomy	Perceived social support from the teacher
In mathematics the lesson often starts late.	0.79
... it takes a very long time at the start of the lesson until the students have settled down and started working.	0.95
... a lot of lesson time is wasted.	0.93
... students mess around the whole time.	0.95
... students talk among themselves the whole time.	0.96
... teaching is very often interrupted.	0.95
... teacher often expects split-second answers.		0.77
Our mathematics teacher does not leave us much time to think when asking questions.		0.99
... often sets problems where it is not just the calculation that is important, but choosing the right approach.			0.71
... sets problems where we have to take time to think.			0.56
... sometimes asks us to demonstrate several different ways of solving a problem.			0.57
... sometimes sets problems that do not have a clear-cut solution and asks us for explanations.				0.57
... changes the setting of problems to find out whether we have really understood the mathematical idea.					0.75
... varies problems to find out what we have understood.					0.83
... lets us use our own strategies to solve difficult problems.				0.76
... lets us solve problems according to our personal preferences.					0.64
In mathematics lessons our teacher sometimes lets us go astray in our working until we notice that something is wrong.				0.59
In mathematics lessons our teacher sometimes accepts mistakes and lets us carry on until we see that something must be wrong.				0.63
Our mathematics teacher immediately knows what someone has not understood.					0.91
... immediately notices if someone has not understood something properly.					0.92
... knows exactly what each of us can accomplish.					0.74
... always notices immediately if a student is not keeping up.					0.92
... is quick to see if students have argued.					0.83
... is quick to see if there’s something wrong between students sitting next to each other.					0.82
... soon notices if a student has problems or worries.					0.84
... can easily tell if someone is really sad.					0.82
... always addresses students’ problems.					0.93
... always takes time to talk if students want to discuss something with him/her.					0.84
Factor	Interfactor correlations
Inefficient classroom management	1.00
Inadequate interaction tempo	0.29*	1.00
Tasks	−0.41*	−0.27*	1.00
Experience of cognitive autonomy	−0.23*	−0.35*	0.57*	1.00
Personal support	−0.23*	−0.57*	0.41*	0.74*	1.00

1. *p < 0.05

Appendix 2

Teacher questionnaire items with loadings on the latent factors and interfactor correlations

Questionnaire item	Factor loadings
	Inefficient classroom management	Inadequate interaction tempo	Cognitive challenging tasks	Variation of tasks	Supporting cognitive autonomy	Use of student errors	Monitoring problems	Providing social support
It is difficult to start the lesson on time in this class.	0.83
In this class, it takes a long time at the beginning of the mathematics lesson until the students are quiet and begin working.	0.82
I often have the impression that a lot of time is wasted in mathematics lessons in this class.	0.69
The students in this class mess around a lot.	0.82
The students in this class talk among themselves a lot.	0.80
Lessons are very often disrupted in this class.	0.85
I expect quick answers to spontaneous questions in this class.		0.42
If a student cannot answer a question, I move on to the next student immediately.		1.00
I set problems where it is not just computation that is important, but choosing the right approach.			0.45
I set problems where students have to take time to think.			0.65
I sometimes ask students to demonstrate several different ways of solving a problem.			0.59
I set problems that do not have a clear-cut solution and ask students for explanations.			0.62
I change the setting of problems so that I can see whether the student has really understood the mathematical idea.				0.67
I vary problems so that I can see what the students have understood.				0.77
I let students apply their own strategies to solve math problems.					0.54
I let students solve mathematics problems according to their personal preferences.					1.00
I sometimes deliberately let the students go astray until they realise that something must be wrong.						0.86
If a student makes a mistake when a new topic is being covered, I initially accept the suggestion without comment, and carry on working through the problem with the students until the mistake becomes obvious.						0.57
It is easy for me to tell whether a student has understood a problem.							0.57
I am very quick to notice if someone has not understood something.							0.60
I know the strengths and weaknesses of individual students.							0.54
I immediately notice if a student is not keeping up.							0.56
I am quick to see if students have argued.							0.69
I am quick to see if there’s something wrong between students sitting next to each other.							0.70
I am quite quick to notice if a student has problems or worries.							0.79
I can easily tell if someone is really sad.							0.72
I take care of my students if they have problems.								0.76
I take time to listen if my students want to discuss something with me.								0.82
Factor	Interfactor correlations
Inefficient classroom management	1.00
Inadequate interaction tempo	0.10	1.00
Cog. Challenging tasks	−0.24*	−0.27*	1.00
Variation of tasks	−0.20*	−0.20*	0.58*	1.00
Supporting cog. autonomy	−0.10	−0.21*	0.31*	0.30*	1.00
Use of student errors	−0.30*	−0.15*	0.37*	0.26*	0.25*	1.00
Monitoring problems	−0.19*	−0.23*	0.17*	0.17*	0.21*	0.18*	1.00
Providing social support	−0.17*	−0.27*	0.13*	0.26*	0.28*	0.22*	0.43*	1.00

1. *p < 0.05