Education briefing

Education briefing

Article Type: Education briefing From: Journal of Property Investment & Finance, Volume 30, Issue 6

Introduction

Exams are almost always the main instrument for gauging a student’s mastery of a course, and therefore, for assigning an equitable grade to the student’s performance. Traditionally in the UK student performance has been assessed by unseen essay based exams. But today many universities are experiencing budget cuts that result in larger classes. The need to teach these larger classes without suffering an excessive reduction in the amount of time available for research and administrative duties exerts pressure on lecturers to make increasing use of multiple-choice tests.

The major advantages of multiple-choice tests are:

• •

  speed of marking leading to quick feedback;

• •

  lack of bias in grading; and

• •

  breadth of coverage of the subject (Saunders and Walstad, 1990).

The disadvantages of multiple-choice tests are:

• •

  lack of depth of thinking;

• •

  a fixed response encouraging guessing (Bridgeman, 1992);

• •

  difficulty of construction, especially to test higher order thinking (Brown et al., 1997; Suskie, 2004); and

• •

  gender-biasness (Hassmen and Hunt, 1994).

The advantages of essay based exams are:

• •

  they give students the freedom to show originality and a greater depth of understanding of the topic (Bridgeman, 1992; Scouller, 1998; Scouller and Prosser, 1994; Walstad and Becker, 1994; Tuckman, 1993; Gijbels et al., 2005);

• •

  they provide a written record for assessing the thought processes employed by the examinee; and

• •

  ease of construction primarily because fewer questions are needed to be prepared.

The disadvantages of essay based exams are however numerous and include:

• •

  the length of time needed to mark the exam leading to slow feedback (Wainer and Thissen, 1993);

• •

  the questionable reliability of the marking process (Tuckman, 1993);

• •

  the issue of whether an open-ended question elicits the intended response;

• •

  racial and cultural biasness (Hassmen and Hunt, 1994);

• •

  the inclusion of only a small number of questions, which does not evaluate student learning over all of the topics covered in the course (Walstad and Becker, 1994); and

• •

  whether writing skills rather than an understanding of the subject may be more highly rewarded by markers (Wolcott, 1988; Powers et al., 1994; Connelly et al., 2005).

A comparison of the advantages and disadvantages of essay based exams and multiple-choice tests suggest that they appear to be measuring different dimensions of knowledge (Becker and Johnston, 1999). Multiple-choice test examining the breadth of knowledge of the students in the subject, while their depth of knowledge of the course can be better assessed through the written exam. In other words, multiple-choice tests and essay based exams are complementary and so a single form of testing should be avoided. As a consequence, on a number of courses in the real estate masters programs at City University London we examine student performance using both methods of assessment.

One important aspect of any test and in particularly multiple-choice tests is the length of time given for the test. Most multiple-choice tests are administered under timed conditions. That is, a predetermined amount of time is allowed for completing the test, and any items not completed within the prescribed time limit are considered to be incorrect. But do the early finishers perform significantly differently than the other students? In other words, is there a systematic relationship between test completion time and multiple-choice test performance?

The effect that completion time has on a student’s test score is uncertain a priori. For instance, it could be argue that good students know the material so they will take less time. Alternatively, it could be argued that good students are diligent in checking their answers during tests so they would take more time. The argument about poor performing students is just as uncertain. One argument would be that poor students take less time because they do not know the answers. Another way of reasoning is that poor performing students take more time because they are struggling to complete the test in time.

The results of previous studies in a number of disparate degrees provide little in the way of resolving this uncertainty; as they produce confusing and contradictory results. For instance, some studies report a positive linear relationship, i.e. the longer the time a student spends on a test the higher the score (Feinberg, 2004; Obligacion, 2004; League, 2008); while others find a negative linear relationship, which suggests that those students who take a shorter time perform better (Takeda, 2007). Some studies find a nonlinear inverted U relationship with higher scores for those finishing during the middle of the test (Terranova, 1972; Hilmer and Hilmer, 2008), while others find a nonlinear relationship with the best scores at the beginning and end of the exam (Senior et al., 2010). On the other hand some studies produce inconclusive results (Landrum et al., 2009), while a number of studies find little or no relationship between completion time and test scores at all (see inter alia, Bridges, 1985; Paul and Rosenkoetter, 1980; Foos, 1989; Wierzbicki, 1994)[1].

This inconsistency in previous studies maybe a reflection of the fact that some studies only examine the correlation between completion time and test scores, while others include control variables for other factors which might explain test performance. For instance, Landrum et al. (2009) find that while older students tend to take more time to complete a test, age was not significantly related to test performance. Landrum et al. (2009) and Feinberg (2004) both find the students who performed well in early tests also tended to perform well in subsequent tests, but not with time to complete the test. Feinberg (2004) also found no gender impact of completion time on tests scores. Yet, as reported by Hassmen and Hunt (1994), numerous studies have shown that men tend to have an advantage on multiple-choice test.

To the best of the author’s knowledge no study has been undertaken on the effect of completion time on multiple-choice tests in real estate courses. In the empirical work below, we attempt to determine the effect that completion time has on a student’s multiple-choice test scores.

The paper is set out as follows. The next section provides summary statistics of test scores and completion times in two courses at the City University London. Section 3 presents the results of a regression of completion time on test scores after controlling for gender and age. The last section concludes the study.

Data and summary statistics

The author is at the City University London and lectures on the two post-graduate degrees: MSc Real Estate Investment and MSc Real Estate at the Cass Business School. The study uses data from two first term courses: Real Estate Investment and Financial Analysis (REIFA) taken by students on MSc Real Estate Investment and the Real Estate Asset Management (REAM) course taken by MSc Real Estate students. The number of students on the REIFA and REAM courses is 47 and 86, respectively[2]. The students in these particular courses were required to complete a multiple-choice class test and one unseen written exam. Each exam was worth 100 points, with the multiple-choice class test accounting for 25 per cent and the final exam counting 75 per cent towards the overall grade.

For each of the class tests the students were required to record the time they finished the test, to their satisfaction, on the covering page of the test. The students completion time was then calculated as the difference between the finish and start time in minutes. Both multiple-choice tests were intended to take a maximum of 60 minutes and in fact most students completed the test in less time (with a mean test time of 56 minutes and 46 minutes for the REIFA and REAM courses, respectively). Scores in the multiple-choice tests were expressed in percentages (number of correct responses divided by the perfect score).

We start by briefly discussing summary statistics for the students’ performance presented in Table I. The average scores on the two tests were 76 and 72 per cent, while the median scores were 78 and 74 per cent, respectively. Looking at the tails of the distributions, the maximum scores were 92 and 86 per cent, while the minimum scores were 52 and 48 per cent, respectively.

When looking at the time spent, Table I shows that the average times on the two tests were 56 minutes and 46 minutes, while the median times were 60 minutes and 45 minutes, respectively. The maximum times were 60 minutes and 75[3] minutes, respectively, while the minimum times were 25 minutes on each test.

Table I Summary statistics

There were 30 students (65 per cent) on the REIFA course who took the maximum 60 minutes to complete the test, which suggests that the time allocated for the exam (along with the average time being 56 minutes) may be too short. By way of a contrast, only 11 (13 per cent) of the students taking the REAM exam took the full 60 minutes. This may be an indication (along with the average time being only 45 minutes) that the time allocated for the REAM exam may have been too long.

Next we analysed the edge points of the data. The four edge points would be:

• •

  the data point where the least amount of time was taken;

• •

  the data point where the most amount of time was taken;

• •

  the best scoring exam papers; and

• •

  the worst scoring exam papers, the results shown in Table II[4].

Looking at the 30 students that took the longest time to finish the REIFA test, their average score was 75 per cent, which is 1 per cent higher then with the overall average (76 per cent). At the opposite end of the spectrum, the student that finished the REIFA test first had an average score 2 per cent higher than the overall average (78-76 per cent). Of the 11 students who took the longest time to finish the REMA test, their average score was 2 per cent lower than the overall average (72 per cent), whereas the two students that finished the REMA test first had an average score of 75 per cent, which is 3 per cent higher than the overall average. However, the 30 students that took the longest time on the REIFA test had scores ranging from 54 to 92 per cent, while the 11 students that took the longest time on the REAM test had scores ranging from 54 to 82 per cent, both of which are very similar to the maximum and minimum scores overall. This implies that those students who use the whole time allocated for the test perform both the best and the worst, while those students who finish first are not necessarily the best or the worst.

Table II Edge point analysis: REIFA and REAM

Looking at the top two performing students on the REIFA test, their average time was 53 minutes, which is 3 minutes quicker than the overall average (56 minutes), whereas the bottom performing student took 55 minutes, which is same as the average of all students. The top two performing students on the REAM test were considerably quicker than the average, by 8 minutes, while the two worst performing students were 5 minutes faster than the overall average (46 minutes). In other words, while the best students tend to finish quicker than the average, the worst students were taking no more time than the average. This implies that while the best students will not necessarily be the first to finish, typically a student that finishes first tends to get a good score, suggesting they know the material, whereas the worst performing students probably take less time because they do not know the answers.

We nexted calculated the correlation between the students’ scores and completion time, together with two other variables also thought to influence test scores[5]. First, gender (male/female) as suggested by Hassmen and Hunt (1994) and second student age, in years, as Landrum et al. (2009) find that older students tend to take more time to complete a test than younger students. The results presented in Table III for both multiple-choice tests.

Panel A of Table III shows there is a negative, but insignificant, correlation between completion time and exam score on the REIFA test, indicating degradation in performance as you take a longer time to finish the test. By way of a contrast, Panel B of Table III shows that the REAM test indicates a positive, but insignificant, relationship between test score and completion time, suggesting that the more time you take in the test the higher your score.

Table III Correlation test scores and completion time: REFIA and REAM tests

Table III also shows that the correlation between age, gender and student performance varies across the tests. In that although Panels A and B of Table III both show that age is positively related to the students’ test score, there is a marked difference between gender and test scores between students taking the REIFA and REAM tests. The results in Panel A suggest a positive relationship between gender and performance, while Panel B indicates a negative relationship.

These results therefore confirm and reject the findings of previous studies. In other words, the simple reliance on the results of a correlation analysis between multiple-choice test scores and completion time can easily lead to conflicting conclusions across degrees and courses. We need to examine the relationship between test scores and completion time in a multivariate setting controlling for additional factors that influence student performance such as gender and age.

Regression results

The primary hypothesis to be tested is that a student’s test score is related to his completion time and given the results in previous studies we posit that the relation is nonlinear; either U shaped or an inverted U relationship (see inter alia, Terranova, 1972; Hilmer and Hilmer, 2008; Senior et al., 2010). Specifically, as the length of time increases, a higher/lower score is expected initially. However, beyond a certain point an increase in the completion time leads to a decrease/increase in test score. To tests this hypothesis we use regression analysis with the logarithm of the students score as the dependent variable and a number of the independent variables: the logarithm of completion time, and its squared variable, the logarithm of the student’s age and a dummy variable for gender (male/female), see equation (1):

Where: the dependent variable, ln(S), is the natural log test score of the student, ln(T) is the natural log of the amount of time that an individual takes to complete the exam, [ln(T)]² is the square of the natural log of the amount of time that an individual takes to complete the test, ln(Age) is the natural log of the age of the individual, MF is a dummy variable taking the value one, if the individual is male, and zero otherwise and β₀, (β₁, β₂, β₃ and β₄ are the regression coefficients to be estimated. The result of the regression analysis is shown in Table IV for both multiple-choice tests.

The results of Model 1 in Table IV show that simply regressing student test scores on completion time alone can easily lead to conflicting results, as suggested by the correlation analysis. The results for the REIFA regression suggesting a negative relationship, while the REAM regression implies a positive relationship, between student test scores and completion time. However, once the square of completion time is included in the regression (Models 2 and 3), the results are consistent across both tests. Models 2 and 3 both show that multiple-choice test scores are positively related to completion time and negatively related to its square; suggestive of an inverted U relationship. In other words, students who finish first or last do not necessarily achieve the highest scores, although the regression coefficients are insignificant at the usual levels. This relationship is consistent with the edge point analysis above and supports the results of Terranova (1972) and Hilmer and Hilmer (2008) but contradicts the findings of Senior et al. (2010).

Table IV also shows that the relationship between test scores and completion time are also influenced by our control variables; age and gender. Test scores are significantly positively related to age, which contradicts the findings of Landrum et al. (2009). The results also show that gender is insignificantly related to test scores in both tests; supportive of the results of Feinberg (2004) but contradictory of the findings of Hassmen and Hunt (1994).

Table IV Regression results of test scores and completion time: REIFA and REAM tests

The adjusted R² values and F-statistics in Table IV, however, show that the various models explain only a small amount of the variability in the data, even with our control variables, and so additional variables thought to explain a student’s performance in tests need to be investigated.

Conclusion and implications

We estimated the relationship between student test completion time and performance on two different multiple-choice exams on two different masters’ course in real estate. The study found a positive relationship between completion time and test performance and a negative relationship between the square of completion time and test performance in both multiple-choice tests, after controlling for age and gender. This implies an inverted U relationship between completion time and test scores with higher scores for those finishing during the middle of the test. In other words, students who finish first or last do not necessarily achieve the highest scores, although the regression coefficients are insignificant at the usual levels. Lack of significance is expected however as students are individuals who do not behave in the same exact way. There are students who do well, whether they finish early or finish late, while there are students that do poorly, but who finish early, as well as some who do poorly when finishing late, as shown by the edge analysis. So we cannot say that by just by taking more time students will do better or worse on multiple-choice tests.

So what do we tell our students about the how to use their time in taking multiple-choice tests? The findings suggest that students should aim to take up most of their allotted time to complete the test, not simply to choose the right answers, but to review their answers before putting down their pens. Nonetheless, there are no substitute to diligent preparation and sound study habits.

Notes

1. 1.

   The studies by Foos (1989) and Wierzbicki (1994) examine completion order, which is not the same as measuring the amount of time to complete the exam.

2. 2.

   One student failed to take the REIFA test, while four failed to take the REAM test due to illness and although these students subsequently took the test at a later date the questions were different and so their results were excluded from the analysis.

3. 3.

   Seven students on the REAM tests were allowed 15 minutes additional time due to reading difficulties. Two of these students completed the test in less than 60 minutes, one student only took additional 5 minutes, while another four took the extra 15 minutes allotted to them which accounts for the 75 minute maximum on the REAM test.

4. 4.

   The students given extra time on the REAM test were excluded from this part of the analysis.

5. 5.

   Additional variables that may explain student test performance such as: positive personality traits (Stoeber and Kersting, 2007); cognitive ability (Rohde and Thompson, 2007); stress coping strategies (Folkman and Lazarus, 1985); and the influence of racial and family background (Haile and Anh Ngoc, 2008) are beyond the scope of this study.

Stephen L. LeeCass Business School, City University London, London, UK