The need for objective measures of student achievement to assess the effectiveness of schools has crossed over from K-12 academic institutions into the realm of higher education. The number of college students being asked to take skills tests is growing with the results having an increasing impact on educational policy.
However, the results of the exams infrequently have any impact on the students themselves, and a new paper from the Educational Testing Service looks at whether that has a bearing on the relevance of the exam results. In other words, if the students don’t care about the outcomes of the exams, does that mean that the exams don’t serve as an accurate reflection of their knowledge?
The study, designed and led by Lydia Liu, Brent Bridgeman and Rachel Adler, seems to point to that conclusion. The authors recruited 757 students to take an abbreviated version of the multiple-choice ETS Proficiency Profile, an essay portion of the same exam, and a brief survey to determine how motivated they were to do well on their exams. Before the tests were administered, students were told that their scores would be used to assess the quality of the school they were attending, and might, as a result, influence how their diploma was perceived.
The students were then assigned into three random groups. One was assured that their scores would remain anonymous and would only be used for further research. The second thought that faculty at their university would be privy to their scores. A third was told that the results would be provided to potential employers to allow them to judge the students’ level of knowledge.
It turned out that being extra motivated did wonders for one’s test scores. Even when other factors were controlled for, students who thought that their scores might have a real impact on their academic careers received scores that was .86 of a standard deviation higher than those who thought their scores would remain anonymous.
Most strikingly, after controlling for SAT scores, the score differences between sophomores and seniors, namely the value-added learning that could be attributed to students’ college education, varied dramatically from negative gain (-0.23 SD) to substantial gain (0.72 SD) across motivational conditions and tests.
The students’ assessment of their own motivations proved to be fairly accurate as well. Those whom the short survey identified as being more motivated tended to perform better. These conclusions have led the researchers to offer several recommendations to schools who hope to use these kinds of tests to assess student knowledge.
Limited college learning reported in prior research is likely an underestimate of student learning, if students received test instructions similar to the instruction used in our control condition. There are practical strategies that institutions can use effectively to enhance students’ test-taking motivation. Institutions that employ different motivational strategies should be compared with each other with great caution, especially when the comparison is for accountability purposes.