Now, in studies that threaten to shake the foundation of high-stakes test-based accountability, Stroup and two other researchers said they believe they have found the reason: a glitch embedded in the DNA of the state exams that, as a result of a statistical method used to assemble them, suggests they are virtually useless at measuring the effects of classroom instruction.
Pearson, which has a five-year, $468 million contract to create the state’s tests through 2015, uses “item response theory” to devise standardized exams, as other testing companies do. Using IRT, developers select questions based on a model that correlates students’ ability with the probability that they will get a question right.
That produces a test that Stroup said is more sensitive to how it ranks students than to measuring what they have learned. Such a design flaw could also explain why Richardson students’ scores on the previous year’s TAKS test were a better predictor of performance on the next year’s TAKS test than the benchmark exams were, he said. The benchmark exams were developed by the district, the TAKS by the testing company.
Stroup, who is preparing to submit the findings to multiple research journals, presented them in June at a meeting of the House Public Education Committee. He said he was aware of their implications for a widely used and accepted method of developing tests, and for how the state evaluates public schools.
“I’ve thought about being wrong,” Stroup said. “I’d love if everyone could say, ‘You are wrong, everything’s fine,’ ” he said. “But these are hundreds and hundreds of numbers that we’ve run now.”
Gloria Zyskowski, the deputy associate commissioner who handles assessments at the Texas Education Agency, said in a statement that the agency needed more time to review the findings. But she said that Stroup’s comments in June reflected “fundamental misunderstandings” about test development and that there was no evidence of a flaw in the test.
About Holt Think: Ed, Creativity, Tech, Administration
Tim Holt shares his views on education, creativity, education administration, technology and the merger of all of them here. Whether it is links, articles, essays, or news, he shares a ton of information. He hopes you can keep up.
My new blog, about my quest for a PhD is located at http://callingdrholt.tumblr.com
All of these entries are the sole opinion of Tim Holt and do not reflect the opinions of his employers, friends, relatives, or anyone he happens to know in passing.
All entries here are copyright (cc) Creative Commons except where noted or obvious.