SAT error highlights imperfections -- and raises questions

~ Critics of standardized testing say this confirms that the industry is stretched too thin.

For the last five years, Hamilton College in upstate New York has been one of a growing number of colleges not to require the SAT exam. The test causes too much anxiety, Hamilton concluded, and there's a risk of missing bright students who don't test well.

On Tuesday night, Hamilton's faculty voted unanimously to make that policy permanent. By coincidence, the next morning brought a reminder that there's another potential downside to standardized tests -- news arrived that 4,000 SAT exams taken last October had been mis-scored.

"They do a lot of things right," Hamilton dean of admission and financial aid Monica Inzer said of the College Board, which owns the exam. "But it shows how vulnerable we all are when we depend too much on one test."

The error affected fewer than 1 percent of test-takers, and shouldn't affect admissions decisions -- though Inzer noted it's too late for students to apply to schools they might have considered with a higher score.

Experts say mistakes are inevitable in any operation on the scale of grading millions of tests. Still, the episode is likely to spark wider discussion about standardized tests -- both college entrance exams and the growing number of high-stakes, state-level exams: Just how much risk of error is tolerable when students' futures are at stake?

Recent years have seen a number of scoring errors on state-level tests, graduate school exams like the Graduate Management Admission Test. Some were small and caught early, others significant. In 2003 and 2004, 4,100 people were incorrectly told by the Educational Testing Service they failed a teacher licensing exam. In 2000, more than 8,000 Minnesota high school students were mistakenly told they had failed a state exam, and dozens missed their class graduation ceremonies.

That mistake prompted a previous incarnation of Pearson Educational Management, which also scores the SAT, to pay a $7 million settlement. On college admissions bulletin boards this week, there was talk of lawsuits in response to the SAT gaffe, along with angry comments from students and parents.

While the SAT error was comparatively small in scale, "it is such a visible program, that people freak out," said Scott Marion, vice president for the New Hampshire-based National Center for the Improvement of Educational Assessment.

"These students and their families are so anxious, and something like this even though it's relatively minor out of the millions of kids who took standardized tests this year, it rattles their sense of anxiety even more," said Wylie Mitchell, dean of admissions at Bates College in Maine, another SAT-optional school.

Critics of standardized testing seized on the error as confirmation the testing industry -- dominated by CTB/McGraw -Hill, Harcourt Assessment and Pearson -- is stretched too thin for the public's good.

"The volume is way up, and the people with the competence to do this don't exist," said Robert Schaeffer of the group Fair Test, which opposes many of the ways standardized tests are used.

A recent report by Education Sector, a Washington, D.C.-based think tank, portrayed a highly competitive industry facing huge pressure from its biggest clients -- the states -- to cut costs and deliver results quickly. That time pressure is sometimes reinforced by contract provision for financial penalties if scores are late coming back.

Pearson says the SAT error may have been caused by excessive moisture that caused answer sheets to expand and some marks to be unreadable. Spokesman David Hakensen said Friday Pearson has invested heavily in quality and capacity; since 2000, it has increased its number of scanners by 66 percent, added 60 percent more processing space and increased its report printing capacity 45 percent.

Most of the incorrect scores were off by fewer than 100 points on the 2,400-point test, and only 16 changed by 200 points or more, the College Board said.

Marion said companies like Pearson are improving their processes, but the increased demand and time pressure may be negating the progress. In any case, perfection is impossible.

"You won't see this mistake from Pearson again, but you'll see a different mistake," he said. "As long as you have humans involved, you're going to have some mistakes."

Fair Test, Schaeffer's group, wants more transparency and expansion of the rights of students to challenge their scores on standardized tests. The SAT error was uncovered because at least one student asked for a hand score. But that request costs $50 (refunded if an error is found), and there's a risk of getting a lower score. The College Board says it gets about 500 such requests per year, most of which reveal no error.