Miscalculation on Scores Shows a Weakness of Tests
By Anemona Hartocollis
[T] he revelation that a national testing company miscalculated the reading and math scores of thousands of New York City students opens a window into the fragility of standardized testing, even as school districts across the country are increasingly using test scores to make high-stakes decisions.
Officials for both the New York City school system and the testing company, CTB/McGraw-Hill, one of the three biggest in the country, have admitted that if not for the persistence of one whistle-blower -- the city's testing director -- the errors might never have been discovered.
Yet those errors led to one of the bleakest chapters in the tenure of Chancellor Rudy Crew, who had made literacy a priority and then in June was forced to announce the original low scores. The Chancellor used the scores to make a host of decisions with huge impact on the lives of children and educators: from taking over failing schools to removing superintendents to sending more than 8,600 children to summer school who should not have been forced to be there.
Now thousands of schoolchildren are trying to salve the ache of being unfairly branded as failures, and the mistake has tarnished the reputation of standardized testing carefully cultivated by the testing companies themselves.
"I don't think these kinds of errors are rampant," said Michael J. Feuer, director of the board on testing and assessment of the National Academy of Sciences. "But obviously, when you're dealing with attempts to analyze large quantities of data, errors are always possible."
The error on the CTB/McGraw-Hill test, which the company said also affected eight other systems, was not the first for the testing industry. In the mid-1990's, California introduced a statewide test and was criticized after improperly releasing scores to hundreds of schools that were based on inadequate samplings of students.
The problems, critics say, graphically illustrate why many educators and testing companies themselves advise against using a single test given on a single day as the sole criterion for making decisions about promotion or graduation.
"It's clear that CTB/McGraw-Hill messed up," said Monty Neill, executive director of the National Center for Fair and Open Testing in Cambridge, Mass. "But CTB did not make the Mayor, the Chancellor or the Board of Education decide to use test scores only, in violation of the standards of the testing industry, to make decisions about kids."
In the New York City episode, the testing company and Dr. Crew find themselves on the defensive. Yet Dr. Crew, who just three months ago said he would make "no excuses" for some of the worst test scores in years, now boasts just the opposite situation: one of the brightest pictures in years. The adjusted statistics show that 48.5 percent of New York City's third, fifth, sixth and seventh graders read at or above grade level, not 44.6 percent as announced in June. After a calculation to make the 1999 test comparable to the 1998 test, Dr. Crew announced on Wednesday that reading scores had risen 4.9 percent from 1998, an unusually high one-year gain for any school system.
Yet instead of basking in the glow of positive headlines, the Chancellor has found that achievement virtually ignored in the atmosphere of skepticism created by the errors. While almost no one -- except the school system's own director of assessment, Robert Tobias -- raised doubts about the dramatic 10-point decline in math scores announced in June, the new reading scores have drawn a different reaction. (The board has not yet recalculated math scores.)
"The fix is too good to believe," said Stephen Ivens, vice president of Touchstone Applied Science Associates, in Brewster, Putnam County, which supplied its reading test, Degrees of Reading Power, to New York City schools for 12 years before it was replaced by CTB/McGraw Hill.
"Could you as a publisher make the data come out more or less favorable?" he asked. "The answer is, yes you can, particularly once I know what the results are."
Dr. Crew said yesterday that he saw no reason to abandon the use of standardized testing in making high-stakes decisions like whether to promote students. "That is a really, really cynical view," he said of comments like Dr. Ivens'. " 'I paid them off'; 'I sold them my first-born.' There's a level of it being absolutely insidious -- when the scores look like they might be anywhere near decent, to just make this a scandal. These teachers and these children, they earned something."
Dr. Crew said he used the reading and math tests to screen for summer school, despite Tobias's suspicions, because the testing company strenuously denied until last week that there was any problem.
David Taggart, the president of CTB/McGraw-Hill, said yesterday that such scoring problems occur rarely, and are normally detected, as in New York City's case, by school district officials who notice something amiss when they put the standardized test scores in the context of everything else they know about their students.
CTB/McGraw-Hill has posted a prominent warning on its Web site, advising school districts that "no single test can ascertain whether all educational goals are being met."
Dr. Crew has removed five superintendents and put four more on probation, again citing low scores.
Now, eight of the nine districts where he removed superintendents or put them on probation show gains in reading instead of declines.
Robert Riccobono, who was ousted by Dr. Crew as superintendent of district 19 in East New York, Brooklyn, said he believed his fate and that of the other superintendents would have been different if the erroneous scores had never been announced. Riccobono is the only superintendent appealing his ouster. The new data, he said, could only help his case.
"Wow," he said, as he learned of the revised scores. "It has to help, because it's clear to me that the only reason we were under scrutiny to begin with had to do with the lower performance on the reading test."
******************************************************** * Jerry P. Becker Dept. of Curriculum & Instruction Southern Illinois University Carbondale, IL 62901-4610 USA Fax: (618) 453-4244 Phone: (618) 453-4241 (office) (618) 457-8903 (home) E-mail: firstname.lastname@example.org