From New York Times; New York; September 15, 1999; p. 1. [East Coast Edition] [New York City Public Schools] ******************************************************************
Chancellor Cites Test Score Errors
By Anemona Hartocollis
Chancellor Rudy Crew revealed yesterday that significant errors were made in scoring the high-stakes tests given last spring to determine who would be assigned to summer school and that some children had been compelled to attend by mistake.
Dr. Crew announced the problem in a news release, and declined to answer questions about it. But an aide said the number of children mistakenly assigned to summer school could range from 2,000 to 3,000.
Dr. Crew attributed the confusion to a miscalculation by the test publisher, CTB/McGraw-Hill, and the company confirmed yesterday that similar scoring errors had occurred on tests given in other parts of the country.
But yesterday's revelation was just the latest in a series of embarrassments for the Chancellor's plan to end automatic promotion of failing students. In mid-August, Dr. Crew proudly announced that he expected most of the 35,000 summer school students to go on to the next grade.
But on Sept. 1, he had to revise that figure, saying that 60 percent -- or 21,000 -- would be held back, largely because 14,000 students either did not show up in summer school or did not take the tests.
Education and testing experts said yesterday that the disarray in New York City was a lesson in how high-stakes testing across the country might encourage educators to make snap judgments about students that turn out to be unfair.
"What's wrong with it is who ends up getting hurt in this process: the kids," said Seymour C. Fliegel, a senior fellow at the Center for Educational Innovation, a New York City research institute.
As a result of the miscalculation, some parents were forced to give up summer vacations -- with immigrants sometimes canceling expensive airline tickets for homeland visits -- so that their children could go to classes.
Jill Chaifetz, executive director of Advocates for Children, a nonprofit advocacy group, said that many parents had been surprised to hear that their children were failing, and now, she said, it might turn out that their surprise was justified.
She said the revelation of the flawed scoring would only compound the distress of parents over what she criticized as the hastiness of the summer school program.
It is now possible, she said, that some children passed the tests last spring, were sent to summer school by mistake, and then failed the tests at the end of summer, forcing them to repeat the grade.
Advocates for Children has challenged the Board of Education's summer school program in court, arguing that it violates the board's own policy of looking at factors besides test scores -- like homework and the judgment of teachers -- in determining promotion. Oral arguments in the case took place yesterday.
New York City's summer school program has been strongly pushed by Mayor Rudolph W. Giuliani, who has argued that one way to improve schools is to get tough with failing students. A spokeswoman for Mr. Giuliani declined to comment on the scoring problems yesterday.
The mistake came in a percentile score that showed how the students compared with a national sample of children across the country. The publisher gave the test questions to a sample of students in different cities. But in some cases, the test company used answers from the wrong questions in computing the norm, or national average.
The company said that the scores were wrong for 4,500 students, or 3 percent of the 150,000 students in the national sample.
Steven H. Weiss, a spokesman for CTB/McGraw-Hill, acknowledged yesterday that the company was responsible for the scoring errors. He refused to say what other school districts had received flawed test results, saying he did not have their permission to release that information. But he said none of the other districts had used the test scores to make high-stakes decisions.
Karen Crowe, a spokeswoman for Dr. Crew, said the test company had told New York City that nine other jurisdictions were affected by the problem.
Mr. Weiss said that New York City's director of testing, Robert Tobias, had been the first person to suspect that there was an error in the reading and math tests, and had urged CTB/McGraw-Hill to investigate. Ms. Crowe said that Mr. Tobias's suspicions were aroused by the precipitous drop in test scores from the year before.
Although the tests were new and scores typically drop on new tests, Mr. Tobias believed, she said, that the drop was abnormally sharp.
Mr. Tobias first alerted the testing company to his concerns last spring, New York City officials said.
None of the children sent to summer school by mistake have been notified yet, because the board is still computing who they are and how many there are. Yesterday's estimate of 2,000 to 3,000 children is very preliminary, officials stressed.
Ms. Crowe said that it had taken until Monday for the testing company to confirm the problem, and that New York City officials were still trying to figure out the extent of the scoring error. Mr. Weiss said the delay occurred because the company had to review tens of thousands of test forms.
Saying that he was "distressed" by the problems, Dr. Crew called for an independent audit of the quality control procedures used by CTB/McGraw-Hill. Ms. Crowe said the school system was considering suing the test company.
"We pay an enormous amount of money to a company to provide accurate test results, which we use to make high-stakes decisions, and they should be accurate," Ms. Crowe said. In 1998, the board spent more than $2 million in contracts with the testing company, and in 1999, it is authorized to spend $3.3 million, she said.
Mr. Weiss said that the underlying integrity of its tests had not been questioned, and that the company believed that tests were one of the most reliable instruments available to measure student performance.
Monty Neill, executive director of the National Center for Fair and Open Testing, an advocacy group in Cambridge, Mass., said that similar scoring problems had occurred at least once before, on a test designed by the same company and used in Indiana. Mr. Neill said that in 1996-97, the superintendent of schools in Fort Wayne noticed a large drop in percentile scores despite an overall rise in the number of correctly answered questions on the test.
Mr. Weiss, the CTB/McGraw-Hill spokesman, said yesterday that he could not confirm or deny that case. But he suggested that Mr. Neill, as a frequent critic of testing practices, was unreliable.
"Given his well-known position against testing, it's not surprising he would choose to capitalize on this situation to advance his own cause," Mr. Weiss said.
But some testing experts said that New York City school officials had violated a widely accepted tenet in education by basing their decisions to promote children on the results of just one test.
"Whenever we focus on one factor alone, we risk making major mistakes like this one," said Jay P. Heubert, a professor at Teachers College at Columbia University who has studied high-stakes tests.
The reading and math tests, given last spring to 300,000 New York City students in grades 3, 5, 6 and 7, showed that 44.6 percent of students were reading at grade level. The board said that after equating, a process to make the tests comparable to the previous year's tests, reading scores across the grades were stable, while math scores dropped 10 percentage points.
Scores in the third grade, which Dr. Crew has made a focus of attention, dropped more than any in other grade in both reading and math.
It was the drop in third-grade scores, despite the resources devoted to that grade, that aroused Mr. Tobias's suspicions, Ms. Crowe said.
********************************************************** * Jerry P. Becker Dept. of Curriculum & Instruction Southern Illinois University Carbondale, IL 62901-4610 USA Fax: (618) 453-4244 Phone: (618) 453-4241 (office) (618) 457-8903 (home) E-mail: email@example.com