---------- > From: firstname.lastname@example.org > Hidden in the "etc." is one that puts it all at question, right, Wayne? > If you teach taking tests, kids will learn to take tests. And the > standardized tests will suggest, falsely, that the kids know reading, > and mathematics, and science, and history, and geography. You've > mentioned the effect yourself, though I can't see why it doesn't bother > you. When a new test is introduced, scores go down. As the district > becomes more and more familiar with the test, the scores go up. But > then when a new test is introduced, they go back down again. This is > prima facie evidence that they learn to take the test--not the content > the test allegedly covers. > > --Lou Talman
Well, one solution to that problem is to give every student every possible test once every year. So you would administer the SAT itself and even the GRE and MCAT to grades K through 12. You'll probably find less than 100 10 year olds that do well on the GRE and the MCAS. A careful researcher would then go and interview them to characterize how well they really knew the material. Perhaps 1 or 2 of the scores would be statistical fluctuation. Some might be found related to Asperger's syndrome. But there really are 10 year olds that bright and well read. The research literature on IQ 180+ kids goes back more than 70 years. THAT pool of kids would max out on most of the tests like ERB and SAT-9. There is also another pool of kids, the profoundly retarded, that would get minimum scores on most of the tests. In those two pools of kids, you could probably identify physiological mechanisms and/or genetic markers that would account for most of the variation. If you then proceeded to administer the same suite of tests every year for several years you could probably statistically characterize the "test-taking" effect Lou refers to with great precision. You would have a very good picture of the degree of correlation between tests and between tests and some genetic markers and between tests and various other social and economic variables. You would also have succeeded in wasting several years of education and hundreds of millions of dollars. There is a simpler solution. Acknowledge that the test taking effect exists alongside real measurement capability. Look at the Olympics: pole vaulting, discus, 100 meter, free style swimming, archery, etc. are all very carefully controlled standardized tests. They all have a component of measuring real ability and also have the test-taking effect problem. It's not likely that anyone naive to the Olympics or similar competitions would do well regardless of innate ability. And it is certainly true that even with a lot of practice in many competitions, some people just don't have what it takes to win even a local marathon. And athletic competitions also do a better job of measuring past experience than future performance: Athletes wear out and the next generation replaces them. That's just the way life is. Why do some people have trouble making similar judgements about academic achievement?
Standardized tests measure exactly what they test. for example: ______ growth in the economy was mainly responsible for the profit _______. (A)Little.. upturn (B)Sluggish.. slump (C)Rapid.. decrease (D)No.. growth (E)Buoyant.. decline If a student has no idea what the word responsible means, they're gonna have a tough time with this one aren't they? So a reasonably large vocabulary is needed. Next you need some vague idea of how profit is calculated. There is a segment of the population that would have serious difficulty with this problem for one or both of those reasons. I can also see how a professional margin trader might get this one 'wrong'. But I have a hard time believing that many readers of this list would have much problem with this question. It does seem to test content to me. I can even see it being used as the basis for a constructivist learning situation. We could discuss a few thousand similar test items on this listserve. The devil's in the details you know. It is easy to lampoon tests in general or a few particular questions, but it is more difficult to rule out the relevance of each and every question.
My son's private school did a small version of the first paragraph this year. The ERB and one of the SAT's was administered to each third grader. I only saw the scores for my son. I was struck by how good the correlation was between the subtest scores. Had they done only the ERB, I might have dismissed two of the lower scores as statistical fluctuations. Were the test scores useful? Again, I'd say yes. I am now motivated to find materials for practicing reading comprehension. I have no intention of teaching to the test. Those tests are gone by now. But I do believe reading comprehension is important and I do believe that it is a skill that can be developed through practice as much as batting or pitching or soccer.