RESPONSES TO RECENTLY-RAISED ISSUES REGARDING 12TH-GRADE TIMSS
Since the release of the TIMSS report on mathematics and science achievement in the final year of secondary school and the corresponding third volume of NCES' Pursuing Excellence, several questions have arisen about the validity of the comparisons and the relevance of the results. Some observers have argued that the methodology was biased against students in the United States in several ways, including the fact that the average age of U.S. students was lower than in other countries, that the populations tested in other countries represented a more selective group of the age cohort, and that not all our students who took the advanced mathematics examination had been exposed to the material on the assessment.
Several arguments have also been advanced about why the results of the assessments do not matter. Some say we should not be so worried about achievement at the end of high school because our students "catch up" in college. Critics cite the fact that our economy is the strongest in the world as evidence that people do not need the type of mathematics and science knowledge and skills measured on the TIMSS assessment. Howard Gardner writes that TIMSS measures only "a kind of lowest common denominator of facts and skills." More crucial, he argues, is the ability to "apply scientific and mathematical concepts to the world around them." Others mention abilities and qualities such as creativity, flexibility, and innovation as more crucial for economic success.
Following are responses to the most common questions and criticisms regarding the validity and relevance of the TIMSS assessment of students at the end of secondary school.
1.Differences in age and grade levels
Q. The students in other countries, on average, were older than the U.S. students. Is this because secondary schools in other countries go beyond grade twelve? If so, how fair or useful is it to compare U.S. students with those in other countries who are one to three years older and have more years of instruction?
A. The purpose of this component of TIMSS was not to compare students of the same age or years of schooling, but rather to compare students at a similar point in the education system: the end of secondary school. In many societies, including the United States, students completing secondary school are deemed "ready" to enter adult society, part of which is the workforce. Thus, a comparison of students at this point sheds light on the level of knowledge expected of and attained by the general population.
While there was a range of ages among students taking the assessment, and NCES found that age was related to achievement, the gap between the average ages of U.S. students and those in other countries, and consequently, the impact it has on the results, has been overstated. At the extreme, the students participating in TIMSS in Iceland had an average age of 21.2 years, 3.1 years higher than the average age of the U.S. students, 18.1. However, the international average was far lower: 18.7 years, much closer to the U.S. average. This gap was even smaller on the physics and advanced mathematics assessments, where the average age of our students was 18.0 years for both assessments. The international averages on these assessments were just a few months higher, 18.4 years in physics and 18.3 years in advanced mathematics.
Regarding the years of schooling, in other countries, both college preparatory and vocational programs may go through grade thirteen or fourteen, but most of those countries also have other secondary programs that end earlier, often before grade 12. While eight countries (out of 21 in the general knowledge assessments) included some students in grades above twelfth grade, all of those countries also included students in twelfth grade and five of them also included students below grade 12. It is also important to note that in some countries, the older average age of students reflects a later school starting age. In Denmark, Slovenia, Norway, Sweden, and parts of the Russian Federation and Switzerland, students start the first grade at the age of seven, compared to our typical starting age of six.
The differences in average ages and years of instruction also need to be considered in light of the content of the general knowledge assessments. If the content were based on high-level curriculum topics, then younger students and students with fewer years of schooling might be at a disadvantage, since it is reasonable to think that they would be less likely to have been exposed to these topics than older students or students with more years of schooling. However, the TIMSS general knowledge assessments did not represent advanced-level content. While the items on these assessments were not based on any one curriculum, TIMSS analysts have found the topics on the mathematics general knowledge assessment to be most similar to topics covered by the seventh grade in most countries, and the topics on the science general knowledge assessment similar to topics covered by the ninth grade. When they looked at the items in terms of U.S. curricula, they found these topics to be introduced at later grade levels, by the ninth grade for mathematics and by the eleventh grade for science. Thus, it appears that students in the United States and in other countries were being tested on topics they should have already covered, several years earlier in most cases.
In the end, it is difficult to use the higher ages of students in other countries to explain our relatively poor performance. While many of the countries which outperformed the U.S. included students whose average age was higher than ours, but we were also outperformed by countries in which the average age of students was lower than ours. For example, students in New Zealand and Australia were, on average, younger than U.S. students but still scored significantly higher than our students on both general knowledge assessments. It is also interesting to note that students in the Russian Federation performed similarly to U.S. students on both general knowledge assessments, but were more than a year younger and were all in eleventh grade.
2.Differences in enrollment rates
Q. In other counties, only the best students are still enrolled in secondary school in the late teenage years. Isn't it unfair to compare our general teenage population-the vast majority of whom are still in school-with elite populations in other countries?
A. There are several ways to look at secondary school enrollment. One way is to divide the number of young people enrolled in secondary school by the total number in the population within the corresponding age cohort. Recognizing that this can often overstate enrollment if young people outside the age cohort are enrolled, OECD data indicate that the majority of countries participating in this component of TIMSS, including the U.S., have over 85 percent of the age cohort enrolled in secondary schools. Thus, while variation in enrollment rates does exist, the countries are roughly comparable, and more so than in previous years.
Looking at enrollment another way, among seventeen-year-olds, the U.S. actually has a smaller proportion enrolled in school than the average for the other TIMSS countries for which this information is available (75 percent vs. 82 percent). If higher school enrollment rates were associated with lower average scores, the bias would be in favor of U.S. students rather than against them. However, in TIMSS, using either of the methods of measuring enrollment, students in countries with higher enrollment rates than the U.S. tended to score significantly higher than the U.S. on both the mathematics and science general knowledge assessments. Furthermore, in TIMSS, the pattern generally appears to be that higher secondary enrollment rates are associated with higher levels of performance, rather than the reverse.
3.The "unfair" definition of our advanced mathematics population
Q. The advanced mathematics test had calculus questions, but U.S. students whose highest mathematics course was precalculus were included in the test sample. Wouldn't it have been a fairer comparison to look only at those students who had actually taken calculus?
A. The advanced mathematics assessment was not primarily a calculus test. Calculus items comprised only about 25 percent of the questions. Interestingly, even with precalculus students included in our sample of advanced mathematics students, our weakest content area was not calculus, but geometry. This is even more surprising when we take into consideration that advanced mathematics students in the United States will typically have studied geometry for a full year by the twelfth grade.
The purpose of the advanced mathematics assessment was to compare advanced mathematics students across countries on content above the level of "general knowledge." For this assessment, countries were asked to create course- or program-based definitions of advanced students such that the resulting population represented from between 10 to 20 percent of the age cohort. To meet this specification, the United States needed to include students in precalculus classes, as only seven percent of the age cohort were taking or had taken a calculus class. (Would it be fair to compare seven percent in the U.S. with over 16 percent in Canada, 20 percent in France, or 33 percent in Austria?) The resulting population in the United States represented 14 percent of the age cohort, compared to the international average of 19 percent. In this respect, the group of U.S advanced mathematics students was somewhat more selective than in some other countries.
Even with precalculus students included in our advanced mathematics population, one cannot assume that U.S. students were at any more of a disadvantage than students in other countries. In none of the countries were students chosen on the basis of whether they had taken calculus. An in-depth curriculum analysis might reveal that students in other countries had more exposure to calculus (or other topics) than U.S. students, but it might also show that U.S. students had relatively more exposure to some topics than students in other countries. If U.S. advanced mathematics students do face a curriculum that includes fewer advanced topics than those faced by their international peers, that would be an important finding.
End of Part 1 of 2 ********************************************************
Jerry P. Becker Dept. of Curriculum & Instruction Southern Illinois University Carbondale, IL 62901-4610 USA Fax: (618)453-4244 Phone: (618)453-4241 (office) E-mail: JBECKER@SIU.EDU