Q. There are many instances of numbers that are not believable-ones that are inconsistent with other data (within or outside of TIMSS) or with common sense. Given that, why should we trust any of the findings of the study?
A. There have been only a handful of numbers, out of the many contained in the published reports about the end of secondary school phase of TIMSS, which have been challenged by critics. In most cases, the seeming problems with the data reflect a misunderstanding or misrepresentation of the data.
For example, TIMSS data about the amount of time U.S. seniors work has been alleged to be inconsistent with similar data from the Bureau of Labor Statistics. However, the data being compared are not comparable, either in terms of the populations being compared (seniors in the case of TIMSS and the entire population ages 16 to 19 from BLS) or the questions used to gather the data (hours worked on a "normal school day" in TIMSS and hours worked per week in the other). Data gathered from 1992 high school seniors in February through April of 1992 by the National Education Longitudinal Survey of 1988 (NELS:88) reveal patterns of working near the end of the senior year in high school that are not inconsistent with those in TIMSS.
Another allegation is that only about half of the eligible population in each country participated in TIMSS, on average, and was ten percent or less in a few countries. However, this was not the case. The statistics being cited do not represent the percentage of the eligible population participating and the very low percentages are for countries that did not participate in the general knowledge assessments. Of the 21 countries participating in the general knowledge assessments, 14 excluded less than 10 percent of the internationally defined eligible population from the TIMSS sample. Five of the seven countries that excluded at least 10 percent of the eligible students performed the same or worse than the U.S. on one or both of the general knowledge assessments.
It is also important to remember that TIMSS is a sample survey. All the estimates have a margin of error (called the standard error). Numbers that seem inconsistent or implausible may not be so when the margin of error (sometimes a rather large margin of error) is taken into account. This can particularly be a factor when the total sample size is relatively small, or when the entire sample is subdivided into smaller subcategories. For example, the standard error for the estimate of the average score for all U.S. students on the science general knowledge assessment was 3.3 points. The standard errors for the average score ranged from 3.5 to 13.5 points for subgroups of U.S. students classified on the basis of how many science courses they were currently taking at the time of TIMSS.
Q. Because of all the differences between education systems and between cultures, we are really comparing apples with oranges. Because countries have different educational goals and philosophies, not to mention curricula and school systems, how valid are these types of comparisons?
A. Far from rendering international comparisons meaningless, cultural differences are often what international comparisons such as TIMSS seek to shed light upon. Some factors may be nearly impossible to measure across cultures, such as pressure to do well in school, but TIMSS offers an excellent opportunity to study several important differences between cultures and education systems (such as part-time employment, TV watching, hours spent doing homework, and factors related to school environment) and their relationship to achievement. What is remarkable is that-at least in initial analyses-so few of the factors examined one-by-one could account for our relatively poor performance. While the U.S. compared unfavorably to the international average on such factors as taking mathematics or science in the senior year, amount of homework, theft of property and personal threats at school (indications of a poor school environment), and hours working at a part-time job, none of these factors had a statistically significant relationship to our relative performance. Perhaps the explanation for our poor performance lies in cultural factors which may be difficult to change, such as pressure to do well in school; but it is also possible that they are due to school-related factors over which we have more control, such as curriculum and instruction.
6.The relevance of the results
Q. The end of high school is too soon to tell whether our young people acquire the necessary skills and knowledge. Don't most of them catch up in college, while their counterparts in other countries may not attend college or may take higher education less seriously than our students?
A. Many people have made the observation that our students do not become serious until they reach college. However, we cannot assume that all our students will attend college and take mathematics and science courses, and that students in other countries are standing still, waiting to be caught. While the majority of our high school graduates do enroll in either two-or four-year postsecondary institutions, a substantial proportion does not. Thirty-eight percent of 1994 high school graduates were not enrolled in a postsecondary institution in the fall after graduation. For those who do enroll, catching up often means taking remedial classes: 24 percent of 1995 freshmen took remedial mathematics.
If students complete college-and many do not, they can graduate having met only the minimum course requirements for mathematics and science, which in many cases may be no more than a single course in each. Data from the NELS:88 study show that of 1992 high school graduates who attended a postsecondary institution by spring of 1994, 32 percent had not taken any mathematics classes above the remedial level. Furthermore, the students who were most in need of "catching up," those with low levels of mathematics achievement at the end of their senior year in high school, were less likely than others to take mathematics above the remedial level in college.
While some of our students may catch up in college, by taking either regular or remedial mathematics classes, many may never close the gap between them and their international peers. Even if it is true, as some have argued, that college students in other countries do not work as hard as our students, we know that students in other countries start out at a higher level than ours, continue to take mathematics and science in college, and are more likely to specialize in mathematics- and science-related fields.
Q. Why should we worry about the results of a paper-and-pencil test? We know that other skills, such as the ability to apply knowledge, critical thinking, and creativity, are more important for success in the real world?
A. TIMSS is not a superficial test of knowledge and skills. The purpose of the general knowledge assessments was to see how well students could apply what they had learned in mathematics and science to situations likely to occur in their everyday lives. Items on all four of the end of secondary school assessments included not only multiple choice questions, but also short and extended free- response items as well. Typical items on the mathematics assessment required students to analyze a stated problem, decide which mathematical tools to use to solve the problem, and then solve it. Science items presented students with an observation, situation, or hypothesis and asked them to use their knowledge of science to either explain the cause, predict the results, or describe how one might go about testing the claim. Thus, the items on the TIMSS general knowledge assessments required a strong base of knowledge, but also a variety of other intellectual skills, including reasoning, application of knowledge, and designing multi-step solutions.
Q. U.S. students may not excel in terms of academic achievement in mathematics and science, but there are more important qualities required to succeed in life, such as creativity, flexibility, innovation, and entrepreneurial skill, none of which were addressed by the TIMSS examination. Why should we be concerned about the results of TIMSS when the strength of our economy shows that we do not have to perform well on these tests to prosper?
A. The strength of an economy or the quality of its workforce will never rest solely on academic achievement. As Robert J. Samuelson points out in "Stupid Students, Smart Economy?" (Washington Post, 3/12/98), there are many factors that contribute to a productive workforce, the education system being only one of them. Other factors include the availability of jobs, the work environment, and corporate practices. Thus, a successful economy does not mean we do not have serious problems in mathematics and science education, or that these problems do not matter, only that currently, our economy is succeeding in spite of these problems. For our long-term economic well-being, nearly all experts in the field of national economic productivity believe that the knowledge and skill levels of America's students in science, mathematics, and technology will become increasingly important.
Q. The majority of us use very little science and mathematics in our jobs. Why do individuals need a strong background in mathematics and science beyond basic computational skills?
A. There are several ways in which individuals benefit from having high levels of mathematics and science skills, both in the labor market and in their daily lives. Higher academic achievement is associated with higher wages and lower unemployment. Data on those currently employed show that those with high levels of mathematics and science achievement earn significantly more and are less likely to face periods of unemployment than those with lower levels of achievement, even among people with the similar levels of educational attainment (NCES, Education and the Economy, NCES97-279, pp. 33-41).
As crucial as academic achievement may be to success in the labor market, the importance of science and mathematics knowledge reaches beyond economic productivity. The TIMSS general knowledge assessments were designed to test students' ability to apply their knowledge to situations they might encounter in everyday life, which is becoming more complicated. As adults, today's students will need to make sense of a rapidly changing world. They will need to make difficult decisions for themselves and their families regarding finances and health care. Collectively, as members of a community, they will need to understand and make decisions regarding how to treat the world in which they live. *************************************************************
Jerry P. Becker Dept. of Curriculum & Instruction Southern Illinois University Carbondale, IL 62901-4610 USA Fax: (618)453-4244 Phone: (618)453-4241 (office) E-mail: JBECKER@SIU.EDU