by clicking on "Download the PDF" under "How to obtain this publication."
I have looked at some of the math items on the PISA, and some of these items do look impressive and would be pretty challenging to many students. For instance, question 10 asks students to calculate perimeters of figures not normally given on tests; other figures are typical but do not give the kinds of information that we can plug right away into a formula.
Some of the estimation questions are pretty clever and are probably challenging to many students. Question 5 on continent areas is one such question. Another one is Question 30 on shapes.
Question 28 on coins is pretty clever. Others are question 7 on the speed of a racing car, question 25 on the best car, question 43 on the lighthouse, and question 45 on the twisted building. These do take some non-routine thinking (at least with respect to traditional American education) to answer these.
Other items are fairly simple and routine. They look no different from standard textbook problems. Some (not all) examples include question 4 on cubes, question 12 on exchange rate, question 14 on colored candies, question 15 on science tests, question 29 on pizza, question 41 on shoes for kids.
Another item looks promising because it appears to be one in which you have to determine a pattern (it is the question 3 about apple and conifer trees). But then the second part of the question reveals the pattern! So the question fails to check if the student has actually found the pattern. If the question is given on MML or some other computer homework/quiz/test system where part 2 is not revealed till the student answers part 1, then this might work. Is the PISA given in this way? Or will students see all parts at one time? I cannot tell, but this question I ask is vitally important; without knowing, we cannot determine if later parts to a question are truly giving away answers to earlier parts. But even part 1 fails to check completely if the student has found the pattern in general because the student is not asked to derive a formula for the pattern.
Does that mean that these basic or routine items are "bad"? Not necessarily so. Basic questions are still good to ask because students will need to know the basics to be able to think critically. It helps to emphasize to students that basic questions are still important, not just the "thinking type" math questions--which are also very important as well. And basic questions help us to distinguish students who have trouble with more involved questions because they have trouble applying what they know from those students who simply lack the basic understanding of mathematical concepts. Without basic questions, we cannot distinguish these two kinds of students, and it is extremely important that we do so. But too much emphasis on basic questions is not good: We should not be content with just the ability of students to answer basic questions if we are to help them learn deeply and meaningfully. A bank of sample items does not really reveal to us what the proportion of basic questions is on the test. That is, we cannot really judge a test just by knowing that some items look like this and some look like that.
The document says that some of these items were experimental items that did not make it to the PISA whereas others are actual items from past PISA exams. But none of the past tests are reproduced in full; the document is just a smattering of various kinds of test questions.
PISA should release the actual test given and not just sample questions. Why is this? Without knowing what the test looks like, we can be fooled into buying the test creators' claims that the test is effective, that the test does check critical thinking and not just rote, blah, blah, blah. We cannot know if the test emphasizes rote and shallow thinking too much and not enough on higher reasoning. Many math tests I am not impressed by simply because the test focuses far too much on shallow thinking, not because the test contains inappropriate questions (though sometimes inappropriate questions will be a significant issue, but avoiding them is not enough to make an effective math test). Sometimes the problem is also with it being multiple choice. In other words, the main problem I usually see with the math tests I have seen is with what they lack.
Furthermore, how do we know that a test which is supposed to check critical thinking using items that the students should not have seen before really accomplishes that goal? Many teachers will use sample items to construct practice tests. If teachers can construct practice tests that are extremely similar to the actual test, then the actual test does not accomplish this stated goal. If teachers use what information PISA releases but does not release the actual test itself, we cannot verify this claim. Thus, I feel it is not a good sign when test creators will not release previously given tests to the public.
At least PISA requires students to explain their reasoning on many of these questions, which is a vital feature to any effective math test. Most standardized math tests are entirely or almost entirely multiple choice, which I abhor for a math test. However, based on my comments above about not being able to determine what any actual PISA test looks like, we cannot determine the proportion of multiple-choice items and free-response items.
I am bothered to see that students can get full credit for reporting answers without units of measurement. This encourages a bad habit in students because there is great difference between an abstract number as an element of an algebraic structure and as a real-world measurement. We cannot report a measurement as just 5 because abstract numbers do not tell us their size. For real numbers, we can determine their size relative to each other, but there is no "real world" size to the number 5 all by itself. There is also a big difference in the arithmetic of measurements and the arithmetic of abstract numbers. We can add the abstract fractions 5/6 and 2/5 as is (of course, we need a common denominator), but we cannot add 5/6 foot and 2/5 yard by adding the fractions as is because the measurements are not given in the same units.
Finally, to determine if PISA really does measure critical thinking in mathematics, we need to know what other countries teach and what they do, if anything, in preparing students for the PISA and how they choose their students who take the test. The comments I had mentioned about "clever" questions above might or might not apply to students in other countries; all I can go by at this time is using what is normally taught in America to make those judgment calls.
In short, PISA may or may not be effective in accomplishing the goals they intend for the test to have, but there is not enough information available to make any solid conclusions in their favor.