Math Forum Discussions

Jonathan Groves

Posts: 2,068
From: Kaplan University, Argosy University, Florida Institute of Technology
Registered: 8/18/05

Is the PISA Exam Effective? Answer Is Inconclusive
Posted: Jan 13, 2011 4:02 AM

Plain Text

Dear All,

A copy of sample PISA items can be downloaded from the website

http://www.pisa.oecd.org/document/31/0,3343,en_32252351_32236191_41942687_1_1_1_1,00.html

by clicking on "Download the PDF" under "How to obtain this publication."

I have looked at some of the math items on the PISA, and some of these
items do look impressive and would be pretty challenging to many students.
For instance, question 10 asks students to calculate perimeters
of figures not normally given on tests; other figures are typical but do
not give the kinds of information that we can plug right away into a
formula.

Some of the estimation questions are pretty clever and are probably
challenging to many students. Question 5 on continent areas is one
such question. Another one is Question 30 on shapes.

Question 28 on coins is pretty clever. Others are question 7 on
the speed of a racing car, question 25 on the best car, question 43
on the lighthouse, and question 45 on the twisted building.
These do take some non-routine thinking (at least with respect to
traditional American education) to answer these.

Other items are fairly simple and routine. They look no different from
standard textbook problems. Some (not all) examples include question 4 on
cubes, question 12 on exchange rate, question 14 on colored candies,
question 15 on science tests, question 29 on pizza, question 41 on shoes
for kids.

Another item looks promising because it appears to be one in which you have to
determine a pattern (it is the question 3 about apple and conifer trees).
But then the second part of the question reveals the pattern! So the question
fails to check if the student has actually found the pattern. If the question
is given on MML or some other computer homework/quiz/test system where part
2 is not revealed till the student answers part 1, then this might work. Is
the PISA given in this way? Or will students see all parts at one time?
I cannot tell, but this question I ask is vitally important; without knowing,
we cannot determine if later parts to a question are truly giving away answers
to earlier parts. But even part 1 fails to check completely if the student has
found the pattern in general because the student is not asked to derive a formula
for the pattern.

Does that mean that these basic or routine items are "bad"? Not necessarily so.
Basic questions are still good to ask because students will need to know the
basics to be able to think critically. It helps to emphasize to students that
basic questions are still important, not just the "thinking type" math
questions--which are also very important as well. And basic questions help us
to distinguish students who have trouble with more involved questions because
they have trouble applying what they know from those students who simply lack
the basic understanding of mathematical concepts. Without basic questions, we
cannot distinguish these two kinds of students, and it is extremely important
that we do so. But too much emphasis on basic questions is not good: We should
not be content with just the ability of students to answer basic questions if we
are to help them learn deeply and meaningfully. A bank of sample items does not
really reveal to us what the proportion of basic questions is on the test. That
is, we cannot really judge a test just by knowing that some items look like this
and some look like that.

The document says that some of these items were experimental items that did
not make it to the PISA whereas others are actual items from past PISA
exams. But none of the past tests are reproduced in full; the document is
just a smattering of various kinds of test questions.

PISA should release the actual test given and not just sample questions. Why is
this? Without knowing what the test looks like, we can be fooled into buying
the test creators' claims that the test is effective, that the test does check
critical thinking and not just rote, blah, blah, blah. We cannot know if the
test emphasizes rote and shallow thinking too much and not enough on higher
reasoning. Many math tests I am not impressed by simply because the test
focuses far too much on shallow thinking, not because the test contains
inappropriate questions (though sometimes inappropriate questions will be
a significant issue, but avoiding them is not enough to make an effective
math test). Sometimes the problem is also with it being multiple
choice. In other words, the main problem I usually see with the math tests
I have seen is with what they lack.

Furthermore, how do we know that a test which is supposed to check
critical thinking using items that the students should not have seen before
really accomplishes that goal? Many teachers will use sample items to construct
practice tests. If teachers can construct practice tests that are extremely
similar to the actual test, then the actual test does not accomplish this stated
goal. If teachers use what information PISA releases but does not release the
actual test itself, we cannot verify this claim. Thus, I feel it is not
a good sign when test creators will not release previously given tests
to the public.

At least PISA requires students to explain their reasoning on many of these
questions, which is a vital feature to any effective math test.
Most standardized math tests are entirely or almost entirely
multiple choice, which I abhor for a math test. However, based on my
comments above about not being able to determine what any actual PISA test
looks like, we cannot determine the proportion of multiple-choice items
and free-response items.

I am bothered to see that students can get full credit for reporting
answers without units of measurement. This encourages a bad habit in
students because there is great difference between an abstract number as
an element of an algebraic structure and as a real-world measurement.
We cannot report a measurement as just 5 because abstract numbers do
not tell us their size. For real numbers, we can determine their
size relative to each other, but there is no "real world" size to the
number 5 all by itself. There is also a big difference in the arithmetic
of measurements and the arithmetic of abstract numbers. We can add the
abstract fractions 5/6 and 2/5 as is (of course, we need a common
denominator), but we cannot add 5/6 foot and 2/5 yard by adding the
fractions as is because the measurements are not given in the same units.

Finally, to determine if PISA really does measure critical thinking
in mathematics, we need to know what other countries teach and what
they do, if anything, in preparing students for the PISA and how
they choose their students who take the test. The comments I had
mentioned about "clever" questions above might or might not apply
to students in other countries; all I can go by at this time is
using what is normally taught in America to make those judgment
calls.

In short, PISA may or may not be effective in accomplishing the goals
they intend for the test to have, but there is not enough information
available to make any solid conclusions in their favor.

Jonathan Groves