Many years ago, this problem appears on a “benchmark” test given to third graders in Philadelphia:

A. 3

B. 9

C. 18

D. 24

Almost half the third graders (46%) in a group of schools I was supporting chose B. The correct answer, C, was chosen by 31% of the students. I don’t recall being all that surprised, but as I have thought about it over the years, I’ve come to see it as one more piece of evidence that many students do not do “sense-making” when they’re doing math. They look for numbers and guess what operation they’re supposed to do (addition is very popular). I have a theory that if you had asked those same third graders to draw a picture of the story, many more of them would get the problem correct. So the problem above becomes this task:

To informally test this theory, I invite anyone with access to students in grades 2-5 (or older kids if you think they might be good subjects) to run a small experiment for me. I’ve written two sets of questions, one math and one drawing. Each has three questions. The math questions are taken directly from some Grade 3 benchmark tests, but I can see them being used in grades 2-5 for the purposes of this experiment (though you are welcome to use them with whomever you want – it’s an experiment, after all, not hard scientific research).

Since I am interested in how students solve problems when they think they’re supposed to “do math” versus when they might not realize they’re engaging in mathematical thinking, I’d ask that the math questions be given during math instructional time and the drawing questions be given during literacy instructional time, or, barring that, any time that isn’t math, and that the two sets of questions be given a couple of weeks apart.

The attached PDF contains 26 copies of each set of tasks (labled Student A through Student Z), instructions, and a roster sheet so that you can keep track of which student corresponds to which letter. (If you have more than 26 students, be creative.) You can report results through an online survey or send me copies of the completed task sheets (as a scanned PDF, or toss them in an envelope) or both.

Download the files: Fetter Sense-Making Experiment v1 [PDF]

So, wanna play? I hope so!

For this exact reason we support the use of Whole Number manipulatives. See the video here: http://masteryed.com/manipulative-block-demonstration

The problems are often poorly stated for the grade levels.

ORIGINAL: “The corner deli sells roses in bunches of 6. If Dylan buys 3 bunches of roses, how many roses does he have?”

Assuming the student knows what a ‘bunch’ of roses means, the problem does not clarify if Dylan bought the roses at the corner deli; it does not say how many roses were in the bunches he bought. Based on the stated question, the best answer would be “3 bunches.”

The question to be asked:

CHANGED: “If Dylan buys 3 bunches of 6 roses each, how many roses did he buy?”

This is a math question.

The rest of the questions are essentially the same.

————————————————

ORIGINAL: “Chelsea’s favorite cupcakes come in packages of 4. Chelsea bought 5 packages of cupcakes. Draw a picture of the story.”

There is no information indicating whether Chelsea bought the packages of four; whether they were her favorites; or whether she just bought some cupcakes. A Suitable picture would be Chelsea holding 5 packages of an indeterminate number of cupcakes.

CHANGED: “Chelsea bought 5 packages of 4 cupcakes each. These were her favorites. Draw a picture showing how many cupcakes she bought.”

————————————————

ORIGINAL: Six flower plants come in one box. Lucy has 1 box. She gets 4 more boxes of flower plants. Draw a picture of the story.

Similar to above. The data says 6 plants can be in a box. Lucy has one box with an unknown number of plants. She gets 4 more boxes. The suitable picture would be Lucy with 5 boxes and an indeterminate number of plants.

CHANGED: “There are six flower plants in every box. Lucy already has 1 box of six plants. She gets 4 more boxes of 6 plants each. Draw a picture showing how many plants Lucy ended up with.”

————————————————

How the problem is stated can change the answer and/or direct the student.

Consider:

20 divided by 5 can also be written as 20/5. 20/5 can be written as

1/5 + 1/5 + 1/5 + 1/5 + 1/5

1/5 + 1/5 + 1/5 + 1/5 + 1/5

1/5 + 1/5 + 1/5 + 1/5 + 1/5

1/5 + 1/5 + 1/5 + 1/5 + 1/5

each row totals 1, therefore the answer is 4

It is important to be sure what is being asked of the student…

Hi, Jeff. Thanks for the thoughtful analysis. This is a whole ‘nother can of worms! I decided to use these math problems because I already had some real student data for them, and because for each one, more students chose the same wrong answer than chose the right answer (in my small dataset of about 160 students). I fabricated the “drawing” problems to mimic the structure of the math problems pretty closely.

Are they problems students are currently expected to be able to answer? Yes. Are there obvious issues with them? Sure, some of which you’ve enumerated. Is writing questions easy? Not in the least! Do we wish that big corporations being paid to generate tests did a better job? For sure. Does the structure of those problems contribute to students’ struggles in answering them? Probably. Have people done research around these issues? No doubt. Do the big corporations persist in producing such questions? Yep.

I’m grateful to be able to hang out with people who love to talk about these things! I wanted to try this experiment because I was curious whether my theory was right, and I’m very interested in the intersection (or not, as the case may be in many settings) of math, literacy, and sense-making. I’m excited that it may bring up many other topics of conversation.

Hi Annie,

This sounds like a wonderful experiment. Your controls make sense. However, unless I missed it, you didn’t specify the order in which to give each question and this might affect your results. For example, giving the “drawing” question before the math question might produce a higher success rate for students answering the math question. Which I hope happens. If students are given the math question first, it might produce a lower success rate.

I’ll ask some colleagues if they can do this experiment.

Hey, Andrew. Sounds like I should edit the survey so that folks can say which one they did first. Unless we want to eliminate variables and ask everyone to do the math set first – so that students are more likely to use their “normal” math problem solving methods (which is what I’m interested in documenting).

I think it would be extremely helpful to have a fair comparison of data from both sides. Maybe alternate the order with the teachers that sign up.

Teacher A: math, then picture

Teacher B: picture, then math

Teacher C: math, then picture

and so on…

I think the data will be a lot more rich. I could see the students who did the picture first, will be more successful when they do the math question. Of course, I could be terribly wrong.

Here’s a follow-up to consider. Give the same teachers a second task to do later in the year, but switch the order to see what happens.

Teacher A: picture, then math

Teacher B: math, then picture

etc.

One last clarification: these are self-contained classrooms where the students have the same teacher for math and literacy, right?

Not sure I can force order on people, since they don’t have to “sign up” – they can just grab the file and go (and never report back, but I hope they won’t!). So I think it makes the most sense to add that question to the survey. (Update: I’ve added that to the survey, and also asked people about how much time elapsed between doing the two sets.)

I could definitely send people who DO submit data a new set to try later in the year, and ask them to do it in the opposite order. Let’s say I’ll do that.

I am assuming that this would be done mostly by people in self-contained classrooms. But I also know some coaches or computer teachers (like @MrsPollardprime) might try it with kids they see (or can borrow) regularly.

Makes sense to add that to the survey. I’m fascinated to hear the results.

This seems like a wonderful experiment that would give some real insight into how kids think about math problems. It’s my opinion that forming a picture of the problem (at least a mental picture if not a physically realized one) is exactly what it means to “understand” the problem. If a student doesn’t actually understand the problem then the answer he gives, even if it happens to be “correct”, is *irrelevant*. As one without much access to kids I really hope people with access to kids will actually do the experiment and report their findings. As for problems with the statement of the problem (I agree that the term “bunches” may not be familiar to many kids) I would expect that to be demonstrated by an inability to form the picture. This would be a practical approach to testing the quality of problem statements.

[...] Annie Fetter’s work at the Math Forum has always been impressive and it’s a total oversight I hadn’t realized she writes a blog until now. [...]

I’m a parent, not a teacher, but will test this with my children. I’m interested in discussing with the kids some of the assumptions/semantic traps that Jeff LeMieux mentioned. For us, messy problem statements are ideal because I’m looking to build their broader thinking skills and don’t really care about assessing (or drilling) any particular computational skill.

I hope this experiment will also lead to some interesting posts on http://mathmistakes.org/ (with whom I have no affiliation). One of the benefits I experienced when I was teaching was seeing misunderstanding and mistakes from my students; it really sharpened my own mathematical thinking. I would love to see more additions to that resource.

And, thanks to dy/dan for pointing out your blog (among all the other great things he does . . .)

Did this with child 1 today and it was rather anticlimactic. He got them all right and in the debrief on how he thought through the questions, I realized that these scenarios are his primary model for multiplication right now. In other words, very difficult for him to understand how anyone could approach the question incorrectly.

He didn’t write anything down, but just circled the answers. When asked why, he said it was more challenging to do it all in his head, so that’s what he selected.

So as long as he is doing sense-making and not “key word = specific operation”, then he’s headed in the right direction. Can he explain why he chose to do the operations and steps that he did? (Obviously I don’t know Child 1, but I have met other kids who were “good at math” and solved stuff in their heads, but weren’t accustomed to articulating their decisions and their process much beyond, “I added, because that’s what you do.”)

On the one hand, these aren’t hard problems. On the other hand, if you have no sense-making habits, they’re hard problems, since it’s hard to discern which operation to use just from the key words available (one of many reasons why the whole “key words” thing is so ridiculous).