A Rubric for Coding Problem Difficulty
[Level 1]
[Level 2]
[Level 3]
[Level 4]
[Level 5]
From: Renninger, K. A. & FeldmanRiordan, C. (in preparation). "Technology as a tool for developing students' mathematical thinking." (The help of Crystal Akers and Alice Henriques in clarifying this coding scheme is gratefully acknowledged.)
Problem difficulty is coded similarly for problems included in each of The Math Forum's Problem of the Week services, although adjustments have been made with respect to the grade band of the problem.
Based on the revised NCTM Standards for problem solving, elementarylevel problems are understood to include work typically undertaken by students in grades 35 (or 6), and the challenge of these problems involves recognizing and applying appropriate strategies involving basic calculations.
Middleschoollevel problems are understood to include work undertaken by students in grades (5)68; the challenge of these problems involves the number of steps included, which may cause the student conceptual difficulties.
Finally, problems at the high school level (algebra, geometry, discrete mathematics, trigonometry/calculus) are understood to include work undertaken by students in grades 912, and the challenge here is in the level of abstraction and the required synthesis of prior knowledge.
Problem difficulty is defined relative to other problems at the particular level being studied (elementary, middle school, geometry, etc.). Identification of the difficulty level of each problem was undertaken with respect to both the concept (the "what" of a problem) and the mathematics (the computation required, or the "how" of a problem). Bonus portions of problems were not included in the difficulty ratings.
Levels of difficulty
Coding of problem difficulty focuses on the mathematical challenges represented by the problem, the difficulty of the mathematical concept, and the difficulty of mathematical calculations for students at a given level of problem solving. The rating scale consists of 5 levels of difficulty, wherein a Level 5 problem is a very difficult problem for students in a given grade band.
Level 1
Only one concept needs to be worked on; the mathematics is rudimentary and represents prior knowledge rather than something new.
Examples:
 Elementary, difficulty level 1: Oh, Brother!
Aidan is how many days younger than T.J.?
The concept is clearly presented in this problem. Solving the problem only involves prior knowledge about the number of days in a year.
 Middle School, difficulty level 1: BacktoSchool Shopping
Mrs. Bender noticed that the total of the items her children purchased was $155. Can you determine the cost of one pair of jeans?
This problem requires that students subtract the cost of several items from the total to find the cost of two pairs of Devin's jeans, then divide the result by two to get the cost of one pair of jeans. Subtraction and division should be prior knowledge for students in the middle school level grade band. The most difficult part of this problem is sorting through the story to find the pertinent information.
 Geometry, difficulty level 1: Divided Rectangle
A rectangle is divided into four rectangles with areas 45, 25, 15, and x. Find x.
This problem asks the student to find the factors of three values and to identify common factors. For geometry students, finding common factors should be prior knowledge.
 Algebra, difficulty level 1: Halloween Night
One hundred candy kisses were being distributed one at a time to five goblins at my door. The phone rang, and when I came back I didn't remember where I had left off. How did I continue distributing without starting over?
The key to this problem is realizing that 100 is evenly divisible by 5. There are no new concepts introduced here. The answer can be found by the guess and check method as long as the process is explained clearly.
 Discrete Mathematics, difficulty level 1: Mother's Chores
A Traveling Salesperson Problem. Tom's mother needs to make several stops around the city and then get home. What is the shortest route without going back to one of the stops?
In this problem, there is only one concept that needs to be worked on, finding the shortest round trip for the mother to complete her errands. The problem can be solved using prior mathematical knowledge.
 Trig/Calculus, difficulty level 1: Environmentally Friendly
What dimensions for a given page will use the least amount of paper?
Students in this grade band should have the mathematics ability to solve this problem. The concept is clearly stated within the problem and no twist is necessary to solve it.
back to top
Level 2
Either a) the concept is clearly stated within the problem, and the mathematics is challenging for students at the given level of the PoW, or b) the concept requires some "stretching" for students at this level, and the mathematics is based on prior knowledge.
(Note: Problems that require attention to explanation are likely to be found at Level 3, rather than Level 2, because of the difficulty involved in explaining mathematical understanding.)
Examples:
 Elementary, difficulty level 2: Charlene Goes Shopping
Charlene needs to add some items to her winter collection. Help her determine where she should shop.
The concept in this problem is easily presented, but the math is challenging because the students have to calculate the percentage reduction before shipping and handling from one of the store orders. There is no "twist" to the problem.
 Middle School, difficulty level 2: How old is Mrs. Nelson?
Ms. Nelson claims that she is three times as old as her little sister. However, six years ago she was five times as old as her sister was. How old is Ms. Nelson?
This problem could be solved by the guessandcheck method, which should be prior knowledge for the student and would ordinarily cause this to be scored as a level 1 problem. The requirement to explain the process and reasoning behind the chosen method, however, stretches the student's mathematical skills, boosting the problem to Level 2.
 Geometry, difficulty level 2: Find the Volume of a Box
Given the surface area of three sides, find the dimensions and volume of a rectangular box.
The math concepts in this problem should be easy, since they involve finding common factors from the surface area, and then using these to find the volume of a box. The problem is difficult, however, because the students are asked to identify and explain their solution path.
 Algebra, difficulty level 2: The Great One
Indiana Jones found a math problem about the life span of "The Great One" engraved on an old tombstone. How old was the Great One when he died?
The concept is clearly stated in this problem, but the mathematics requires slightly more work than that expected for a Level 1 problem. The problem does not contain a twist.
 Discrete Mathematics, difficulty level 2: The Locker Problem
This classic number theory problem investigates properties of prime numbers, perfect squares, and counting in a problem that involves opening and closing locker doors.
This problem could be solved by creating a model and simulating the opening and closing of the lockers. The mathematics involved, factors and multiples, is prior knowledge for students in this grade band. The requirement to "discover" a pattern and to determine which lockers are touched exactly twice, however, makes this a level 2 rather than a level 1 problem.
 Trig/Calculus, difficulty level 2: Building Boxes
Find the maximum volume of a rectangular prism constructed from an 8.5 in. by 11 in. piece of paper.
In this problem, the concept of finding the maximum volume of the box is clearly stated. The mathematics in the problem is challenging for students, however, because they need to create a function for the volume of a threedimensional object that is being constructed from a twodimensional object.
back to top
Level 3
The problem (a) contains a "twist" or additional problem requirement that students in this grade band may overlook even though they can complete the problem accurately, and (b) requires discourse knowledge of mathematical concepts and basic mathematical ability appropriate to students at this level.
(Note: At the elementary level, multiple parts within a problem make what may initially appear to be a Level 3 problem into a Level 4 problem.)
Examples:
 Elementary, difficulty level 3: Chocolate, Orange, Megamint Swirl
How many half gallon cartons of ice cream can fit in the freezer?
This problem is appropriate for students in this grade band but it contains a twist. Because you have to fit whole boxes into the freezer, students cannot just divide the volume of the individual boxes into the volume of the freezer and answer with a decimal.
 Middle School, difficulty level 3: Wraparound Universe
Explore the game of tictactoe using a cylindrical tictactoe board.
The level of discourse and basic mathematical ability required by this problem are appropriate for students in this grade band. The main difficulty the problem represents involves clearly explaining why one player can always win.
 Geometry, difficulty level 3: Congruent Chords
Two congruent circles are drawn, and four congruent chords are drawn, two in each circle, all perpendicular to the diameter through both circles. The distance between the two furthest chords is 20, and the distance between two chords of the same circle is 8. What's the area of one of the circles?
In this problem, students may overlook the need to explain why the congruent chords are equidistant from the centers of their respective circles. In addition, the student needs to figure out how to get the information needed to provide the area of a circle, since this information is not provided.
 Algebra, difficulty level 3: ProductPlus1
The product of any four consecutive integers, increased by one, is always a square number. Give at least three instances of that statement and prove that this will always occur by finding an algebraic expression for that "square number."
The first part of the problem is fairly simple, but extrapolating from the patterns to write a generalized equation is a challenge for students in this grade band. Some students may overlook the need to actually prove that this is the correct expression.
 Discrete Mathematics, difficulty level 3: The Fourth of July Parade
After the parade, the people on the float I was on shook hands with each other. The Mayor came over and shook hands with only the people he knew. How many people did he know if there were 1625 handshakes altogether?
Although this problem may at first appear to be the traditional handshake problem, it has a slight "twist": the mayor only shakes hands with the people that he knows. Providing an additional challenge, solving the problem requires that students explain mathematically how to account for this "twist."
 Trig/Calculus, difficulty level 3: A Parabola Proof
Prove that the area of a parabola is 2/3 the product of its width and height.
Students may be challenged by this problem. Although it does not contain a true "twist," it does have an additional problem requirement that students are likely to overlook: that the formula describes the area of any given parabola.
back to top
Level 4
The problem includes the elements listed in difficulty Level 3 and contains an algorithm new to students in this grade band; students may miss the problem by getting bogged down in the math but not by missing the concept; students may not finish the problem or may not attempt all parts of the problem.
Examples:
 Elementary, difficulty level 4: Garden Fresh Fruit
Help the poor farmer predict his revenue (if he can sell all his fruit).
The expected discourse knowledge of mathematical concepts and the basic mathematics ability required to solve this problem is appropriate to students in this grade band, but they may get bogged down in the many calculations it requires. The twist in this problem is that students have to deal with many different fractions of a row.
 Middle School, difficulty level 4: Office Tile Choice
Find the total cost of purchasing tiles for my office.
The concept behind this problem is easily within the range of ability for students in this grade band, but the problem contains smaller components that students can overlook or answer incorrectly; for example reporting the cost of individual tiles rather than the cost of a complete box of tiles.
 Geometry, difficulty level 4: Splitting up a Triangle
Take any triangle ABC. Construct D and E as the midpoints of BC and AB, respectively. Now construct DF and EF, where F is any point on AC. How are the areas of the triangles related to the area of the quadrilateral?
It is unlikely that students in this grade band will expect that the two areas may be the same. They also need to provide a proof and a clear explanation, adding to the difficulty of the problem. Here students could get bogged down in the math, or not know how to reach the correct conclusion.
 Algebra, difficulty level 4: Motorcycle Daredevil
A motorcycle stunt person needs some math equation work to calculate her most daring feat: jumping between ramps at high speeds.
This problem requires students to apply what they know about parabolas to a physicsoriented math problem. There is a lot of math here in which students may get bogged down, and there are four parts to the problem.
 Discrete Mathematics, difficulty level 4: First Presidential Veto
This problem involves the Jefferson, Adams, and Webster apportionment methods.
This problem is complicated by the fact that students must consider three different apportionment methods. Students could become bogged down in the calculations in any of the methods. An added requirement is that students recognize that the U.S. Constitution guarantees that every state is entitled to at least one representative. Missing this element may lead students astray.
 Trig/Calculus, difficulty level 4: The Ninja Mission
Find the measurement of a spike's angle, given the star's perimeter.
In this problem, the student must be careful to include the three points rather than just the perimeter of a disk when calculating the perimeter of the shuriken. It is a problem in which the student may get bogged down with the math, or not know what to do with all the information given (i.e., the density of the titanium).
back to top
Level 5
The problem includes the elements listed in difficulty Level 3 and requires discourse knowledge of mathematical concepts and mathematical ability above the level of the students in this grade band; or it contains a concept, theorem, or algorithm that a rater familiar with this mathematics topic does not recognize.
Examples:
 Middle School, difficulty level 5: Exploring Area
This twoweek learning experience challenges students to find information about Pick's Theorem and to solve an area problem using this method.
This problem requires that students learn about a new concept, Pick's Theorem, and understand it well enough to apply it to the problem. The problem also includes many smaller questions that students might overlook.
 Discrete Mathematics, difficulty level 5: The Dating Game
This problem is based on the Stable Marriage Algorithm, which requires students to make the best match possible between a set of girls and a set of boys desiring to date each other.
This problem requires that students learn about a new concept, the Stable Marriage Algorithm, and understand it well enough to apply it to the problem. The problem also includes many smaller questions that students might overlook.
 Trig/Calculus, difficulty level 5: Needles and Lines
Use trigonometry and calculus to find the probability of a needle touching a line.
In this problem, the student is challenged to use prior knowledge of probability, trigonometry, and calculus together in new ways, pushing the mathematical ability required for the problem above the level of students in this grade band. As a result, students may have difficulty figuring out how to set up the problem.
back to top
