Challenges in the Use of Technology Active Assessment Some Lessons Learned in AP Calculus

Thomas Dick
Department of Mathematics
Oregon State University
Corvallis, OR 97331-4605

What role should technology play in the assessment of a student's mathematical learning or achievement? There are (at least) two very different ways to consider this question: 1) in terms of the delivery of the assessment itself, and 2) the availability of technology as a tool during assessment.

By technological delivery, I mean the physical means by which assessment tasks are presented to the student. As such, a sharp distinction should be made between the messenger and the message. For example, very standard "basic skills" questions could be presented to a student over computers linked to the internet (i.e., low-tech tasks delivered through a high-tech medium). There are a number of important and exciting issues related to technological delivery and formative assessment, including immediacy of feedback and the intelligent diagnostic linking to remediation and/or extension tasks. However, the focus of this brief paper is on the second aspect and the notion of technology active assessment tasks that require the student to make use of sophisticated calculators or software as a mathematical tool to successfully complete. In particular, my comments will address issues related to the use of graphing calculators and computer algebra systems at the secondary and collegiate levels, especially in calculus instruction, but I hope they might have some value that goes beyond this arena.

What's in the box? QCAS as representational toolkit

The term Computer Algebra System (CAS) tends to draw attention to the symbolic algebra capabilities. However, most CAS's incorporate a wide variety of tools, including numerical routines (such as curve-fitting and robust solution approximations for algebraic and differential equations), spreadsheet capabilities for dealing with data, powerful graphics with tracing and dynamic zooming, as well as symbolic algebra and other features. Thus, a CAS would be more accurately called a Computer Math System. Also, while some folks tend to think of CAS's and graphing calculators as being miles apart in terms of power, the distance between the two is rapidly being blurred beyond distinction. The latest generation of graphing calculators have all of the features I just mentioned above, including significant symbolic algebra capabilities.

Both the NCTM Standards at the secondary level and the calculus reform movement at the collegiate level have made strong cases for a multiple representation approach to the central concept of function. I think in this vein it is useful to think of the CAS as a "toolkit" for moving (transforming) both within and among function representations, including symbolic, tabular, and graphic representations. The CAS provides the technological key for exploiting the "Rule of Three."

The figure below illustrates how we can think of some of the most common features of graphing calculators and CAS's in these representational terms.

(NOTE: The term "Rule of Four" is now commonly used to embrace the importance of verbal representations of functions. I realize that the word "representation" carries very different meanings in different contexts in mathematics education, especially those having to do with coginitive structures. In this paper, I am employing the term as it has been used as an organizational philosophy for curricular and assessment development.)

The Advanced Placement (AP) Calculus program sits at the crossroads of secondary and collegiate mathematics and has made some significant changes in response to developments in calculus reform over the last decade. In considering issues of technology active assessment, there may be some valuable lessons to be learned from the experience of the AP program in its efforts to implement changes in their testing.

LESSON 1: Providing a tool that students have had little experience with during their mathematics education may confound assessment of their mathematics performance.

The AP program first allowed scientific calculators on calculus examinations in 1983 (prior to that year, no calculators were allowed). The policy was discontinued after only two years because of very real concerns that many students were performing less well on the examinations, not because of poorer understanding of the calculus concepts and procedures, but because of poor "calculator management" skills (for example, wasting time attempting to use the calculator on tasks where its use would be inefficient or inapplicable). When scientific calculators were again allowed starting with the 1993 examinations, these same problems did not resurface. It seems reasonable to suggest that the increasingly regular use of calculators in mathematics classrooms in the interim had helped students to be better managers of the machine.

In 1995, the AP program first required the use of graphing calculators on the examinations. A drop in scores was noted for that year, but I would be loathe to suggest that this was an entirely similar repeat of the phenomenon experienced in 1983. (There were other competing explanatory factors at least as plausible, including a significant increase in the number of students taking the examination.)

LESSON 2: To encourage the use of technology in the classroom, it should play a significant role in assessment.

Actually, this is a lesson in human nature. Philosophically, the discussion of pedagogical issues in using technology is quite independent of the discussion of assessment issues. One can make a strong case for the new learning opportunities made possible by a CAS in the classroom, especially in building a more robust conceptual understanding of function. If we wish to assess whether students' understanding is indeed more robust, the CAS need not play a role. For example, a calculus task which provides the graph of a derivative f' and asks the student to make interpretations regarding the behavior of f would not involve the use of a CAS at all. However, there are classroom activities involving a CAS that could help a student build the understanding necessary for success.

In fact, the AP program actively encouraged the use of graphing calculators in the classroom long before the machines were allowed on the examinations. The value these tools could have in building conceptual understanding was recognized, but the challenges of equity (both in economic access and in the "playing field" given the wide spread of capabilities in the machines) were worrisome. Alas, many teachers were reticent to employ graphing calculators in their teaching because they were not permitted on the examination. Thus, to effectively encourage teachers to take advantage of the opportunities afforded by graphing calculators, the AP program began requiring them on the examinations in 1995.

LESSON 3: Technology-active assessment tasks are difficult to create!

It is true that the AP program has additional concerns in formulating assessment tasks that are not faced by a classroom teacher. The concerns over equity resulted in the program setting a "floor" of capabilities necessary for machines allowed on the exams: function graphing in an arbitrary window, numerical solutions to equations, numerical approximations of values of derivatives and definite integrals were required. Effectively there has been no ceiling of capabilities, though the non-QWERTY keyboard requirement has been interpreted by many as an attempt to thwart symbolic algebra capabilities (actually, the non-QWERTY keyboard requirement has much more to do with test security). Hence, assessment items must be carefully designed so that a student having a more powerful calculator does not have some distinct advantage over the student who has a less powerful calculator.

Currently, the examinations include a multiple-choice section split into two parts: one part where no calculators are permitted and one part where graphing calculators are permitted. The examinations also include a free-response section where students must provide written work and explanations that are graded by readers. Graphing calculators are permitted for the free-response section. In those parts of the examination where graphing calculators are permitted, not all of the tasks require the use of a graphing calculator. In fact, roughly a third of the items are technology active (in the sense that they require the use of a graphing calculator). Another third might be characterized as technology inactive (in the sense that there is no role for the machine) and the remaining items might be called technology neutral (the use of technology might be helpful but is not required for successful completion of the task). Because of the spread of capabilities in graphing calculators, especially regarding symbolic manipulation, what one would consider as standard differentiation and integration items appear in the non-calculator part of the exam. (Of course, if every student had access to such machine capabilities, there would still be little assessment value in having such items on the open-calculator part of the exam.

The challenge of creating "authentic" technology tasks for the purposes of the AP examination has been great. By an authentic task, I mean one which requires the student to make intelligent use of the technology rather than artificially coercing the student to do so with gratuitously messy numbers or functions.

Example: Find the equation of the tangent line to

y = 3.947e^.27x at x = 1.392.

Yuk! This is a standard procedural question dressed up in ugly clothes. The differentiation required is easy, but one would reach for a calculator just to carry out "non-calculus" computations.

There seem to be two natural kinds of authentic technology active items that have worked well on the AP examinations. I would distinguish between them as to whether technology plays a role on the "back-end" or "front-end" of the task.

"Back-end" Tasks

This is a task that requires a student to use a calculus-based analysis to model a solution, but the technology is required in the final stage to carry out calculations that would defy paper-and-pencil techniques.

For example, an application leading to a definite integral might require the numerical integration capabilities of the machine. It is true that one could make this into a technology-inactive task by only requiring the expression instead of the calculation of the necessary definite integral, but it is much more natural to ask follow-up questions regarding an interpretation of the result if the calculation is actually carried out.

"Front-end" Tasks

This is the kind of task where the technology must be used to open the door to using calculus-based analysis.

For example, given a symbolic expression for f', a student might be asked to analyze the behavior of f (extrema, inflection points, intervals of concavity, etc.). One does not need to go far out of one's way to find examples where no closed form antiderivative expression can be found (with or without a CAS!). Yet, the technology could be used here to switch representations - by graphing the derivative function the student can use a conceptual understanding of the connections between the behaviors of the two functions to solve the problem.

Again, one could turn this into a technology inactive task by simply providing the graph of f' (a very nice assessment task, in my opinion), but I believe that the task described above assesses an aspect of student understanding beyond the graphically presented stimulus.

In short, I think the AP program is making progress in the area of technology active assessment and I have learned much from its experience.

Concluding Remarks

The question of allowing calculators or computers in on-demand assessments of student achievement is a "hot button" issue that advocates of technology must be braced to deal with head-on. Unfortunately, the arguments used by opponents of technology is a straw man.

Twenty years ago, Zalman Usiskin (Mathematics Teacher, May 1978) used the term "crutch premise" to characterize the notion that allowing students to use a calculator for arithmetic would render them unable to do arithmetic when the calculator is absent. In the years that followed, despite a wide range of studies that had refuted the crutch premise (including a large meta-analysis published in JRME in 1986 by Hembree & Dessart) the controversy shows no sign of dying down.

Indeed, with the subsequent emergence of affordable and powerful graphic calculators and, more recently, of hand-held computer algebra systems, the controversy is seemingly revisited at every level of the K-14 curriculum. It is hard not to get a feeling of deja vu when hearing a debate about factoring skills in the first-year algebra curriculum or about differentiation skills in the calculus curriculum when a computer algebra system is available. It is as if one could simply recycle the declarations, filling in the blanks with the appropriate grade-level skill.

A political lesson to be learned from the continuing controversy over use of such basic technology in assessment is that we should not expect to satisfy some critics of the use of CAS in the classroom no matter how compelling the research evidence compiled might be. Frustration with those circumstances might lead some to question the utility of performing such research at all, but I believe that would not be a wise stance to take. Especially with regards to the issues of exploiting symbolic manipulation capabilities of the CAS, we should proceed with some caution. I am not convinced that the parallels between "symbol sense" and "number sense" are very easy to draw. Indeed, mathematics education research could help us in formulating a good definition of "symbol sense."

Charles Patton, a colleague of mine in calculus curriculum development, is fond of pointing out a more accurate perspective on the role of technology in calculus: The question is not so much how curriculum should change in light of the presence of technology, but how the lack of technology in the past has warped the mathematics curriculum. There is more to the discussion than just rehashing the crutch debate with Maple in place of a four-function calculator. The new tools also make new mathematics accessible to students and the discussion needs to transcend a preoccupation with performing old tasks with new tools.