When Does The Use Of Computer Games And Other Interactive Multimedia Software Help Students Learn Mathematics?

(DRAFT June 14, 98)

Maria M. Klawe, vpsas@unixg.ubc.ca

Department of Computer Science

The University of British Columbia

Vancouver, Canada, V6T 1Z4

Abstract

This paper presents an overview of research findings from the E-GEMS project on how various factors influence the effectiveness of educational computer games and other multimedia software activities in enhancing mathematics learning. The insights described here result from quantitative and qualitative studies examining issues related to software and hardware design, methods of use in classrooms and computer labs, and gender differences. Our findings suggest that computer games can be highly effective in increasing children’s learning and enjoyment of mathematics. The extent of the effectiveness, however, depends on many things including details of the software design such as interface styles and scaffolding, teacher and student expectations, the level of integration with other learning activities, and the setting and pattern of use. In addition, our studies have frequently revealed gender differences with respect to children’s attitudes towards and interactions with computer games.

1. The E-GEMS Project

E-GEMS, the Electronic Games for Education in Math and Science project, is a collaborative project centered at the University of British Columbia (UBC). E-GEMS involves researchers in computer science and mathematics education as well as teachers, children and professional game developers. In addition to UBC, the E-GEMS participants include Queen’s University, Electronic Arts, and several schools in British Columbia and Ontario. E-GEMS was created in late 1992 to explore the potential of specially designed electronic games to increase learning and appreciation of mathematics and science by children in grades 4-8. This age range was chosen because studies in British Columbia [BH89] indicated that this is when most children lose interest in these subjects. We were interested in electronic (i.e. computer and video) games because of their appeal to children and because they offer excellent opportunities for visualization and exploration of complex concepts. On the other hand, we had serious concerns about using such games. Most girls, especially aged 10 and older, seemed to be less interested in playing electronic games than boys, and less interested in using computers in general. Moreover, there was little concrete evidence that traditional electronic games were effective in enhancing mathematics and science learning for purposes beyond memorization drills. E-GEMS research has focused primarily on mathematics education although some of the software we have worked with also addresses other subject areas.

The E-GEMS use of the term "educational computer game" encompasses a wide range of computer-based activities. Specifically, we use it to refer to educational software incorporating some popular electronic game elements such as a high degree of interactivity, exploration, puzzles, challenges, scoring, graphics, music, sound effects, or narrative. This use is consistent with children’s use of the term "computer game" which they use for any enjoyable computer-based activity. Such activities typically include drawing and painting programs, multimedia word processors and spread sheets, encyclopedias, simulations, and puzzles as well as the usual kinds of computer games.

Over the last five years E-GEMS research has ranged from basic studies on how children interact with commercially available computer and video games, to developing fully-featured educational computer game prototypes, to conducting focused quantitative and qualitative studies to evaluate the effectiveness of various design and use options. Throughout we have paid particular attention to trying to understand how to ensure this type of software is effective for girls as well as boys. E-GEMS research draws upon results and methodology from many areas. These include general theories of learning and motivation, mathematics education, computer-aided learning, as well as several areas in human-computer interaction such as user-interface design and gender issues. Because of the number and breadth of these areas we refrain from providing a comprehensive review of the relevant work here, but excellent surveys can be found in [S98, PE96, In97, M96].

E-GEMS research began with a seminal study in 1993 on children’s interactions with computer and video games. Up to that point there had been little concrete research on this topic though several authors had suggested the potential of video games as an educational tool [e.g. M80, B82, DB86, S86]. Our study, conducted over a two month period in a science museum, observed over ten thousand children as they interacted with video and computer games in an exhibit created and staffed by E-GEMS researchers. One frequent observation was children’s lack of awareness of any connection between the skills and knowledge learned during game-play and the skills and knowledge associated with school work. Two other central themes that emerged from these observations were the popularity of collaborative play for both girls and boys, and gender differences in game preferences [IUK+94, LUK+95]. In particular, the game elements most commonly identified as important by girls were story line, characters, worthwhile goals, social interactions, creative activities, and challenge. Most boys, on the other hand, valued entertainment, fast action, adventure, challenge, and violence.

Based on the results of the 1993 study, E-GEMS research has focused on attempting to answer three questions:

- How should mathematical computer games be designed and used so that students engage in conscious reflective exploration of mathematical concepts?
- How should mathematical computer games be designed and used so as increase achievement, confidence, and enjoyment in mathematics for girls as well as boys?
- How might collaborative play of educational computer games enhance learning and motivation?

This paper focuses on what we have learned in relation to the first two questions, but information on our work related to the third question can be found in [I97, IBKU95, IGBK95, IMBK97, In97, GK97, GR98] as well as in the other E-GEMS publications available from our website www.cs.ubc.ca/nest/egems/home.html. Our approach to answering these questions is based on three components. The first is the key role played by teachers and students in several classrooms who participate in the design of E-GEMS computer games as well as the design of strategies for incorporating educational computer games into mathematics education. These classrooms also serve as ongoing long term research sites to test the effectiveness of games and strategies. The second is an iterative process used to develop mathematical computer games and strategies, alternating among design, implementation and classroom evaluation. The third is the use of more focused quantitative and qualitative studies to explore specific design and use issues.

Most of the points discussed in this paper are based on our research experiences with two E-GEMS games, Super Tangrams and Phoenix Quest. Thus we begin with a summary of our work with each of these games (sections 2 and 3, respectively). We then cover more general topics: ways in which software can help in learning mathematics (section 4), game design (section 5), game use (section 6) and gender issues (section 7). We close with a brief description of our current research.

2. Super Tangrams

Super Tangrams was created by E-GEMS Ph.D. student Kamran Sedighian as part of his thesis research [S98] on the effect of various design strategies on children’s learning of and attitudes towards complex mathematical concepts. Sedighian’s design process started with several months of classroom observations on students’ use of existing software related to two-dimensional geometry. These software activities attempted to help children learn about transformations such as rotation, reflection and translation, by requiring the student to use the transformations to move geometric shapes into specific configurations. Based on this foundational work, Sedighian formed a number of hypotheses. The first was that conventional "direct object manipulation" (DOM) interfaces in which students manipulate the shape being transformed would be less effective than a "direct concept manipulation" (DCM) interface in which students manipulate a mathematical representation of the transformation being applied to the shape. The second was that an appropriate use of scaffolding, particularly the gradual removal of visual feedback aids and requiring the use of specific transformations to achieve some configurations, would stimulate students’ reflective thinking about the transformations. He named the combination of this scaffolding approach with the DCM interface "Reflective Direct Concept Manipulation", RDCM. Finally, Sedighian conjectured that by providing "on-demand" access to an instructional component that adapted to the level of challenge facing the student, it would be possible to keep the level of challenge carefully balanced with the knowledge and skill being acquired by the student, and that this would result in an activity that was intrinsically highly motivating. Sedighian referred to this strategy as "Challenge-driven learning" [S97].

In order to test the merits of these hypotheses, Sedighian used an iterative process alternating between classroom testing and development to create six versions of a fully-fledged educational computer game. For each of the three interface designs, RDCM, DCM and DOM, there were two versions: one with music, colorful graphics and cartoons, and one without any of these embellishments. In addition the RDCM and DCM versions had adaptive and non-adaptive instructional modules respectively. In 1996, Sedighian conducted a study with 116 grade 6 students at a different school to evaluate the effectiveness of the six versions in terms of their impact on students’ understanding of two dimensional transformational geometry, and attitude towards the subject area and various aspects of the game design and experience. The students were assigned to seven groups (two for the embellished RDCM version, and one for other each version) and each group played their version of Super Tangrams for forty-minutes per day over a period of ten consecutive school days. In each group students had been assigned to pairs by their teachers, and each pair shared a single computer throughout the ten sessions. In order to explore the effect of mediation, Sedighian mediated the playing sessions of one of the embellished RDCM groups. There was no adult mediation during the playing sessions of the other groups though Sedighian was present to help with technical problems, and there was often a significant level of interaction and dialogue between the different pairs of students.

All students completed a written test on transformations, containing fifty-two questions on translation, rotation and reflection, before the first playing session. After the final playing session the students wrote the same test, and completed a questionnaire on attitudes towards mathematics, computer games and their experience playing Super Tangrams. Video taped interviews were then conducted with approximately 20% of the students in each group to explore their questionnaire answers more fully. In addition, a control group of twenty grade 6 students at another school wrote the transformation test two times to assess the learning effect of writing the test twice.

The results of this study strongly confirmed Sedighian’s hypotheses. In particular, the performance gains of students using RDCM challenge-driven learning versions were very significantly better than those of students using DCM versions, which were again significantly better than those of students using versions with conventional direct manipulation (DOM) interfaces [S96, SK96, SW97, S98]. There was no effect found for the presence of embellishments (music, graphics, cartoons) nor for the mediation. In addition, there was no learning effect found in the control group’s test results. Table 1 below shows the mean pre-test and post-test scores for each group. The notations "+E" and "-E" denote the presence or absence of embellishments, and "+M" denotes the presence of mediation. We attribute the higher performance of the control group on the pre-test to the fact that students in the control were recent immigrants from countries whose mathematics curriculum covers transformational geometry in earlier grades.

Version of Super Tangrams |
pre-test mean |
post-test mean |
change |

RDCM+E+M |
22% |
76% |
54% |

RDCM+E |
25% |
76% |
51% |

RDCM-E |
37% |
74% |
37% |

DCM+E |
22% |
38% |
16% |

DCM-E |
25% |
45% |
20% |

DOM+E |
23% |
19% |
-4% |

DOM-E |
30% |
22% |
-8% |

Control |
45% |
44% |
-1% |

Table 1. Mean pre- and post-test scores of students playing different versions of Super Tangrams for ten 40-minute sessions on consecutive school days.

The qualitative data from questionnaires and interviews strongly supported the quantitative data on performance gains. Although each group said they found the playing sessions very enjoyable, the students using the RDCM versions indicated that they had had to think very hard and felt they had learned a great deal. The students using DCM versions expressed this to a lesser degree, and the others to a much lesser degree. A much more detailed description of the design of Super Tangrams and analysis of the results of Sedighian’s study is given in his thesis [S98].

Given the dramatic nature of the results in Sedighian’s study, we were anxious to determine the extent to which the effectiveness of the RDCM+E version of Super Tangrams might have been due to some of the other characteristics of his study. We were especially interested in the possible influence of four factors:

- the type of school (upper middle class public school with few immigrant students),
- the presence of Sedighian (an interested and enthusiastic observer) during all the playing sessions,
- the tightly packed arrangement of the computers in a small space (8 computers in approximately 120 sq. ft.), and
- the high frequency of the playing sessions (one per school day).

As a result we decided to conduct further studies involving students playing the RDCM+E version of Super Tangrams in more typical school settings.

The second Super Tangrams study was conducted in the late fall of 1996 with two grade 6/7 classes in a school located in one of the lowest income neighborhoods in Vancouver. Both classes were entirely comprised of ESL (English as a Second Language) students. The playing sessions took place in the usual bi-weekly computer lab period, with students again playing in pairs sharing a single computer. An E-GEMS researcher with no previous involvement with Super Tangrams was present during all the playing sessions, but did not interact with the students except to resolve minor technical problems with the computers. Before the study started both classroom teachers were very enthusiastic about Super Tangrams, and were given a half-day introduction to the software. However, although the teachers were normally responsible for supervising their class during the computer lab periods, the presence of the E-GEMS researcher during the Super Tangrams sessions seemed to reduce the need (in their eyes) for their being present in the lab at those times. This, combined with the multitude of extra teacher responsibilities occurring in the pre-Christmas season, resulted in neither teacher being present in the lab for the majority of the time that their students played Super Tangrams. Moreover, despite the teachers’ original intentions of doing some classroom activities to integrate Super Tangrams with other modes of learning geometry, this did not happen.

Class |
pre-test mean % |
post-test mean % |
change |

Class A (n=16) |
38% |
50% |
12% |

Class B (n=16) |
32% |
51% |
19% |

Combined (n=32) |
35% |
50% |
15% |

Table 2. Playing Super Tangrams (RDCM+E) for ten 40-minute sessions twice per week in a computer lab with little teacher involvement and no integration with classroom activities.

The separate and combined pre- and post-test results for the two classes are shown in Table 2 above. In order to better compare the learning outcomes with Sedighian’s study, only the grade 6 students in each class wrote the tests. In addition, the results of three students in class A were not included because, for some unknown reason, they erased most of the answers on their post-test before handing it in. While this study also showed performance gains by the students, the gains are lower than those exhibited by the RCDM+E group in the Sedighian study. The researcher present during the sessions had observed that the students had little awareness that the purpose of playing Super Tangrams was to learn mathematics. We felt that the lack of teacher involvement and student awareness, as well as the lack of connection with other geometry activities, were likely to be contributing factors to these outcomes.

As a result we conducted a ten week study with a grade 6 class at another school in the spring of 1997. The majority of students in this school are the ch ildren of students attending the University of British Columbia, including a large number of international graduate students. Most of the family incomes are very low and many families do not speak English as a first language. As before students were assigned to a pairs. Each pair was allotted one forty-minute playing session per week on one of four computers placed in their classroom. Finally, the students were not allowed to play Super Tangrams at any time other than their allotted weekly sessions.

During the ten week study, the teacher taught one math class per week on geometry using her usual materials. Some of these sessions covered transformational geometry but used different representations and examples from those in Super Tangrams. As in the previous study, the teacher had received a half-day introduction to Super Tangrams but did not have time to prepare materials based on Super Tangrams. She did, however, regularly tell her students that the purpose of playing Super Tangrams was to learn geometry, and sometimes asked students to identify similarities and differences between the game and what was being covered in class. An E-GEMS researcher was present to observe each of these math classes, and the playing sessions for eight of the students. The mean test scores for this class are shown in Table 3. The level of improvement in test scores falls midway between the first two studies. We conjecture that the primary reason the post-test scores remain lower than in the Sedighian study is the low frequency of the playing sessions (once per week). However, the other differences in location and adult involvement as well as the large fraction of ESL students may also have been contributing factors. Table 4 summarizes the results for playing RDCM+E over the three studies, looking at the following factors: location, frequency, and adult influence.

Class |
pre-test mean % |
post-test mean % |
change |

Class C (n=29) |
36% |
63% |
27% |

Table 3. Playing Super Tangrams (RDCM+E) for ten 40-minute sessions once per week in a classroom with strong teacher involvement and some integration with classroom geometry activities.

Group |
n |
Location |
Frequency |
Adult influence |
Pre-test |
Post-test |
Change |

Sedighian +M |
29 |
Small room |
Once/day |
Active researcher mediation |
22% |
76% |
54% |

Sedighian |
15 |
Small room |
Once/day |
Active researcher present |
25% |
76% |
51% |

Class A+B |
32 |
Computer lab |
Twice/week |
Passive researcher present |
35% |
50% |
15% |

Class C |
29 |
Classroom |
Once/week |
Active teacher present |
36% |
63% |
27% |

Table 4. Playing Super Tangrams (RDCM+E) for ten 40-minute sessions in different situations.

3. Phoenix Quest

Our goal for Phoenix Quest was to combine our findings on girls’ preferences in computer games [IUK+94], with our educational design knowledge from research on Super Tangrams and an earlier game, Counting on Frank [KSW96], to create a mathematical computer game that strongly appealed to girls aged 10-14. After several years of development and evaluation, we feel we have succeeded. Most girls love Phoenix Quest (and most boys like it). Moreover, when used in classrooms with a moderate level of support from the teacher it is effective in increasing interest, enjoyment and achievement in mathematics.

Phoenix Quest emphasizes three primary game elements: a story, interactive communication between the player and the story characters, and mathematical puzzle activities embedded in the story context. The main character is a fourteen-year old girl, Julie, who has fallen into the Phoenix Archipelago, an alternate world in which someone has stolen a feather from the Golden Phoenix. The player must help Julie and Darien, a boy of fifteen who has also fallen into this world, overcome many challenges to recover the feather, including dealing with Saffron and the Keeper, two enigmatic adults who inhabit the Phoenix Archipelago. The player progresses through Phoenix Quest gathering story chapters and mathematical puzzles by succeeding at various tasks. These tasks include solving mathematical puzzles, finding words in the already collected chapters that represent the answers to clues sent in correspondence from story characters, and writing to the story characters. The mathematical puzzles cover a wide range of concepts: fractions, ratios, negative numbers, rectangular and polar coordinates, graph algorithms (minimum spanning tree and travelling salesman problems), logic, and number sequences. Each puzzle has multiple levels of gradually increasing difficulty to help the player gain understanding of the underlying concepts. In addition, several of the puzzles use scaffolding techniques similar to those in the RDCM version of Super Tangrams.

We developed Phoenix Quest over a three-year period alternating between development and classroom evaluation phases. Starting with a very small prototype containing our first attempt at a correspondence activity, we tested each phase of Phoenix Quest in classrooms before adding further components and features. After each round of testing we modified most of the already developed elements of the game to increase the game’s effectiveness. The final version contains a full-length novel by a children’s author, Julie Lawson, correspondence activities with the four main characters, 15 interactive mathematical puzzles, a collectible card game called Strife, and a variety of riddles, cloze passages, and other language arts activities. Finishing the entire game takes between 15 and 25 hours.

Over the past two years we have observed several hundred students in grades 4 — 8 using Phoenix Quest in school classrooms and computer labs. Phoenix Quest is extremely popular with girls who often identify it as one of their top choices among all computer games. It is also popular with boys, though many boys tell us they wish it had more fast action and violence. Girls and boys tend to play the game differently [KWSD96, KUIK97, DUKY98]. Girls spend more time savoring each activity. They engage in extended sequences of postcard exchanges with Julie and Darien, and discuss the characters’ personalities at length with their friends. When writing to Julie and Darien, they try to prolong the fantasy that Julie and Darien are real people who are communicating with them, whereas boys try to prove that Julie and Darien are not real. Boys play through the game more quickly than girls, racing each other to build large collections of chapters, puzzles, and cards.

One of Phoenix Quest’s strengths is that it supports a wide range of playing styles and interests by giving players access to many different activities at once, yet manages to motivate students to (eventually) do virtually all the mathematical activities. At each point in the game the player has access to the puzzles, story chapters, characters, and Strife cards obtained so far. Once a player has obtained a puzzle, the levels can be played in any order, in contrast to most games that allow access to level n+1 after level n has been completed. Each level has a large number of different instances so players can replay the same level many times without doing the same puzzle twice. We incorporated these elements because our early classroom indicated that a large degree of player choice and "replayability" was highly desirable to students.

On the other hand, the goal of providing so much choice seemed initially to conflict with our learning objectives. In particular, we wanted players to do all the levels of each puzzle to complete the carefully scaffolded sequences of activities. In the early stages of

Phoenix Quest development we observed that most students were only playing the easy levels of each puzzle and hence avoiding most of the intended mathematical learning. At that point we were using story chapters as the primary reward system, but there were too many puzzle levels to be able to give a different story chapter as the reward for completing each puzzle level. We solved this by creating a card game (Strife) as the final mathematical challenge of the game, and giving a Strife Card as a reward for completing each puzzle level.

Strife cards come in three types: character, action and gear. Each card has an illustration, a description and graph of its function in the game, and some "flavor text" adding irrelevant but entertaining information about the character or item. In order to play Strife, the player must select a team of at most 4 character cards (e.g. Pinnacle Pixie, Bog Beast, Fiona the Waif) and a deck of at least 20 action and gear cards (e.g. Lost Puppy, Fireball, Spirit Switch, Butterfly, Influenza) from the cards they have managed to collect. Players can win the game by using either a "nice" or "nasty" strategy. Players using the nice strategy win by raising the karma values of their own characters to the ultimate level. Players using the nasty strategy win by reducing the stamina values of their opponent’s characters to zero. The value of the cards given as rewards increases with puzzle level, but the particular card received is randomly chosen from a small set so that players will replay levels several times in order to collect all possible cards. Strife Cards have proved enormously effective as a reward mechanism; players now try to complete all levels of every puzzle to ensure they get every possible card.

When used with supporting classroom activities such as related pencil and paper worksheet activities, large and small group discussions, and journal writing, we have observed significant increases in interest, enjoyment and achievement in learning the mathematics concepts in the puzzles. On the other hand, without these supporting activities, despite enthusiastic game playing the increases have been much more modest. Table 5 shows the pre- and post-test results of a seven-week study in which the students in four classrooms played PQP, a subset of Phoenix Quest containing 6 of the math puzzles. None of the students in three of the classrooms had played Phoenix Quest before the study, while half the students in the fourth classroom had played it the year before. Three tests (pre-, mid- and post-) covering concepts from four of the puzzles (number patterns and sequences, fractions, angles, and finding minimum spanning trees) were given to the students at the beginning, three weeks into, and at the end of the study. The tests were also given to a control class not participating in playing PQP.

During the study students played in pairs for 30 minutes twice per week on computers in their classroom. During the first three weeks no supporting activities were done in any of the classrooms. In the last four weeks of the study (after the mid-test had been given), supporting activities were done in each classroom. In three of the classrooms, the students did pencil and paper worksheet activities using a format closely modeled on the math puzzles, and then discussed the outcomes of the activities in whole class sessions. In the fourth classroom, a guest teacher lectured on the concepts and pointed out their role in the mathematical puzzles. The mid-test results showed no significant improvement over the pre-test results in any of the classes. In some of the classes the mid-test scores were slightly lower than the pre-test scores.

Class |
Play sessions |
Supporting activities |
Pre-test |
Post-test |
Change |

Grade 5 control |
none |
none |
31% |
28% |
-3% |

Grade 5 class A |
2 per week |
worksheets, discussion |
31% |
52% |
21% |

Grade 5 class B |
2 per week |
worksheets, discussion |
36% |
53% |
17% |

Grade 5 class C |
2 per week |
worksheets, journals, discussion |
30% |
53% |
23% |

Grade 6/7 class |
2 per week |
lectures |
56% |
56% |
0% |

Table 5. Results of the seven-week study of playing 6 Phoenix Quest puzzles.

All three of the grade five classes show substantial improvement over the four-week period between the mid-test and post-test, while the grade 6/7 class showed no improvement. We suspect that the primary reasons for the lack of improvement in the scores of the grade 6/7 class was the higher starting scores, lower interest in PQP because of previous exposure to Phoenix Quest, and lack of interest in the lectures from the guest teacher.

4. How Can Software Help Students Learn Mathematics?

The degree to which using interactive multimedia software can help students develop skills and understanding with respect to a particular topic in mathematics is obviously related to the topic as well as the kind of skills and understanding in question. Most of the early mathematical computer games focused on drill and practice of simple number operations and concepts. Such games are easy to develop. Moreover, playing such games are an effective and motivating method of increasing fluency for many students. However, drill and practice is only one of many components of mathematics learning, and can also be achieved via a variety of non computer-based methods. E-GEMS research therefore focused on exploring the effectiveness of the use of computer games in facilitating learning more complex concepts and skills. We have found that the most useful approach to answering this question has been to identify things that software can do particularly well and that are valuable inassisting students in learning some concepts and skills. We now discuss four such software capabilities.

a. Provide essentially unlimited numbers of examples and problems.

This is, of course, one of the reasons for the success of computer-based drill and practice activities, but is also valuable in more complex situations. Software greatly facilitates access to very large numbers of examples because of the ability to use randomization to generate examples and problems on the fly, and the ability to store large amounts of data.

b. Facilitate visualization and manipulation; link visualization with symbolic representations.

Software can be especially valuable in cases where concrete manipulatives are difficult to create, or where it is difficult to tie the appropriate symbolic representation with the physical manipulation of a concrete manipulative. For example, tangram pieces are often used as manipulatives in learning 2 dimensional transformational geometry, but it is difficult for students to associate the symbolic representation of a reflection or rotation with a particular physical flipping or turning of a piece. Software can support the association by requiring the students to interact with the symbolic representation in order to apply the transformation. The E-GEMS research studies on Super Tangrams described in section 2 indicate the effectiveness of the use of such an approach.

c. Provide adaptive sequencing and feedback.

A common approach to helping students deal with complexity and/or abstraction is to provide a carefully designed sequence of activities in which the level of difficulty steadily increases, and the student is gradually introduced to the key aspects of the concept being learned. Software can facilitate this process in a number of ways. By progressively reducing visual or auditory feedback in a task, students can be required to gradually assume greater cognitive responsibility. Moreover, the sequence and help modules can be designed so as to adapt to the student’s performance of the tasks. Again the effectiveness of these approaches is demonstrated in the results of the Super Tangrams studies described in section 2.

d. Provide sustained contextualization in a meaningful and engaging application.

Almost every mode of mathematics learning (e.g. textbooks, lectures, paper and pencil problems, hands-on projects) offers excellent opportunities for presenting mathematical ideas, problems, and structures within an interesting and motivating context. Game software has a number of obvious advantages because of its interactive and multimedia capabilities, and its ability to keep students deeply immersed and engaged for sustained periods of time. One of its difficulties, however, perhaps due to the immersive effect, is that students often fail to have conscious awareness of the concepts, structures, and algorithms they encounter and use in educational computer games, and fail to be able to transfer what they have learned to other contexts. This common problem with educational games is the reason for the E-GEMS focus on the first research question (A) listed in section 1. Fortunately this problem can be addressed by careful game design and by integration of game play with other modes of learning, as outlined in the sections on game design and use, and illustrated in the sections on Super Tangrams and Phoenix Quest.

5. Game Design

There are many choices that must be made in designing a computer game aimed at enhancing mathematics learning:

- content to be learned,
- activity in which the learning is to occur
- underlying model(s) of learning
- representations of the concepts
- interfaces used to manipulate concepts and objects
- navigational structure and sequencing of activities,
- feedback and reward systems
- entertainment elements such as graphics, sound, story, characters, and humor

As mentioned above our choice of content has been based selecting topics that would benefit from one or more of advantages listed in the preceding section. We now briefly mention a few observations and recommendations with respect to some of the other design issues. While these remarks certainly do not apply in all cases, they have applied sufficiently often in our experience that we have found them worth taking into account in game design. Many are illustrated in the earlier sections describing our work with Super Tangrams and Phoenix Quest.

a. Activity

- A high degree of interactivity is essential. Students want to spend the majority of their time continuously interacting with the program in ways that influence the outcome of the activity. Moreover, they want immediate and frequent feedback on the results of their actions. In most cases students are not interested in engaging in passive activities (reading, listening, watching) for extended periods of time (i.e. more than a minute). Exceptions occur when students need information to achieve an important goal towards which a significant degree of effort has already been expended, or the passive activity provides a relaxing but brief reward after a completing an arduous challenge in the game.
- Most successful interactive activities are analogues of non-computer versions, e.g. simulations, problem solving, puzzles, quizzes, matching or grouping items, creating artifacts, conducting conversations, having an adventure, exploring a new environment. If the activity is not motivating in real life it is not likely be motivating on the computer.
- Activities with explicit goals work well for most students. However, many (especially girls) also appreciate access to more open-ended activities that can be done while stuck in trying to achieve a particular goal.

b. Representation of concepts

The following are obvious but they are ignored surprisingly often.

- It is important to use a representation that reflects what you want the student to think about.
- Using the same representation as used in other accompanying modes of mathematics education (textbook, lectures, worksheets) will help students transfer and integrate understanding between the different modes.

c. Interfaces used to manipulate concepts and objects

- Interfaces that involve direct interaction with the representation of the concept to be learned (Direct Concept Manipulation) are better than those that manipulate an object being used to illustrate the concept (Direct Object Manipulation) [SW97, S98].
- (Slightly) awkward interfaces are better than intuitive, easy interfaces for directing the learner’s attention [Sv91,Go96, SK96]. For example, if you want a student to think about the choice of a numeric value, having them type in the value works better than a graphical interface such as clicking on the value or using a slider to set the value.

d. Navigational structure and sequencing of activities

- Scaffolding, i.e. having the student complete a sequence of related activities that gradually increase in cognitive challenge, is very effective in enhancing learning [S98, SK96, S97].
- Many students strongly prefer a wide degree of choice of activities to a tightly constrained sequence. For example, if a puzzle has several levels of difficulty they want to be able to attempt the levels in any order rather than be forced to do the levels in increasing order. Similarly they like games that allow them to choose among several different kinds of activities at any point. This conflicts with the desire to have students complete scaffolded sequences of activities, but carefully structured reward systems can be effective in inducing students to complete sequences of activities, even in games with enormous freedom of choice (section 3, [KWSD96]).
- Using a single multiple choice question to verify that the student understood akey concept as a necessary step to gaining the reward for completing a puzzle or simulation involving the concept, can be very effective in focusing the students’ attention on that concept in the puzzle or simulation. This tactic also useful in fore-shadowing upcoming concepts by including the concepts among the wrong answers to the multiple choice question. For example, in the Hexagon puzzle in Phoenix Quest students have to find a path through a hexagonal tiling such that the set of numbers on the tiles on the path share a common property. After finding the path students are asked which one of four properties the sequence of numbers satisfied (e.g. multiples of 2, perfect squares, prime numbers, or numbers that have 5 as a remainder when divided by 6). Before adding this question, we observed that students often found the path by trial and error. After we added the question, we found that students were much more focused in thinking about which properties and patterns might be present when looking for the path.
- Students are more likely to want to take advantage of instructional modules or practice areas when they get stuck in the middle of a highly motivating challenge than before they have started the challenge. On the other hand they are reluctant to leave the challenging activity if leaving will cause them to lose the progress they have made so far. Thus it is important to allow students to access instructional and practice modules while maintaining their state in the midst of challenging activities.

e. Feedback and rewards

- Some form of scoring is important; students want ways to measure their achievement.
- Other forms of extrinsic rewards (e.g. sound effects, animation, money, items providing additional power or functionality) are also very effective in increasing students’ motivation, and in most cases do not seem to detract from the intrinsic motivation of the activity nor from the learning outcomes.
- Multiple reward systems in which the player receives several different rewards (e.g. a score, sound and graphic feedback, and a number of items with different properties) for completing an activity work well because different people are motivated by different rewards. Moreover, multiple reward systems encourage repeated playing of particular components since the same person is often motivated by different rewards at different stages of understanding a concept and in progressing through a game.
- Partial reinforcement, in particular incorporating a degree of randomness in providing rewards, is particularly effective in encouraging students to keep playing or repeat an activity. This fact is well documented in the field of psychology (e.g. B.F. Skinner’s theory of behavior), and the technique is abundantly applied in commercial video and computer games. For some reason it is seems to be less commonly used in educational software.

f. Entertainment elements

The presence of entertainment elements that are well integrated with the activity is more important than the level of polish and sophistication. Most students passionately appreciate the presence of colorful graphics, animation, sound effects and music in educational software, and express dissatisfaction when they are absent. However, our experience has been that students are quite happy with fairly "low-tech" levels of these elements (contrary to the beliefs of most commercial game developers). For example, most of the animation in Phoenix Quest is very primitive, and the illustrations in the story are still pictures, yet few students have expressed the need for better graphics. There was little sound in the early versions of Phoenix Quest, and we got endless requests for sound effects and music. Adding fairly low fidelity sound seems to have completely satisfied the vast majority of students. In Sedighian’s Super Tangrams study [S98, SeS97, SS97] the "enjoyment" scores of the students playing the plain versions with bland colors and no sound effects or music were almost as high as those playing the embellished versions. For example the average score for the statement "I like to learn math from computer games like Super Tangrams" was 4.6 for students with embellished versions and 4.4 for students with the plain versions, where 5 represented "strongly agree" and 1 represented "strongly disagree". Similarly, the scores for the question "Compared to other educational games you have played, how much did you like playing Super Tangrams?" were 4.3 (fully-featured) and 4.1 (plain) where 5 represented "loved it" and 1 represented "hated it". On the other hand 86% of the students playing the embellished versions either "loved" or "liked" having the music. Similarly, 97% of these students loved or liked the colorful patterns and cartoons, but only 40% of the students playing the plain versions loved or liked the graphics.

6. Game Use

How games are used is often as important as how games are designed in influencing their effectiveness in enhancing mathematics learning. As in game design, the primary issue is having students recognize, think about and value the mathematics embedded in the computer game. When this has failed to occur we have seen little evidence of mathematical learning resulting from game play. We have found three factors to be particularly important in focusing students’ attention on the mathematics: teacher attitudes, supporting activities and collaborative play. We will briefly discuss each of these below. There are also more pragmatic issues related to the length and frequency of playing sessions. Based on the Super Tangrams studies described in section 2, our sense is that playing a game several times a week for a few weeks is likely to be more effective than playing once or twice a week for several weeks, but more research is needed to confirm this. Unfortunately one or two periods per week is currently the most common form of computer access in schools.

a. Teacher attitudes

Students are more likely to see a game as helpful in learning mathematics and are more likely to want to play it (particularly girls), if they receive clear indications that the teacher believes this is the purpose of the game and expects them to play it. These indications can and should take several forms: making explicit statements about the reasons for playing the game, establishing a playing schedule and keeping a record of who has played, and engaging in various other activities that integrate the playing experience with other classroom experiences.

b. Supporting activities

We have found whole class and smaller group discussions about the game to be particularly effective in focusing attention on the mathematical ideas, as well as helping the less experienced computer game players catch up with the more experienced players. Typical questions from the teacher include "what have you discovered so far?", "what have you done?", "how did you do it?", and "what kind of mathematics did you need to use to do it?". Pencil and paper activities modeled on the game activities have been also been extremely popular, especially when the activity asks students to design their own version of part of the game [KP95]. One of our teachers has had her students (grades 4 and 5) keep "research" journals recording their game playing experiences. Despite many entries of the form "First I played game A, and then I played game B, etc.", over time these journals have provided the teacher with many insights about how her students thought about the mathematics in the game.

c. Collaborative play

For single-user games we strongly recommend having a pair of students share each computer, rather than having students work alone in a one-to-one configuration.

Many studies have shown that children generally prefer to play computer games collaboratively. Some studies have shown that pairs sharing a computer made more progress through a game (e.g. solved more problems) that students working alone, though there are also studies showing no significant difference in the level of progress (see [In97] for an overview of the literature). In addition to the motivational effects of sharing a computer, and the possible benefits in performance, our classroom observations have indicated that the continuous dialogue between partners about how to play the game helps students recognize and articulate the mathematical concepts in the game.

7. Gender Issues

Throughout our research we have observed many differences in how girls and boys interact with computers and computer games. As mentioned before, we saw significant differences in our 1993 study on children’s interactions with video and computer games [IUK+94, LUK+95]. In addition to having different game preferences from those of boys, girls were much less assertive in competing for access to the video games and computer games. They spent more time creating ideas for new games at the game design area than in playing or watching video and computer games. In comparison to boys, girls spent less time per day playing video and computer games at home, owned fewer games, and were less interested in and knowledgeable about the game industry.

We have seen similar differences in our classrooms. In several grade 4 and 5 classes, girls were initially much less interested in playing the games on the computers in their classroom. Few girls volunteered answers in the class discussions about the games. They were even reluctant to speak when asked their opinion in the discussions. In each of these cases, however, the situation changed dramatically after minor interventions. For example, after establishing a day of the week for girl-only computer play during lunch-time (plus a corresponding day for boys), and holding a "girl-only" discussion followed by an analogous discussion for the boys, girls became equally enthusiastic about playing the games and participating in the whole group discussions. During the girl-only discussion the girls told us that they had felt uncomfortable participating before because the boys seemed to be so much better at computer games. Our response was that it was important to us that they played because we wanted to find out how to make games that girls liked to play. After these changes the girls were. The fact that such small actions could remedy the situation indicates the importance of teachers making a conscious effort to ensure that all students feel encouraged to use computers. The E-GEMS researchers in Ontario have explored gender differences in slightly older (grade 7 and 8) students’ attitudes towards computer games, and have found similar results [see DUKY98, UK96, U98].

Our studies have also revealed differences in how girls and boys play computer games. In several studies examining the impact of specific software and hardware interfaces on performance (e.g. drag-and-drop versus point-and-click [In97], turn-taking protocols for mouse-sharing [IGBK95, IMBK97], and two player communication [GK97]) there have been statistically significant differences between the results for boys and girls. In general, the interface or protocol that works best for boys is not always what works best for girls and vice versa. In addition, boys seem much more concerned with making rapid progress (e.g. number of puzzles or levels completed) through the game, and in most cases where we have measured progress over a fixed period of time the boys have indeed made more progress [In97, GK97, KWSD96]. Whether these gender differences are due to different levels of game playing experience, different attitudes towards game play, or other factors, the fact that they exist makes it important for game developers and teachers to be aware of the potential differences and to provide flexibility in how games are used.

8. Current and future research

The goal of our current prototype development efforts is to investigate the educational potential of multi-player networked computer games for students in grades 6-10. Over the past two years we have been developing two prototypes, Island and Avalanche, using different platforms and game formats in order to compare the effectiveness of various design and use issues. In Island (MAC, LAN), players freely explore an island, obtaining building resources as rewards for solving mathematical puzzles, creating buildings either alone or collaboratively, and viewing those of other players. There is no explicit story or overall goal in Island other than those created by the players themselves. Last year, we completed our first formal multi-user study using the "Builder" component of Island to investigate the effect on learning and attitude outcomes of three variables: communication mode, gender, and how the task was specified [GK97, GR98]. In Avalanche (Java, Web) four players assume the roles of the leading citizens in a mountain ski town, and work together to deal with the problems caused by a series of avalanches. In contrast to Island, the story and its specific challenges are central to Avalanche. In addition, Avalanche has lower levels of graphics, animation and real-time interactivity than Island because of its web platform. We are currently in the midst of classroom observations of the first (limited) versions of Island and Avalanche, and expect to complete the next phase of development by the early fall and continue with more focused studies. We are also continuing to further investigate many of the other design and use issues mentioned in this paper.

Acknowledgments

I want to thank the multitude of people involved in E-GEMS, including my children Janek and Sasha who got me interested in this, colleagues in education Rena Upitis, Ann Anderson, Marv Westrom and David Pimm; teacher researchers Eileen Phillips, Doug Super, Carolyn Dymond and Carolyn Varah; graduate students Kori Inkpen, Kamran Sedighian, and David Graves; game industry collaborators Paul Lee, Don Mattrick, and Greg Bestick; the twenty or more UBC undergraduate computer science students who have done most of the game prototype development, and the hundreds of school students in grade 4 — 7 who have participated in our studies. Thanks is also due to the many funders of E-GEMS research: the Natural Sciences and Engineering Research Council of Canada, B.C. Advanced Systems Institute, Electronic Arts Canada, Creative Wonders, Apple Canada, Hewlett-Packard Canada, and IBM Canada.

Bibliography

[B82] Blank, D.E. (1982). Pac Man Goes to School: Teaching Problem Solving in the Electronic Age, Computing Teacher, 10 (1), 50-53.

[BH89] P. Blackstone and G. Hamilton, Gender Equity in Mathematics, Science and Technology (1989), A Survey of Vancouver Island Schools in Breaking the Barriers Software and Information Kit for Grades 3-7 , Software Training Associates, Victoria BC Canada V8X 3Y1.

[DB86] David, A., Ball, M.P. (1986). The Video Game: A Model for Teacher-Student Collaboration, Momentum, 17 (1), 24-26.

[DGS95] A. Durndell, P. Glissov, and G. Siann, Gender and computing: Persisting differences, Educational Research 37(3):219-227.

[DUKY98] J. DeJean, R. Upitis, C. Koch and J. Young, The story of Phoenix Quest: how girls respond to a prototype language and mathematics computer game, Gender and Education, in press.

[GK97] D. Graves and M. Klawe, Supporting learners in a remote computer-supported collaborative learning environment: the importance of task and communication, Proc. CSCL ‘97, Toronto, 1997.

[Go96] D. Golightly, Harnessing the interface for domain learning, Proc. CHI’96, Conf. Comp. 37-38.

[Gr98] D. Graves, Supporting learners in a remote computer-supported collaborative learning environment: the importance of task and communication, M.Sc. thesis, Dept. of Comp. Sci. UBC, www.ubc.ca/nest/egems/home.html

[Ho96] S. Holst, Directing learner attention with manipulation styles, Proc. CHI’96, Conf. Comp. 43-44.

[I97] K. Inkpen (1997). Drag-and-Drop vs. Point-and-Click Mouse Interaction Styles in Interactive Learning Environments for Children. Submitted to Trans. Computer-Human Interaction, July, 1997.

[IBKU95] K. Inkpen, K.S. Booth, M. Klawe and R. Upitis, Playing together beats playing apart, especially for girls, Proc. CSCL'95. Lawrence Erlbaum Associates, 177-181.

[IGBK95] K. Inkpen, S. Gribble, K.S Booth, and M. Klawe, Give and take: children collaborating on one computer", Proc. CHI'95, ACM press, 258-259.

[IMBK97] K. Inkpen, J. McGrenere, K.S. Booth, and M. Klawe, Turn-Taking Protocols for Mouse-Driven Collaborative Environments, Proc. Graphics Interface '97, 138-145.

[In97] K. Inkpen (1997). Adapting the Human Computer Interface to Support Collaborative Learning Environments for Children. Ph.D. Dissertation, University of British Columbia, August 1997.

[Ink97] K. Inkpen (1997). Three Important Research Agendas for Educational Multimedia: Learning, Children and Gender. ED-MEDIA 97, Calgary, Canada.

[IUK+94] K. Inkpen, R. Upitis, M. Klawe, J. Lawry, A. Anderson, M. Ndunda, K. Sedighian, S. Leroux, and D. Hsu, We have never-forgetful flowers in our garden: girls' responses to electronic games, Journal of Computing in Mathematics and Science Education

[Kl92] M. Klawe, Bringing mathematical research to life in the schools, Proc. 7th Int. Conf. Math. Education, Quebec, 1992.

[KP95] M. Klawe and E. Phillips, A classroom study: Electronic games engage children as researchers, Proc. CSCL ‘95, Indiana, 1995.

[KSW96] M. Klawe, D. Super and M. Westrom, Counting on Frank: Postmortem of an Edutainment Product, www.ubc.ca/nest/egems/home.html.

[KUIK97] M. Klawe, R. Upitis, K. Inkpen, and C. Koch, The Story of Phoenix

Quest: Developing an Appealing Mathematical Computer Game for Girls (and

Boys), Proc. Grace Hopper Memorial Conf. on Women in Computing, San Jose, 1997.

[KWSD96] M. Klawe, M. Westrom, D. Super, and K. Davidson, Phoenix Quest: Lessons from developing an educational computer game, Int. Conf. on Multimedia Tech. Man., Hong Kong, 1996.

[LUK+95] J. Lawry, R. Upitis, M. Klawe, A. Anderson, K. Inkpen, M. Ndunda, D. Hsu, S. Leroux, and K. Sedighian, Exploring common conceptions about boys and electronic games, Journal of Computers in Mathematics and Science Teaching, 14(4):439-459.

[M96] J. McGrenere (1996). Design: Educational Electronic Multi-Player Games -- A Literature Review. Technical Report 96-12, Department of Computer Science, UBC.

[M80] Malone, T.W. (1980). What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games. Palo Alto, California: Xerox Corporation, Palo Alto Research Center.

[Ma82] T. Malone, Heuristics for designing enjoyable user interfaces: lessons from computer games, Human Factors in Computer Systems, 63-68, Gaithersburg, MA, 1982.

[PE96] T. Plomp and D. Ely (Eds.), International Encyclopedia of Educational Technology (2nd Ed.). Oxford, UK: Elsevier Science, 1996.

[RaMWW92] J. Randel, B. Morris, C. Wetzel, B. Whitehill, the effectiveness of games for educational purposes: A review of recent research, Simulation and Gaming 23(3):261-276, 1992.

[Re92] M. Resnick, Collaboration in simulated worlds: Learning through and about collaboration, SIGCUE Outlook 21(3):36-38, 1992.

[S96] K. Sedighian (1996). An Investigation of Design Factors of Game-Based Electronic Learning Environments. Proc. 2nd International Conference on the Learning Sciences, Evanston, USA.

[S97] K. Sedighian (1997). Challenge-driven learning: A model for children's multimedia mathematics learning environments. ED-MEDIA 97, Calgary, Canada. (Best-Paper Award)

[S98] K. Sedighian (1998). Interface style, flow and reflective cognition: issues in designing interactive multimedia mathematics learning environments for children, Ph.D. Dissertation, University of British Columbia, February 1998.

[SeK96] K. Sedighian and M. Klawe, Super Tangrams: A child-centered approach to designing a computer supported mathematics learning environment, Proc. ICLS '96, 490-495.

[SeS97] K. Sedighian and A. Sedighian (1997). Aesthetic Response: Children's Reactions to Colour and Graphics in Educational Software. ED-MEDIA 97, Calgary, Canada.

[Sh93] B. Shneiderman (Ed.), Sparks of innovation in HCI, Norwood NJ, Ablex Publishing, 1993

[S86] Silvern, S.B. (1986). Classroom Use of Videogames. Educational Research Quarterly, 10(1), 10-16.

[Sk86] R. Skemp, The psychology of learning mathematics (2nd Ed.), Middlesex UK, Penguin, 1986.

[SK96] K. Sedighian and M. Klawe, An interface strategy for promoting reflective cognition in children, CHI’96, Conf. Comp. 179-180.

[So96] E. Soloway (Ed.), Interactions: New visions of human-computer interaction, NY, ACM 1996.

[SS96] K. Sedighian and A. Sedighian (1996). Can Educational Computer Games Help Educators Learn About the Psychology of Learning Mathematics in Children? Proc. 18th Ann. Mtg. Int. Group for the Psychology of Mathematics Education -- the North American Ch

[SS97] K. Sedighian and A. Sedighian (1997). Use of Background Music in Electronic Learning Environments. ED-MEDIA 97, Calgary, Canada.

[Sv91] G. Svendsen, Influences of interface style on problem solving, Int. J. Man-Mach. St. 35:379-397.

[SW97] K. Sedighian and M.Westrom (1997). Direct Object Manipulation vs. Direct Concept Manipulation: Effect of Interface Style on Reflection and Domain Learning. HCI 97: Premier European Conference on Human-Computer Interaction, Bristol, UK.

[SWK96] D. Super, M. Westrom, and M. Klawe, Design issues involving entertainment click-ons, Proc. CHI’96, Conf. Comp. 177-178.

[UK96] R. Upitis and C. Koch, Is equal computer time fair for girls? Potential Internet Inequities. Proceedings of the 6th annual Conference of the Internet Society, INET '96, Montréal, Québec.

[U98] R. Upitis, From Hackers to Ludites, Game Players to Game Creators:

Profiles of Adolescent Students Using Technology in a Classroom Setting.

Journal of Curriculum Studies, 30(3) 293-318.