Math Forum Annual NSF Grant Progress Report

REC-9618223 (February 24, 1999)
Eugene A. Klotz (
Stephen Weimar (

The Math Forum continues to make strong progress with regard to its three strategic objectives for this grant, even as these have evolved over the course of the last two years:

In the first year of this grant the Forum focused on developing the promising services that had emerged from the proof-of-concept phase, and on experimenting with other applications of Forum resources, including teacher leadership development in the context of Urban Systemic Initiatives. In the current (second) year, the Forum has concentrated on core services and relations: scaling key projects, laying the foundation for future growth, developing collaborations with key research and organizational partners, positioning the Forum to realize self-sustaining opportunities, and beginning to study the impact of the Forum on mathematical thinking and learning.

Our approach to the third and last year of this grant evolves directly from the resulting strengths and opportunities of this prior work.

The Math Forum has developed a knowledge-building environment that integrates and capitalizes on activity in three areas: mentoring, community building, and establishing a resource center that provides easy access to mathematics, learning activities, and powerful tools. Support for reform and improvement takes place through a 'bottom-up' approach in which insights, technologies, materials from research, and national standards are infused into our projects and services. The Forum disseminates and promotes many improvements in mathematics education through its choice of problems and activities, communication and modeling that take place in mentoring, the selection of associates and organizations with which to conduct projects and develop resources, and through the structure and featured elements of our resource center.

Part of the Forum's success can be attributed to its non-compulsory character as a rich resource and an open community. Forum services attract users by freely meeting strong needs of key audiences. For students and parents this means help with homework or opportunities for enrichment and independent study. For teachers it involves deepening their mathematical understanding, providing access to activities for the classroom, supporting the integration of technology into instruction, and enabling conversations and resource-sharing among peers to overcome isolation. Having attracted users, Forum projects take advantage of their involvement to provide the critical elements for improved mathematics education: experience with interesting and powerful mathematics, reflection on practice and communities of support, integration of practice with research and evolving standards, and easy access to new tools and materials.

The near-term challenge for the Forum is twofold:

Secondary issues that follow on these two key points are:

For the third year of this grant three key efforts will run in parallel:

By focusing on a limited number of key projects, the Forum can conserve staff resources and solidify the position of these services in the Internet education community. This is particularly important as the Forum adds new work in cultivating a larger user base and developing relations with corporate partners. The success of the approach hinges in part on using this limited set of projects to meet the wider set of Forum goals: aiding professional development, community building, and the generation and publishing of activities and tools developed in the classroom.

The descriptions contained later in this report of what we have done during the past year, together with the attached evaluation summary, indicate why we believe these goals can be accomplished.

As will be outlined in more detail below, the Forum is aggressively pursuing relations with corporate partners in order to establish a sustainable venture. Growing our user base is a central part of this activity that will benefit the other strands as well, serving at least three critical functions:

Establishing a high profile and a critical mass of users is critical for creating an environment in which talented, energetic individuals wish to share what they have developed and contribute their expertise. In turn, a high level of activity and a significant amount of content attract researchers, funders, professional organizations, Internet companies, and publishers who can pursue their goals in the context of this community. With the growth in usage comes the need to expand capacity in terms of the underlying software and hardware systems; thus the Forum systems and programming staff must increasingly focus on adapting applications developed elsewhere, rather than on writing their own (this is doubly appropriate as more Internet tools are developed and their quality improves).

While the Forum's attention must turn to the matters of growth and other sources of financial support, there remain large areas requiring further research. Several of these will be pursued because of they are critical to the success of the other third-year goals: how to generate more and better mathematics and on-line explorations, improve access to resources, deepen the level of reflection and discourse, respond effectively to school and district professional development needs, and serve well the entire K-12 community, particularly younger students.

As will be shown, the Forum is well-positioned to achieve its objectives for the coming year and to continue translating its research into immediate and long-lived contributions to the learning and teaching of mathematics.

    I. Sustainability

One key objective for the funded research was to develop self-sustaining versions of the research programs as they became proven services, capable of continuing after the period of this NSF grant. We have continued to pursue this effort aggressively along the two lines indicated in the first year of work: business plans for revenue generating Internet services, and partnerships with mathematics education organizations and researcher programs.

Business plans: The principal investigators of the Math Forum, with the cooperation of Swarthmore College, are forming a new company that will assume the lead responsibility for development of a self-sustaining future for the Math Forum. It will seek corporate partners to develop and implement a commercially viable interface. In coordination with the Math Forum staff, it will develop business and operational plans that establish a brand identity, increase market share, lead to revenue models, and scale projects to accommodate 10-100 times existing usage. The services developed under this grant will then be poised to continue and ultimately achieve a new self-sustaining existence.

During this next year the work on the grant will continue as described in the grant under the direction of the principal investigators and Swarthmore College. It will probably take most, if not all, of this year to secure the business relationships and conduct the preparatory efforts described above. Once the new company has been created, new staff has been hired, and equity funding has been raised, this new company will pursue business in coordination with the existing Math Forum group. Once this grant has expired, the new company will assume responsibility for the bulk of the Math Forum projects and staff members as appropriate. The College and the new company also anticipate pursuing some continued research efforts jointly based on the developing services.

Currently we have retained the Dallas-based investment banking firm Capital Alliance to informally solicit interest from a wide range of potential strategic partners and investors (see material delivered to Eamonn Kelly, NSF program officer, in February). Forum business advisors suggested that this would be a good way to evaluate indications of interest that the Forum was already receiving and to locate opportunities to work with others who share our goals for development of Internet education resources.

One of the first proposals that the Forum is evaluating involves Encore Software, a West Coast company that sells retail math and other education software packages. Encore was looking to enter the Internet education arena, and found the Math Forum already doing what it planned to create. Encore would bring capital to invest and a management team with experience in marketing and starting up new companies. This represents one way for the Math Forum to strengthen its position in the market and to develop to the point where it has reasonable revenue models that will enable it to seek effective collaborations with larger corporate partners.

Swarthmore College has also expressed the intent to provide the Math Forum with funds (currently proposed at $120,000), housing, and other resources as cost-sharing in the transition to independent status. The College, while unaccustomed to such activity, is proud of the Forum and very supportive of its efforts to become self-sustaining, which may provide a model for future projects as well.

Please see the business planning documents given to Eamonn Kelly at the end of February for more information on the future development of corporate partnerships and revenue generation.

Partnerships: On other fronts, the Forum has continued to develop relations that generate income with other researchers, professional organizations, and schools. NCTM has contracted with the Forum for a number of moderated discussion groups, as has Teaching Mathematics for American Democracy, a small project out of the University of Wisconsin-Madison, and CESAME, which is conducting the professional development support for the Investigations curriculum.

Through the TappedIn and ESCOT projects, SRI has subcontracted for Forum services and participation in research, as have TERC (Bridging Research and Practice) and the Journal of Online Mathematics and its Applications (JOMA), out of the MACMATC project run by Gerald Porter at the University of Pennsylvania.

The Union City School District and a number of individual schools paid the Forum for professional development workshops. JPBM pays the Forum to manage the Web site for the annual Mathematics Awareness Month.

Out of all of this activity, the research subcontracts most effectively and significantly support the Forum's mission where the funds from other contracted services total approximately $20,000.

Clearly this does not indicate the success of the original plan to form a consortium funding the Forum as an R&D arm of professional mathematics organizations. It does suggest, however, that the Forum is cultivating a critical and supporting network among these groups, and the new business model that the Forum is developing contemplates that these research activities will be a core mission of the new organizations, providing the basis for its ability to offer cutting edge and high value added services through the Internet. The relations with NCTM, JPBM, and others add significant value to the Forum in terms of credibility, opening access to people and resources, connecting Forum users to developments in the field, and creating an integrated community in which the Forum is the K-12 "user group" for the wider mathematical community. Recently the Forum consulted with NCTM to improve the mathematics and design of its DoMath site, another indicator of the kind of reciprocity that is developing.

At this point the path to sustainability clearly lies along the lines of the business plans sketched out earlier; however, as noted above, these business plans are built upon the scaling up of the existing Math Forum services to reach a far larger universe of users, and upon continuing to provide leadership in math education research. The network of professional researchers and organizations plays a critical role in the viability of our plans because the success of this undertaking will hinge, to a large extent, on the continued visibility and position of the Math Forum in the larger educational arena, as well as its access to high-quality mathematics, personnel, and innovations.

Growth, Visibility, and Site Development Support: The Forum continues to be a well-respected and popular site, often appearing in articles on the educational use of the Internet. Recently we were nominated for a 1999 Webby, as one of five sites in the Education category. One of the big challenges this year was handling the growth in usage, which has doubled since August 1998 to 200,000 visits a week (a more meaningful statistic for us than the oft used 'hits'). This has required a focused systems development effort that includes:

  1. system administration: adding inexpensive servers to optimize load, allowing us to grow our services without a large up-front investment in hardware. We are committed to Apache/mod-perl as a Web development platform.

  2. designing and developing a Web database system to manage mentored POWs.

  3. designing and developing a Web database system to manage a T2T Ask-an-Expert system. Our doctor's office system was the first Ask-an-Expert system software design to receive certification from the White House-funded Virtual Reference Desk project.

  4. designing and developing a moderated Web discussion environment, combining our discussion management software with dynamic Web page techniques.

  5. changing our systems to make maximum use of third-party mechanisms that have matured over the past year for embedding dynamic functionality in HTML pages. Ultimately, this will save time and money by allowing others to maintain and enhance these mechanisms.

  6. The availability of powerful, inexpensive database backends has increased dramatically in the past year. At the same time, tools have become available that allow these databases to be more easily and efficiently integrated into Web site design. These changes have occurred in fortunate parallel with our site growth. Some of our effort in the past year has gone into harnessing this database backend power, preparing to use commercial systems. This will allow us to dramatically increase the number of customers we can serve with the personnel and hardware available.

  7. The systems group has worked to learn and incorporate new technologies where appropriate. We have found new XML technologies well suited to support our Pages of Math project. We have worked on developing a Document Type Definition that lays out the structural components of the online textbook. We have discovered and adapted XSL tools to convert XML-coded content pages to dynamic HTML for display on Web browsers in a flexible, easily modified way. We have experimented with new dynamic HTML and Javascript techniques for navigating the online curricular resource.

  8. Over the last year, we have examined and cooperated with attempts by GEM and others to develop metadata standards that allow meaningful data exchange across disparate areas of the Web. We have worked toward a uniform application of metadata across the various services and areas on our own site. See the Forum Software Tools.

    II. What We Have Done

To facilitate comparison, the headings in the following sections correspond to those in the original proposal.

  1. Community-building

    The Forum has pursued community development in several directions during the second year. First, we have attempted to help communities locate themselves within the Math Forum. We have created opportunities to host discussion groups for NCTM, NCSM, ASSM, TMAD, CESAME, EDGE (Enhancing Diversity in Graduate Education), and others. We have also worked to host Web sites for others, including NCSM, ASSM, and the Friends Council on Education. The Council of Presidential Awardees staffs the Teacher2Teacher service, which includes public discussions associated with the question-and-answer exchanges in the archives, in order to provide focused, substantive settings for dialogue around teaching issues.

    Second, we have tried to help teachers develop materials and projects and share them online. As part of this effort in 1998 we conducted another advanced summer institute to foster development of online projects. An example of the sort of community that can emerge from such efforts is the Class2Class project, which facilitates networking between classrooms and teachers wishing to join with others in projects at a distance.

    Third, we maintain an active help desk that supports users in finding resources, learning how to use the Internet for mathematics education, making their sites available to others, and connecting to other users and communities.

    Finally, the Forum has also maintained an area for teacher workshop announcements and one for undergraduate research projects. The workshop database represents a collaboration with NTCM, MAA, and AMS.

    Enriching the medium.
    The Math Forum developed its use of TappedIn and included it in our community-building and professional development activities, as outlined below.

    • Workshops and Networking: We educated teachers attending Forum workshops and participating in Forum projects to the use of TappedIn, notably those in the San Diego and Philadelphia Urban Systemic Initiatives. These workshops included our two USI summer programs, our 1998 Advanced Institute, and the monthly follow-up workshops. (See our workshop page for more detail on the respective agendas.) We also conducted workshops at a few conferences, including the NCTM annual conference in which we introduced TappedIn as one element of the Math Forum environment.

      In our workshops we have used TappedIn to hold conversations linking teachers in the USIs and elsewhere around the common issues they face, such as the integration of standards-based testing with curriculum reforms. We also use TappedIn sessions to enable virtual participation in workshop conversations.

    • Support Services: Math Forum staff kept regular office hours for two hours each week in the TappedIn suite to conduct tours for visitors and experiment with this as an alternative to our online email-based support of Forum community members. We also spent time each week at TappedIn, making additions to the Math Forum Suite or talking to other users.

    • Professional Development Presentations: We would like to experiment with online professional development courses. This year, for twelve weeks of the spring semester, we hosted a weekly discussion lasting approximately 1.5 hours once a week, usually for 1-3 participants. Prior to the discussion we prepared a set of notes containing Web sites and dialogue pertaining to the discussion topic. These notes were shown to the participants periodically to steer the, and are available for other visitors to the site. (See the Math Forum Transcripts in the Math Forum Suite on TappedIn.)

      These presentations have been advertised through the regular TappedIn electronic bulletin, Math Forum newsletters, and occasional announcements to mailing lists and newsgroups.

      In the next year we would like to incorporate TappedIn into work with Roberta Schorr and Dick Lesh around using reflection on pieces of work done by children in order to learn more about mathematical thinking and the implications for classroom practice.

    • Development of Local Infrastructure: As we work with groups we create age-level or project-specific rooms in the Forum suite for focused conferencing. At least one Forum workshop participant has set up her own office in TappedIn and is planning on coordinating classroom projects with other teachers through this medium.

    • Internal Communication: The Math Forum regularly uses TappedIn to include remote staff and colleagues in meetings.

    Raising the public profile.
    We have continued to manage the Web site for JPBM's Mathematics Awareness Month. The Forum has consulted with NCTM to improve its DoMath Web site and we host NCTM's discussions on the proposed revisions for the NCTM Standards. A series of public discussions is under development in coordination with NCTM to promote and support the use of technology in mathematics education. These high-profile efforts to promote mathematics, along with Forum services such as the weekly newsletter in which new mathematics and Internet projects are promoted, the What's New and Key Issues Web pages, and the Student Showcase and other Showcases represent Forum efforts to make mathematics more visible to the general public and generate discussion of the critical issues affecting the mathematics education community.

    Including parents and concerned citizens.
    Our collaboration with BBN for a parent education program was not funded and we hope in the future to develop our own Parent Forum built around a question-and-answer area staffed by Presidential Awardees, analogous to our Teacher2Teacher service, as well as a focused library of resources addressing parental concerns and support for at-home learning through Ask Dr. Math.

    Bridging the gulf between mathematicians and math education.
    The Mathematicians in Mathematics Education page has been launched, in collaboration with Susan Addington and Judy Roitman and Mathematicians and Education Reform. Through this page the Forum attempts to foster involvement of mathematicians in K12 education by publishing materials and case studies of successful programs. Gene Klotz has given a number of related talks and organized sessions in the math education sections of AMS and MAA meetings.

    The Forum is also working with JPBM to conduct an electronic version of a questionnaire to get baseline data on current practices among mathematics faculty in order to document current practices and promote models for the educational improvement of faculty and effective reward structures. Ultimately this is expected to lead to a Web site that contains important resources and ongoing versions of the task force's products as well as supporting.

    Facilitating the coordination of Internet math providers.
    In August the Math Forum organized a meeting of the Eisenhower National Clearinghouse (ENC) and NCTM, joined by the Corporation for Public Broadcasting, to facilitate communication and coordination. ENC and the Forum are implementing standards for metadata (based on work of IMS, GEM, and others) to enable shared resources and indexes. NCTM and the Forum are coordinating on many levels, from public discussion of the revised Standards to hosting user group services on the Forum such as public submissions of workshop announcements and Web pages. Gene Klotz has attended many meetings, served on committees, and designed a number of Internet programs to further build relations between the Forum and professional associations as part of our effort to bridge the professional mathematics community and K12 education:

      Annual Meeting of the MAA/AMS in Baltimore, January: Co-chaired a session on using the Web to teach mathematics with Earl Fife and Larry Husch of Math Archives and spoke briefly about the Math Forum at the Mathematicians in Education Reform Banquet.

      MAA: Chair of the Ad-hoc Committee on Databases in Publications and member of the Committee on Electronic Services.

      Gave invited lectures on the Math Forum at local AMS meetings at Kansas State and Temple University.

      Participated in the International Conference on Teaching Mathematics, Samos, Greece and the International Conference on Technology in Collegiate Mathematics, New Orleans.

      Member, NCTM's Electronic Format Group, charged with constructing a hypertext version of Standards 2000. The draft links to the Math Forum and there is a Standards 2000 discussion group hosted by the Math Forum. The committee had a session on the electronic version at the NCTM Annual Meeting, and maintained a booth to gather user feedback. EFG put together a conference on technology and teaching, which featured many of the leading experts in this area. Papers and discussion are hosted on the Math Forum.

      Chair, the Electronic Resources Committee, charged with informing NCTM members, staff, and board about technology and the teaching of mathematics. The committee has put together two minicourses for the NCTM Annual, one on new technology for teaching in primary schools, the other for secondary schools. ERC is also developing, jointly with the Math Forum, a Web service whereby teachers will send in their experiences in using various technologies in their teaching.

      The Math Forum will serve as publisher of the Journal of Online Mathematics and its Applications, an outgrowth of the NSF-sponsored MACMATC project.

  2. Facilitating User Construction, Organization, Evaluation, and Use of Web Materials

    The incredible growth of materials available through the Forum is largely a result of leveraging the activity of those who use them and participate in our services and discussion groups. One challenge is to turn these resources into structured materials that are easily used and whose value is readily apparent. Another is to cultivate leaders and corresponding models of resource development and classroom application that promote effective use of the Internet and generation of higher quality activities and projects. Described below are some of the ways in which the Forum is addressing these challenges.

    ESCOT: The Forum is a partner in the ESCOT project, coordinating integration teams in which teachers work with software developers, and curriculum disseminators to use component software to fill gaps in the curriculum where technology can play an effective role. Can the use of less costly and quicker-to-develop components by such teams lead to more significant and directly applicable materials for the classroom? We're focusing initial integration efforts around Problem of the Week type services. This is an easy environment with a ready audience, successful operations, and research already underway into the impact on the classroom and mathematical thinking; it is attractive to teachers and easy to integrate into existing classrooms and sequences. The plan is to develop a set of problems, supporting software environments and activities, that are presented over the course of a year and afterwards reside in a Web resource in which they are mapped to areas of the middle school curricula. As much as possible, the problems would be chosen to address areas of the curricula with concepts that are hard to teach or that could benefit from the use of technology.

    We will begin preparing a problem environment with each of the software developers that will be featured in the late spring Middle School POW, after the end of the scheduled sequence but before schools are out (May 24 - June). At the same time we will have recruited a group of a dozen teachers who will be working with us for the next year or more. They and others will do these POWs with their classes and we will collect information on how they fared (mathematical concepts developed, ease of use, learning strategies, integration logistics, use of presented technology, etc.).

    From this information and the experiences of our lead teachers we will prepare for a week-long summer workshop in which the teachers work with developers to plan a series of problems to run over the following school year. Each teacher will be responsible for developing approximately two problems with related curriculum mappings and integration activities, and for overseeing the mentoring for the weeks in which their problems are featured.

    During the year developers and teachers will continue working together as needed with the aid of environments such as Timbuktu, NetMeeting, and TappedIn, introduced over the summer.

    While we believe that it is important to have a close relationship between what is developed in ESCOT and the five NSF middle school curricula, the POW effort will give us some data about how much participation is driven by users of those curricula, and what the benefits are for others.

    One possible useful outcome is to incorporate the lessons and activities from these POWs into the Pages of Math curriculum resource that the Forum is developing.

    BRIDGING RESEARCH AND PRACTICE: The Forum has been awarded a subcontract to work with TERC on the use of multimedia articles (embedded video) in professional development. In the first year the Forum will be one of several groups producing articles with teachers; we will then experiment in subsequent years with online professional activities built around these articles. Approximately six Forum Teacher Associates will come together for several two-day workshops in which they reflect on videos of student problem-solving, study teacher intervention and facilitation of inquiry and the development of mathematical communication and thinking, and plan activities for their classrooms to further investigate issues raised by the video. To assess the impact on classrooms of participation in such research, the Associates will be videotaped and observed in the classrooms as part of the process. One of the key goals for the Forum is to create an online professional development environment that is grounded in discussion of actual practice and informed by research.

    Developing our Resource Center.
    The Math Forum Web site now contains over 400,000 web pages. Hundreds of items are added each month to the archives of Ask Dr. Math and the Internet Resource Collection, soon to be the Forum Internet Mathematics Library. Thousands of postings are archived for the many mathematics and education discussion groups hosted on the Forum. This year we implemented a link submission process that is actively used by the public to add another dimension to the Forum strategy of building resources through the involvement of our users.

    We are close to finishing the development of a more detailed category structure that will underlie all of our resources, adding many math subcategories, establishing a tree for pedagogical topics, and reevaluating over 5,000 entries for quality and categorization. The new structure positions us to meet the needs of users who seek Internet resources organized according to standards and curriculum sequences and to make it easier to find specifically what one wants, out of the mountain of available information.

    The availability of powerful, inexpensive database backends has increased dramatically in the past year. At the same time, tools have become available that allow these databases to be more easily and efficiently integrated into Web site design. These changes have occurred in fortunate parallel with our site growth. Some of our effort in the past year has gone into harnessing this database backend power. As stated above, this will allow us to dramatically increase the number of customers we can serve with the personnel and hardware available. It will also make it feasible to tie in all of our archives and services so that a more effective portal environment can be constructed for users, presenting information and services by level, topic, and educational needs and interests.

    There are also a number of new and updated areas presenting selected resources for target audiences, including:

    Our Webmaster function is a critical triage and community-building element to round out our Resource Center, providing a research assistant for teachers seeking to make use of the Internet and a friendly guide for new users learning their way around.

    Forum Teacher Associates.
    The Forum seeks to build leadership capacity in the teaching corps through recruitment of lead teachers and involving them in design and administration of Web projects and services. As predicted in the original proposal, the Forum is developing Associates to the point where they are able to continue their Internet-related efforts with non-Forum support. Suzanne Alejandre, recruited through a Forum advanced institute several years ago, has become a workshop leader and materials developer extraordinaire, not only for the Forum but for the Rialto schools and the SCORE project in California.

    We also seek to develop systems that increase the number of Associates that can be cultivated through the Forum, and three mechanisms are proving their worth: workshops and ongoing support for project development, moving teachers into leadership roles in Forum projects, and providing Web hosting facilities for educators producing materials or projects.

    This past summer the Forum conducted the most successful advanced institute yet, as one can see from the quality of the resources and projects developed by the fifteen participants. One of those projects, Class2Class, has continued to develop into a full-fledged Web service facilitating connections between teachers seeking to run projects between classes at a distance over the Internet.

    Three new teachers have also been cultivated as Associates to run the Elementary, Algebra, and Calculus Problems of the Week. Two teachers have been recruited to manage the Teacher2Teacher project, with one also providing support for the Middle School Problem of the week. The new research projects, ESCOT and Bridging Research and Practice, are also likely to result in new Forum Teacher Associates.

    Increasingly we are able to capitalize on the quality of support, level of traffic, and visibility of the Forum to attract users seeking to publish high-quality materials. Perhaps the most noteworthy example is Alexander Bogomolny's Cut-the-Knot, which is now hosted by the Math Forum and being integrated into our site. Stan Wagon's college-level Problem of the Week has been moved to the Forum site, and Carol Mortensen's MathWorld Interactive project is also now hosted by the Math Forum.

    Other materials developed on the Forum Web site over the last year include:

    Systemic initiatives and reform.
    The Math Forum Forum's involvement with the San Diego and Philadelphia USIs was described as heroic but "incorrectly capitalized, intellectually, programmatically, and financially. The projects in the sites, while valuable, are not structured to make a significant or sustainable contribution to the local systemic reform effort. That is, they may be appropriate as a higher ed. Eisenhower professional development program, for example, but they are not structurally embedded in the system's reform operation. The initial discussions with Art St. George last year suggested that ESR provide a higher investment in the connection. There is some merit in that line of reasoning -- a truer impact will likely require a higher level of funding."

    The Forum also conducted a workshop series in Union City with the district high school teachers. In conjunction with this effort we piloted a highly successful college visitation program, hosting a group of students at Swarthmore College for two days of mathematics and an introduction to college life. As in San Diego and Philadelphia, while well-run, the program with the high school teachers was less productive than desired, due to insufficient Internet access and readiness as well as a lack of strategic context and internal leadership to motivate application of what is learned. The plan for going forward is to shift attention to the middle school where these critical elements are more developed.

    In moving forward, the Forum is seeking partnerships with agencies working in systemic contexts where new projects or interventions would involve planning as a team from the start so that the Forum can play an integrated role, providing Internet services that are coordinated with and support other professional development efforts. Contacts have been made in Detroit and plans are being developed with SRI (TappedIn) for a project with Kentucky.

    Professional development programs.
    While a number of projects with which the Forum planned to work have faded away, such as the Online Internet Institute, we have continued to conduct a number of workshops, including:

    • Math Forum Advanced Summer Institute
    • Council of Islamic Schools in North America Second Mid-Atlantic Conference
    • Sr. Clara Muhammad School, Philadelphia
    • AERA annual meeting
    • NCTM Regional Conference
    • NJ Math Teachers Conference
    • Toyota TIME grant workshop at NCTM in VA
    • MathConn at Cedar Crest College
    • NCTM Conference within a Conference at the annual meeting
    • Precalculus Conference at Dimacs
    • Discrete Math Leadership Program at Dimacs

    These workshops provide valuable opportunities for the Forum staff and evaluators to understand the needs of teachers and students and to observe the challenges faced in using the Math Forum effectively. Some of them also connect us to the research communities in mathematics and education, opening lines of communication and collaboration. Through such efforts the Forum is able to maintain a high profile and effectively network in order to promote use of the Forum and other NSF-funded activities.

    College and university contributions.
    The Math Forum had two projects with students developing Web pages. One, "Famous Problems in Mathematics," has been favorably cited and linked to by outside groups such as NCTM. The other, an elementary discussion of how to use the two most popular computer algebra systems on the Web, is still under development.

    Gene Klotz also attended the Sketchpad User's Group meeting at the AMS/MAA Annual Meeting and another session on retirement, telling them about opportunities at the Forum. He regularly cultivates contributions to the Key Issues pages and tries to maintain good relations with folks like Dick Askey, Davide Cervone, Andrew Odlyzko, Jeff Weeks, and Laurie Snell.

  3. Online Educational Mentoring and Professional Development

    Rather than pursuing the development of online courses and formal learning environments, the Forum has focused its professional development efforts on informal education through activities such as Teacher2Teacher, as well as through extension of key services such as Ask Dr. Math and the Problem of the Week. The goal is to create a culture of learning and self-improvement through the online community that supports efforts elsewhere to professionalize teaching with better knowledge of mathematics, reflection on practice, infusion of research, and sharing of ideas and materials.

    Professional development programs.
    Our development of online professional development programs includes our work with TappedIn, as described above, as well as on the materials prepared for and resulting from the work with the summer advanced institute. See the Teacher Associates section earlier in this report.

    We have focused a lot of attention on Web-based discussion groups using our Epigone software, to which we have added moderation features. Many organizations and users want effective online discussions rather than the free-wheeling forums that seem to dominate. We are working closely with many such as NCTM, CESAME, and others who recognize that focused, productive discussions require seeding with thoughtful postings and a protected environment that wards off personal attack and irrelevant messages. Our moderated environment also enables discussions to be reorganized and edited as lines of conversation are defined. The next set of features will hook these discussions into the same category structure defining other archives so that we can study whether such organization widens participation while establishing focused communities of interest.

    Pre-service education.
    John Olive's group at UGA has administered a revised version of last year's survey to all the student teachers BEFORE they went to their field placements this semester. They are currently analyzing those responses and following up with key students at their field placement. They will also follow-up with those students who have indicated a willingness to be interviewed. Tom Cooney will be leading the effort on the analysis of the interviews and surveys this coming summer.

    The students' responses to the revised questionnaire, administered last semester before these students went out to student teaching, indicate a strong intention to use technology with their students during student teaching whenever possible, with Geometer's Sketchpad (GSP) being the most preferred software. There are also generally positive attitudes towards the Math Forum although not as clear a message regarding use of the Forum with their students. Almost all of the math ed students have made use of the Forum as a support for their own learning, not only in Math Ed courses. Access to the Internet from their field placement schools during their student teaching is still a barrier for several students. UGA is confident that this will change over the next couple of years as both the Federal and State governments have universal access as an educational priority.

    Building on this investigation, we are planning a pre-service mentor program for our Problems of the Week through which college and graduate students intending to be teachers can have early interactions with K12 students that will enable them to study mathematical thinking, practice communication, and design activities without having to arrange for on-site access to classrooms.

    Mentoring services for students.

    Ask Dr. Math has experimented with several strategies for scaling, and has successfully doubled its performance over the last seven months. Currently we answer over 2,000 questions a month, with many more users benefiting from the over 4,400 archived responses. We attempt to add more value by writing FAQS that present more thorough and organized treatment of the more commonly asked questions, including: Geometric Figures & Formulas, Learning to Factor, Learning Multiplication Facts, Dividing by 0, False Proofs, Number Bases, Proofs, Analytic Geometry Formulas, Trigonometry Formulas, Polar Coordinates, Cylindrical Coordinates, and Spherical Coordinates.

    The two most promising directions for meeting increased demand are in recruiting and tenuring more volunteers with the addition of administrative staff, and in enabling questioners to use the archives more effectively. "Math doctors" can't answer enough questions from scratch to make it economical to scale up through hired respondents, at least until there are improvements in the archiving system, related triage functions that can direct users to existing resources, and ways of identifying questions requiring personal attention. Hired administrators may not only have more of a multiplier effect but also better support improvements in the pedagogical qualities of responses.

    At a certain point we contemplate experimenting with fee-based services for professionals using Ask Dr. Math in the workplace and for users requiring faster response rates. In the meantime, we are planning a research project with Dr. George Tecuci, at George Mason, using his Disciple system to develop agents that would aid in directing users to appropriate resources for the questions. The Forum is also now refocusing on its Pages of Math experiment to develop a more coherent inquiry environment out of the sort of material found in the Ask Dr. Math archives.

    Another strategy that may serve this project is to cultivate existing doctors as local administrators, responsible for recruiting and supervising their own teams. One doctor in a government job with community service requirements experimented with this approach and was able to recruit five new members, three of whom remain involved. It remains to be seen whether this approach is more widely applicable.

    We have also begun to publish an internal math doctors' newsletter and to design a community center to acknowledge doctors making substantial contributions, enabling them to share effective responses and build the quality of the service through reflection on their practices.

    Problems of the Week: The Problem of the Week Project was designed to be both an enrichment opportunity for students and a way in which teachers and students could be engaged in activities reflecting evolving standards for mathematics education. The objectives include:

    • providing non-routine problems
    • improving problem-solving skills
    • encouraging multiple solution methods
    • fostering mathematical communication
    • building an online mathematics community

    As you will see in the numbers and comments presented below, as well as in the evaluation reports, the project has been successful on all of these counts. The critical challenges currently facing us are to increase the impact of the POWs on professional development and to implement systems to handle increases in participation.

    During the spring and summer of 1998 a major revision of the puzzler software that runs the Problems of the Week project was undertaken. We took a look at input from teachers and students using our project and realized that the greatest user need was to simplify the submission process. We have accomplished this by requiring the students to submit their school, name, age, class, and email information only once. On subsequent visits they can find their name on a pull-down menu.

    With the increased ease of use, the addition of Algebra and Calculus POWs in the Fall of 1998, and the tremendous traffic to the site it has become necessary to rewrite the entire puzzler system again to speed up the process. Work on the new system began in November and continues. Testing is expected to begin in late March. The biggest improvement is expected to be speed and easing of the server load, which will in turn improve the working conditions for the mentors.

    As the POW project pulls in more students we need to find ways to scale up the mentoring end of the project. This year we have introduced a cavalry program. Our cavalry mentors can gain access to the office and reply to student submissions at any time, rather than being responsible for a week's problem. They choose the number of submissions that they can answer at a given time.

    We are also working on a certification program for pre-service teachers. In this program we will award a certificate to pre-service teachers who complete a prescribed program of mentoring. We are doing some work this year following pre-service classes who are mentoring, and expect to have a program ready for September.

    One last thing that we have been trying with the POWs this year is to teach students how to use other Forum resources, particularly Dr. Math. When terms that are unfamiliar are used in problems, we suggest they look in the Dr. Math archives, or we provide links to specific Dr. Math threads. Archive comments will also lead students to discover more about specific problems.

    The numbers listed here reflect the number of submissions starting with the problems posted Sept 7, 1999

        Elementary Problem of the week:
            Total to date: 7131
            Average per week: 310
            Greatest number of submissions in one week: 612

        Middle School Problem of the week:
            Total to date: 10020
            Average per week: 436
            Greatest number of submissions in one week: 808

        Geometry Problem of the week:
            Total to date: 10305
            Average per week: 448
            Greatest number of submissions in one week: 815

        Algebra Problem of the week: project began week of Oct 5
            Total to date: 1629
            Average per week: 86
            Greatest number of submissions in one week: 185

        Trig/Calculus Problem of the week: project began week of Oct 5
            Total to date: 377
            Average per week: 20
            Greatest number of submissions in one week: 54

            Elempow - 9 cavalry mentors; 16 active mentor groups
            Midpow - 15 cavalry mentors; 3 active mentor groups
            Geopow - 16 cavalry mentors; no mentor groups
            Algpow - 8 cavalry mentors; 3 mentor groups
            Calcpow - 1 cavalry mentor/administrator; 6 active mentor groups
            [where mentor groups take responsibility for a given week's problem]

    Additional POW facts:

    • many new countries reached (Singapore, China, El Salvador, Croatia, Greece, to name a few) in addition to regular participants from Turkey, Canada, and Australia.

    • expanding & improving the mentor program - Elempow mentors include classes of students from sixth grade through preservice teachers. We also have teachers, former teachers, and "interested others" working as cavalry mentors, and two younger cavalry members - sixth grade (continuing from the enjoyment she found when her class did it in Sept.) and eleventh grade (a young man interested in sharing his love of math). This type of mentoring is now possible with the clearance levels we added this year. This learning experience is invaluable at both ends of the scale. Younger mentors learn what it's like to "be the teacher," learning about problem development, creating grading guidelines, and seeing different approaches to problem solving. Preservice teachers are given the opportunity to interact with students before entering the classroom and to experience responses from/to students with all levels of ability.

    • teachers using the elempow in their classrooms have commented on the benefits: the opportunity to use technology, interactive problems, math outside the classroom, teaching students to revise if not correct the first time, resource for elementary teachers not so mathematically inclined, etc.

    • we are also being used by a fair number of home-schooled kids to supplement their math.

    We have had comments from teachers including:

    • I enjoy the problems and have used many in my classroom. They are great for non routine problem solving.

    • ... It has greatly increased their awareness of what it takes to be a good problem solver and we are making progress with each problem we attempt. We really enjoy reading the solutions from other groups and this gives us ideas for our own work.

    • Thank you for the time and effort put into the site. My kids like the fact that they are on a national scale.

    • I have given students the go ahead to use your site for independent extra credit work.

    • ... a big thank you to everyone involved with this site. The kids are getting hooked and really enjoy the problems.

    • Congratulations for this site in the Internet, it's very interesting and practical. Unfortunately, here in Brazil, we have so many structural problems that this kind of projects are not found here. I am very happy with my participation in this project. Thanks for all.

    • Our students have really enjoyed participating in your Elementary Problem of the Week and have benefitted from it, especially in learning how to write a narrative to explain the logic behind solving a math problem.

    • My students are very excited about getting a response from you.

    • Thanks for a really challenging site that has added to the interest in mathematics

    • I have just come across your Problem of the Week site and I'm much impressed. I am a mathematics teacher, specializing in enrichment, and I will certainly get my students involved with your site. I had always wanted to put together something similar but I doubt that I have the expertise to do it as well as you!

    • I am a fourth grade teacher and I am really impressed by your Web site. I feel my students will benefit from an interactive opportunity to express mathematical ideas and be challenged in math.

    • ... We were looking for problems to test our students' ability to understand and respond to problems written in English. The problems were probably not intended for this purpose, but I find some of the them quite suitable. (from an instructor at a University in Japan)

    • I want to thank you again for providing such a wonderful program

    • This is a great problem! Thank you for the service to our students. I am referring to: <>

    • I love the POW. I use old ones in class for challenge problems weekly and we also have a math club after school that is participating each week. It has greatly enriched our math program. Thanks.

    • Mostly, I want to thank you and your co-mentors for your thoughtful replies on the POW's. The students really enjoy hearing what they did right or wrong. Parents, also, are very impressed by the Math Forum. Thanks for your support, educationally!

    From parents:

    • I just wanted to take a sec to say thank you for this project. My son entered the midpow forum for the first time, this past week. He is also working the same problems in his advanced math class at Benold Middle School of Georgetown, Texas. He loves working problems and, as his father, I wish to extend my support and gratitude.

    • Thank you again for all the encouragement you gave him last year. He was so thrilled with his certificate. I put it away with all the "special" papers that parents save.

    From students:

    • I really like your idea about having the one a week thing for math problems. This week the problem was fun and I wish to play more. My math teacher thought about assigning us these every week. We think it will be fun.

    • I would like to know if you can make next weeks problem a little easy. Thank you if you can help. Answer back as soon as possible. Please!

    • I think that this contest is a great idea and as my math teacher says, It gives the math teachers and kids new problems that are not seen too often.

    • My advanced geometry teacher handed my class a question from your Web site. Eventually, I found the time to check out your site and see it for myself. I love your site!! It's a very great site dedicated to something I love: geometry! I hope you keep up the great work.

    Teachers mentoring teachers.
    Teacher2Teacher was officially launched March 11, 1998 in cooperation with the Council of Presidential Awardees in Mathematics (CPAM). Nearly 200 questions and answers have been archived since then. The service has been growing slowly, in large part because CPAM is still developing as an organization and this project has not received sustained organizational investment. While awardees Claudia Carter and Mary Lou Derwent, among others, have provided valuable leadership, the Forum has decided to expand its role in this project in order to provide increase support for the volunteers staffing the service and to drive the reflection that may lead to higher quality in the responses. We are also considering the expansion of this service by recruiting teachers from outside of CPAM. Others have expressed interest and we have seen among some of the CPAM members, that participation can be important as another professional activity on resumes when seeking jobs and other professional opportunities.

    One of the key elements of this service, public discussions attached to archived questions and answers, has already indicated the potential value of this strategy for generating more focused conversations on the Internet when grounded in concrete teaching issues and substantive interactions. Additionally, the new ask-an-expert software implemented here has also provided a testing ground for automated retrieval of appropriate archive entries when questions are submitted.

    Curriculum Development Projects.
    Of the three curriculum projects that had committed to work with us in the original proposal, only Investigations has followed through, and we recently completed a discussion group area for CESAME as part of their professional development environment supporting implementation of the curriculum. We also developed some professional development Web pages for Key Curriculum's IMP project.

    While these publishers may be less than gung ho about the Web, there are many signals that publishers are looking for supplemental services, such as help/tutoring services to differentiate their products from others, and they are anxious about the Internet leading to a change in the textbook market. The Forum is well-positioned to capitalize on our lead in Internet mathematics services and to make good use of our active community in meeting the needs of publishers and providing opportunities for teachers and students to influence the development of resources to suit their needs.

    Summary: The Math Forum Evaluation, Year 2
    Report of the Local Evaluation Team1

    1Norman L. Webb is the external evaluator for the Math Forum.
    Hard copy of his report was submitted in the Fall of 1998.)

    K. Ann Renninger
    Wesley Shumar

    Evaluation of The Math Forum (NSF Grant: The Math Forum, #9618223) has two components: a quantitative formative (and to some extent summative) component that addresses user participation, contributions, and integration of Internet activity into math education, and a more qualitative component that tracks use and change in the virtual community. Evaluation is used in program design decisions, not just for analysis of final results.2

    2A combination of methods is employed in the evaluation of the Forum. These include: ethnographic evaluation, structured interview, focus groups, and open and forced-choice questionnaires. In addition, traditional qualitative methods are stretched to include: (a) tracking the "trails" left by Forum users, and (b) analysis of change in students' participation in the Forum's interactive projects such as Ask Dr. Math and the Problems of the Week. Additionally, a subsamples of teachers and their students are being studied in more depth through classroom observation, interviews, and face-to-face and online discussions with other teachers about specific topics in mathematics, including student learning and instructional practice.

    Data on which this summary draws include: questionnaires addressing general project usage, POW usage by teachers and by students, Dr. Math usage by teachers and by students, Math Dr. feedback, feedback from those who received answers to Dr. Math, feedback from those who did not receive answers from Dr. Math, and feedback from users of the searcher; interviews with teachers and forum staff; workshop observation and feedback; and staff focus group summaries from spring and fall.

    Questions guiding evaluation of the Forum include:

    1. Does the Forum provide a meaningfully different type of learning tool/enhancement for mathematics education?

      1. What kind of a resource does the Internet provide for teachers, students, and/or parents and other citizens?

      2. What kind of facilitation, adjustments of existing resources and/or mentoring relationships need to be made to meet the strengths and needs of different users?

      3. What is the impact of the Internet on the development of mathematical thinking?

    2. What role does the concept of community play in Forum development? Additionally, what leads to institutional sustainability and growth, volunteer participation and collaboration, and partnerships for a virtual resource center?

      1. What are the usage and changed usage patterns of different groups of participants?

      2. What adjustments are necessitated by the pressures of scaling up and simultaneously developing a self-sustaining infrastructure?

      3. What is the impact of scaling up on project development, partnerships, and Forum staff organization?

    3. Taking the Forum as an exemplar of a link between research and practice, what new potential does the Internet provide for linking research and practice? What is the relation between research in both mathematics learning and student cognition and ongoing practice as reflected in Forum-based: project development, student learning, curriculum development, professional development?

    Evaluation Summary

    The following is a summary of data collected in year 2 during the evaluation of The Math Forum. The summary addresses in turn each of the research questions guiding the evaluation process.

    1. Does the Forum provide a meaningfully different type of learning tool/enhancement for mathematics education?


      From its beginnings, users have reported that the Forum provides them with resources that they have not previously had readily available. Users also report the importance of using these to address their own questions.

      The Forum models ways in which to practice reform mathematics. It articulates a language for thinking mathematically and it endeavors to enable its users to make connections between their experience with math and the contents of the Forum.

      The limitations of the Internet have afforded the Forum with a particular kind of strength. It stretches users to find a language that can be shared as they interact across the Internet. This leads to an attention to language, the need for connections, and perspective taking that has an impact on users' mathematical thinking.

      The Forum provides opportunities for a wide range of users to engage the Forum since it includes a variety of levels of mathematics and types of projects. Links between pages on the site enable users to grow in the level and nature of their work with mathematics.

      Presumably because scaling up at such a rapid rate has entailed the need to enlist the help of others, there has been an increase in the numbers of users who are also contributors to the Forum. There continue to be contributors who are very creative, some of whom even bring their own projects to the Forum. There also is an increasing cadre of people whose contributions are recruited to fill particular roles (such as helping out with the Geometry POW when Annie, its facilitator, is overwhelmed with too many responses).

      1. What kind of a resource does the Internet provide for teachers, students, and/or parents and other citizens?


        The Forum's work with teachers suggests enriched and cost-effective opportunities for professional development. It provides teachers with a wide variety of mathematics and examples of mathematical thinking. It supports teachers in improving their reflection on the mathematics they teach and the process of their instruction. It informs teachers about new development in technologies and their application in the classroom. It also provides an infusion of reform practice since Forum projects integrate research findings in practice. From the perspective of involving teachers in reform-based mathematics instruction, the role of the Forum as an extension of rather than a replacement for curriculum has meant that teachers could incorporate Forum resources into their classes as they are ready to do so. The richness of the possibilities and the interest of their students appears to enable them to increase their use of these resources over time.

        Teachers overwhelmingly report that the Forum provides them with materials that they can download and use in their classes right away. Their big complaint is time and the Forum provides them with tools they want--interesting mathematics and options for teaching it--when they need them. Teachers appreciate that the Forum allows them to interact with others, to ask questions, and share ideas with others about their own work. They also like the fact that the Forum provides different kinds of learning environments for their students, environments that encourage them to seek resources, problem-solve together, and to think about the mathematics in their lives. Speaking of the POW in particular, teachers cite the problem solving required as increasing their students' problem solving skills, making them better able to approach new problems and explain their solutions.


        The Forum provides students with a resource that enables them to seek out challenging mathematics and engage in conversation with mathematicians. Questionnaires directed to users of the POW indicated a substantial number of students submitting answers to the POW outside of class assignments.

        Students report that they like finding resources on their own. They can find people who like math and develop relationships with them. They are able to ask questions that would be hard to ask in a class. They discover that they really like math. They also find that the Forum provides an alternative to their teachers and textbooks.

        Parents and other citizens

        The Forum provides parents and other citizens with a resource that is informed by research and enables them, like the teachers, to work with mathematics at a variety of levels and develop their mathematical thinking. It informs them about developments in new technologies. It also enables them to become informed about reform-based practice.

        Parents report that they appreciate that the Forum gives them a way to help their kids that they have not had before. They like being able to use good math resources and they appreciate being able to think about math.

      2. What kind of facilitation, adjustments of existing resources and/or mentoring relationships need to be made to meet the strengths and needs of different users?


        The Forum continues to provide two types of workshops for user groups. These include workshops that provide a broad introduction to use of the Internet and the Forum and more focused advanced user workshops that address new technological developments and their applications to classroom projects. The Forum has run regional workshops for NCTM, the USI clusters in San Diego and Philadelphia, and groups from schools and districts in many states. Their more advanced workshops are conducted in real time, and also are subscribed to by online users. The content of these workshops is archived.

        While the Forum has found that some combination of face-to-face interaction and virtual interaction is most beneficial in terms of enabling users to engage and contribute to Forum projects, recently a great percentage of its contributors have not had face-to-face interaction with Forum staff. One of the strategies that continues to work for them as they scale up is to recruit teachers (for the POWs) and citizens (mathematicians, pipe fitters, etc. for Ask Dr. Math) to assist them by providing well-defined contributions. Contributions such as those from the calvary that assists the Geometry POW, do not require that one "know" the Forum way of doing something necessarily. Rather, the archive provides effective models of how to handle the job and the staff provide a check on its execution.

        Adjustments of existing resources

        In addition to recruiting assistance for projects, developments on the programming side of the Forum have meant that projects provide teachers with assessment tools that they have not had in the past. Thus, for example, teachers whose students are using the POW are able to focus in on one students' contributions over time, compare their students' contributions by problem, etc. Similarly, the searchers are enhanced to cope with misspelling and misuse of terms enabling users to find what they seek (as well as the spelling and correct terminology), and the archive and resource center have each been organized to enable people to more easily access resources. Each of these developments has enabled users to have enhanced use of the site, and enhanced opportunities to develop their understanding of mathematics.


        Another strategy for scaling has involved cross-age mentoring projects. The Forum staff is experimenting with different models of mentoring projects. In the Elementary POW, for example, cross-age tutoring has been a very effective experience for both the classes of students who are mentoring the POW as well as the students who are doing the POW. Both the Middle School POW and the Geometry POW now include back-up "calvary" to help handle the hordes of problems received. In the POW projects, Forum staff oversee mentoring by classes and cavalry members and intercede to enable their helpers grow as mentors and to ensure that the users are encouraged to grow as well.

        The Ask Dr. Math project, on the other hand, has a tenuring process in place that means that the "Doctor" has previously demonstrated his or her ability to engage a student in thinking about the question posed--as opposed to providing "the" answer--or, is considered to be "in training" and does not directly answer students but instead works with a staff member to develop answers. For each of the projects, the archives provides a record of mentoring that enables teachers and others to learn from others, answer their own questions and/or to clarify them.

        In addition to explicit training and oversight in responding to students/users, data have also been gathered to provide project staff with information about users' needs from their mentors. Findings from study of student responses to the POWs, suggest that there are developmental differences in the type of feedback students are able to use. At the younger ages, and/or when students have little interest in mathematics, they like appreciation for effort only if they get the answer correct. They also like suggestions about how to work on a problem, and getting information that extends what they know. Older students, and those who are both correct and incorrect in their problem solutions, on the other hand, appreciate acknowledgment of their efforts in problem solution whether they have been accurate or not. Those who are accurate also are interested in receiving suggestions and getting information that extends their knowledge.

      3. What is the impact of the Math Forum on the development of mathematical thinking?

        As an extension of classroom learning, the Forum provides a particular set of experiences that cannot readily be obtained in a classroom or through work with a text. In each of the projects, a person is asked to explain how he or she arrived at an answer (or they are asked to do so after they submit an answer). For the Ask Dr. Math project, users are encouraged to explain not only the nature of the problem they are asking, but how they have tried to answer the problem, including specifying the resources they've used. Forum users are individually accountable for explanations and encouraged to ask questions and engage in discussions. They are encouraged to use resources and to explain how they have done so such that another person might be able to use this information to solve the problem. They also are able to peruse the solution pages, lesson plans, etc. of others, in turn, finding models for themselves. The design of the projects and their facilitation are intended to enhance mathematical thinking. Thus, in order to talk about the impact of the Forum on teachers and students and other users, we have begun studying in their mathematical thinking.

        Among students and teachers, indicators of mathematical thinking include the ability of the individual to connect math to past experience and the real world, to identify multiple paths for problem solution, and to describe solutions in a way that is reproducible. From in-depth interviews with teachers, it appears that change in mathematical thinking is enabled by: interest (the stored knowledge and stored value that an individual brings to his or her understanding of mathematics), a willingness to take risks, and resilience. Goals for project design and facilitation include enabling those who lack interest in mathematics, risk-taking, and resilience to develop these characteristics.

    2. What role does the concept of community play in Forum development? Additionally, what leads to institutional sustainability and growth, volunteer participation and collaboration, and partnerships for a virtual resource center?


      The Forum represents a vision about the possibilities for an Internet community that extends the collegiality found in schools, classrooms, or the work place. The Forum's use of the term "community" stems from a sense of what could be and this is the way it is engaged by its users. In this sense, the Forum is an intentional community. For the staff, the concept of community translates itself into recognition of the importance of stimulating involvement and investment, a sense of belonging, so that users feel comfortable contributing and sharing materials with others. In its project design and facilitation, the Forum staff has taken leadership in modeling ways to engage mathematics, articulating a language for thinking mathematically (and developing this thinking), and making connections between a person's experience with math and the math in the real world. The Forum has provided many different ways for people to interact with one another. Each of these reflects different points of access for people of differing strengths and needs.

      In terms of its response to the incredible pace at which the Forum community has grown, there is increased awareness among project leaders that scaling strategies are essential to meet the demands for their projects and service; and sustainability is a recognized necessity. At the same time, the emphasis and purpose of the Forum continues to focus on providing high quality mathematical resources and instructional support for users. Needs for scaling up, however, have required additional support from both programmers and users of the Forum community.

      1. What are the usage and changed usage patterns of different groups of participants?

        Across the board, there has been a tremendous increase in use of the Forum resources by teachers, students, and parents or other citizens. This size has doubled again in the last six months to 180,000 visitors per week. Several of the POWs, for example, have received their 10,000th submission for the year.

      2. What adjustments are necessitated by the pressures of scaling up and simultaneously developing a self-sustaining infrastructure?

        With increased usage and the need to scale up, the Forum staff has rallied. There is an emphasis on efficiency, the need to meet scaling issues, a concern for system, order, and communication. One of the hallmarks of Forum work in the past was its creativity. The staff were a group of bright people, who were supported in pursuing their own projects. There continues to be license for and appreciation of staff members' creativity and ability to take initiative, but this now is channeled in the direction of enhancing existing projects the staff are focused on making scaling work, and are building issues and learning that emerges from this effort into new collaborations/partnerships.

        As part of the evaluation process, the staff engages in focus group discussions to address goals and think together about emerging concerns. These discussions have enabled those working on different aspects of projects to keep abreast of developments and to work in concert. (Over the past year, almost as a matter of practice, regular meetings of these groupings have begun to be held that are facilitated within group.) The focus group meetings are used to generate questions for users (as well as information that users need to know) and to identify other sources of data to answer their questions. Once these data have been collected and summarized, they become the subject of subsequent focus group discussions.

        Of particular concern in the process of gathering information about the design and facilitation of the site has been the issue of how to get users to read directions. For example, it is a huge complication for the Ask Dr. Math project, if a teacher decides to have all five of his or her classes each send individual questions in to Dr. Math. The complication is that there are a lot of questions that are not genuine and that typically have not involved any resource gathering/effort on the students' part--one of the requirements in submitting a question is to explain where the complication in one's understanding is and the efforts you have taken on your own to answer the question. The directions for the project clearly provide teachers with information about what makes a good question, but many teachers were seemingly unaware of this. With scaling up, this kind of misuse was causing problems for most of the projects. A strategy that has served to alleviate a lot of these problems has been to nest information about effective project use into the questions that are posed to users. The strategy of nesting information that the staff needs its users to have was also used recently to provide users with advance notice of its decision to move to Web-based availability of the POW.

        The focus group methodology, together with the staff's independent interactions with users, enable the Forum staff to be clear about the needs of users. It also ensures that their knowledge of how to work with users continues to improve. In response to questions about scaling and usage generated by the focus groups and examined through questions to users and "tracking" users' mistakes, the archive has been recataloged, an index and site map has been developed to provide a better structure for the site, and searches have been made more powerful and able to account for inaccurate spelling and logic.

        The Forum has continued to create and hone different ways for users to interact, linking discussion lists to Web pages, making services easy to browse, developing partnerships that involve teachers in focused discussions about particular mathematics problems, the development of revised Standards, and feedback to programmers about teachers' needs, etc. Research on interaction suggests the importance of people being able to see each other in order to interact. The Forum is working to solve problems emerge in interactions across the net. They too have found that a combination of face-to-face and online support is useful in community building, but they have an ever-increasing pool of users who do not have face-to-face contact and they are exploring how to make this work.

        In their own practice the Forum staff models efforts to find the possibilities afforded by such problems as no face-to-face interaction. This kind of problem-solving has spawned online versions of workshops and use of Tapped-In for focused discussion sessions. This vision or sense of possibility is recognized implicitly if not explicitly by the teachers with whom they work. Most importantly, perhaps, it is recognized by both high-end teachers and those who barely have the technological and mathematical skills to use the Forum well.

        Part of scaling up and developing a self-sustaining infrastructure has required a redefinition of the Forum's relationship with Swarthmore College as a business partner. The focus of this partnership is growing the Forum.

      3. What is the impact of scaling up on project development, partnerships, and Forum staff organization?

        Project development

        The immediate impact of scaling up is that scaling has become an ongoing concern for the staff. Scaling up has led the staff to place in an emphasis on efficiency as enabled by programming, increased solicitation of volunteer support, and strategizing about how to maintain quality. Knowledge of the needs of users has enabled programming to be responsive to users, as well as to needs of project staff in their efforts to recruit and train increased numbers of volunteers.

        The Forum pulls people into the community by making space for them, whether this is space to learn and/or space to create and through this to grow. With an increased number of discrete doable volunteer jobs, there are more possible ways in which people can do work with the Forum. The archives provide a database that facilitates the training of individuals to work on Forum projects; programming developments facilitate their oversight.


        Initially, the Forum was in the position of courting partnerships. Some of these partnerships have remained at the level of shared links, while others have developed into research collaborations where the Forum contributes to projects that necessitate input from teachers and which, in turn, afford them the opportunity to learn more about teachers' needs, how to facilitate and seed discussions, and the use of its own projects.

        Forum staff organization

        The Forum staff has had to address scaling of its staff as well as the scaling up of its projects. The Forum as a site has grown and so has the staff that supports it. Although the office moved from two rooms adjacent to the Mathematics and Statistics Department at the College into its own house, the staff numbers too many to all work on the premises. Some come in regularly, others irregularly. As noted above, the staff are focused in their efforts. These efforts, however, have become more complex than they were even a year ago. The programmers share a common language and different programmers have added specialties which lead to challenges in maintaining communication across projects. The specifics of each project's coordination have been delegated to a particular individual or group of individuals. While their work bears on that of others, and they are all contributing to what the Forum community is and can be, the staff's work efforts are increasingly streamlined.

        Weekly lunch presentations during which different efforts at the Forum are summarized and current questions and concerns specific to these identified provide staff with a sense of what others are doing and creates for them a sense of the Forum as a whole. Focus group meetings provide a similar function, although these typically are attended only by those working on the concern or topic under discussion.

    3. Taking the Forum as an exemplar of a link between research and practice, what new potential does the Internet provide for linking research and practice? What is the relation between research in both mathematics learning and student cognition and ongoing practice as reflected in Forum-based: project development, student learning, curriculum development, professional development?

      Use of the Forum by definition involves dissemination of research about reform-based mathematics. Forum project design and facilitation reflects Standards-based suggestions for both curriculum and instructional practice. It provides models of tasks informed by research for its users. It also provides opportunities for users to learn through apprenticeship at a pace that matches their readiness. The Forum, perhaps because it is an Internet resource, has been able to "break set" with traditional practice by introducing reform-based instruction while maintaining a high level of quality mathematics.

      The Forum offers students and teachers opportunities to engage with reform mathematics in a variety of ways. Models and resources necessary for learning and teaching are available and easily searched. They are available when a person is ready to work with them and a person can take as much time as he or she needs in order to do so. The facilitators of projects (as well as the archives and the resource center) work with each person submitting answers or questions to understand or explain their answers/questions and encourage them to be resourceful about using the other types of support that are also available to them on the Forum in their work. Teacher2Teacher and Class2Class, two recently developed Forum projects, also provide opportunities for people to discuss or follow discussions concerning what is known about learning and mathematics. Since the projects and resource collections are continuously being updated, this means that they too provide users with a way to stay abreast of research developments either explicitly through the discussion groups, or more implicitly in the projects.

      The Forum continues to cultivate and strengthen its partnerships with groups that are interested in bridging research and practice. These collaborations between teachers and researchers about mathematical thinking (with TERC), teacher discussions with programmers on component software development for curriculum enhancement (with SRI), and teacher and professional organizations about the revised Standards (with NCTM) include opportunities for the users of the Forum to continue to enhance their understanding of mathematics and current issues in reform pedagogy through the process of thinking with others. Study of their participation, in turn, informs the Forum about needed directions they might take in their reform-based practice: how to most effectively mentor others, seed discussions, and so forth.

      The use of focus group methodology as the basis of the Forum's formative evaluation is informed by research on the power of metacognition in learning. Reflecting on action has become a conscious strategy for addressing the complexities of scaling. It serves the purpose of enhancing the staff's reflective awareness of their practice and enables them to continue to work together as a unit despite increasingly singular contributions to the Forum enterprise. This methodology also serves to involve the staff in the kind of engagement that they in turn are facilitating for their users.

      Information about the Forum has largely spread by word of mouth. For example, a teacher finds the Forum and downloads one of the Problems of the Week (POW). Another teacher sees that students are challenged and engaged in doing the POW and asks where she/he can find it, and so forth. Information about the Forum has been the focus of news articles and practitioner reports in journals; it has also been disseminated through: the Forum newsletter, Forum-sponsored professional development workshops, and talks by Forum staff to practitioner-based organizations. This type of outreach has led users to talk with others about the possibilities they see in the Forum projects, including the resource center and archives.

      Results of Forum evaluation efforts have intentionally been disseminated in two ways: to the Forum staff and its partners who then in their design and facilitation of projects model and/or apprentice users in reform-based practice, and to professionals. Formative data collection has been instrumental to Forum staff members' work on project design and facilitation. Forum staff members regularly work together with the evaluation team in focus groups, questionnaire development, and interpretation of data. Thus, data collected reflect the questions of the staff, as well as those that the evaluation team have introduced based on research findings in the fields of anthropology, cognitive science, developmental psychology, educational psychology, evaluation, and the learning sciences. These data provide the basis for decision-making about the design and facilitation of projects, as well as the collaborations undertaken with partners.

      Forum evaluation efforts are being disseminated to professional audiences through conferences and journals. An edited volume, Building Virtual Communities, is in preparation. Central to this dissemination effort is a concern for bridging what have been discrete research literatures, and articulating links between theory, research, and practice.

      Training Researchers:

      Forum staff have become practitioner-researchers because the design of the evaluation involves them fully in the formative evaluation process. They help to identify research questions, design instruments, and interpret data.

      Teachers who collaborate with Forum staff as cavalry members, class mentors, and/or Math Doctors have also become practitioner-researchers. Each of these roles involves identifying user strengths and needs and the nature of the scaffolding necessary to enable users to grow in their mathematical understanding.

      Teachers and evaluators who participated in Forum sessions on using multiple methods in evaluation research at the CREATE conference were able to ask questions, conference, and correspond with Forum evaluators who are working with these methods. The methodology of the Forum evaluation also is a case study for the Swarthmore College Honors Seminar, Child Psychology and Practice.

      Anthropologists working to describe the use of ethnographic methods to study virtual community building have learned from panel discussions on this topic held at both the International Congress of Anthropological and Ethnological Sciences and the University of Pennsylvania Ethnography Conference.

      College students, most of whom are pre-service teachers, work as research assistants on Forum evaluation. Through this they learn about the role of formative evaluation in informing project design and facilitation and they further develop their skills in quantitative and qualitative methods.


      Renninger, K. A., Weimar, S. A. & Klotz, E. A. (1998). Teachers and students investigating and communicating about geometry: The Math Forum. In R. Lehrer & D. Chazan (Eds.) Designing learning environments for developing understanding of geometry and space (pp. 465-487). Mahwah, NJ: Lawrence Erlbaum Associates.

      Renninger, K. A. & Shumar, W. (1998). Why and how students work with The Math Forum's Problem(s) of the Week: Implications for design. In A. S. Bruckman, M. Guzdial, J. L. Kolodner, and A. Ram (Eds.), Proceedings of ICLS 98 (pp. 348-350). Charlottesville, VA: AACE.

      Renninger, K. A. & Shumar, W. (Eds.) (In preparation). Building virtual communities. New York: Cambridge University Press.


      The Math Forum as a project integrates research and practice in project development and facilitation. As such, users directly engage models and apprentice with those concerned about best practices. In their outreach, the PIs and evaluators have participated in focus group discussions and the review process for manuscripts for NSF, OERI, NRC, school districts, academic journals, and scholarly panels. In each of these contacts, Forum staff use their own practice as a basis for contributing to thinking about educational practice, standards development, and student learning.

      Focus Group Discussion, Mathematica Policy Research, U.S. Department of Education-sponsored study on knowledge transfer. February 18, 1999.


      Renninger, K. A. & Shumar, W. (July, 1998). Combining methods to inform practice at the Math Forum. Paper presented as part of the National Evaluation Institute. Denver, CO.

      Renninger, K. A. & Shumar, W. (December, 1998). Why and How Students Work with The Math Forum's Problem(s) of the Week: Implications for Design. Poster presented at the International Conference of the Learning Sciences. Atlanta, GA.

      Renninger, K. A. & Shumar, W. (April 1999). Interest, risk taking, and resourcefulness as indicators of change at The Math Forum. Paper to be presented as part of the symposium, Indicators of change in computer-based community building.

      Shumar, W. & Renninger, K. A. (July, 1998). Studying Virtual Communities Ethnographically: The Math Forum. Paper presented at the 14th International Congress of Anthropological and Ethnological Sciences. College of William and Mary, Williamsburg, VA.

[Privacy Policy] [Terms of Use]

Home || The Math Library || Quick Reference || Search || Help 

© 1994- The Math Forum at NCTM. All rights reserved.