The JOMA Applet Project: Applet Support for the Undergraduate Mathematics Curriculum

_____________________________________
Back to communities || On to dissemination
_____________________________________

Evaluation

Building on a model of combined methodology used for evaluating The Math Forum, evaluation of the Digital library is designed to generate both quantitative and qualitative data that both (a) enables the design, development, and facilitation of projects to be mounted for each of the three user groups: faculty, students, and developers, and (b) assesses the impact of these projects on user participation, contributions and the integration of applets and related teaching units into the mathematics curriculum.

Evaluation will be undertaken by a local team in consultation with an external evaluator. A combination of methods will be employed. These include: ethnographic evaluation, structured interview, focus groups, analysis of library feedback and discussion groups, questionnaires, and face-to-face discussions.

The evaluation plan for the proposed project has three components: identification and response to user needs, review of applets and teaching units, and use (interoperability and reusability) of applets. Identification of and response to user needs is the primary focus of the digital library evaluation team. The review of applets and related teaching units will be undertaken in accord with data collected on user needs and in conjunction with NEEDS personnel, and participants on related digital library and SMETE projects. It consists of establishing criteria for applets and teaching units. Study of the use of applets also will include data collected on user needs. The interoperability and reusability of applets will be studied by the ESCOT group at SRI more specifically.

1. Identification of and Response to User Needs

Three sources of data will be used to identify the needs of faculty, students, and developers. These include focus groups, analysis of library feedback and discussion groups, questionnaires, and an in-depth analysis of usage by a subsample of faculty and their students, and goals of developers.

Focus groups will be used to enable digital library personnel (including archivists, programmers, Web personnel, evaluators, and PIs) to design, develop, and facilitate services for the different user groups. These will be held three times a year for each service. During the focus group, work goals will be established, plans will be discussed, and questions for users will be identified. Notes from these meetings will be posted and used to guide work on each service. They also will guide discussion in subsequent focus group meetings.

Questionnaires will be used to gather demographic and learning preference information from the three user groups. Questions generated by the focus groups for each service will be posted to the public discussions in addition.

Faculty members who teach single-variable calculus and their students from each of three types of undergraduate settings (liberal arts, undergraduate university, and community college) will be studied in depth using observation, interviews, and face-to-face and online discussions. Calculus has been selected as the focus of this portion of the evaluation in order to provide a basis of comparison between institutions in faculty and student use of applets.

A total of nine faculty and their classes will be studied twice a year during each of the three years of the grant (a teacher and class from each of three liberal arts colleges, a teacher and class from each of three universities, and a teacher and class from each of three community colleges). The faculty will be followed across the three years in order to enable consideration of applet use and any curricular adjustments in their teaching. A new cohort of students in each of these faculty classes will be studied each year in order to evaluate the effect of applet use and adjustment on student performance. Classes will be observed, faculty will be interviewed, and three subsamples of students will participate in a focus group discussion of applet usage and recommendations for applet development.

A subsample of nine developers will be interviewed twice a year over the three-year period of the grant.

2. Review of Applets and Teaching Units

Criteria for applets and teaching units need to be established for the content areas of pre-calculus, calculus, and elementary statistics, along with the non-basic couses. Following this, a review process will be established. The development of review protocols and procedures will be led by NEEDS project personnel in conjunction with Digital Library and SMETE personnel and twelve mathematics faculty.

The NEEDS Quality Review of Courseware effort was developed to advance the field of technology-enhanced learning review as well as to continually raise-the-bar for excellence. The evaluation criteria are divided into three categories, Instructional Design: interactivity, cognition/conceptual change, content, multimedia use and instructional use/adaptability; Software Design: engagement, learner interface and navigation and technical reliability; and Engineering Content: accuracy of content, organization of content and consistency with learning objectives.

Following development of criteria for observational user studies will be conducted with a selected sample of the broader SMETE user community. These studies, which are critical to the development of an effective library, will be based on protocols pilot tested during Stage I at the IEEE Advances in Digital Libraries Conference in May 2000 and at the American Society for Engineering Education Annual Conference in June 2000. The observation studies will help in the development of the system interfaces and will specifically address questions of community identification, barriers to use and ease of use. Digital library feedback and discussion groups will provide feedback on the SMETE Information Portal, a multi-disciplinary resource, into an existing disciplinary community.

Twelve faculty (three for each of the different content areas who currently teach in that content area, with some representation from each type of educational setting) will be recruited to (a) generate a table of contents for each content area and (b) establish criteria for applet and teaching unit review. They will work together online to develop the table of contents, and then face-to-face and online to establish an applet and teaching unit review process.

Two review boards will then be formed to review collected applets and teaching units. Two summer reviewing workshops will be held during each of the three years of the grant.

3. Re-use and Interoperability of Applets

Re-use and interoperability of applets in the digital library will be investigated by the ESCOT group at SRI. These efforts will be evaluated in terms of their effectiveness in generating increased levels of re-use and interoperability of applets in the digital library.

[Privacy Policy] [Terms of Use]

_____________________________________
Home || The Math Library || Quick Reference || Search || Help 
_____________________________________

© 1994-2014 Drexel University. All rights reserved.
http://mathforum.org/
The Math Forum is a research and educational enterprise of the Drexel University School of Education.The Math Forum is a research and educational enterprise of the Drexel University School of Education.

The Math Forum