Evaluation
Building on a model of combined methodology used for evaluating The Math
Forum, evaluation of the Digital library is designed to generate both
quantitative and qualitative data that both (a) enables the design,
development, and facilitation of projects to be mounted for each of the
three user groups: faculty, students, and developers, and (b) assesses the
impact of these projects on user participation, contributions and the
integration of applets and related teaching units into the mathematics
curriculum.
Evaluation will be undertaken by a local team in consultation with an
external evaluator. A combination of methods will be employed. These
include: ethnographic evaluation, structured interview, focus groups,
analysis of library feedback and discussion groups, questionnaires, and
face-to-face discussions.
The evaluation plan for the proposed project has three components:
identification and response to user needs, review of applets and teaching
units, and use (interoperability and reusability) of applets. Identification
of and response to user needs is the primary focus of the digital library
evaluation team. The review of applets and related teaching units will be
undertaken in accord with data collected on user needs and in conjunction
with NEEDS personnel, and participants on related digital library and SMETE
projects. It consists of establishing criteria for applets and teaching
units. Study of the use of applets also will include data collected on user
needs. The interoperability and reusability of applets will be studied by
the ESCOT group at SRI more specifically.
1. Identification of and Response to User Needs
Three sources of data will be used to identify the needs of faculty,
students, and developers. These include focus groups, analysis of library
feedback and discussion groups, questionnaires, and an in-depth analysis of
usage by a subsample of faculty and their students, and goals of developers.
Focus groups will be used to enable digital library personnel (including
archivists, programmers, Web personnel, evaluators, and PIs) to design,
develop, and facilitate services for the different user groups. These will
be held three times a year for each service. During the focus group, work
goals will be established, plans will be discussed, and questions for users
will be identified. Notes from these meetings will be posted and used to
guide work on each service. They also will guide discussion in subsequent
focus group meetings.
Questionnaires will be used to gather demographic and learning preference
information from the three user groups. Questions generated by the focus
groups for each service will be posted to the public discussions in addition.
Faculty members who teach single-variable calculus and their students from
each of three types of undergraduate settings (liberal arts, undergraduate
university, and community college) will be studied in depth using
observation, interviews, and face-to-face and online discussions. Calculus
has been selected as the focus of this portion of the evaluation in order to
provide a basis of comparison between institutions in faculty and student
use of applets.
A total of nine faculty and their classes will be studied twice a year
during each of the three years of the grant (a teacher and class from each
of three liberal arts colleges, a teacher and class from each of three
universities, and a teacher and class from each of three community
colleges). The faculty will be followed across the three years in order to
enable consideration of applet use and any curricular adjustments in their
teaching. A new cohort of students in each of these faculty classes will be
studied each year in order to evaluate the effect of applet use and
adjustment on student performance. Classes will be observed, faculty will be
interviewed, and three subsamples of students will participate in a focus
group discussion of applet usage and recommendations for applet development.
A subsample of nine developers will be interviewed twice a year over the
three-year period of the grant.
2. Review of Applets and Teaching Units
Criteria for applets and teaching units need to be established for the
content areas of pre-calculus, calculus, and elementary statistics, along
with the non-basic couses. Following this, a review process will be
established. The development of review protocols and procedures will be led
by NEEDS project personnel in conjunction with Digital Library and SMETE
personnel and twelve mathematics faculty.
The NEEDS Quality Review of Courseware effort was developed to advance the
field of technology-enhanced learning review as well as to continually
raise-the-bar for excellence. The evaluation criteria are divided into three
categories, Instructional Design: interactivity, cognition/conceptual
change, content, multimedia use and instructional use/adaptability; Software
Design: engagement, learner interface and navigation and technical
reliability; and Engineering Content: accuracy of content, organization of
content and consistency with learning objectives.
Following development of criteria for observational user studies will be
conducted with a selected sample of the broader SMETE user community. These
studies, which are critical to the development of an effective library, will
be based on protocols pilot tested during Stage I at the IEEE Advances in
Digital Libraries Conference in May 2000 and at the American Society for
Engineering Education Annual Conference in June 2000. The observation
studies will help in the development of the system interfaces and will
specifically address questions of community identification, barriers to use
and ease of use. Digital library feedback and discussion groups will provide
feedback on the SMETE Information Portal, a multi-disciplinary resource,
into an existing disciplinary community.
Twelve faculty (three for each of the different content areas who currently
teach in that content area, with some representation from each type of
educational setting) will be recruited to (a) generate a table of contents
for each content area and (b) establish criteria for applet and teaching
unit review. They will work together online to develop the table of
contents, and then face-to-face and online to establish an applet and
teaching unit review process.
Two review boards will then be formed to review collected applets and
teaching units. Two summer reviewing workshops will be held during each of
the three years of the grant.
3. Re-use and Interoperability of Applets
Re-use and interoperability of applets in the digital library will be
investigated by the ESCOT group at SRI. These efforts will be evaluated in
terms of their effectiveness in generating increased levels of re-use and
interoperability of applets in the digital library.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.