[Home]MethodologySymposium

HomePage | RecentChanges | Index | Preferences | Upload

This is a workspace for ideas for the Symposium.

The Symposium is based on the [Group Cognition Workshop at KMRC, Oct 2006]

Symposium Structure

Productive Tensions

[overview slides]

Reducibility vs. Group Cognition

Gerry Stahl [slides] & Friedrich Hesse [slides]
chair Jürgen Buder

Predictive Models vs. Studies of Practice

Ulrike Cress [slides] & Tim Koschmann & Peter Reimann [slides]
chair Friedrich Hesse

Coding & Counting vs. Analysis

Dan Suthers [slides] & Jürgen Buder [slides]
chair Gerry Stahl

Panel Statement

An ijCSCL-sponsored event at the CSCL Conference

Invited Panel: Making use of productive tensions in CSCL

As might be typical for a rather new and interdisciplinary field like CSCL, there are currently many different suggestions as to how our field could be conceptualized. Similar to other fields in education, productive tensions arise on various levels, e.g.:

  1. The practice of analysis vs. the practice of design: Should our practices be geared towards analyzing and understanding CSCL phenomena, or should there be a stronger focus on finding workable solutions in everyday educational practice?
  2. Nomethetic vs. idiographic approach: If we are to focus on the practice of analysis, should CSCL research be directed at identifying generalizable patterns of behavior, or should it aim at a deep understanding of singular events?
  3. Reducibility vs. emergent entities: When theorizing about CSCL, can we reduce the group phenomena involved in CSCL by way of individual constructs, or should we conceptualize emergent, group-level structures?

While powerful arguments can be brought forth to support each of the theoretical and methodological stances described above, the question remains how we can best make use of this diversity. This problem was raised on a workshop held in Tübingen in October, 2006, where the people on this panel and colleagues discussed these issues from various viewpoints. This panel seeks to outline ideas on how to weave different strands of CSCL research together. Using examples from their own research, panelists will discuss how to create “interfaces” between different research approaches to facilitate a culture of knowledge building in our field, and how to employ hybrid methodologies to tackle problems and questions related to CSCL.

Participants:

Initial Draft Position Statements

Following are spaces to put ideas for initial position statements

Juergen's Position Statement for the Symposium

1) Message: To advance theoretical understanding and practical design it is preferable (for me at least) to use quantifiable data instead of narrative descriptions of single cases. Relevant issues are replicability, generalizability, and unbiased selection of data.
2) When it comes to analyzing group discussions, the quantitative approach of coding and counting can be justifiably criticized. It does not allow us to discover interesting patterns in discussions.
3) To get a better understanding of discussion content one should use a "catch and toss" approach to analysis, i.e. switch between qualitative and quantitative analyses.
4) Example: After doing some initial coding, group discussions are visualized (somewhat similar to Dan 's uptake graphs). The visualizations show elements like topic, message length, references, authorship etc. (I will hopefully provide some examples until then). This will help to get a "feel" for discussion, and it could point at things that deserve additional quantitative analyses. E.g., it might be that "successful" groups were focusing on one particular topic (many interrelated, lengthy messages), and this might account for their success.
5) Patterns found by this inductive method would be used as a starting point for additional quantitative analyses. In other words, the qualitative analysis serves as a tool for yielding quantifiable data, but will not be the end product.

Ulrike's Position Statement for the Symposium

CSCL aims at searching for (design-)factors which facilitate learning in groups. Thus, if we want to be able to generalize, we have to make use of an experimental approach which allows us to attribute learning (as dependent variable) to the design factors we manipulate (as independent variable).
The problem is that the design factors’ effect on learning is mediated by processes on two different levels which are different conceptually and statistically: on the individual level (the learners’ cognition, motivation, emotion etc.) and on the group’s level (could be called group cognition, group motivation, interaction etc.). And also “learning” can be measured individually (a person is able to do something she could not do before) or on group level (a group is able to do something which it could not do before).
The variables on both levels – the individual one and the group one - must be described by different variables and measures. The processes of both levels show a mutual interdependency: individual behaviour influences the group’s processes, and the group’s processes influence the individuals’ behaviour. Both levels may be closely coupled, but the group-level variables normally can not be calculated by a simple transformation of the individual level variables. Especially, if collaboration has any emergent effect (and this is what we hope and what we look for in CSCL), then the group level effects can not be reduced to individual-level variables.
A full model of CSCL would have to address all these relevant effects: the design factors, the individual-level processes, the group-level processes, and their interactions regarding the learning outcomes on individual and on group level. Only very advanced statistical methods like hierarchical linear modelling or structural equation modelling would allow modelling all those processes. The problem is that these analyses need much larger sample sizes than we normally have in CSCL.
As long as we can’t reach such large samples we have to rely on more reductionist approaches and we have to focus on processes we can deal with our statistical methods. For example if a study aims at focussing on individual level processes it may be necessary to fake the group’s members’ interactions in order to hold constant or explicitly vary the group influences. Or if a study aims at investigating group level processes it has to restrain to the group-level variables, but then it will not be able to generalize its results to individual behaviour. Because of these restrictions, we have to design our experiments more as “spotlights” which focus on specified processes than as “floodlight” which try to light up all the processes which take place in computer-supported collaborative learning.

Dan's Position Statement for the Symposium

(In progress)

Peter's Position Statement for the Symposium

Main Message: Quantitative analysis is fine, but a variance model may be the wrong starting point.
Assumption:
(1) We are assuming that research in CSCL can/should make contributions to generalized explanatory theories of learning; we are not seeing CSCL research being confined to the improvement of particular socio-technical designs and/or to a purely descriptive program.The question is then how to justify claims as to explanatory power, validity, generalizability, rather than worrying if such kind of claims are relevant at all.
(2) The kind of explanation we are after is a ‘causal’ one: We want to explain an outcome (Y) as being brought about by one or more precursors (X), where X is at least necessary, but ideally necessary and sufficient to bring about Y.

Main claims:
(1) We need to distinguish validity of an explanation from generalization (and comparison). A case study, a single-N or small-N study, can yield perfectly valid causal explanation. A narrative account (invoking only specific agent actions, motives, and contingent changes in the actors environment) can be a valid causal account without referring to any external regularities. Which is the central limitation to a narrative account: it cannot refer to external singular causes, unless invoking regularities (general laws). The causality that is well covered in narrative approaches could be called ‘action causality’.
Note that on this level, important types of analysis can be performed. For instance, if one was of a structuralist persuasion, one could explore to which extent the narrative account can be reduced a set of small principles, or rules, and one can engage in counter-factual reasoning, thus increasing the fertility of the model.

(2) The need for generalizing arises in CSCL both because it is a design discipline—we want to design methods and tools of value for many learners—and because of its aspiration to contribute to learning research. The dimension for generalisation is typically interpreted as across individuals, but there is also the dimension of situations (tasks, settings, etc.) across which one might wish to generalize.
(3) The typical way generalizations are performed in the social sciences is using the variance model, which builds on necessary and sufficient causation and is necessarily quantitative: The precursor(s) X is related to the “dependant” Y mathematically as Y = f(X), with each value of X corresponding to one and only one value in Y. In empirical sciences, measurement error needs to be taken into account, but that does not change the basic logic. This logic requires the researcher interested in generalization to use only variables that satisfy the basic measurement assumptions of variance analysis. The basic form of claims that can be tested in this manner are of the kind “an increase/decrease in X causes and in/decrease in Y”.

(4) While the variance model is fine for many research questions, it has a number of problems when applied in CSCL. To mention the most important ones:
a) No notion of sequential dependencies; variance theory assumes instantaneous causation. This is highly problematic in light of the sequential nature of communication and learning. See my longer paper on this.
b) No notion of combinatorics: It seems to me that what we mostly do in CSCL is discovering how to bring entities (people, ideas, misconceptions, etc.) together so that ‘constructive dynamics’ can ensue (people talking to each other, ideas being exchanged, misconceptions being confronted, …). This is why concepts such as attention management, turn-taking, reciprocity, take-up, coordination etc. are so important and why we think so much about social, representational and technical affordances to support their realization. But variance theory has no concepts for the inherently combinatorical (in addition to sequential) nature of human interaction. A quantitative variable such as “communication density” does not ‘summarize’ the quality and combinatorics of interactions on the level of individual actors, nor can it be ‘reduced’ to these and vice versa. (A quantitative factor such as “communication density” could perhaps be seen as ‘emerging’ from the lower level interactions between the ‘particles’, which would be one way to think about the relation between individual and group processes, but that’s another story.)
c) The selection of variance theory as the main road to generalization makes it more difficult to capitalize on single-N and small-N studies when performing regularity-searching research. What matters for the small-N researcher, the detailled sequence of events, does not matter to the variance-model researcher and vice versa. In other words, it becomes more difficult than need be to realize a cumulative, integrated research strategy in CSCL.

(5) Luckily, there is an alternative to pursue generalizations without having to (solely) rely on variables: process theories that combines (a) necessary conditions (as typical for a narrative), (b) necesssary probabilistic processes and c) external forces that affect the probabilistic processes. Within this framework, we can in a generalizable manner explain why a certain cooperation script, for instance, positively affects learning without having to refer to variables in the variance model sense. (The explanation would entail how the script affects the combinatorics so that certain event sequences become more probable, hence we see more learning).

Friedrich's Position Statement for the Symposium

Comments starting from point three of our general description:

1 Practice of analysis vs. practice of design: Should our practices be geared towards analyzing and understanding CSCL phenomena, or should there be a stronger focus on finding workable solutions in everyday educational practice?

2 Nomothetic vs. idiographic approach: If we are to focus on the practice of analysis, should CSCL research be directed at identifying generalizable patterns of behavior, or should it aim at a deep understanding of singular events?

3 Reducibility vs. emergent entities: When theorizing about CSCL, can we reduce the group phenomena involved in CSCL by way of individual constructs, or should we make assumptions about emergent, group-level entities?

Ad 3: Research in the field of CSCL deals with „collaboration“ and „learning“ in a socio-technological environment. More precisely this research is interested more in (collaborative) learning than in collaboration itself. Collaboration is part of the setting or an intentionally seeked situation to do better learning. This leads to the question: what do we mean with learning (or „uptaking“ or „meaningmaking“ or even cognition)? The process or the outcome? Do we mean individual learning (of a single person) or „group learning“ (even if it is part of the individuals there is something beyond that, and existing only when the group is together) or is it changing the environment e.g. in terms of new or adapted tools?

I assume we have to talk about all of them as well as their interactions. However, we have to be clear about the nature of each of them. It is not that complicated with learning in the individual and with changes in the environment (e.g. the tools). It is more challenging to define the nature of something, which has been labeled „group learning“ (or sometimes “group cognition”) before. Obviously there are phenomena, which only exist, if there are a certain number of persons together as a group. Let us just consider two different examples as outcomes of some sort of learning or doing something together: a piece of music played by a trio and hooliganism.

The music piece: Of course one can only listen to this music if the trio is acting in a special way together. It cannot be done the same way by a single person. However the way they have to interact has been written up by a composer before and how it sounds depends on their skills and the instruments (tools). Would we call this „group learning“? Would it be important that exactly the same three people come together and even then, would it sound the same?

Hooliganism: Persons being quite „normal“ in their everyday life could be part of a group of hooligans and reacting quite strange in that context. Is this something which can seen as an emergent phenomena coming out of the group interacting/situation? Or is it probably something, which is part of these individuals and just needs some special conditions to become overt?

Without giving definite answers in these two cases we have to differentiate between components which are part of the individuals and a composition which is only existing when the components are put together. From an analytical point of view this asks for a) knowing, what the components are and b) in which way they are interacting and c) to which degree to outcome is predictable or has some degree of freedom

The questions remains, is it this we mean with „group learning“ and do we need this term or another label to address this? How far away is this from terms like „division of labor“ and from products resulting from that?

Ad 1 and 2: In any case we have to analyze the conditions, processes and outcomes as well in single cases as across a bigger number of persons and we are interested to understand what is going on in terms of causes. We also want to make use of our understanding and apply it in similar situations. So we need to generalize (see point 1). And „workable solutions“ can profit from understanding why they work (see point 2).

Tim's Position Statement for the Symposium

So here is an approach to how to conduct CSCL research: a researcher develops a notion concerning a new way of doing instruction and theorizes that it will lead to improved learning. To test this theory, the innovation is introduced into several classrooms. Following the experience, a test is administered and the results are compared with results from other classrooms that are similar in all obvious respects, but in which the innovation was not introduced. Appropriate statistical testing reveals that the difference in average scores across the treatment and control classrooms was statistically significant and the effect size was appreciable. Would this represent a publishable finding in CSCL and would its design represent an appropriate framework for conducting work in the future?

I would argue that the answer is 'no' in both cases and I base my judgment on several observations:

1) In clinical trials with new drugs, one must first do bioavailability studies to determine an adequate dosage range prior to testing efficacy. But, what is the equivalent of an effective dose of an instructional innovation? Is it a single exposure? The completion of a lesson? Some more extended period of instruction? This seems to be particularly problematic for initiatives designed not just to teach some matter, but rather to restructure the learner's thinking or problem solving (e.g., PBL). Does it make sense to even talk about an effective dose when the treatment variable is a socially-constituted practice?

2) Psychologists have long acknowledged the problem of bias caused by experimenter and subject expectations. To address this problem, drug trials in medical research are commonly done using a double-blind design. When talking about an instructional practice, are there blinding procedures that would conceal from the subject and the experimenter just what kind of treatment is being received?

3) The models employed in inferential statistics presume that treatments are applied uniformly across all subjects. One could quite plausibly satisfy this condition if the treatment involves ingesting 50 mg. of Drug X. On the other hand, where researchers have examined implementations in different classroom settings, there often appears to be great variability across "realizations" of the innovation. Does it even make sense to talk about uniformability when discussing forms of contingently-organized practice?

What we see in all three instances is that this particular experimental design, borrowed from the biological sciences, is not well suited to the task of understanding or evaluating a socially-constituted form of practice. Variation in practice is treated as an error source, something to be eliminated, if possible. Variation in practice, however, is the phenomenon of interest here (or, at least, it should be!). Adopting more sophisticated statistical models, multivariate models, for example, or a process-based models might render some of the enumerated problems irrelevant but would likely introduce new assumptions and requirements that would be equally difficult to satisfy. What we need is a methodology that treats instructional practice, not as peripheral or as taken-for-granted, but rather as the central focus of investigation.

I think it might be overly ambitious to expect that we will be able to negotiate this afternoon a mutually-acceptable framework for conducting future research. The first step, in effecting a cure, however, is always one of naming the problem. I name the problem of past research in education as one of systematically failing to treat instructional practice as the central topic of inquiry.

Gerry's Position Statement for the Symposium

Introduction

This panel is a presentation of ijCSCL. The journal is dedicated to publishing innovative research in CSCL from all perspectives and scientific methodologies. The question of methodology is a fundamental and complex one within the CSCL research effort. The journal is particularly interested in fostering new ideas on this topic. We hope that the opinions voiced during this panel will contribute to that effort. Specific positions taken should in no way be seen as positions of the journal, its Board of Editors or its reviewing perspective.

The historical context of CSCL

CSCL builds on the work of its predecessors:
  1. Research on cooperative learning (e.g., Johnson & Johnson) focused on learning as a psychological process of individuals who happened to be in group contexts.
  2. Computer support of group learning (e.g., computer-mediated communication) took face-to-face communication as the gold standard and tried to duplicate its characteristics.
But the potential of CSCL goes essentially beyond these:
  1. To overcome the limitations of the individual mind, and to inter-animate multiple personal perspectives in achievements of group cognition.
  2. To allow people around the world to build knowledge in collectivities not confined by geographic and traditional boundaries.

The historical context of the learning sciences

Modern society has some entrenched ideologies:
  1. The ideology of the individual treats the learner as a consumer and learning as acquisition of factual knowledge.
  2. The ideology of science treats facts as discoverable by established methods and science as objective knowledge.

But recent theorists provide new ways of conceptualizing learning and science:

  1. Vygotsky and Mc Luhan argue that our learning is fundamentally mediated by social interactions and forces.
  2. Kuhn and Latour argue that sciences follow unpredictable paths of inquiry, rather than adhering to the ideal picture of science promoted by politicians and bureaucrats.

CSCL needs research approaches that:

  1. Do not necessarily focus on the individual as the “learner.”
  2. Do not necessarily rely on methods of established and certified sciences.
  3. Explore the potential of groups to accomplish things as groups.
  4. Explore the potential of software to open new opportunities for collaborative learning.

Methodological problems:

  1. We have discovered already that the potential of CSCL is still distant. It requires technologies, pedagogies, facilitators and student motivations that we do not yet have, so we cannot simply observe it on a large scale.
  2. Learning in CSCL requires a combination of very different attitudes, tasks and settings, so it cannot be studied in controlled independent-variable comparisons to individual face-to-face learning.
  3. Interesting occurrences in CSCL settings are highly situated, un-reproducible and sparse, so they cannot be subjected to automated or statistical analysis.
  4. Characteristics (“variables”) of a controlled setting are enacted (understood, interpreted, constructed) by the individuals and groups involved, responding to the unique sequentiality of interactions and open-ended resources and possibilities.

Design-based research

  1. We need to create innovative CSCL settings where we can observe group interactions that inform us about the nature of collaborative learning and about the design of computer support for it. Our methods for the analysis of the results will need to be invented in response to our particular research questions, but should generally be oriented to understanding the interactions that take place and informing the re-design of the technologies that mediate those interactions.
  2. This will integrate “the practice of analysis and the practice of design” in the analytic design of practice.
  3. From the analysis of informative case studies and of collections of related cases, we will build and gradually generalize an understanding of collaborative learning. This understanding can provide hypotheses for testing specific points.
  4. The goal of CSCL is to go beyond individual learning to group knowledge building. Just as there can be no group cognition without individual cognition, we should recognize that individual learning is at heart a social product and that collaborative learning generally incorporates individual learning—not by being reducible to it, but as a result of building shared meaning.

Note on editing

To get editing permission, Click on the "Preferences" link and enter "mathforum" for your Administrator Password. Then you should see the "Edit text of this page" link at the bottom of the page. This will open an edit window. Below the edit window is a "HowToUseThis?" link to simple instructions on editing this wiki.

HomePage | RecentChanges | Index | Preferences | Upload
This page is read-only | View other revisions
Last edited July 21, 2007 10:31 pm by Gerry (diff)
Search: