SampleChatData

Sample Chat Transcripts

Information about the Coding Process:

We are working with 3 independent coders. (The handles M,N and R denote each coder) The coding process basically includes 2 phases. In the first phase we form an agreed conversation thread for the given transcript. First, the coders individually assign their codes for the conversation thread dimension. Subsequently, a face to face meeting is held in an effort to produce an agreed c-thread for the transcript.

In the second phase, each coder independently codes remaining dimensions depending on the agreed c-thread. Finally, one of the coders forms the final version by checking the codes assigned by other coders. During this merging process we usually rely on majority checking, i.e. if at least 2 out of 3 coders assigned the same code to a particular utterance then that assignment is used for the final version. In the event of disagreements the merger picks one of the assignments depending on their suitability to the coding scheme.

During our coding calibration sessions we observed that the way a coder interprets the conversation (i.e. the order of events, who is replying to whom etc.) heavily affects the reliability of the final coding assignments. This motivated us to employ a stratified coding procedure. The flow of conversation is essentially captured by the conversation thread dimension. This is mainly why we agree on the conversation thread first before going over other dimensions.

The rules and regulations that we employed during the process can be reached at: [Coding Scheme (pdf)]

An Overview of both Sessions:

• Both groups were exposed to the problem at the same time.
• Both sessions were planned to last about an hour. The first group’s session took around 90 minutes whereas the other group’s session took about 75 minutes.
• The groups were asked to solve a geometry problem [Problem Definition & Solution].
• The first group formed a shared image [(gif)], whereas the other group chose not to do so.
• The first group couldn’t get the correct solution, whereas the other group was able to provide correct answers for both parts of the problem. The second group’s members seem to be mathematically more skillful as compared to the other group’s members.
• The first group’s members exchanged a lot of messages; they raised questions, offered ideas, helped each other to catch up etc. On the contrary, there was less interaction in the other session. The second group’s members individually solved the problem.

Powwow2 Group 1:

Coding Session Statistics:

This table shows inter-coder agreements. The last column shows the percentage of agreement between all 3 coders.

	Powwow2 gr_1 (723)	M vs R	M vs N	N vs R	M-N-R	
	Conversation Thread	58.64%	67.08%	63.35%	47.86%
	Conversation		66.53%	79.11%	65.28%	58.51%
 	Social			71.23%	81.74%	75.66%	66.39%
 	System Support		94.88%	96.54%	96.40%	93.91%
 	Problem Solving Thread	70.40%	84.23%	73.31%	65.42%
 	Problem Solving		87.83%	89.21%	86.31%	83.13%
 	Math Moves		89.90%	92.12%	94.47%	89.07%

Assigned codes [Group 1 (xls)]

Session summary [(html)]

Powwow2 Group 2:

Coding Session Statistics:

This table shows inter-coder agreements. The last column shows the percentage of agreement between all 3 coders.

	Powwow 2 gr_2 (204)	M vs R	M vs N	N vs R	M-N-R	
	Conversation Thread	66.50%	67.98%	64.04%	53.20%
	Conversation		70.44%	77.34%	70.94%	62.07%
 	Social			83.74%	89.16%	79.80%	76.85%
 	System Support		87.68%	95.07%	84.73%	83.74%
 	Problem Solving Thread	63.55%	75.86%	73.89%	60.10%
 	Problem Solving  	70.44%	76.85%	68.97%	64.04%
 	Math Moves		75.37%	81.77%	75.86%	69.95%

Assigned codes [Group 2 (xls)]

Session summary [(html)]