Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.


Math Forum
»
Discussions
»
sci.math.*
»
sci.stat.math
Notice: We are no longer accepting new posts, but the forums will continue to be readable.
Topic:
Measuring InterRater Agreement
Replies:
3
Last Post:
Jun 6, 2013 1:15 PM




Re: Measuring InterRater Agreement
Posted:
Jun 6, 2013 9:21 AM


On 06/06/2013 3:15 AM, Ray Koopman wrote: > Consider two raters assigning the same objects to a set of nominal > categories. How should an ideal measure of agreement change when the > data changes? For instance, if one rater assigns an object to category > A, and the other rater assigns that object to category B but then > changes his mind and puts it in A, it's clear that a proper agreement > measure ought to increase with that change. But what if both raters > change their minds and put the object in category C? How should an > ideal measure respond to that change? Should it ever decrease? >
Here's an illustration of what Ray has described (view with fixed font).
R2A R2B R2C R1A aa ab ac R2B ba bb bc R3C ca cb cc
In a second table (after the change), one observation switches from cell ab to cell cc. (More generally, one observation switches from an offdiagonal cell to a cell on the main diagonal.)
Intuition (or my intuition, at least) suggests that the measure of agreement should increase, because the total number of agreements has increased.
So why do I have the feeling we're being set up? ;)
 Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/Home "When all else fails, RTFM."



