Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » sci.math.* » sci.stat.math.independent

Topic: Measuring Inter-Rater Agreement
Replies: 3   Last Post: Jun 6, 2013 1:15 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Bruce Weaver

Posts: 736
Registered: 12/18/04
Re: Measuring Inter-Rater Agreement
Posted: Jun 6, 2013 9:21 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

On 06/06/2013 3:15 AM, Ray Koopman wrote:
> Consider two raters assigning the same objects to a set of nominal
> categories. How should an ideal measure of agreement change when the
> data changes? For instance, if one rater assigns an object to category
> A, and the other rater assigns that object to category B but then
> changes his mind and puts it in A, it's clear that a proper agreement
> measure ought to increase with that change. But what if both raters
> change their minds and put the object in category C? How should an
> ideal measure respond to that change? Should it ever decrease?
>


Here's an illustration of what Ray has described (view with fixed font).

R2-A R2-B R2-C
R1-A aa ab ac
R2-B ba bb bc
R3-C ca cb cc

In a second table (after the change), one observation switches from cell
ab to cell cc. (More generally, one observation switches from an
off-diagonal cell to a cell on the main diagonal.)

Intuition (or my intuition, at least) suggests that the measure of
agreement should increase, because the total number of agreements has
increased.

So why do I have the feeling we're being set up? ;-)

--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/Home
"When all else fails, RTFM."



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.