The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » sci.math.* » sci.math

Topic: Is there a way to calculate an average ranking from uneven lists?
Replies: 15   Last Post: Oct 30, 2013 12:18 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]

Posts: 219
Registered: 5/27/08
Re: Is there a way to calculate an average ranking from uneven lists?
Posted: Oct 29, 2013 2:09 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

Jennifer Murphy <JenMurphy@jm.invalid> wrote:
> JohnF <> wrote:
>>Jennifer Murphy <JenMurphy@jm.invalid> wrote:
>>> James Waldby <not@valid.invalid> wrote:
>>>>For the given problem, averages of ranks probably aren't a statistically
>>>>sound approach. For example, see the "Qualitative description" section
>>>>of article <>, which says:
>>>>"User ratings are at best ordinal categorizations. While it is not
>>>>uncommon to calculate averages or means for such data, doing so
>>>>cannot be justified because in calculating averages, equal intervals
>>>>are required to represent the same difference between levels of perceived
>>>>quality. The key issues with aggregate data based on the kinds of rating
>>>>scales commonly used online are as follow: Averages should not be
>>>>calculated for data of the kind collected." (etc.)

>>> Yes, I did feel a little uneasy about averaging numbers that are not
>>> really numerical in the usual sense.

>>Yes, I think that approach would be wrong, based on the following
>>extreme case counterexample:
>> Suppose you have 100 different lists, 99 of which identically contain
>>the same two books, and only those two books, in the same ranking,
>> Lists 1-99 contain: Book#1=The Bernie Madoff Story, #2=The Ken Lay Story
>>Finally, the 100th list contains, say 100 different books, including
>>our above two losers, but ranked
>> List 100 contains: Book#1=The complete works of Shakespeare,
>> #2=..., #3=..., ..., and finally,
>> #99=The Bernie Madoff Story, #100=The Ken Lay Story
>>Clearly, Bernie and Ken suck, but when they're the only two
>>books on a list, then they have to rank #1 and #2.
>>So you need a methodology that avoids a combined ranking giving
>> Bernie: 99 #1-scores and one #99-score, and giving
>> Ken: 99 #2-scores and one #100-score.
>>That would significantly overestimate them.

> This example is badly skewed.

Well, yeah, that's kind of the point. A less skewed list may still
contain the same general problem (yes, you get my point (I think)),
just not so evidently. That's why I said "extreme case counterexample"
above. Moreover, this general problem is already well-known...
Simpson's Paradox
or just read's_paradox
In particular, compare and contrast those tables in the wikipedia
article with the tables from one of your preceding replies...

> Let's consider some actual data. Here are 3 sample lists each containing
> 5 books, but not the same 5 books:
> Rank List 1 List 2 List 3
> 1 A B F
> 2 B A H
> 3 C E C
> 4 D G D
> 5 E D A
> When listed by book, the data looks like this:
> List 1 List 2 List 3
> Books Rank Rank Rank
> Book A 1 2 5
> Book B 2 1
> Book C 3 3
> Book D 4 5 4
> Book E 5 3
> Book F 1
> Book G 4
> Book H 2
> How would you calculate average rankings?

What you're trying to do is find a way to "combine" those lists
into one list. See how Simpson's Paradox illustrates the
counter-intuitive problems that can arise trying to do that?
And that's what I tried to (way less authoritatively) illustrate
with my skewed example above.

> First, all of the lists I will use will be
> from authoritative sources, they will all have a lot more than 2 books,
> and no two lists will be even close to identical. So I have to quibble
> with your point about the Bernie and Ken books being that terrible. If
> these 100 lists are from reputable sources (as opposed to, say, from
> some crackpot on usenet ;-)), then the Bernie and Ken books do not
> "suck". If the last list is the 100 greatest books of all time, then
> even making it to #99 and #100 would be well above the "suck" level. ;-)
> But I get your point (I think).
> I might include a list containing only science fiction books. This would
> exclude 95% of all books, but, as you say, some book on that list will
> have to be #1. The #1 book on the sci fi list would probably also be on
> at least some of the other lists, so it's ranking there would be
> factored in. And I can apply a weighting factor to each list so that the
> most substantive lists have a greater influence on the results.

John Forkosh ( mailto: where j=john and f=forkosh )

Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2017. All Rights Reserved.