Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Is there a way to calculate an average ranking from uneven lists?
Replies: 15   Last Post: Oct 30, 2013 12:18 PM

 Messages: [ Previous | Next ]
 JohnF Posts: 219 Registered: 5/27/08
Re: Is there a way to calculate an average ranking from uneven lists?
Posted: Oct 29, 2013 2:09 AM

Jennifer Murphy <JenMurphy@jm.invalid> wrote:
>>Jennifer Murphy <JenMurphy@jm.invalid> wrote:
>>> James Waldby <not@valid.invalid> wrote:
>>>>
>>>>For the given problem, averages of ranks probably aren't a statistically
>>>>sound approach. For example, see the "Qualitative description" section
>>>>of article <http://en.wikipedia.org/wiki/Rating_scale>, which says:
>>>>"User ratings are at best ordinal categorizations. While it is not
>>>>uncommon to calculate averages or means for such data, doing so
>>>>cannot be justified because in calculating averages, equal intervals
>>>>are required to represent the same difference between levels of perceived
>>>>quality. The key issues with aggregate data based on the kinds of rating
>>>>scales commonly used online are as follow: Averages should not be
>>>>calculated for data of the kind collected." (etc.)

>>>
>>> Yes, I did feel a little uneasy about averaging numbers that are not
>>> really numerical in the usual sense.

>>
>>Yes, I think that approach would be wrong, based on the following
>>extreme case counterexample:
>> Suppose you have 100 different lists, 99 of which identically contain
>>the same two books, and only those two books, in the same ranking,
>> Lists 1-99 contain: Book#1=The Bernie Madoff Story, #2=The Ken Lay Story
>>Finally, the 100th list contains, say 100 different books, including
>>our above two losers, but ranked
>> List 100 contains: Book#1=The complete works of Shakespeare,
>> #2=..., #3=..., ..., and finally,
>> #99=The Bernie Madoff Story, #100=The Ken Lay Story
>>Clearly, Bernie and Ken suck, but when they're the only two
>>books on a list, then they have to rank #1 and #2.
>>So you need a methodology that avoids a combined ranking giving
>> Bernie: 99 #1-scores and one #99-score, and giving
>> Ken: 99 #2-scores and one #100-score.
>>That would significantly overestimate them.

>
> This example is badly skewed.

Well, yeah, that's kind of the point. A less skewed list may still
contain the same general problem (yes, you get my point (I think)),
just not so evidently. That's why I said "extreme case counterexample"
above. Moreover, this general problem is already well-known...
In particular, compare and contrast those tables in the wikipedia
article with the tables from one of your preceding replies...

> Let's consider some actual data. Here are 3 sample lists each containing
> 5 books, but not the same 5 books:
> Rank List 1 List 2 List 3
> 1 A B F
> 2 B A H
> 3 C E C
> 4 D G D
> 5 E D A
> When listed by book, the data looks like this:
> List 1 List 2 List 3
> Books Rank Rank Rank
> Book A 1 2 5
> Book B 2 1
> Book C 3 3
> Book D 4 5 4
> Book E 5 3
> Book F 1
> Book G 4
> Book H 2
> How would you calculate average rankings?

What you're trying to do is find a way to "combine" those lists
into one list. See how Simpson's Paradox illustrates the
counter-intuitive problems that can arise trying to do that?
And that's what I tried to (way less authoritatively) illustrate
with my skewed example above.

> First, all of the lists I will use will be
> from authoritative sources, they will all have a lot more than 2 books,
> and no two lists will be even close to identical. So I have to quibble
> with your point about the Bernie and Ken books being that terrible. If
> these 100 lists are from reputable sources (as opposed to, say, from
> some crackpot on usenet ;-)), then the Bernie and Ken books do not
> "suck". If the last list is the 100 greatest books of all time, then
> even making it to #99 and #100 would be well above the "suck" level. ;-)
> But I get your point (I think).
>
> I might include a list containing only science fiction books. This would
> exclude 95% of all books, but, as you say, some book on that list will
> have to be #1. The #1 book on the sci fi list would probably also be on
> at least some of the other lists, so it's ranking there would be
> factored in. And I can apply a weighting factor to each list so that the
> most substantive lists have a greater influence on the results.

--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Date Subject Author
10/27/13 Ben Bacarisse
10/28/13 Jennifer Murphy
10/30/13 Jennifer Murphy
10/27/13 James Waldby
10/28/13 Jennifer Murphy
10/28/13 Graham Cooper
10/28/13 Graham Cooper
10/28/13 Graham Cooper
10/28/13 Graham Cooper
10/28/13 Graham Cooper
10/28/13 Jennifer Murphy
10/28/13 JohnF
10/28/13 Jennifer Murphy
10/29/13 JohnF
10/30/13 Jennifer Murphy