JohnF
Posts:
219
Registered:
5/27/08


Re: Is there a way to calculate an average ranking from uneven lists?
Posted:
Oct 29, 2013 2:09 AM


Jennifer Murphy <JenMurphy@jm.invalid> wrote: > JohnF <john@please.see.sig.for.email.com> wrote: >>Jennifer Murphy <JenMurphy@jm.invalid> wrote: >>> James Waldby <not@valid.invalid> wrote: >>>> >>>>For the given problem, averages of ranks probably aren't a statistically >>>>sound approach. For example, see the "Qualitative description" section >>>>of article <http://en.wikipedia.org/wiki/Rating_scale>, which says: >>>>"User ratings are at best ordinal categorizations. While it is not >>>>uncommon to calculate averages or means for such data, doing so >>>>cannot be justified because in calculating averages, equal intervals >>>>are required to represent the same difference between levels of perceived >>>>quality. The key issues with aggregate data based on the kinds of rating >>>>scales commonly used online are as follow: Averages should not be >>>>calculated for data of the kind collected." (etc.) >>> >>> Yes, I did feel a little uneasy about averaging numbers that are not >>> really numerical in the usual sense. >> >>Yes, I think that approach would be wrong, based on the following >>extreme case counterexample: >> Suppose you have 100 different lists, 99 of which identically contain >>the same two books, and only those two books, in the same ranking, >> Lists 199 contain: Book#1=The Bernie Madoff Story, #2=The Ken Lay Story >>Finally, the 100th list contains, say 100 different books, including >>our above two losers, but ranked >> List 100 contains: Book#1=The complete works of Shakespeare, >> #2=..., #3=..., ..., and finally, >> #99=The Bernie Madoff Story, #100=The Ken Lay Story >>Clearly, Bernie and Ken suck, but when they're the only two >>books on a list, then they have to rank #1 and #2. >>So you need a methodology that avoids a combined ranking giving >> Bernie: 99 #1scores and one #99score, and giving >> Ken: 99 #2scores and one #100score. >>That would significantly overestimate them. > > This example is badly skewed.
Well, yeah, that's kind of the point. A less skewed list may still contain the same general problem (yes, you get my point (I think)), just not so evidently. That's why I said "extreme case counterexample" above. Moreover, this general problem is already wellknown... Google Simpson's Paradox or just read http://en.wikipedia.org/wiki/Simpson's_paradox In particular, compare and contrast those tables in the wikipedia article with the tables from one of your preceding replies...
> Let's consider some actual data. Here are 3 sample lists each containing > 5 books, but not the same 5 books: > Rank List 1 List 2 List 3 > 1 A B F > 2 B A H > 3 C E C > 4 D G D > 5 E D A > When listed by book, the data looks like this: > List 1 List 2 List 3 > Books Rank Rank Rank > Book A 1 2 5 > Book B 2 1 > Book C 3 3 > Book D 4 5 4 > Book E 5 3 > Book F 1 > Book G 4 > Book H 2 > How would you calculate average rankings?
What you're trying to do is find a way to "combine" those lists into one list. See how Simpson's Paradox illustrates the counterintuitive problems that can arise trying to do that? And that's what I tried to (way less authoritatively) illustrate with my skewed example above.
> First, all of the lists I will use will be > from authoritative sources, they will all have a lot more than 2 books, > and no two lists will be even close to identical. So I have to quibble > with your point about the Bernie and Ken books being that terrible. If > these 100 lists are from reputable sources (as opposed to, say, from > some crackpot on usenet ;)), then the Bernie and Ken books do not > "suck". If the last list is the 100 greatest books of all time, then > even making it to #99 and #100 would be well above the "suck" level. ;) > But I get your point (I think). > > I might include a list containing only science fiction books. This would > exclude 95% of all books, but, as you say, some book on that list will > have to be #1. The #1 book on the sci fi list would probably also be on > at least some of the other lists, so it's ranking there would be > factored in. And I can apply a weighting factor to each list so that the > most substantive lists have a greater influence on the results.
 John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

