Analyzing Survey Results

Date: 05/04/2000 at 15:34:05
From: Diane Davis
Subject: Survey statistics

A satisfaction survey was given that asked people to rank, in order of 
importance, seven factors. I need to determine the meaning of the 
results (e.g. what is the most important factor, second most important 
factor, etc.) I've tabulated the results (see below) for each 
category, and determined the percentage it represents of the ranking 
and of the category as a whole.
 
               1st  2nd  3rd  4th  5th  6th  7th  8th
     Factor 1   15   36   13    9   12    4    9    0
     Factor 2   26    9   23   13   12    9    1    4
     Factor 3    3    4    9   18   20   21   17    2    
     Factor 4    3    8    5   16   16   20   16    9
     Factor 5    7    6   12    8   12   11   19   21
     Factor 6    7   14    9   18   12   11   19    6
     Factor 7   16   16   16   12   10   17    9    0

How do you make a determination of priority when, for example, Factor 
2 received the greatest number of responses for the first and third 
ranks?

Any help you give is appreciated. Thank you.

Date: 05/04/2000 at 20:50:58
From: Doctor Schwa
Subject: Re: survey statistics

In general, this problem is impossible to solve. Otherwise we'd have 
voters rank the candidates in an election and then use the rankings to 
pick an unambiguous overall favorite. But there's a nice little proof 
out there that, according to a few simple rules, there is no possible 
system that will always give the "best" answer.

So, we have to make do somehow. Statistically the easiest thing to 
work with would be the average rank assigned to each factor, so why 
not use that?

By that criterion, Factor 1 edges out 2 slightly; 7 is a distant 
third, followed by 3, 4, 5, and 6.

By the way, why are there 8 places when there are only 7 factors? And 
if people had to rank each thing, why isn't there the same number of 
people total each time? Did everybody leave one rank spot blank? Also 
each factor didn't get ranked by the same number of people...

- Doctor Schwa, The Math Forum
  http://mathforum.org/dr.math/   

Date: 05/09/2000 at 15:00:57
From: Diane Davis
Subject: Re: survey statistics

Thank you for your help.  

The reason there were 8 places is that there was an "other" response 
where an answer could be written in. Some chose to use this, but there 
weren't more than 2 of the same response.

Also, some respondents may have only indicated their 1st through 5th 
choices, etc. which is why the total responses were not equal for each 
factor.

This was clearly a very flawed instrument; however, it was never 
really intended to be used statistically. That issue came up after the 
fact.

How would you recommend designing a more useful "attitude" survey tool 
in which results could be more accurately tabulated? If you could 
recommend some resource books, Web sites, etc. I would appreciate it.

Thanks again for all of your help and quick response.

Date: 05/11/2000 at 13:16:49
From: Doctor Schwa
Subject: Re: survey statistics

>The reason there were 8 places is that there was an "other" response 
>where an answer could be written in. Some chose to use this, but 
>there weren't more than 2 of the same response.

"Other" is always dangerous if you want to get consistent data that 
you can compare from person to person. But "other" is really useful if 
you want to get qualitative data; people will write in things that are 
very useful anecdotally, but very hard to analyze quantitatively.

>Also, some respondents may have only indicated their 1st through 5th 
>choices, etc. which is why the total responses were not equal for 
>each factor.

Yes, so I recommend that you ask people, instead of ranking them in 
order of importance, to rate each one on a scale. For instance: 
"choose a rating from 1 to 10, with 1 being totally unimportant and 10 
being vitally important."

Then taking the average clearly makes sense. Ranks are much harder to 
work with. Ranks also don't show so much when people have a clear 
favorite or when there are two or three that are about equally 
important to them.

>This was clearly a very flawed instrument; however, it was never 
>really intended to be used statistically. That issue came up after 
>the fact.
>
>How would you recommend designing a more useful "attitude" survey 
>tool in which results could be more accurately tabulated? If you 
>could recommend some resource books, websites, etc. I would 
>appreciate it.

Survey design is a difficult topic. It's not taught particularly well 
in most college stat courses. I haven't seen any particularly great 
books on survey design. I recommend visiting the apstat-l discussion 
list, archived by the Math Forum:

   http://mathforum.org/kb/forum.jspa?forumID=67   

and searching the archives of the list for words like "survey design" 
- if you don't find anything good there, post a message asking for a 
suggested reference. We may be the math doctors, but they are the real 
experts on statistics.

>Thanks again for all of your help and quick response.

No problem!

- Doctor Schwa, The Math Forum
  http://mathforum.org/dr.math/