On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich <email@example.com> wrote:
>You give no hint, that I notice, of what it is that you >are trying to accomplish. > >For most purposes of inference that come to my mind, >the extreme cases -- the ones that you seem to propose >to drop -- are the most informative and most interesting. >So I conclude that your interests are probably the opposite >(in some fashion) from what my naive interests would be. > >I repeat-- What are you trying to do?
I am trying to calculate for each word the relative likeliness that it would be encountered by an average well-educated person in their daily activities: reading the paper, listening to the news, attending classes, talking to other people, reading books, etc.
The raw scores that I have already do that, but I question the weighting.I do not think that the average person encounters the types of words typically found in academic journals at the same frequency as they would those found in newspapers or magazines. Therefore, I want to re-weight the five sources to reflect a more average experience.