On Feb 23, 11:20 am, Jennifer Murphy <JenMur...@jm.invalid> wrote: > On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich > > <rich.ulr...@comcast.net> wrote: > >You give no hint, that I notice, of what it is that you > >are trying to accomplish. > > >For most purposes of inference that come to my mind, > >the extreme cases -- the ones that you seem to propose > >to drop -- are the most informative and most interesting. > >So I conclude that your interests are probably the opposite > >(in some fashion) from what my naive interests would be. > > >I repeat-- What are you trying to do? > > I am trying to calculate for each word the relative likeliness that it > would be encountered by an average well-educated person in their daily > activities: reading the paper, listening to the news, attending classes, > talking to other people, reading books, etc. > > The raw scores that I have already do that, but I question the > weighting.I do not think that the average person encounters the types of > words typically found in academic journals at the same frequency as they > would those found in newspapers or magazines. Therefore, I want to > re-weight the five sources to reflect a more average experience.
The "average" well-educated person will never read an academic journal. Whether or not a well-educated person will read a novel (or fiction in general) will depend strongly on whether that person is male or female.