On Feb 23, 3:20 pm, Jennifer Murphy <JenMur...@jm.invalid> wrote: > On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich > > <rich.ulr...@comcast.net> wrote: > >You give no hint, that I notice, of what it is that you > >are trying to accomplish. > > >For most purposes of inference that come to my mind, > >the extreme cases -- the ones that you seem to propose > >to drop -- are the most informative and most interesting. > >So I conclude that your interests are probably the opposite > >(in some fashion) from what my naive interests would be. > > >I repeat-- What are you trying to do? > > I am trying to calculate for each word the relative likeliness that it > would be encountered by an average well-educated person in their daily > activities: reading the paper, listening to the news, attending classes, > talking to other people, reading books, etc. > > The raw scores that I have already do that, but I question the > weighting.I do not think that the average person encounters the types of > words typically found in academic journals at the same frequency as they > would those found in newspapers or magazines. Therefore, I want to > re-weight the five sources to reflect a more average experience.
Seems to me what you are looking for is a kind of `basket of information` that the `average well-educated person` would encounter in their `daily activities`. So, I guess, those should be your weights. In other words, you need to assess the volume of spoken sources, etc. the average person you are interested in is exposed to in each of the five groups. You could use that as a basis for your weighting. Whether it beats the raw (equal) weights is not immediately clear.