Chris
Posts:
4
Registered:
11/1/11


Re: information theory?
Posted:
Nov 3, 2011 11:39 AM


On Nov 2, 2:51 pm, RichD <r_delaney2...@yahoo.com> wrote: > On Nov 2, Chris <chris.sant...@gmail.com> wrote: > > > Be careful going down this path. For me it was > > much more interesting in theory than in practice. > > > Me in 2008: "Like, what *is* the nature of information, > > man?" (waves hand to clear smoke from the room) > > > Then I signed up for Info Theory, and I spent > > a whole semester taking integrals of logs of > > probability distributions and proving that one > > was bounded by the other. > > It's too bad you got so little out of it. > It's not 'just a theory', it's a field > where theory and practice coincide. > The theorists concoct the recipes, the > engineers follow the instructions, and > it works, bang on. Fukengrooven. > And it's the root of many a Silly Con > Valley fortune. > > I strongly suggest, if you have some > free neurons and time and nothing's good on > tv, read Shannon's original paper. It's > utterly fucking brilliant. There's no doubt, > in Valhalla, Shannon shares a table with Einstein. > > It's sort of funny, he called it communication > theory, it was his disciples who popularized > 'information theory'. > >  > Rich
One of the coolest ideas I remember learning about is directed divergence:
http://en.wikipedia.org/wiki/Kullback?Leibler_divergence
and being able to estimate how much it will cost you in bits if your estimate of the underlying probability distribution of the information you're trying to code wrong. I'd cite the fact that  even though I'm not a super expert on all the topics  the fact that I can pick up these topics again rather quickly as evidence that dipping my toes in the field was worth it. School's not about becoming an expert  it's about working the muscles so you can become an expert if you need to.

