Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » sci.math.* » sci.math.research

Topic: Claimed approximation for Kullback-Leibler distance D(p||q) when p
and q are close, apparently based on Taylor series

Replies: 0  

Advanced Search

Back to Topic List Back to Topic List  
jdm

Posts: 16
Registered: 11/22/10
Claimed approximation for Kullback-Leibler distance D(p||q) when p
and q are close, apparently based on Taylor series

Posted: Jan 1, 2012 6:36 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

The following claim featured in a research paper I've been studying -
however, no proof accompanied it beyond a statement that the
approximation could be obtained using Taylor series at order 2 - and
it wasn't clear what the variable was supposed to be or around which
point.

Let p and q be discrete probability distributions of random variables
taking on values from a set with M+1 elements;

p=(p_0, ..., p_M)

(p_i = P(random variable with distribution p is equal to i))

Likewise, q=(q_0, ..., q_M)

Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
i - let e_i denote the value (p_i - q_i).

Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}
^{M}(e_{i}^{2}/q_{i})/2.

I haven't been able to verify this approximation for myself, as I
stated, and if anyone reading this can help (even by arguing that the
approximation isn't in fact valid) it would be much appreciated!

Many thanks,

James McLaughlin.




Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.