Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Claimed approximation for Kullback-Leibler distance D(p||q) when p
and q are close, apparently based on Taylor series

Replies: 0

 jdm Posts: 16 Registered: 11/22/10
Claimed approximation for Kullback-Leibler distance D(p||q) when p
and q are close, apparently based on Taylor series

Posted: Jan 1, 2012 6:36 PM

The following claim featured in a research paper I've been studying -
however, no proof accompanied it beyond a statement that the
approximation could be obtained using Taylor series at order 2 - and
it wasn't clear what the variable was supposed to be or around which
point.

Let p and q be discrete probability distributions of random variables
taking on values from a set with M+1 elements;

p=(p_0, ..., p_M)

(p_i = P(random variable with distribution p is equal to i))

Likewise, q=(q_0, ..., q_M)

Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
i - let e_i denote the value (p_i - q_i).

Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}
^{M}(e_{i}^{2}/q_{i})/2.

I haven't been able to verify this approximation for myself, as I
stated, and if anyone reading this can help (even by arguing that the
approximation isn't in fact valid) it would be much appreciated!

Many thanks,

James McLaughlin.