Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
Drexel University or The Math Forum.


jdm
Posts:
16
Registered:
11/22/10


Claimed approximation for KullbackLeibler distance D(pq) when p and q are close, apparently based on Taylor series
Posted:
Jan 1, 2012 6:36 PM


The following claim featured in a research paper I've been studying  however, no proof accompanied it beyond a statement that the approximation could be obtained using Taylor series at order 2  and it wasn't clear what the variable was supposed to be or around which point.
Let p and q be discrete probability distributions of random variables taking on values from a set with M+1 elements;
p=(p_0, ..., p_M)
(p_i = P(random variable with distribution p is equal to i))
Likewise, q=(q_0, ..., q_M)
Where p and q are close  defined as p_{i}  q_{i} << q_{i} \forall i  let e_i denote the value (p_i  q_i).
Then, according to the paper, D(pq) \approx D(qp) \approx sum_{i=0} ^{M}(e_{i}^{2}/q_{i})/2.
I haven't been able to verify this approximation for myself, as I stated, and if anyone reading this can help (even by arguing that the approximation isn't in fact valid) it would be much appreciated!
Many thanks,
James McLaughlin.



