Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
Drexel University or The Math Forum.
|
|
jdm
Posts:
16
Registered:
11/22/10
|
|
Claimed approximation for Kullback-Leibler distance D(p||q) when p
and q are close, apparently based on Taylor series
Posted:
Jan 1, 2012 6:36 PM
|
|
The following claim featured in a research paper I've been studying -
however, no proof accompanied it beyond a statement that the
approximation could be obtained using Taylor series at order 2 - and
it wasn't clear what the variable was supposed to be or around which
point.
Let p and q be discrete probability distributions of random variables
taking on values from a set with M+1 elements;
p=(p_0, ..., p_M)
(p_i = P(random variable with distribution p is equal to i))
Likewise, q=(q_0, ..., q_M)
Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
i - let e_i denote the value (p_i - q_i).
Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}
^{M}(e_{i}^{2}/q_{i})/2.
I haven't been able to verify this approximation for myself, as I
stated, and if anyone reading this can help (even by arguing that the
approximation isn't in fact valid) it would be much appreciated!
Many thanks,
James McLaughlin.
|
|
|
|
