Math

设

$$
P(x) \sim \mathcal{N}(\mu_1, \sigma_1^2), \quad Q(x) \sim \mathcal{N}(\mu_2, \sigma_2^2)
$$

定义

$$
D_{\mathrm{KL}}(P\lVert Q) = \int P(x) \log \frac{P(x)}{Q(x)} dx
$$

则有

$$
\boxed{
D_{\mathrm{KL}}(P\lVert Q) = \ln \frac{\sigma_2}{\sigma_1} + \frac{(\sigma_1^2 - \sigma_2^2) + (\mu_1 - \mu_2)^2}{2\sigma_2^2}
}
$$

Step 1. 分布比值

$$
\frac{P(x)}{Q(x)} = \frac{\sigma_2}{\sigma_1} \exp \left[ \frac{\sigma_1^2(x - \mu_2)^2 - \sigma_2^2(x - \mu_1)^2}{2\sigma_1^2\sigma_2^2} \right]
$$

因此

$$
\log \frac{P(x)}{Q(x)} = \ln \frac{\sigma_2}{\sigma_1} + \frac{(\sigma_1^2 - \sigma_2^2)x^2 + (2\sigma_2^2\mu_1 - 2\sigma_1^2\mu_2)x + (\sigma_1^2\mu_2^2 - \mu_1^2\sigma_2^2)}{2\sigma_1^2\sigma_2^2}
$$