推导两个一维正态分布的KL散度

设

$$
P(x) \sim \mathcal{N}(\mu_1, \sigma_1^2), \quad Q(x) \sim \mathcal{N}(\mu_2, \sigma_2^2)
$$

定义

$$
D_{\mathrm{KL}}(P\lVert Q) = \int P(x) \log \frac{P(x)}{Q(x)} dx
$$

则有

$$
\boxed{
D_{\mathrm{KL}}(P\lVert Q) = \ln \frac{\sigma_2}{\sigma_1} + \frac{(\sigma_1^2 - \sigma_2^2) + (\mu_1 - \mu_2)^2}{2\sigma_2^2}
}
$$

Step 1. 分布比值

$$
\frac{P(x)}{Q(x)} = \frac{\sigma_2}{\sigma_1} \exp \left[ \frac{\sigma_1^2(x - \mu_2)^2 - \sigma_2^2(x - \mu_1)^2}{2\sigma_1^2\sigma_2^2} \right]
$$

因此

$$
\log \frac{P(x)}{Q(x)} = \ln \frac{\sigma_2}{\sigma_1} + \frac{(\sigma_1^2 - \sigma_2^2)x^2 + (2\sigma_2^2\mu_1 - 2\sigma_1^2\mu_2)x + (\sigma_1^2\mu_2^2 - \mu_1^2\sigma_2^2)}{2\sigma_1^2\sigma_2^2}
$$

Step 2. 逐项积分

1. 二次项

$$
\int P(x) (\sigma_1^2 - \sigma_2^2) x^2 dx
= (\sigma_1^2 - \sigma_2^2) \mathbb{E}[x^2]
= (\sigma_1^2 - \sigma_2^2)(\mu_1^2 + \sigma_1^2) \equiv A
$$

2. 一次项

$$
\int P(x) (2\sigma_2^2\mu_1 - 2\sigma_1^2\mu_2) x dx
= (2\sigma_2^2\mu_1 - 2\sigma_1^2\mu_2)\mathbb{E}[x]
= (2\sigma_2^2\mu_1 - 2\sigma_1^2\mu_2)\mu_1 \equiv B
$$

3. 常数项

$$
\int P(x)(\sigma_1^2\mu_2^2 - \mu_1^2\sigma_2^2) dx
= \sigma_1^2\mu_2^2 - \mu_1^2\sigma_2^2 \equiv C
$$

Step 3. 合并结果

$$
A + B + C
= \sigma_1^2\bigl(\sigma_1^2 + \mu_1^2 - \sigma_2^2 - 2\mu_1\mu_2 + \mu_2^2\bigr)
$$

因此

$$
D_{\mathrm{KL}}(P\lVert Q)
= \ln \frac{\sigma_2}{\sigma_1} + \frac{A + B + C}{2\sigma_1^2\sigma_2^2}
$$

化简后得到最终结果：

$$
\boxed{
D_{\mathrm{KL}}(P\lVert Q) = \ln \frac{\sigma_2}{\sigma_1} + \frac{(\sigma_1^2 - \sigma_2^2) + (\mu_1 - \mu_2)^2}{2\sigma_2^2}
}
$$