$$

Suppose you have some energy function $E(\cdot )$ and you to find a distribution $q$ which strikes a balance between minimizing energy while being reasonably spread. Then a reasonable choice of objective is to subtract the entropy, giving

$$\mathcal{L}(q):=\underset{q(\mathit{z})}{\mathrm{E}}[E(\mathit{z})]-\mathrm{H}(q(\mathit{z})).$$

This is called the variational free energy: allegedly, in physics, it represents the part of the energy of system $q$ which is actually available for doing useful work.

Minimizing the free energy

Let $q^{\ast }$ be the distribution which minimizes the free energy. Since probabilities must sum to $1$, there must be a constant $\lambda$ such that for all $z$,

$$\begin{array}{rl}0& =\frac{\mathrm{d}}{\mathrm{d}{q}^{\ast }(z)}(\mathcal{L}\left(q^{\ast }\right)+\lambda \sum _{z^{\prime }}{q}^{\ast }\left(z^{\prime }\right))\\ & =E(z)+\log q^{\ast }(z)+1+\lambda ,\end{array}$$

so the unique solution has $q^{\ast }$ be exponentially sensitive to the energy:

$$\begin{array}{rl}{q}^{\ast }(z)& =\exp (-E(z)-\lambda -1)\\ & \propto \exp (-E(z)),\end{array}$$

and more precisely given by $q^{\ast }(z)=\frac{\exp (-E(z))}{Z}$ where

$$Z:=\int \exp (-E(z))\mathrm{d}z$$

is the partition function.

Interestingly, minimizing the original loss $\mathcal{L}(q)$ is equivalent to minimizing the information divergence between $q$ and $q^{\ast }$:

$$\begin{array}{rl}\mathcal{L}\left(q\right)& =\underset{q(\mathit{z})}{\mathrm{E}}[E(\mathit{z})]-\mathrm{H}(q(\mathit{z}))\\ & =\underset{q(\mathit{z})}{\mathrm{E}}[-\log q^{\ast }(\mathit{z})-\log Z]+\underset{q(\mathit{z})}{\mathrm{E}}[\log q(\mathit{z})]\\ & =\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{(q(\mathit{z}))}q(\mathit{z})}\\ \\ {\displaystyle \phantom{(q^{\ast }(\mathit{z}))}{q}^{\ast }(\mathit{z})}\end{array}\right)+\log \frac{1}{Z}.\end{array}$$

Non-uniform priors

If instead we encourage $q$ to be spread with respect to some reference distribution (or “prior”) $p$, i.e. if we try to minimize

$$\mathcal{L}(q):=\underset{q(\mathit{z})}{\mathrm{E}}[E(\mathit{z})]+\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{(q(\mathit{z}))}q(\mathit{z})}\\ \\ {\displaystyle \phantom{(p(\mathit{z}))}p(\mathit{z})}\end{array}\right),$$

then we can reduce to the previous case by folding $p(z)$ into the energy term:

$$\underset{q(\mathit{z})}{\mathrm{E}}[E(\mathit{z})]+\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{(q(\mathit{z}))}q(\mathit{z})}\\ \\ {\displaystyle \phantom{(p(\mathit{z}))}p(\mathit{z})}\end{array}\right)=\underset{q(\mathit{z})}{\mathrm{E}}[E(\mathit{z})-\log p(\mathit{z})]-\mathrm{H}(q(\mathit{z})).$$

This means that the minimizer $q^{\ast }$ is given by $q^{\ast }(z)=\frac{p(z)\exp (-E(z))}{Z}$ where

$$Z:=\int p(z)\exp (-E(z))\mathrm{d}z$$

and we still have $\mathcal{L}(q)=\mathrm{D}\left(\begin{array}{c}\phantom{(q(\mathit{z}))}q(\mathit{z})\\ \\ \phantom{(q^{\ast }(\mathit{z}))}{q}^{\ast }(\mathit{z})\end{array}\right)+\log \frac{1}{Z}$.

In particular, if $E(z)$ represents a negative log likelihood $\log \frac{1}{p\left(x\right|z)}$, then $\mathcal{L}(q)$ is precisely the surprise upper bound, and $q^{\ast }(z)$ is given by the posterior $p\left(z\right|x)$.