$$

The cross-entropy of $Q$ relative to a prior $P$ measures the “average surprise” you will experience when seeing draws from $Q$ if you were expecting draws from $P$:

$$\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right):=\underset{x\sim Q}{\mathrm{E}}[\log \frac{1}{P(x)}].$$

Interpretation

It’s called “cross-entropy” because it mixes and matches elements from the entropies of $Q$ and $P$: the expectation is taken over $Q$, but the probability values are from $P$. More formally,

Quantity	Relation with $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$
$\mathrm{H}[Q]$	As we’ll see in the next section, $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)\ge \mathrm{H}[Q]$. This is intuitive: you ought to be the least surprised if you were expecting the correct distribution, and $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(Q\right)}Q\end{array}\right)$ is just $\mathrm{H}[Q]$.
$\mathrm{H}[P]$	There is no general relation between $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ and $\mathrm{H}[P]$, but when $P$ is uniform over some set $\mathcal{X}$, then $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)=\log \left|\mathcal{X}\right|=\mathrm{H}[P]$ no matter what $Q$ is.

Just like the entropy of $Q$ measures the number of bits you need to describe draws from $Q$ using an optimal encoding scheme, the entropy of $Q$ relative to $P$ is the number of bits you need to describe draws from $Q$ using an encoding scheme that was designed to be optimal for $P$ (and thus may be suboptimal for $Q$).

#to-write express as I[X at E; X]?

Relation to information divergence

We can decompose cross-entropy into the entropy of $Q$ (the “inherent uncertainty” of draws $Q$) and the information divergence between $Q$ and $P$ (the “additional surprise” from expecting distribution $P$ but getting $Q$):

$$\begin{array}{rl}\stackrel{\text{``total surprise''}}{\overbrace{\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)}}& =\underset{x\sim Q}{\mathrm{E}}[\log \frac{1}{P(x)}]\\ & =\underset{x\sim Q}{\mathrm{E}}[\log \frac{1}{Q(x)}]+\underset{x\sim Q}{\mathrm{E}}[\log \frac{Q(x)}{P(x)}]\\ \text{(1)}& & =\underset{\text{``inherent uncertainty''}}{\underbrace{\mathrm{H}[Q]}}+\underset{\text{``additional surprise''}}{\underbrace{\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)}}& \ge \mathrm{H}[Q],\end{array}$$

with equality iff $P$ and $Q$ are identical.

#to-write chain rule (and observe that it’s not “truly” conditioned in the LHS)

Minimization

When training a generating model, we want to get a model distribution $P$ to approach some target distribution $Q$, so it makes sense to try to minimize a statistical distance like the information divergence

$$\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)=\underset{x\sim Q}{\mathrm{E}}[\log \frac{Q(x)}{P(x)}].$$

However, we typically don’t have access to the probability values $Q(x)$ (they’re exactly what we’re trying to learn); we can only sample from $Q$. Fortunately, equation (1) tells us that it’s equivalent to minimize the cross-entropy $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$, since the difference $\mathrm{H}[Q]$ is fixed, and

$$\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)=\underset{x\sim Q}{\mathrm{E}}[\log \frac{1}{P(x)}]$$

only requires knowledge of $P(x)$.

When working on a finite dataset $\mathcal{D}$, minimizing cross-entropy also corresponds to maximizing the likelihood of observing $\mathcal{D}$, assuming the data points are independent:

$$\underset{x\in \mathcal{D}}{\mathrm{E}}[\log \frac{1}{P(x)}]=-\frac{1}{\left|\mathcal{D}\right|}\log (\prod _{x\in \mathcal{D}}P(x)).$$

The use of cross-entropy as a loss has a few interesting consequences:

• Since $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)\ge \mathrm{H}[Q]$, cross-entropy loss can never drop to zero. But as the loss approaches $\mathrm{H}[Q]$, the divergence $\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ tends to $0$, which forces $P$ to mimic $Q$ with ever more accuracy.
  • On the other hand, we usually don’t know the inherent uncertainty $\mathrm{H}[Q]$ of the real-world distribution, so we can never know for sure how close we are to matching it, and how small the information divergence is.
• Because cross-entropy is the cost of encoding draws from $Q$ using the optimal scheme for $P$, any model with low cross-entropy loss gives a cheap encoding scheme for the real world-distribution $Q$, and vice versa: generative models are compression algorithms.

Gradient over logits

Suppose that the learned distribution $P$ is generated by first computing a logit ${\tau }_x$ for each possible value $x$, then taking the softmax

$$P(x):=\frac{e^{{\tau }_x}}{\sum _z{e}^{{\tau }_z}}.$$

Then direction of greatest decrease of $\mathrm{H}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ as a function of the logits $\tau$ is given by

$$-\mathrm{\nabla }{}_{\tau }\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)=\underset{\mathit{y}\sim Q}{\mathrm{E}}[\mathrm{\nabla }{}_{\tau }\log P(\mathit{y})],$$

which, using the expression for the logarithmic derivatives of softmax, becomes

$$\begin{array}{rl}-{(\mathrm{\nabla }{}_{\tau }\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right))}_x& =\underset{\mathit{y}\sim Q}{\mathrm{E}}[\mathbb{1}(\mathit{y}=x)-P(x)]\\ & =Q(x)-P(x).\end{array}$$

When learning a real world distribution $Q$, since we don’t have access to the values of the density $Q$, we can’t use the expression $Q(x)-P(x)$ on the second line. Instead, we can use the expression on the first line:

• draw some $\mathit{y}\sim Q$;
• push ${\tau }_{\mathit{y}}$ up in by $\propto 1-P(\mathit{y})$;
• push every other ${\tau }_x$ down by $\propto P(x)$.

Backpropagation

If the logits themselves depend on some underlying parameters $\theta$, then

$$\begin{array}{rl}-\mathrm{\nabla }{}_{\theta }\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)& =-\sum _x\mathrm{\nabla }{}_{{\tau }_x}\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)\mathrm{\nabla }{}_{\theta }{\tau }_x\\ & =\sum _x(Q(x)-P(x))\mathrm{\nabla }{}_{\theta }{\tau }_x\\ & =\underset{x\sim Q}{\mathrm{E}}[\mathrm{\nabla }{}_{\theta }{\tau }_x]-\underset{x\sim P}{\mathrm{E}}[\mathrm{\nabla }{}_{\theta }{\tau }_x].\end{array}$$

In particular, if the logits come from some energy function

$${\tau }_x:=-E_{\theta }(x),$$

then

$$-\mathrm{\nabla }{}_{\theta }\mathrm{H}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)=-\underset{x\sim Q}{\mathrm{E}}[\mathrm{\nabla }{}_{\theta }{E}_{\theta }\left(x\right)]+\underset{x\sim P}{\mathrm{E}}[\mathrm{\nabla }{}_{\theta }{E}_{\theta }\left(x\right)].$$

Pushing in this direction means trying to decrease the energy over the target distribution $Q$ but compensating by increasing the energy over the current model distribution $P$, hence the name contrastive divergence.

#to-write the loss isn’t computable in that case but you can get a proxy loss back