$$

The entropy¹ of a random variable is a quantitative notion of how uncertain we are about its outcome. More precisely, it’s the “average surprise”² you get from samples, where surprise is measured as $\log \left(\frac{1}{\text{probability of getting that outcome}}\right)$:

$$\mathrm{H}[\mathit{X}]:=\underset{x\sim \mathit{X}}{\mathrm{E}}[\log \frac{1}{\mathrm{Pr}[\mathit{X}=x]}].$$

It’s the average “amount of information” you get from observing $\mathit{X}$: the number of bits you need to describe draws from $\mathit{X}$, amortized over many draws.

By extension, we define the entropy of a distribution $P$ as

$$\mathrm{H}\left(P\right):=\underset{x\sim P}{\mathrm{E}}[\log \frac{1}{P(x)}].$$

Properties

• Entropy ranges between $0$ (if $\mathit{X}$ is deterministic) and $\log |\mathcal{X}|$ (if $\mathit{X}$ is uniform), where $\mathcal{X}$ is the support of $\mathit{X}$.
• It’s subadditive: $\mathrm{H}[\mathit{X},\mathit{Y}]\le \mathrm{H}[\mathit{X}]+\mathrm{H}[\mathit{Y}]$, with equality iff $\mathit{X}$ and $\mathit{Y}$ are independent.
  • Intuitively, you can’t be more surprised by the combination of $\mathit{X}$ and $\mathit{Y}$ than you were by seeing both outcomes separately.
  • The “correlated part” of $\mathit{X}$ and $\mathit{Y}$ is counted once in $\mathrm{H}[\mathit{X},\mathit{Y}]$ but twice in $\mathrm{H}[\mathit{X}]+\mathrm{H}[\mathit{Y}]$.

We will prove $\mathrm{H}[X]\le \log |\mathcal{X}|$ by looking at entropy deficit, and subadditivity by looking at mutual information.

Binary entropy

In particular, let $H(p):=p\log \frac{1}{p}+(1-p)\log \frac{1}{1-p}$ denote the entropy of a Bernoulli with mean $p$. Then

• $H(p)$ is concave,
• $H(p)$ is symmetric around $\frac{1}{2}$: $H(p)=H(1-p)$,
• when $p\approx \frac{1}{2}$, $H(p)=1-\mathrm{\Theta }\left({(p-\frac{1}{2})}^2\right)$,
• when $p\le \frac{1}{2}$, $H(p)=\mathrm{\Theta }(p\log \frac{1}{p})$.

Conditional entropy

#to-write make the expectation explicit Similarly, we can define conditional entropy as the average surprise you get from $\mathit{Y}$ if you’ve already seen $\mathit{X}$, or the number of bits you need to describe draws from $\mathit{Y}$ to someone who knows $\mathit{X}$:

$$\begin{array}{rl}\mathrm{H}\left[\mathit{Y}\right|\mathit{X}]:=& \underset{x\sim \mathit{X}}{\mathrm{E}}[\mathrm{H}[{\mathit{Y}|}_{\mathit{X}=x}]]\\ =& \underset{(x,y)\sim (\mathit{X},\mathit{Y})}{\mathrm{E}}[\log \frac{1}{\mathrm{Pr}[\mathit{Y}=y|\mathit{X}=x]}]\end{array}$$

where ${\mathit{Y}|}_{\mathit{X}=x}$ denotes the random variable obtained by conditioning $\mathit{Y}$ on $\mathit{X}=x$. Confusingly, $\mathrm{H}\left[\mathit{Y}\right|\mathit{X}]$ denotes a fixed value, in contrast with the conditional expectation $\mathrm{E}\left[\mathit{Y}\right|\mathit{X}]$, which is a random variable depending on $\mathit{X}$.

Chain rule

The conditional entropy gives a chain rule:

$$\begin{array}{rl}\mathrm{H}[\mathit{X},\mathit{Y}]& =\underset{(x,y)\sim (\mathit{X},\mathit{Y})}{\mathrm{E}}[\log \frac{1}{\mathrm{Pr}[\mathit{X}=x\wedge \mathit{Y}=y]}]\\ & =\underset{(x,y)\sim (\mathit{X},\mathit{Y})}{\mathrm{E}}[\log \frac{1}{\mathrm{Pr}[\mathit{X}=x]\mathrm{Pr}[\mathit{Y}=y|\mathit{X}=x]}]\\ & =\mathrm{H}[\mathit{X}]+\mathrm{H}\left[\mathit{Y}\right|\mathit{X}],\end{array}$$

which immediately shows that adding data can only increase the entropy

$$\mathrm{H}[\mathit{X},\mathit{Y}]\ge \mathrm{H}[\mathit{X}].$$

Together with subadditivity, this shows that conditioning can only decrease entropy

$$\begin{array}{}\text{(subadditivity)}& \mathrm{H}\left[\mathit{Y}\right]& \ge \mathrm{H}[\mathit{X},\mathit{Y}]-\mathrm{H}[\mathit{X}]\text{(chain rule)}& & =\mathrm{H}\left[\mathit{Y}\right|\mathit{X}],\end{array}$$

with equality iff $\mathit{X}$ and $\mathit{Y}$ are independent.

Mixtures and concavity

If we see $\mathit{Y}$ as a mixture³ ${\mathit{Y}}_{\mathit{X}}$ of the variables ${\mathit{Y}}_x:={\mathit{Y}|}_{\mathit{X}=x}$ weighted by $\mathit{X}$, then the above is saying that entropy is strictly concave:

$$\underset{\text{average of entropies}}{\overset{\mathrm{H}\left[\mathit{Y}\right|\mathit{X}]}{\overbrace{\underset{x\sim \mathit{X}}{\mathrm{E}}\left[\mathrm{H}\left[{\mathit{Y}}_x\right]\right]}}}\le \underset{\text{entropy of mixture}}{\overset{\mathrm{H}\left[\mathit{Y}\right]}{\overbrace{\mathrm{H}\left[{\mathit{Y}}_{\mathit{X}}\right]}}}\le \underset{\text{entropy of ``disjoint mixture''}}{\overset{\mathrm{H}[\mathit{X},\mathit{Y}]}{\overbrace{\mathrm{H}[\mathit{X},{\mathit{Y}}_{\mathit{X}}]}}},$$

with equality respectively iff the variables ${\mathit{Y}}_x$ are all indentical, and iff they have disjoint supports.

If we take a multiplicative view of entropy, then the entropy of a disjoint mixture is upper-bounded by the sum of the entropies:

$$\exp \mathrm{H}[\mathit{X},{\mathit{Y}}_{\mathit{X}}]\le \sum _x\exp \mathrm{H}\left[{\mathit{Y}}_x\right],$$

and this is achieved if $\mathit{X}$ is distributed according to $P(x)\propto \exp \mathrm{H}\left[{\mathit{Y}}_x\right]$.

Use as a prior

#to-write term like $-\mathrm{H}(P)$ to the loss

Gradient

Over logits

Suppose that the learned distribution $P$ is generated by first computing a logit ${\tau }_x$ for each possible value, then taking the softmax

$$P(x):=\frac{e^{{\tau }_x}}{\sum _z{e}^{{\tau }_z}}.$$

Then the direction of greatest increase of $\mathrm{H}\left(P\right)$ as a function of the logits $\tau$ is given by the gradient of the expectation that defines $\mathrm{H}(P)$,

$$\begin{array}{rl}\mathrm{\nabla }{}_{\tau }\mathrm{H}\left(P\right)& =\mathrm{\nabla }{}_{{\tau }_x}\left(\underset{\mathit{y}\sim P}{\mathrm{E}}[\log \frac{1}{P(\mathit{y})}]\right)\\ & =\underset{\text{relative probability gradient}}{\underbrace{\underset{\mathit{y}\sim P}{\mathrm{E}}[\frac{\mathrm{\nabla }{}_{\tau }P}{P}(\mathit{y})\log \frac{1}{P(\mathit{y})}]}}+{\cancel{\underset{\text{direct gradient}}{\underbrace{\underset{\mathit{y}\sim P}{\mathrm{E}}[\mathrm{\nabla }{}_{\tau }\log \frac{1}{P(\mathit{y})}]}}}}^0,\end{array}$$

where the direct gradient term goes to zero because

$$\begin{array}{rl}\underset{\mathit{y}\sim P}{\mathrm{E}}[\mathrm{\nabla }{}_{\tau }\log \frac{1}{P(\mathit{y})}]& =-\underset{\mathit{y}\sim P}{\mathrm{E}}\left[\frac{\mathrm{\nabla }{}_{\tau }P(\mathit{y})}{P(\mathit{y})}\right]\\ & =-\int \mathrm{\nabla }{}_{\tau }P(y)\mathrm{d}y\\ & =-\mathrm{\nabla }{}_{\tau }\int P(y)\mathrm{d}y\\ & =-\mathrm{\nabla }{}_{\tau }1.\end{array}$$

(It makes sense that this term goes to zero, since it roughly corresponds to trying to increase the entropy by “decreasing the probability uniformly everywhere at once”, which is impossible.)

Using the expression for the relative derivatives of softmax, this further becomes

$$\begin{array}{rl}{(\mathrm{\nabla }{}_{\tau }\mathrm{H}\left(P\right))}_x& =\underset{\mathit{y}\sim P}{\mathrm{E}}[(\mathbb{1}(\mathit{y}=x)-P(x))\log \frac{1}{P(\mathit{y})}]\\ & =P(x)\log \frac{1}{P(x)}-P(x)\underset{\mathit{y}\sim P}{\mathrm{E}}[\log \frac{1}{P(\mathit{y})}]\\ & =P(x)(\log \frac{1}{P(x)}-\mathrm{H}(P)).\end{array}$$

Intuitively, this says that logit ${\tau }_x$ should increase if $x$ is rarer than a typical draw from $P$ (which is operationalized as $\log \frac{1}{P(x)}>\mathrm{H}(P)$), but that this only matters proportionally to the current probability $P(x)$. Also note that this expression has mean $0$ over the possible values $x$, which seems desirable: shifting all the logits by a constant has no effect on $P$, so we might as well keep the mean of the logits fixed.

Over probabilities

Suppose we increase $P(x)$ and shrink the other probabilities ${\left(P\left(y\right)\right)}_{y\ne x}$ while keeping their proportions. This can be modeled by increasing the logit ${\tau }_x$ while keeping the other logits ${\left({\tau }_y\right)}_{y\ne x}$ constant. So the change in entropy by an infinitesimal change like this is given by

$$$$

In addition, we have

$$\mathrm{\nabla }{}_{{\tau }_x}P(x)=P(x)(1-P\left(x\right)),$$

so the effect of increasing $P(x)$ by an infinitesimal amount by tweaking ${\tau }_x$ only (which means the other probabilities $P(y)$ for $y\ne x$ will get shrunk while keeping their proportions) is

$$\begin{array}{rl}\mathrm{\nabla }{}_{P(x)}\mathrm{H}(P)& =\frac{\mathrm{\nabla }{}_{{\tau }_x}\mathrm{H}\left(P\right)}{\mathrm{\nabla }{}_{{\tau }_x}P(x)}\\ & =\frac{\log \frac{1}{P\left(x\right)}-\mathrm{H}\left(P\right)}{1-P(x)}.\end{array}$$

---

#to-write

• the “amount of information” thing makes sense? because it’s the ==information you gain== if you started from a prior of $\mathit{X}$’s distribution then you actually observe $\mathit{X}$? i.e. ${\mathrm{E}}_{\mathit{x}\sim [\mathit{X}]}\left[\mathrm{D}\left[\begin{array}{c}\phantom{\left({\mathit{X}|}_{\mathit{X}=\mathit{x}}\right)}{\mathit{X}|}_{\mathit{X}=\mathit{x}}\\ \\ \phantom{\left(\mathit{X}\right)}\mathit{X}\end{array}\right]\right]$
• okay I’m now thoroughly KL-pilled, it’s the source of everything. $\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ is the information you’ve gained if you started from a prior of $P$ and somehow your prior is now $Q$
  • but I want to understand the sense if which $\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ is a lower bound on the “number of questions you must have asked in order to get there”?
    • i.e. the information $\mathit{X}$ gains you about $\mathit{X}$ is limited by the information that it gives about itself
    • i.e. ${\mathrm{E}}_{\mathit{x}\sim [\mathit{X}]}\left[\mathrm{D}\left[\begin{array}{c}\phantom{\left({\mathit{Y}|}_{\mathit{X}=\mathit{x}}\right)}{\mathit{Y}|}_{\mathit{X}=\mathit{x}}\\ \\ \phantom{\left(\mathit{Y}\right)}\mathit{Y}\end{array}\right]\right]\ge \mathrm{H}[\mathit{X}]$?
    • actually I believe you’ve just rediscovered $\mathrm{I}[\mathit{X};\mathit{Y}]\ge \mathrm{I}[\mathit{X};\mathit{X}]=\mathrm{H}[\mathit{X}]$
      • darn information theory is so cool
    • and btw you can write this as $\mathrm{E}\left[\mathrm{D}\left[\begin{array}{c}\phantom{\left(\mathit{Y}|\mathit{X}\right)}\mathit{Y}|\mathit{X}\\ \\ \phantom{\left(\mathit{Y}\right)}\mathit{Y}\end{array}\right]\right]$
      • and this directly shows that it’s equal to $\mathrm{I}[\mathit{X};\mathit{Y}]=\mathrm{H}[\mathit{Y}]-\mathrm{H}\left[\mathit{Y}\right|\mathit{X}]$ I guess
        • though maybe you should swallow the pill and not talk about entropy ever again!
        • note: I guess it also means $\mathrm{I}[\mathit{X};\mathit{Y}]=\mathrm{I}[\mathit{Y};\mathit{Y}]-\mathrm{I}[\mathit{Y};\mathit{Y}|\mathit{X}]=-\mathrm{I}[\mathit{Y};\mathit{Y};\mathit{X}]$?
          • cool story bro
          • wait is it the case that in general if you add a variable that’s already present it just flips the sign?
          • $\mathrm{I}[\mathit{X};\mathit{Y};\mathit{Z};\mathit{Z}]=\mathrm{I}[\mathit{X};\mathit{Y};\mathit{Z}|\mathit{Z}]-\mathrm{I}[\mathit{X};\mathit{Y};\mathit{Z}]$
            • and somehow I’m supposed to just know that $\mathrm{I}[\mathit{X};\mathit{Y};\mathit{Z}|\mathit{Z}]=0$?
            • I guess this just suggests that if any variable is a constant then the interaction is $0$, which would make a whole lot of sense tbh
            • very similar to Wick in that case!
      • really, entropy should be negative…
  • okay I’m this close to just replacing all occurrences of $\mathrm{D}$ by $\mathrm{I}$
    • and to have mutual information as a more core concept than entropy
    • and to literally just define $\mathrm{H}[\mathit{X}]$ as $\mathrm{I}[\mathit{X};\mathit{X}]$
      • this would explain why you have to somehow bother to use two copies of $\mathit{X}$ in the definition of entropy!
        • (and same thing in basically all of the power entropies)
      • ultimately ==information is about something providing information about another thing==
        • there is no such thing as “the information of one thing”
        • and the information divergence is the core of what information means
        • and it’s also what makes information theory so nice: it provides all the nontrivial inequalities!
      • information divergence measures the size of a partial update, whereas entropy measures the (average) size of a total update
    • does $\mathrm{I}[\mathit{X};\mathit{X};\mathit{X}]$ make sense? it’s $\mathrm{I}[\mathit{X};\mathit{X}|\mathit{X}]-\mathrm{I}[\mathit{X};\mathit{X}]=0-\mathrm{H}[\mathit{X}]=-\mathrm{H}[\mathit{X}]$ hmm maybe this is slightly cursed?
    • and $\mathrm{I}[\mathit{X};\mathit{X};\mathit{X};\mathit{X}]=\mathrm{I}[\mathit{X};\mathit{X};\mathit{X}|\mathit{X}]-\mathrm{I}[\mathit{X};\mathit{X};\mathit{X}]=\mathrm{H}[\mathit{X}]$? so it keeps alternating, hmmm…
      • actually this is support for defining $\mathrm{I}[\mathit{X}]:=-\mathrm{H}[\mathit{X}]$ instead of $\mathrm{D}[\mathit{X}]$
        • indeed, $\mathrm{I}[\mathit{X};\mathit{X}]=\mathrm{I}\left[\mathit{X}\right|\mathit{X}]-\mathrm{I}\left[\mathit{X}\right]$
          • and I’m not sure what this first quantity $\mathrm{I}\left[\mathit{X}\right|\mathit{X}]$ is supposed to mean but it sure seems like it should be $0$?
            • if you set $\mathrm{I}[\mathit{X}]:=\mathrm{D}[\mathit{X}]$ then this quantity is $\mathrm{H}[\mathit{X}]+\mathrm{D}[\mathit{X}]=$ entropy of the prior on $\mathit{X}$?
          • actually very confusing
        • and $\mathrm{I}[\mathit{X}]=\mathrm{I}\left[\right|\mathit{X}]-\mathrm{I}\left[\right]$
          • i.e. “how much does $\mathit{X}$ help you predict nothing”??
          • so we’d have to accept that $\mathrm{I}\left[\right|\mathit{X}]=\mathrm{I}[\mathit{X}]+C=C-\mathrm{H}[\mathit{X}]$
            • i.e. $\mathit{X}$ is actively detrimental to the task of predicting nothing? (???)
        • actually feels like we should have $\mathrm{I}\left[\right|\mathit{X}]=\mathrm{I}[]$ given that $\mathit{X}$ is not dependent with anything in the LHS (this is definitely a rule right?)
          • which would mean $\mathrm{I}[\mathit{X}]=0$
          • but then $\mathrm{I}\left[\mathit{X}\right|\mathit{X}]=\mathrm{I}[\mathit{X};\mathit{X}]=\mathrm{H}[\mathit{X}]$, even though $\mathrm{I}\left[\mathit{X}\right|\mathit{X}]={\mathrm{E}}_{\mathit{x}\sim [\mathit{X}]}\left[\mathrm{I}\left[{\mathit{X}|}_{\mathit{X}=\mathit{x}}\right]\right]$ and ${\mathit{X}|}_{\mathit{X}=x}$ is literally a point mass no matter what $x$ is
            • actually that’s a general argument for $\mathrm{I}\left[\mathit{X}\right|\mathit{X}]$ being a constant $C$ (not clear which constant, but $0$ would seem like a reasonable choice here since we get zero whenever there is any deterministic constant in the expression?)
            • so $\mathrm{I}[\mathit{X}]$ should be $C-\mathrm{H}[\mathit{X}]$
        • to summarize:
          • given that $\mathrm{I}[\mathit{X}]=C-\mathrm{H}[\mathit{X}]$ is the only tenable position, we would have to have $\mathrm{I}\left[\right|\mathit{X}]=C^{\prime }-\mathrm{H}[\mathit{X}]$
          • but then that violates the rule of “if $\mathit{X}$ is independent from the joint distribution of the things in the LHS then the information should be $0$”
        • okay I vote for $\mathrm{I}[\mathit{X}]$ just being undefined!! $\mathrm{I}[\mathit{X};\mathit{Y}]$ is clearly the core quantity here
          • and it seems like this makes things better in the case of continuous random variables as well
      • it’s in fact somewhat baffling that the interaction information is even well-defined (that it’s consistent and symmetric)
    • actually I think I’d be on board to keep $\mathrm{D}[\cdot ]$ only for the deficit
• information theory is specifically about making dimensionless statistics additive
• KL divergence is to entropy the same as squared distance (!!) is to variance?

1. More precisely “Shannon entropy” or “information entropy” to separate it from the wider class of dimensionless uncertainty measures based described in Generalized entropies. ↩

2. Note that this is a pretty stupid notion of surprise. If I ask you to draw me numbers between $1$ and $1000$ and everytime you do so I go “woah that’s crazy, this number only have a $1/1000$ chance of appearing”, then you would be justified in questioning my sanity. ↩

3. a convex combination of distributions ↩