$$

Entropies are dimensionless notions of how spread out a random variable is (as opposed to dispersions, which care about distance in space). Concretely, for any convex function $f$ such that $f(0)=f(1)=0$ we can define the $f$-entropy of a distribution $P\in \mathrm{△}(X)$ as

$$\begin{array}{rl}{H}^f(P)& :=-\sum _{x\in X}f(P(x))=-\underset{\mathit{x}\sim P}{\mathrm{E}}\left[\frac{f(P(\mathit{x}))}{P(\mathit{x})}\right].\end{array}$$

Range

By convexity, $f(p)\le 0$ for any $p\in [0,1]$, so $H^f(P)$ is nonnegativew. On the other hand, if we let $n:=|X|$, then by convexity we have

$$\begin{array}{rl}\frac{H^f(P)}{n}& =-\underset{\mathit{x}\in X}{\mathrm{E}}[f(P(\mathit{x}))]\\ \text{(convexity)}& & \le -f\left(\underset{\mathit{x}\in X}{\mathrm{E}}[P(\mathit{x})]\right)& =-f\left(\frac{1}{n}\right),\end{array}$$

so the $f$-entropy takes a maximum of $-nf\left(\frac{1}{n}\right)$ when $P$ is uniform.

Links to other quantities

Informally, $\sum _{x\in X}f(P(x))$ can be understood as the $f$-divergence $D^f\left(\begin{array}{c}\phantom{\left(P\right)}P\\ \\ \phantom{\left(1\right)}1\end{array}\right)$, where $1$ is an imaginary prior distribution that puts probability $1$ on every point. In this sense, the $f$-entropy is (the negative of) the “absolute” version of the $f$-divergence, for a neutral prior. The closer $P$ is to uniform, the lower $D^f\left(\begin{array}{c}\phantom{\left(P\right)}P\\ \\ \phantom{\left(1\right)}1\end{array}\right)$ is, and the higher the entropy $H^f\left(P\right)$ is.

More broadly, we’ll call “$f$-entropy” any (usually decreasing) function of $H^f\left(P\right)$. For example,

• the Shannon entropy is given by $\mathrm{H}(P):=H^f(P)$ for $f(p):=p\log p$ and ranges from $0$ (single point) to $\log n$ (uniform);
• the power entropies are given by $H_{\alpha }(P):=H^f(P{)}^{\frac{1}{1-\alpha }}$ for $f(p):=p^{\alpha }$ and they all range from $1$ (single point) to $n$ (uniform).