$$

Direct relations

We can directly “read off” the table for the nonnegative form that:

• $d_{\mathrm{JS}}^2(P,Q)$, $d_{\mathrm{Hel}}^2(P,Q)$, $\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$, ${\chi }^2\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$, $D_{\alpha }\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ all behave similarly in the “tweak” regime;
• $d_{\mathrm{JS}}(P,Q)$ is linearly equivalent to $d_{\mathrm{Hel}}(P,Q)$;¹
• $d_{\mathrm{Hel}}^2(P,Q)\le O(\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right))\le O({\chi }^2\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right))$;
• $d_{\mathrm{Hel}}^2(P,Q)\le O(d_{\mathrm{TV}}(P,Q))$.

In addition, using the tilted definition of $d_{\mathrm{TV}}$, we have

$$\begin{array}{rl}{d}_{\mathrm{TV}}^2(P,Q)& \le \underset{x\sim P}{\mathrm{E}}{\left[\{\begin{array}{ll}O\left(|\frac{Q(x)}{P(x)}-1|\right)& \text{if }\frac{Q(x)}{P(x)}\le 1\\ 0& \text{otherwise}\end{array}\right]}^2\\ \text{(convexity)}& & \le \underset{x\sim P}{\mathrm{E}}\left[\{\begin{array}{ll}O\left({(\frac{Q(x)}{P(x)}-1)}^2\right)& \text{if }\frac{Q(x)}{P(x)}\le 1\\ 0& \text{otherwise}\end{array}\right]\text{(pointwise)}& & \le O(d_{\mathrm{Hel}}^2(P,Q)),\end{array}$$

so $d_{\mathrm{TV}}(P,Q)$ and $d_{\mathrm{Hel}}(P,Q)$ are polynomially equivalent.

Relations that depend on the entropy of the prior

remark like “for the missing links, can be arbitrarily bigger but only if prior has tiny probability values”

Information divergence vs total variation distance

Let

• $p:=\underset{x}{min}P(x)$ be the smallest probability in the prior,
• ${\delta }_x:=Q(x)-P(x)$ be the vector of the probability differences, so that $\frac{1}{2}{\Vert \delta \Vert }_1$ is the total variation distance between $P$ and $Q$.

Then

$$\begin{array}{rl}\mathrm{D}\left(\begin{array}{c}{\displaystyle \phantom{\left(Q\right)}Q}\\ \\ {\displaystyle \phantom{\left(P\right)}P}\end{array}\right)& =\sum _{x}Q(x)\log \frac{Q(x)}{P(x)}\\ & =\sum _x(P(x)+{\delta }_x)\log (1+\frac{{\delta }_x}{P(x)})\\ & \le \sum _x(p+|{\delta }_x|)\log (1+\frac{|{\delta }_x|}{p})\\ \text{(superadditivity)}& & \le (p+\sum _x|{\delta }_x|)\log (1+\frac{\sum _x|{\delta }_x|}{p})& =(p+{\Vert \delta \Vert }_1)\log (1+\frac{{\Vert \delta \Vert }_1}{p})\\ & \le \{\begin{array}{ll}O\left({\Vert \delta \Vert }_1\right)& \text{if }{\Vert \delta \Vert }_1\le O(p)\\ O({\Vert \delta \Vert }_1\log \frac{{\Vert \delta \Vert }_1}{p})& \text{otherwise.}\end{array}\end{array}$$

#to-write generalize it (e.g. to chi squared)

Summary

#figure using Levels of closeness notation

#to-write add the link $d_{\mathrm{TV}}\le D_{\mathrm{\infty }}$, which is so cute

1. In particular, as a divergence measure, $d_{\mathrm{JS}}^2(P,Q)$ doesn’t have that much in common with the information divergence $\mathrm{D}\left(\begin{array}{c}\phantom{\left(Q\right)}Q\\ \\ \phantom{\left(P\right)}P\end{array}\right)$ despite being based on it. It only looks information-theoretic in the way it approaches $1$. On the other hand, I believe it inherits the niceness of information divergence when it comes to taking gradients. ↩