$$

The $l_2$-norm of a random variable is shared between the mean and the variance:

$$\mathrm{E}\left[{\mathit{X}}^2\right]=\mathrm{E}[\mathit{X}{]}^2+\mathrm{Var}[\mathit{X}]={\mu }^2+{\sigma }^2,$$

Three categories

#to-write restrict to positive-valued

Depending on how the $l_2$-norm is split, we can place nonnegative random variables into three scaling-invariant categories:

1. Fat-tailed: $\sigma \gg \mu$.
   • These variables have some rare but crazy-high values that make the variance freak out.
   • e.g. a bit $\mathit{X}\in \{0,1\}$ that is very rarely $1$
2. Well-rounded: $\sigma =\mathrm{\Theta }(\mu )$.
   • These variables are overall civilized, and vary roughly equally above and below their mean.
   • e.g. a uniform random bit, any exponential random variable, the absolute value or square of any normal of mean $0$, most real world things
3. Concentrated: $\sigma \ll \mu$.
   • These variables are tightly concentrated around a specific non-zero value.
   • e.g. a bit $\mathit{X}\in \{0,1\}$ that is almost always $1$, (the absolute value of) a normal with mean $1$ and tiny variance

#figure

Relative variance

We can also define the relative variance $r=\frac{\mathrm{Var}[\mathit{X}]}{\mathrm{E}[\mathit{X}{]}^2}=\frac{{\sigma }^2}{{\mu }^2}$ (also scaling-invariant), with the correspondance

• fat tailed: $r\gg 1$,
• well-rounded: $r=\mathrm{\Theta }(1)$,
• concentrated: $r\ll 1$.

The shape parameter of a gamma distribution is $k=\frac{1}{r}=\frac{{\mu }^2}{{\sigma }^2}$.

The ratio $\sqrt{r}=\frac{\sigma }{\mu }$ is known as the “coefficient of variation”.

Summing independent copies

As we sum independent copies, fat-tailed variables eventually become well-rounded, which quickly become concentrated. Indeed, $\mathit{Y}$ is the sum of $m$ independent copies of $\mathit{X}$, then

$$\mathrm{E}[\mathit{Y}{]}^2=m^2\mathrm{E}[\mathit{X}{]}^2\text{but}\mathrm{Var}[\mathit{Y}]=m\mathrm{Var}[\mathit{X}],$$

so the variance loses ground, and the relative variance becomes $m$ times smaller:

$$\frac{\mathrm{Var}[\mathit{Y}]}{\mathrm{E}[\mathit{Y}{]}^2}=\frac{m\mathrm{Var}[\mathit{X}]}{m^2\mathrm{E}[\mathit{X}{]}^2}=\frac{1}{m}\times \frac{\mathrm{Var}[\mathit{X}]}{\mathrm{E}[\mathit{X}{]}^2}.$$

Mixtures

#to-write Eric points out that when you take mixtures, by jensen’s you can only increase the relative variance!