$$

This note uses Named tensor notation.

The singular values decomposition of a matrix $A\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}\times \phantom{\mathsf{fg}}\mathsf{right}}$ is a complete description of how it scales different directions in the spaces ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$ and ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{right}}$. Any matrix $A$ can be expressed as the sum of $r:=\mathrm{rank}(A)$ outer products

$$A=\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma UV,$$

where

• $U\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{sing}\times \phantom{\mathsf{fg}}\mathsf{left}}$ is made of $r$ orthonormal vectors that span $A$ within ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$;
• $V\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{sing}\times \phantom{\mathsf{fg}}\mathsf{right}}$ is made of $r$ orthonormal vectors that span $A$ within ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{right}}$;
• the singular values $\sigma \in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{sing}}$ describes how these vectors are scaled by $A$.

Notes:

• This shows that any matrix is diagonal up to a rotation of the spaces it acts on!
• Unlike for eigenvalues, the vectors $U_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$ and $V_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$ do not need to point in the same direction (and indeed they’re not part of the same space!).
• Since we limited the sum to $r$ terms, $\sigma$ only contains only the nonzero singular values (this is known as the compact SVD).

What it says about $A$ as a transformation

Linear map

When we apply $A$ to a vector $x\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$, $x$’s component in the direction of $U_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$ gets scaled by a factor ${\sigma }_i$ and transformed into direction $V_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$:

$$\begin{array}{rl}A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x& =\left(\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma UV\right)\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\\ & =\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\underset{\text{scaling}}{\underbrace{\sigma }}\underset{\text{``coordinates of }x\text{''}}{\underbrace{\left(U\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\right)}}\underset{\text{new directions}}{\underbrace{V}}.\end{array}$$

By symmetry, $A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }y$ transforms direction $V_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$ into direction $U_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$ and scales by ${\sigma }_i$.

Geometric interpretation

In particular, the transformation $x\mapsto A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x$ turns the unit ball

$$B:=\{x\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}|{\Vert x\Vert }_{\phantom{\mathsf{fg}}\mathsf{left}}\le 1\}$$

into an ellipsoid of dimension $r$ within ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{right}}$, whose semi-axes are ${\sigma }_i{V}_{\phantom{\mathsf{fg}}\mathsf{sing}(i)}$. In particular, the length of the semi-axes are given by the singular values. Indeed, we have

$$\begin{array}{rl}A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x& =\left(\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma UV\right)\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\\ & =\left(U\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\right)\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{sing}\end{array}}{\phantom{fg}\odot }\left(\sigma V\right),\end{array}$$

and the only constraint that $x$ being in $B$ imposes on $U\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{sing}}$ is that it should have norm at most $1$.

Bilinear form

In a similar vein, when we look at the components of $x$ and $y$ relative to $U$ and $V$, the singular values give the coefficients of the bilinear form associated with $A$:

$$\begin{array}{rl}x\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }y& =x\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }\left(\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma UV\right)\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }y\\ & =\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma \underset{\text{``coordinates of }x\text{''}}{\underbrace{\left(U\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{left}\end{array}}{\phantom{fg}\odot }x\right)}}\underset{\text{``coordinates of }y\text{''}}{\underbrace{\left(V\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }y\right)}}.\end{array}$$

Relation to eigenvalues

The singular values of $A$ are the square roots of the eigenvalues of

$$A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }{A}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})}\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}\times {\phantom{\mathsf{fg}}\mathsf{left}}^{\prime }},$$

which corresponds to the linear map that uses $A$ to take a vector $x\in {\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$ into ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{right}}$ and then back to ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$. Indeed,¹

$$\begin{array}{rl}A\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }{A}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})}& =\left(\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma UV\right)\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }\left(\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}\sigma U^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})}V\right)\\ & =\sum _{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{sing}\\ {\phantom{\mathsf{fg}}\mathsf{sing}}^{\prime }\end{array}}\sigma UV\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }{\left(\sigma U^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})}V\right)}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{sing})}\\ & =\sum _{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{sing}\\ {\phantom{\mathsf{fg}}\mathsf{sing}}^{\prime }\end{array}}\sigma U{\left(\sigma U^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})}\right)}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{sing})}\underset{I_{\phantom{\mathsf{fg}}\mathsf{sing},{\phantom{\mathsf{fg}}\mathsf{sing}}^{\prime }}}{\underbrace{\left(V\underset{\begin{array}{c}\phantom{\mathsf{fg}}\mathsf{right}\end{array}}{\phantom{fg}\odot }{V}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{sing})}\right)}}\\ & =\sum _{\phantom{\mathsf{fg}}\mathsf{sing}}{\sigma }^{2}U{U}^{\mathsf{T}(\phantom{\mathsf{fg}}\mathsf{left})},\end{array}$$

which is an eigendecomposition since the vectors of $U$ (in ${\mathbb{R}}^{\phantom{\mathsf{fg}}\mathsf{left}}$) are linearly independent.

1. This particular computation is admittedly simpler in the usual matrix multiplication notation: taking $A\in {\mathbb{R}}^{n\times m}$, we have $A{A}^{\mathsf{T}}=\left(U\mathrm{\Sigma }{V}^{\mathsf{T}}\right){\left(U\mathrm{\Sigma }{V}^{\mathsf{T}}\right)}^{\mathsf{T}}=U\mathrm{\Sigma }{V}^{\mathsf{T}}V\mathrm{\Sigma }{U}^{\mathsf{T}}=U\mathrm{\Sigma }{I}_m\mathrm{\Sigma }{U}^{\mathsf{T}}=U{\mathrm{\Sigma }}^2{U}^{\mathsf{T}}$. ↩