Projections and least squares
Projecting a vector onto a line, a line through points by least squares, the normal equations for an inconsistent system, and singular values.
How to solve it
If a system has no solution, you can find an “almost solution” — the one with the smallest error, the sum of the squared residuals. Geometrically it is a projection: the point of the subspace closest to $\mathbf b$, with the residual perpendicular to it.
Step by step
- The projection of $\mathbf b$ onto the line of $\mathbf a$: the coefficient $\frac{\mathbf a \cdot \mathbf b}{\mathbf a \cdot \mathbf a}$ times $\mathbf a$. The rest $\mathbf b - \mathbf p$ is perpendicular to $\mathbf a$.
- A line $y = C + Dt$ through points: find the means $\bar t$ and $\bar y$, the slope $D = \frac{\sum (t - \bar t)(y - \bar y)}{\sum (t - \bar t)^2}$, then $C = \bar y - D\bar t$.
- An inconsistent system $A\mathbf x = \mathbf b$: solve the normal equations $A^{\mathsf T}A\hat{\mathbf x} = A^{\mathsf T}\mathbf b$.
- Singular values: the square roots of the eigenvalues of $A^{\mathsf T}A$.
Common mistakes
- Dividing by the length of the vector instead of its square: the denominator is $\mathbf a \cdot \mathbf a = |\mathbf a|^2$.
- Forgetting to subtract the means in the slope formula.
- Trying to solve $A\mathbf x = \mathbf b$ as it is — it is inconsistent; solve the normal equations.
- Taking the eigenvalues of $A$ as singular values instead of the roots of the eigenvalues of $A^{\mathsf T}A$.