Orthogonality and orthogonal complements
The dot product brings geometry into linear algebra: lengths, angles, and perpendicularity. Orthogonal vectors and subspaces make many computations simple.
Orthogonal sets
A set of vectors is orthogonal if every pair is orthogonal, with dot product zero. Any orthogonal set of non-zero vectors is automatically linearly independent: dotting a zero combination with \(\mathbf{u}_i\) kills every term except \(c_i\|\mathbf{u}_i\|^2\), forcing \(c_i = 0\). So n non-zero orthogonal vectors in \(\mathbb{R}^n\) form an orthogonal basis.
Coordinates in an orthogonal basis
In an orthogonal basis, coordinates need no system of equations: the weight on \(\mathbf{u}_i\) is \(c_i = \frac{\mathbf{x} \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\). If the basis is orthonormal, with unit vectors, the weight is simply \(\mathbf{x} \cdot \mathbf{u}_i\). This is a major reason orthonormal bases are prized in computation.
Orthogonal complements
The orthogonal complement \(W^\perp\) of a subspace W consists of all vectors orthogonal to every vector in W. It is a subspace, \(W \cap W^\perp = \{\mathbf{0}\}\), and \(\dim W + \dim W^\perp = n\) in \(\mathbb{R}^n\). In \(\mathbb{R}^3\), the complement of a plane through the origin is the normal line to it, and the complement of a line is the perpendicular plane.
The fundamental relationship
For any matrix A, the null space is the orthogonal complement of the row space, \((\operatorname{Row} A)^\perp = \operatorname{Nul} A\), because \(A\mathbf{x} = \mathbf{0}\) says that \(\mathbf{x}\) is orthogonal to every row. Likewise, \((\operatorname{Col} A)^\perp = \operatorname{Nul} A^T\). These four subspaces, and their dimensions, form what Gilbert Strang popularised as the fundamental theorem of linear algebra.
Projection onto a line
The orthogonal projection of \(\mathbf{y}\) onto the line spanned by \(\mathbf{u}\) is \(\hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\mathbf{u}\). The remainder \(\mathbf{y} - \hat{\mathbf{y}}\) is orthogonal to \(\mathbf{u}\), so \(\mathbf{y}\) splits into a part along the line and a part perpendicular to it. The projection is the point of the line closest to \(\mathbf{y}\).
Orthogonal matrices
A square matrix Q with orthonormal columns is orthogonal: \(Q^TQ = I\), so \(Q^{-1} = Q^T\). Orthogonal matrices preserve lengths and angles, \(\|Q\mathbf{x}\| = \|\mathbf{x}\|\), so they represent rotations and reflections, with determinant +1 or -1. Numerical algorithms favour them because they never amplify rounding errors.
Inner products in general
An inner product generalises the dot product to any vector space: a rule \(\langle \mathbf{u}, \mathbf{v} \rangle\) that is symmetric, linear in each argument, and positive for non-zero vectors. For continuous functions on [a, b], \(\langle f, g \rangle = \int_a^b f(t)g(t)\,dt\) is an inner product. With it, functions have lengths and angles, and sines and cosines of different frequencies turn out to be orthogonal, the basis of Fourier series.
Worked example
Let \(\mathbf{u}_1 = (1, 1, 0)\), \(\mathbf{u}_2 = (1, -1, 1)\), \(\mathbf{u}_3 = (-1, 1, 2)\). All pairwise dot products are 0, so they form an orthogonal basis of \(\mathbb{R}^3\). For \(\mathbf{x} = (3, 1, 4)\): \(c_1 = \frac{4}{2} = 2\), \(c_2 = \frac{3 - 1 + 4}{3} = 2\), \(c_3 = \frac{-3 + 1 + 8}{6} = 1\). Check: \(2(1, 1, 0) + 2(1, -1, 1) + (-1, 1, 2) = (3, 1, 4)\).
Key takeaways
Orthogonal sets of non-zero vectors are independent, so n of them in R^n form a basis. Coordinates in an orthogonal basis are (x . u_i)/(u_i . u_i), and simply x . u_i in an orthonormal basis. The orthogonal complement W-perp contains all vectors orthogonal to W, with dim W + dim W-perp = n. Nul A is the complement of Row A, and Nul A^T of Col A. Projection onto a line through u is ((y . u)/(u . u)) u, the closest point. Orthogonal matrices satisfy Q^T Q = I, preserve lengths, and are numerically stable. General inner products, such as integrals of products of functions, extend geometry to function spaces.
Flashcard
Every pair has dot product zero
Check your understanding
Why is it easy to find coordinates in an orthonormal basis?