Home page for accesible maths 3 Inner products 3.C The Cauchy-Schwarz inequality 3.E The Gram-Schmidt process

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

3.D Orthogonality

The Cauchy-Schwarz inequality tells us that if we have an inner product space $V$ (that is, a real vector space with a positive definite symmetric bilinear form), then we can use 3.1 to define a notion of “angle” between two vectors. In particular, two vectors $\vec{x},\vec{y}\in V$ are said to be orthogonal (also known as perpendicular, or at right angles) if:

\langle\vec{x},\vec{y}\rangle=0

This is an important concept in a variety of different contexts. In 3D video games, whenever the perspective of the user rotates, the program must rotate the standard basis to a new one. Since the standard basis vectors in ${\mathbb{R}}^{n}$ are each orthogonal to each other, the resulting basis vectors must still be pairwise orthogonal.

In statistics, the idea of orthogonality is used to describe when two random variables are uncorrelated (i.e. $\operatorname{Cov}(X,Y)=0$ ).

Definition 3.21:

A sequence of vectors $(\vec{x_{1}},\cdots,\vec{x_{r}})$ in an inner product space $V$ is said to be orthogonal, if they are pairwise orthogonal; in other words $\langle\vec{x_{i}},\vec{x_{j}}\rangle=0$ whenever $i\neq j$ . If these vectors also all have unit norm (i.e. $||\vec{x_{i}}||=1$ for every $i$ ), then the sequence is called orthonormal.

Example 3.22.

i.

The sequence $(\begin{bmatrix}1&1\end{bmatrix}^{T},\begin{bmatrix}1&-1\end{bmatrix}^{T})$ is orthogonal, since $\langle(1,1),(1,-1)\rangle=0$ , but is not orthonormal, since their norms are $\sqrt{2}$ .
ii.

The two vectors $\begin{bmatrix}1&1&1\end{bmatrix}^{T},\begin{bmatrix}2&-1&-1\end{bmatrix}^{T}$ are orthogonal. Find a third vector in ${\mathbb{R}}^{3}$ which is orthogonal to both of those.
Solution: We will set up a system of equations whose variables are the coordinates $\begin{bmatrix}x&y&z\end{bmatrix}^{T}$ of our desired vector. Then we need
1. $x+y+z=0$
2. $2x-y-z=0$
The solution set is

$\{\begin{bmatrix}0&y&-y\end{bmatrix}^{T}\;|\;y\in{\mathbb{R}}\}=\operatorname{% span}\{\begin{bmatrix}0&1&-1\end{bmatrix}^{T}\}.$

So if we take the third vector to be $\begin{bmatrix}0&1&-1\end{bmatrix}^{T}$ , then this will create an orthogonal sequence of three vectors.

Exercise 3.23:

Find all unit vectors in ${\mathbb{R}}^{3}$ that are orthogonal to both $\begin{bmatrix}-1&-3&2\end{bmatrix}^{T}$ and $\begin{bmatrix}0&1&1\end{bmatrix}^{T}$ .

[End of Exercise]

Exercise 3.24:

Find an orthonormal sequence of three vectors in ${\mathbb{R}}^{3}$ such that none of them have any zero coordinates (in the standard basis).

[Hint: First find a sequence of orthogonal vectors without zero coordinates, and then scale.]

[End of Exercise]

Exercise 3.25:

Let $A=\begin{bmatrix}3&1\\ 1&1\end{bmatrix}$ . You may assume that $({\mathbb{R}}^{2},\langle\cdot,\cdot\rangle_{A})$ is an inner product space. Find two non-zero vectors in ${\mathbb{R}}^{2}$ which are orthogonal with respect to this inner product.

[End of Exercise]

Definition 3.26:

If $W\subset V$ is a subspace of an inner product space, then we define the orthogonal complement of $W$ in $V$ as follows:

W^{\perp}:=\{\vec{x}\in V\;|\;\langle\vec{x},\vec{y}\rangle=0\textup{ for all % $\vec{y}\in W$}\}.

So $W^{\perp}$ is the set of all vectors orthogonal to all of $W$ . The symbol ${}^{\perp}$ is supposed to make you think of perpendicular lines; it is pronounced “perp”. You should visualize the orthogonal complement as in the following examples.

Example 3.27.

i.

Let $W$ be a 1-dimensional subspace of ${\mathbb{R}}^{2}$ . Then $W^{\perp}$ is the line through the origin which is orthogonal (perpendicular) to $W$ .
ii.

Let $W$ be a 1-dimensional subspace of ${\mathbb{R}}^{3}$ . Then $W^{\perp}$ is the plane through the origin, whose normal vector lies in $W$ .
iii.

Let $W$ be a 2-dimensional subspace of ${\mathbb{R}}^{3}$ . Then $W^{\perp}$ is the line spanned by a normal vector to $W$ .
iv.

Let $W=V$ . Then the orthogonal complement is the zero subspace $W^{\perp}=\{\vec{0}\}$ . To prove this, assume a vector $\vec{x}\in V$ is orthogonal to every vector in $V$ . Then it must be orthogonal to itself. So $\langle\vec{x},\vec{x}\rangle=0$ . But since we assumed $V$ was an inner product space, this implies $\vec{x}=\vec{0}$ .

Several exercises ask to find an expression or basis for the orthogonal complement $W^{\perp}$ of a given subspace $W$ . If you already know a basis for $W$ , then it’s quickest to use:

(\operatorname{span}\{\vec{v_{1}},\cdots,\vec{v_{r}}\})^{\perp}=\{\vec{x}\in V% \;|\;\langle\vec{x},\vec{v_{i}}\rangle=0\;\forall i\}.

Exercise 3.28:

Let $W\subset{\mathbb{R}}^{n}$ be a subspace. Prove that $W^{\perp}$ is a subspace.

[End of Exercise]

Exercise 3.29:

Find a basis for $W^{\perp}$ , where $W:=\operatorname{span}\{\begin{bmatrix}1&1&-1\end{bmatrix}^{T}\}\subset{% \mathbb{R}}^{3}$ .

[End of Exercise]

Theorem 3.30.

Let $V$ be an inner product space (possibly infinite dimensional).

i.

(Triangle inequality) $||\vec{x}+\vec{y}||\leq||\vec{x}||+||\vec{y}||$ for any $\vec{x},\vec{y}\in V$ .
ii.

(Generalized Pythagorean theorem) If $(\vec{x_{1}},\cdots,\vec{x_{n}})$ is an orthogonal sequence, then $\sum_{i=1}^{n}||\vec{x_{i}}||^{2}=||\sum_{i=1}^{n}\vec{x_{i}}||^{2}$ .
iii.

(Parallelogram law) $||\vec{x}+\vec{y}||^{2}+||\vec{x}-\vec{y}||^{2}=2||\vec{x}||^{2}+2||\vec{y}||^% {2}$ for any $\vec{x},\vec{y}\in V$ .

Proof.

(Triangle inequality) For non-negative real numbers, $a\leq b$ if and only if $a^{2}\leq b^{2}$ . Since it is always true that $||\vec{x}||\geq 0$ , it is equivalent to prove the inequality: $||\vec{x}+\vec{y}||^{2}\leq(||\vec{x}||+||\vec{y}||)^{2}$ . Expanding the left hand side, for any $\vec{x},\vec{y}\in V$ , by the definition of $||\vec{x}+\vec{y}||$ :

||\vec{x}+\vec{y}||^{2}=\langle\vec{x}+\vec{y},\vec{x}+\vec{y}\rangle.

By bilinearity of $\langle\cdot,\cdot\rangle$ the above is equal to

\langle\vec{x},\vec{x}\rangle+\langle\vec{y},\vec{y}\rangle+2\langle\vec{x},% \vec{y}\rangle.

By the Cauchy-schwarz inequality the above is less than or equal to

\langle\vec{x},\vec{x}\rangle+\langle\vec{y},\vec{y}\rangle+2||\vec{x}||\cdot|% |\vec{y}||,

which by definition of $||\vec{x}||$ is equal to

||\vec{x}||^{2}+||\vec{y}||^{2}+2||\vec{x}||\cdot||\vec{y}||=(||\vec{x}||+||% \vec{y}||)^{2}.

Notice that above we used that $\langle\vec{x},\vec{y}\rangle\leq|\langle\vec{x},\vec{y}\rangle|$ . The Cauchy-Schwarz inequality was the key step in this proof.

For the proofs of the other two parts, see Exercise 3.32. ∎

Exercise 3.31:

For each of the identities in Theorem 3.30, draw an appropriate diagram of labelled vectors, which allows you to state the identities in terms of lengths and / or angles. For part (ii), assume $n=3$ .

[End of Exercise]

Exercise 3.32:

Prove the identities in Theorem 3.30(ii) and (iii).

[Hint: For Part (ii), use induction. For Part (iii), expand the left hand side, and manipulate the expression, in a similar way to Part (i).]

[End of Exercise]