This site is being phased out.

Inner product spaces: part 2

From Mathematics Is A Science
Jump to navigationJump to search

The angle between $u$ and $v$ is $\alpha$ as: ${\rm cos \hspace{3pt}} \alpha = \frac{< u , v>}{\lVert u \rVert \lVert v \rVert}$

Cauchy-Schwarz inequality implies $|{\rm cos \hspace{3pt}} \alpha| = \frac{|< u , v>|}{\lVert u \rVert \lVert v \rVert} \leq 1$, so $\alpha = {\rm arccos \hspace{3pt}} \frac{< u , v>}{\lVert u \rVert \lVert v \rVert}$ makes sense.


Projection

A familiar construction:

Projection.png

Here ${\bf R}^2$, $v$ is a linear combination of $e_1,e_2$: $$v=a_1e_1+a_2e_2$$ Note: $a_1e_1$ is a multiple of $e_1$.

We can interpret $v$ as (${\rm multiple \hspace{3pt} of \hspace{3pt}} e_1) + {\rm something}$.

Generally, such a representation isn't unique. But what if we require $e_1 \perp v-a_1e_1$? Then it is...

Why is this important?

Practical example: Given $f(x) = \sin x$, find the best quadratic approximation of $f$.

Best in what sense? We mean that the "area" between the two graphs, $\sin$ and $g$ is minimized. In other words, minimize

$$\int_{[-\pi,\pi]}|\sin x - g(x)|^2 dx = \lVert \sin - q \rVert ^2.$$

This is the distance between functions.

Bird's eye view...

Analysis: The set of differential functions $C^1({\bf R})$ and we want to find the functions closest to $\sin$... closest in the sense of the norm $\lVert \sin -q \rVert$.

ClosestQuadraticPolynomialToSin.png

That's the analytic approach.

Algebraically, we have a much better picture.

Observe: quadratic polynomials is subspace of $C^1({\bf R})$.

Linear Algebra and Geometry:

ClosestQuadraticPolynomialToSin2.png

Meaning $\sin q \perp p$, for any $p \in {\rm P}_2$.

So, it's all about orthogonalty...


We now generalize this setup in ${\bf R}^2$:

Projection2.png

Given vector space $V$, $u,v \in V$, $u \neq 0$, projection of $v$ onto $u$ is a vector $p$ such that

  • 1) $p$ is a multiple of $u$, and
  • 2) $v-p \perp u$.

Same as the projection of $v$ on $U={\rm span}\{u\}$.


Theorem: The projection of $v$ on $u$ is $$p = \frac{<v , u >}{< u , u>}u.$$

Proof: Clearly this is a multiple of $u$. Next,

$$\begin{align*} <v-p, u > &= <v, u > - < p , u > \\ &= <v , u > - <\frac{<v, u > } { < u , u > } u , u > \\ &= <v, u > - \frac{<v, u >}{< u , u >} < u , u > \\ &= <v , u > - < u , v> = 0 \end{align*}$$ $\blacksquare$

Example: Find the best linear "approximation" of $\sin$ on $[-\frac{\pi}{2},\frac{\pi}{2}]$.

Then we use the above with: $v$ is $\sin$, $u$ is $x$. Now find the projection $p$ of $\sin$ on the space of linear functions, $y=mx$.

Compute: $$\begin{array}{} < u , v > &= \int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \sin x \cdot x dx \\ &\stackrel{ {\rm parts} }{=} \left( \left. -\cos x \cdot x - \int -\cos x dx\right) \right|_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \\ &= -x \cos x + \sin x |_{\frac{\pi}{2}}^{\frac{\pi}{2}} \\ &= (-\frac{\pi}{2} \cos \frac{\pi}{2} + \sin \frac{\pi}{2}) - (-(\frac{\pi}{2} \cos(-\frac{\pi}{2}) + \sin(-\frac{\pi}{2})) \\ &= 1 - (-1) = 2 \end{array}$$

$$\begin{array}{} < u , u > &= \int_{-\frac{\pi}{2} }^{\frac{\pi}{2} } x \cdot x dx &= \frac{x^3} {3}|_{-\frac{\pi} {2} }^{\frac{\pi}{2} } &= \frac{2}{3}(\frac{\pi}{2})^3 . \end{array}$$

BestApproxOn-Pi-Pi.png

BestApproxOn-PiHalves-PiHalves.png

So, by the formula: $$\begin{array}{} p &=\frac{2}{\frac{2}{3}(\frac{\pi}{2})^3}\cdot x \\ &= \frac{8\cdot 3}{\pi^3}x. \end{array}{}$$ $\blacksquare$

What's the advantage of having an inner product?

Vectors $u,v$ are orthogonal if $< u , v>=0$.

Example: Solve a linear combination problem: given a basis $u_1,u_2,u_3$ express $v \in V$ in terms of these.

$$\begin{array}{} u_1 &= (2,2,-1) \\ u_2 &= (2,-1,2) \\ u_3 &= (-1,2,2) \\ v = (2,3,5) \end{array}$$

Past: Write $(2,3,5) = a_1(2,2-1)+a_2(2,-1,2) + a_3(-1,2,2)$, which is a system of three linear equations, solve it...

Present: Notice $u_1, u_2, u_3$ are orthogonal!

Indeed: $$\begin{array}{} <u_1, u_2> &= <(2,2,-1),(2,-1,2)> \\ &= 4-2-2 = 0, etc \end{array}$$

Now, there is a shortcut for this problem if the basis is orthogonal. Idea: use the projection.

FindTheProjections.png

Find the projections of $v$ on $u_1,u_2$. They will give us $v$ as a linear combination of $u_1, u_2$. That's the plan.

Carry it out here: take inner product of $v$ and $u_1,u_2,u_3$:

$$v=a_1u_1+a_2u_2+a_3u_3.$$

Consider

$$\begin{array}{} <v,u_1> &= <a_1u_1+a_2u_2+a_3u_3,u_1> \\ &= a_1<u_1,u_1> + a_2<u_2,u_2> + a_3<u_3,u_1> \\ &= a_1<u_1,u_1>. \end{array}$$

So,

$$\begin{array}{} a_1 &= \frac{<v,u_1>}{u_1,u_1} \\ a_2 &= \frac{<v, u_2>}{<u_2,u_2>} \\ a_3 &= \frac{<v,u_3>}{<u_3,u_3>}. \end{array}$$

which are projections of $v$ onto $u_i$. Done.


Theorem: An orthogonal set of non-zero vectors (in an inner product space) is linearly independent.

Proof: Suppose $u_1,\ldots,u_n \in V$ are orthogonal to each other: for all $i \neq j$, $$<u_i,u_j> = 0.$$

Suppose $a_1u_1+a_2u_2+\ldots+a_nu_n=0$, are all of these $a_i$ equal to zero?

Multiply the equation by $u_i$ using the inner product:

$$<a_1u_1+\ldots+a_nu_n,u_i> = <0,u_i> = 0.$$

Use linearity of $< , >$.

$$a_1<u_1,u_i>+\ldots+a_n<u_n,u_i>=0$$

Note that $n-1$ of these are $0$.

We're left with:

$$a_i<u_i,u_i>=0.$$

By Axiom 1, $<u_i,u_i>=0$ only if $u_i=0$. Hence $a_i=0$ for each $i=1,\ldots,n$. $\blacksquare$


What to do if this is an infinite set?

Well, an orthogonal set is still defined as: $S \subset V$ and $< u , v>=0$ if $u,v \in S$, $u \neq v$.

How do we change the proof?

...No change.

Orthogonalization

We know that in $V$, with ${\rm dim \hspace{3pt}} V = n$, every set of $n$ orthogonal vectors is a basis.

Suppose $u_1,\ldots,u_n$ is an arbitrary basis in ${\bf R}^n$. How do we construct an orthogonal basis from it?

In ${\bf R}^n$...

We start with a simple case:

  • $u_1,u_2$ linearly independent, convert to $v_1,v_2$ orthogonal.

LinearIndependentToOrthogonal.png

Of course we can choose $v_1=u_1$.

Find $v_2$.

We use $u_2$ to find $v_2 \perp u_1$. How?

Find the projection of $p_2$ of $u_2$ on $u_1$, subtract from $u_2$, the result ("defect") $v_1$ is orthogonal to $u_1$.

That's one of the two part of the definition of projection.

Result: PerpendicularVectors.png

What if we have $3$ vectors?

Ignore $u_3$, for now, and orthogonalize $\{u_1,u_2\}$ to $\{v_1,v_2\}$. Next bring $u_3$ into this.

Orthogonalize3Vectors.png

Idea:

  • $\{u_1,u_2,u_3\}$ are linearly independent, hence
  • $\{v_1,v_2,v_3\}$ are linearly independent.

Why?

In the left image, $u_3$ is not a linear combination of $u_1,u_2$, or: $u_3 \not\in {\rm span}\{u_1,u_2\}$.

In the right image, same. Why? Because $u_3 \not\in {\rm span}\{v_1,v_2\}$, if we make sure that these spans coincide.

Plan: At every step,

  • ${\rm span}\{u_1,\ldots,u_k\}=V_k$

a subspace of $V$...

  • $={\rm span}\{v_1,\ldots,v_k\},$

an orthogonal basis of $V_k$.

Then

  • $u_{k+1} \not\in {\rm span}\{u_1,\ldots,u_k\}=V_k$ and
  • $u_{k+1}$ is linearly independent from $\{v_1,\ldots,v_k\}$.

Same space: $V_k={\rm span}\{u_1,\ldots,u_k\} = {\rm span}\{v_1,\ldots,v_k\}$. The "old basis" is $\{u_1,\ldots,u_k\}$ and the "new basis" is $\{v_1,\ldots,v_k\}$.

Inductive step, above: $v_1,\ldots,v_k$ are orthogonal.

Next: add one more vector $u_{k+1}$, and make it orthogonal to $V_k$.

AddOrthogonalVector.png

Here $u_{k+1} \not\in V_k$:

We need to choose $v_{k+1}$ so that

  • 1. $v_{k+1} \perp v_k$, and
  • 2. ${\rm span}\{v_1,\ldots,v_{k+1}\} = {\rm span}\{u_1,\ldots,u_{k+1}\}$.

How? Find the projection $p$ of $u_{k+1}$ onto $V_k$.

Let's define this.

Definition: Given a subspace $U$ of $V$ and a vector $v \in V$, the projection of $v$ onto $U$ is a vector $p$ such that

  • $p \in U$, and
  • $v-p \perp U$

ProjectionToSubspace.png

Last one: $<v-p, u >=0$ for all $u \in U$.

Theorem: Suppose subspace $U$ has an orthogonal basis $\{u_1,\ldots,u_k\}$, and $v \in V$ (note: possibly $v \in U$, then $p=v$.). Then $$p=<v,u_1>u_1+ \ldots + <v,u_k>u_k.$$

Proof: $p$ is a linear combination of the basis $\in U$:

$$v-p = v-(<v,u_1>u_1+\ldots+<v,u_n>u_k),$$

verify, $i=1,\ldots,k$.

$$<v-p,u_i> = <v,u_i>-(<<v,u_1>u_1,u_i> + \ldots + <<v,u_k>u_k,u_i>)$$

$<u_i,u_j>=0$ if $i \neq j$, so one term left:

$$\begin{array}{} &=<v,u_i> - <<v,u_i>u_i,u_i> \\ &=<v,u_i>-<v,u_i>\cdot 1 = 0 \end{array}$$

We use this to do Gram-Schmidt" orthogonalization.

Summary:

$$\begin{array}{} v_1 &= u_1 & \\ p_2 &= < v,u_2>u_2 &\rightarrow v_2=u_2-p_2 \\ p_3 &= <v_1,u_2>u_2 + <v,u_3>u_3 &\rightarrow v_3 = u_3-p_3 \\ \vdots & & \\ p_n &= <v_1,u_2>u_2+\ldots+<v,u_n> &\rightarrow v_n=u_n-p_n. \end{array}$$

To normalize: $e_k = \frac{v_k}{\lVert v_k \rVert}$, $k=1,\ldots,n$, this gives us orthonormal basis.

Fourier Analysis

From above: projection of ${\rm sin}$ on ${\rm span}\{x\}$ is the best linear approximation.

Orthonormal set in $C[-\pi,\pi]$ is:

$$e_0=\frac{1}{\sqrt{2\pi}}, e_1=\frac{1}{\sqrt{\pi}}{\rm cos \hspace{3pt}}x; e_2=\frac{1}{\sqrt{\pi}}{\rm sin \hspace{3pt}}x; \frac{1}{\sqrt{2}}{\rm cos}(2x);\frac{1}{\sqrt{2}}{\rm sin}(2x),\ldots$$

so that $\displaystyle\int_{-\pi}^{\pi} e_ie_j dx = \delta_{ij}$.

Find first $5$ elements and approximate $f(x)=x$.

How? The projection of $f$ onto ${\rm span}\{e_1,\ldots,e_5\}$:

$$p=<e_0,f>e_0+<e_1,f>e_1+\ldots+<e_5,f>e_5$$

Compute each $<e_i,f>$ by integration:

$$\begin{array}{} <e_0,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi} } dx &= 0 \\ <e_1,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi} }{\rm cos \hspace{3pt} }xdx &\stackrel{ {\rm parts} }{=} 0 \\ \vdots & & \\ <e_k,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi}}{\rm sin \hspace{3pt}}kxdx &=(-1)^{k+1}\frac{2\sqrt{\pi}}{k}. \end{array}$$

$\rightarrow$ Fourier series.