This site is being phased out.
Difference between revisions of "Inner product spaces: part 2"
imported>WikiSysop (→Fourier Analysis) |
(No difference)
|
Latest revision as of 17:59, 13 October 2011
The angle between $u$ and $v$ is $\alpha$ as: ${\rm cos \hspace{3pt}} \alpha = \frac{< u , v>}{\lVert u \rVert \lVert v \rVert}$
Cauchy-Schwarz inequality implies $|{\rm cos \hspace{3pt}} \alpha| = \frac{|< u , v>|}{\lVert u \rVert \lVert v \rVert} \leq 1$, so $\alpha = {\rm arccos \hspace{3pt}} \frac{< u , v>}{\lVert u \rVert \lVert v \rVert}$ makes sense.
Contents
Projection
A familiar construction:
Here ${\bf R}^2$, $v$ is a linear combination of $e_1,e_2$: $$v=a_1e_1+a_2e_2$$ Note: $a_1e_1$ is a multiple of $e_1$.
We can interpret $v$ as (${\rm multiple \hspace{3pt} of \hspace{3pt}} e_1) + {\rm something}$.
Generally, such a representation isn't unique. But what if we require $e_1 \perp v-a_1e_1$? Then it is...
Why is this important?
Practical example: Given $f(x) = \sin x$, find the best quadratic approximation of $f$.
Best in what sense? We mean that the "area" between the two graphs, $\sin$ and $g$ is minimized. In other words, minimize
$$\int_{[-\pi,\pi]}|\sin x - g(x)|^2 dx = \lVert \sin - q \rVert ^2.$$
This is the distance between functions.
Bird's eye view...
Analysis: The set of differential functions $C^1({\bf R})$ and we want to find the functions closest to $\sin$... closest in the sense of the norm $\lVert \sin -q \rVert$.
That's the analytic approach.
Algebraically, we have a much better picture.
Observe: quadratic polynomials is subspace of $C^1({\bf R})$.
Linear Algebra and Geometry:
Meaning $\sin q \perp p$, for any $p \in {\rm P}_2$.
So, it's all about orthogonalty...
We now generalize this setup in ${\bf R}^2$:
Given vector space $V$, $u,v \in V$, $u \neq 0$, projection of $v$ onto $u$ is a vector $p$ such that
- 1) $p$ is a multiple of $u$, and
- 2) $v-p \perp u$.
Same as the projection of $v$ on $U={\rm span}\{u\}$.
Theorem: The projection of $v$ on $u$ is
$$p = \frac{<v , u >}{< u , u>}u.$$
Proof: Clearly this is a multiple of $u$. Next,
$$\begin{align*} <v-p, u > &= <v, u > - < p , u > \\ &= <v , u > - <\frac{<v, u > } { < u , u > } u , u > \\ &= <v, u > - \frac{<v, u >}{< u , u >} < u , u > \\ &= <v , u > - < u , v> = 0 \end{align*}$$ $\blacksquare$
Example: Find the best linear "approximation" of $\sin$ on $[-\frac{\pi}{2},\frac{\pi}{2}]$.
Then we use the above with: $v$ is $\sin$, $u$ is $x$. Now find the projection $p$ of $\sin$ on the space of linear functions, $y=mx$.
Compute: $$\begin{array}{} < u , v > &= \int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \sin x \cdot x dx \\ &\stackrel{ {\rm parts} }{=} \left( \left. -\cos x \cdot x - \int -\cos x dx\right) \right|_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \\ &= -x \cos x + \sin x |_{\frac{\pi}{2}}^{\frac{\pi}{2}} \\ &= (-\frac{\pi}{2} \cos \frac{\pi}{2} + \sin \frac{\pi}{2}) - (-(\frac{\pi}{2} \cos(-\frac{\pi}{2}) + \sin(-\frac{\pi}{2})) \\ &= 1 - (-1) = 2 \end{array}$$
$$\begin{array}{} < u , u > &= \int_{-\frac{\pi}{2} }^{\frac{\pi}{2} } x \cdot x dx &= \frac{x^3} {3}|_{-\frac{\pi} {2} }^{\frac{\pi}{2} } &= \frac{2}{3}(\frac{\pi}{2})^3 . \end{array}$$
So, by the formula: $$\begin{array}{} p &=\frac{2}{\frac{2}{3}(\frac{\pi}{2})^3}\cdot x \\ &= \frac{8\cdot 3}{\pi^3}x. \end{array}{}$$ $\blacksquare$
What's the advantage of having an inner product?
Vectors $u,v$ are orthogonal if $< u , v>=0$.
Example: Solve a linear combination problem: given a basis $u_1,u_2,u_3$ express $v \in V$ in terms of these.
$$\begin{array}{} u_1 &= (2,2,-1) \\ u_2 &= (2,-1,2) \\ u_3 &= (-1,2,2) \\ v = (2,3,5) \end{array}$$
Past: Write $(2,3,5) = a_1(2,2-1)+a_2(2,-1,2) + a_3(-1,2,2)$, which is a system of three linear equations, solve it...
Present: Notice $u_1, u_2, u_3$ are orthogonal!
Indeed: $$\begin{array}{} <u_1, u_2> &= <(2,2,-1),(2,-1,2)> \\ &= 4-2-2 = 0, etc \end{array}$$
Now, there is a shortcut for this problem if the basis is orthogonal. Idea: use the projection.
Find the projections of $v$ on $u_1,u_2$. They will give us $v$ as a linear combination of $u_1, u_2$. That's the plan.
Carry it out here: take inner product of $v$ and $u_1,u_2,u_3$:
$$v=a_1u_1+a_2u_2+a_3u_3.$$
Consider
$$\begin{array}{} <v,u_1> &= <a_1u_1+a_2u_2+a_3u_3,u_1> \\ &= a_1<u_1,u_1> + a_2<u_2,u_2> + a_3<u_3,u_1> \\ &= a_1<u_1,u_1>. \end{array}$$
So,
$$\begin{array}{} a_1 &= \frac{<v,u_1>}{u_1,u_1} \\ a_2 &= \frac{<v, u_2>}{<u_2,u_2>} \\ a_3 &= \frac{<v,u_3>}{<u_3,u_3>}. \end{array}$$
which are projections of $v$ onto $u_i$. Done.
Theorem: An orthogonal set of non-zero vectors (in an inner product space) is linearly independent.
Proof: Suppose $u_1,\ldots,u_n \in V$ are orthogonal to each other: for all $i \neq j$, $$<u_i,u_j> = 0.$$
Suppose $a_1u_1+a_2u_2+\ldots+a_nu_n=0$, are all of these $a_i$ equal to zero?
Multiply the equation by $u_i$ using the inner product:
$$<a_1u_1+\ldots+a_nu_n,u_i> = <0,u_i> = 0.$$
Use linearity of $< , >$.
$$a_1<u_1,u_i>+\ldots+a_n<u_n,u_i>=0$$
Note that $n-1$ of these are $0$.
We're left with:
$$a_i<u_i,u_i>=0.$$
By Axiom 1, $<u_i,u_i>=0$ only if $u_i=0$. Hence $a_i=0$ for each $i=1,\ldots,n$. $\blacksquare$
What to do if this is an infinite set?
Well, an orthogonal set is still defined as: $S \subset V$ and $< u , v>=0$ if $u,v \in S$, $u \neq v$.
How do we change the proof?
...No change.
Orthogonalization
We know that in $V$, with ${\rm dim \hspace{3pt}} V = n$, every set of $n$ orthogonal vectors is a basis.
Suppose $u_1,\ldots,u_n$ is an arbitrary basis in ${\bf R}^n$. How do we construct an orthogonal basis from it?
In ${\bf R}^n$...
We start with a simple case:
- $u_1,u_2$ linearly independent, convert to $v_1,v_2$ orthogonal.
Of course we can choose $v_1=u_1$.
Find $v_2$.
We use $u_2$ to find $v_2 \perp u_1$. How?
Find the projection of $p_2$ of $u_2$ on $u_1$, subtract from $u_2$, the result ("defect") $v_1$ is orthogonal to $u_1$.
That's one of the two part of the definition of projection.
What if we have $3$ vectors?
Ignore $u_3$, for now, and orthogonalize $\{u_1,u_2\}$ to $\{v_1,v_2\}$. Next bring $u_3$ into this.
Idea:
- $\{u_1,u_2,u_3\}$ are linearly independent, hence
- $\{v_1,v_2,v_3\}$ are linearly independent.
Why?
In the left image, $u_3$ is not a linear combination of $u_1,u_2$, or: $u_3 \not\in {\rm span}\{u_1,u_2\}$.
In the right image, same. Why? Because $u_3 \not\in {\rm span}\{v_1,v_2\}$, if we make sure that these spans coincide.
Plan: At every step,
- ${\rm span}\{u_1,\ldots,u_k\}=V_k$
a subspace of $V$...
- $={\rm span}\{v_1,\ldots,v_k\},$
an orthogonal basis of $V_k$.
Then
- $u_{k+1} \not\in {\rm span}\{u_1,\ldots,u_k\}=V_k$ and
- $u_{k+1}$ is linearly independent from $\{v_1,\ldots,v_k\}$.
Same space: $V_k={\rm span}\{u_1,\ldots,u_k\} = {\rm span}\{v_1,\ldots,v_k\}$. The "old basis" is $\{u_1,\ldots,u_k\}$ and the "new basis" is $\{v_1,\ldots,v_k\}$.
Inductive step, above: $v_1,\ldots,v_k$ are orthogonal.
Next: add one more vector $u_{k+1}$, and make it orthogonal to $V_k$.
Here $u_{k+1} \not\in V_k$:
We need to choose $v_{k+1}$ so that
- 1. $v_{k+1} \perp v_k$, and
- 2. ${\rm span}\{v_1,\ldots,v_{k+1}\} = {\rm span}\{u_1,\ldots,u_{k+1}\}$.
How? Find the projection $p$ of $u_{k+1}$ onto $V_k$.
Let's define this.
Definition: Given a subspace $U$ of $V$ and a vector $v \in V$, the projection of $v$ onto $U$ is a vector $p$ such that
- $p \in U$, and
- $v-p \perp U$
Last one: $<v-p, u >=0$ for all $u \in U$.
Theorem: Suppose subspace $U$ has an orthogonal basis $\{u_1,\ldots,u_k\}$, and $v \in V$ (note: possibly $v \in U$, then $p=v$.). Then $$p=<v,u_1>u_1+ \ldots + <v,u_k>u_k.$$
Proof: $p$ is a linear combination of the basis $\in U$:
$$v-p = v-(<v,u_1>u_1+\ldots+<v,u_n>u_k),$$
verify, $i=1,\ldots,k$.
$$<v-p,u_i> = <v,u_i>-(<<v,u_1>u_1,u_i> + \ldots + <<v,u_k>u_k,u_i>)$$
$<u_i,u_j>=0$ if $i \neq j$, so one term left:
$$\begin{array}{} &=<v,u_i> - <<v,u_i>u_i,u_i> \\ &=<v,u_i>-<v,u_i>\cdot 1 = 0 \end{array}$$
We use this to do Gram-Schmidt" orthogonalization.
Summary:
$$\begin{array}{} v_1 &= u_1 & \\ p_2 &= < v,u_2>u_2 &\rightarrow v_2=u_2-p_2 \\ p_3 &= <v_1,u_2>u_2 + <v,u_3>u_3 &\rightarrow v_3 = u_3-p_3 \\ \vdots & & \\ p_n &= <v_1,u_2>u_2+\ldots+<v,u_n> &\rightarrow v_n=u_n-p_n. \end{array}$$
To normalize: $e_k = \frac{v_k}{\lVert v_k \rVert}$, $k=1,\ldots,n$, this gives us orthonormal basis.
Fourier Analysis
From above: projection of ${\rm sin}$ on ${\rm span}\{x\}$ is the best linear approximation.
Orthonormal set in $C[-\pi,\pi]$ is:
$$e_0=\frac{1}{\sqrt{2\pi}}, e_1=\frac{1}{\sqrt{\pi}}{\rm cos \hspace{3pt}}x; e_2=\frac{1}{\sqrt{\pi}}{\rm sin \hspace{3pt}}x; \frac{1}{\sqrt{2}}{\rm cos}(2x);\frac{1}{\sqrt{2}}{\rm sin}(2x),\ldots$$
so that $\displaystyle\int_{-\pi}^{\pi} e_ie_j dx = \delta_{ij}$.
Find first $5$ elements and approximate $f(x)=x$.
How? The projection of $f$ onto ${\rm span}\{e_1,\ldots,e_5\}$:
$$p=<e_0,f>e_0+<e_1,f>e_1+\ldots+<e_5,f>e_5$$
Compute each $<e_i,f>$ by integration:
$$\begin{array}{} <e_0,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi} } dx &= 0 \\ <e_1,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi} }{\rm cos \hspace{3pt} }xdx &\stackrel{ {\rm parts} }{=} 0 \\ \vdots & & \\ <e_k,f> &= \int_{-\pi}^{\pi} x \cdot \frac{1}{\sqrt{\pi}}{\rm sin \hspace{3pt}}kxdx &=(-1)^{k+1}\frac{2\sqrt{\pi}}{k}. \end{array}$$
$\rightarrow$ Fourier series.