This site is being phased out.
Eigenvalues and eigenvectors of linear operators
Motivation
Consider a linear operator given by $A=\left[ \begin{array}{} 2 & 0 \\ 0 & 3 \end{array} \right]$, $A \colon {\bf R}^2 \rightarrow {\bf R}^2$, so $A$ is a transformation of the plane.
What is it geometrically?
Consider where $A$ takes the basis vectors:
$$\begin{array}{} e_1 = \left[ \begin{array}{} 1 \\ 0 \end{array} \right] \stackrel{A}{\rightarrow} \left[ \begin{array}{} 2 \\ 0 \end{array} \right] \\ e_2 = \left[ \begin{array}{} 0 \\ 1 \end{array} \right] \stackrel{A}{\rightarrow} \left[ \begin{array}{} 0 \\ 3 \end{array} \right] \end{array}$$
(These are columns of $A$ by the way.)
Then $A(e_1)=2e_1$, $A(e_2)=3e_2$. So $A$ stretches horizontally by 2x and vertically by 3x.
In fact, any stretch of this kind will have this form:
$$A = \left[ \begin{array}{} h & 0 \\ 0 & v \end{array} \right],$$
where $h$ horizontal coefficient, $v$ vertical coefficient.
What about a stretch but along other axes, i.e., with respect to another basis?
Given a basis $\{v_1,v_2\}$, define a linear operator with matrix $A = \left[ \begin{array}{} a & 0 \\ 0 & b \end{array} \right]$ with respect to this basis.
If this is not the standard basis, then the matrix is not diagonal.
How to find a basis that makes the matrix diagonal, if possible?
Let's analyze.
Observation: $A$ acts as scalar multiplication:
- $A(v_1)=\lambda_1v_1$ and
- $A(v_2)=\lambda_2v_2$,
where $\{v_1,v_2\}$ is the basis we are looking for.
Definition
Definition: Given a linear operator $A \colon V \rightarrow V$, a number $\lambda$ is called an eigenvalue of $A$ if
$$A(v)=\lambda v$$
for some non-zero vector $v \in V$. Then, $v$ is called an eigenvector of $A$ corresponding to $\lambda$.
Example: $A=I_n$, the equation to be solved:
$$\begin{array}{} Av &= \lambda v \\ I_nv=\lambda v \\ v = \lambda v \end{array}$$
So $\lambda=1$. This is the only eigenvalue.
What about eigenvectors?
All of them except $0$.
Example: $A = \left[ \begin{array}{} 2 & 0 \\ 0 & 3 \end{array} \right]$, the equation is:
$$\left[ \begin{array}{} 2 & 0 \\ 0 & 3 \end{array} \right] \left[ \begin{array}{} x \\ y \end{array} \right] = \lambda \left[ \begin{array}{} x \\ y \end{array} \right].$$
Rewrite
$$\left[ \begin{array}{} 2x \\ 3y \end{array} \right] = \left[ \begin{array}{} \lambda x \\ \lambda y \end{array} \right] \rightarrow \begin{array}{} 2x &=\lambda x \\ 3y &= \lambda y.\\ \end{array}$$
Hence
- $x(2-\lambda )=0$, and
- $y(3-\lambda )=0$.
Now,
- $x \ne 0 \Rightarrow 2-\lambda =0 \Rightarrow \lambda =2$,
- $y \ne 0 \Rightarrow 3-\lambda =0 \Rightarrow \lambda =3$.
These are the only two possibilities.
So,
$\lambda = 2$, then $y=0$, so its eigenvectors are $\left[ \begin{array}{} x \\ 0 \end{array} \right]$, $x \neq 0$.
$\lambda=3$, then $x=0$, so its eigenvectors are $\left[ \begin{array}{} 0 \\ y \end{array} \right]$, $y \neq 0$.
Except for the restriction $x \neq 0$, $y \neq 0$, eigenvectors corresponding to the eigenvalue form a subspace:
For $\lambda = 2$, $V_1 = \left\{ \left[ \begin{array}{} x \\ 0 \end{array} \right] \colon x \in {\bf R} \right\}$.
For $\lambda=3$, $V_2= \left\{\left[ \begin{array}{} 0 \\ y \end{array} \right] \colon y \in {\bf R} \right\}$.
More generally...
Eigenspaces
Definition: For a given eigenvalue $\lambda$ of $A$, the eigenspace of $A$ corresponding to $\lambda$ is
$$E_A(\lambda) = \{ v \in V \colon A(v)=\lambda v \}.$$
It's all eigenvectors plus $0$...
Theorem: An eigenspace is a subspace of $V$.
Proof: We use the Subspace Theorem, and need to show:
1. It's non-empty: $A(0)=0 = \lambda \cdot 0$.
2. It's closed under addition:
- $u,v \in E_A \Rightarrow$
- $A(u) = \lambda u$,
- $A(v) = \lambda v$.
Hence $$\begin{array}{} A(u+v) &= A(u)+A(v) \\ &= \lambda u + \lambda v \\ &= \lambda(u+v), u+v \in E_a(\lambda)$. \end{array}$$
3. Similar for scalar multiplication. $\blacksquare$
Example: $E_{I_2}(1) = {\bf R}^2$.
Example: $A = \left[ \begin{array}{} 2 & 0 \\ 0 & 3 \end{array} \right]$, then
- $E_A(2) = {\rm span}\{e_1\}$, $x$-axis.
- $E_A(3) = {\rm span}\{e_2\}$, $y$-axis.
Example: For the zero matrix, $A=0$,
- $Av=\lambda v$, or
- $0 = \lambda v$.
So, $\lambda =0$, hence
- $E_A(0)={\bf R}^2$.
Example: Projection on the $x$-axis, $P = \left[ \begin{array}{} 1 & 0 \\ 0 & 0 \end{array} \right]$,
$$ \left[ \begin{array}{} 1 & 0 \\ 0 & 0 \end{array} \right] \left[ \begin{array}{} x \\ y \end{array} \right] = \lambda \left[ \begin{array}{} x \\ y \end{array} \right], {\rm or}$$
$$\left[ \begin{array}{} x \\ 0 \end{array} \right] = \left[ \begin{array}{} \lambda x \\ \lambda y \end{array} \right], {\rm or}$$
$$x=\lambda x, 0 =\lambda y$$
so, the only possible values are $0$ and $1$.
1) $\lambda = 0 \Rightarrow $
- $x=0 \cdot x \Rightarrow x=0$, and
- $0=0 \cdot y \Rightarrow y$ any.
Hence $E_A(0) = \{ \left[ \begin{array}{} 0 \\ y \end{array} \right] \colon y \in {\bf R} \} = {\rm span}\{e_2\}.$
This is the $y$-axis.
2) $\lambda = 1 \Rightarrow $
- $x=1 \cdot x \Rightarrow x$ any, and
- $0=1 \cdot y \Rightarrow y=0$.
So
- $E_A(1)={\rm span}\{e_1\}$.
This is the $x$-axis.
Example: $A$ is the $90^o$ rotation:
$A = \left[ \begin{array}{} 0 & -1 \\ 1 & 0 \end{array} \right]$.
Then
$$\left[ \begin{array}{} 0 & -1 \\ 1 & 0 \end{array} \right] \left[ \begin{array}{} x \\ y \end{array} \right] = \lambda \left[ \begin{array}{} x \\ y \end{array} \right],$$
or
$$\left\{ \begin{array}{} -y &= \lambda x \\ x = \lambda y \end{array} \right.$$
Solve it.
$$\left\{ \begin{array}{} -xy &= \lambda x^2 \\ xy &= \lambda y^2 \end{array} \right.,$$
so $\lambda x^2 = -\lambda y^2$.
Two cases:
- $\lambda = 0$, then $x=y=0$; and
- $\lambda \neq 0$, then $x^2=-y^2$.
At least one would have to be imaginary.
Since we are talking about $A \colon {\bf R}^2 \rightarrow {\bf R}^2$, $x,y$ have to be real...
Alternative formula
Shortcut: $\lambda$ is an eigenvalue for $A$ (basis fixed). For some $v \in {\bf R}^n \setminus \{0\}$,
$$\begin{array}{} {\rm definition} &\Longleftrightarrow A(v)=\lambda v \\ &\Longleftrightarrow A(v)-\lambda v = 0 \\ {\rm A \hspace{3pt} is \hspace{3pt} a \hspace{3pt} matrix} &\Longleftrightarrow Av-\lambda v=0 \\ &\Longleftrightarrow (A-\lambda)v=0 ({\rm \hspace{3pt} but \hspace{3pt}} \lambda {\rm \hspace{3pt} is \hspace{3pt} a \hspace{3pt} number!}) \\ &{\rm better: \hspace{3pt}} (A- \Lambda)v=0, \\ \Lambda=\left[ \begin{array}{} \lambda & \ldots & 0 & 0 \\ 0 & \lambda & \ldots & 0 \\ \vdots \\ 0 & \ldots & 0 & \lambda \end{array} \right] = \lambda I_n \\ &{\rm or \hspace{3pt}} (A-\lambda I_n)v=0 {\rm \hspace{3pt} for \hspace{3pt} some \hspace{3pt}} v \in {\bf R}^n \setminus \{0\}. \end{array}$$
This is a matrix equation
- matrix $A - \lambda I_n$
- $v$ is the unknown
- right hand side is $0$
Recall theorem: there is no non-zero solution if and only if rank of the matrix $=n$.
So this is an isomorphism and the matrix is invertible...
Use the theorem: $\det (A-\lambda I_n) \neq 0$.
Then, a non-zero solution exists if and only if ${\rm det}(A-\lambda I_n)=0$.
Theorem: Given a linear operator $A \colon {\bf R}^n \rightarrow {\bf R}^n$, bases fixed. Then $\lambda$ is an eigenvector if and only if $\det (A-\lambda I_n)=0$.
Example: $A = \left[
\begin{array}{}
0 & -1 \\
1 & 0
\end{array}
\right]$.
$$\det \left( \left[ \begin{array}{} 0 & -1 \\ 1 & 0 \end{array} - \lambda \left[ \begin{array}{} 1 & 0 \\ 0 & 1 \end{array} \right] \right] \right) = \det \left[ \begin{array}{} -\lambda & -1 \\ 1 & -\lambda \end{array} \right] = \lambda^2 + 1 = 0$$
Solve, then $\lambda = \pm i$.
Try $Av=\lambda v$, for $x, y \in {\bf R}$,
$$\left[ \begin{array}{} 0 & -1 \\ 1 & 0 \end{array} \right] \left[ \begin{array}{} x \\ y \end{array} \right] = i \left[ \begin{array}{} x \\ y \end{array} \right]$$
$$-y = ix,$$
$$x = iy,$$ $x,y$ real.
Note: "Eigen" value = characteristic value.
The characteristic polynomial of matrix $A$ is
$$\chi_A(\lambda) = \det(A - \lambda I_n).$$
This is a polynomial of $\lambda$!
Example: $\lambda^2+1$ for above.
Example: $A = \left[ \begin{array}{} 2 & 0 \\ 0 & 3 \end{array} \right]$, $$\chi_A(\lambda) = \det \left[ \begin{array}{} 2-\lambda & 0 \\ 0 & 3-\lambda \end{array} \right] = (2-\lambda)(3-\lambda) = 0$$
Solve, so $\lambda=2,3$.
Example: $A=\left[ \begin{array}{} 1 & 0 \\ 0 & 0 \end{array} \right]$, solve it. So $\lambda=1,0$.
Properties
- Fact 1: If $A$ is $n \times n$, then the degree of $\chi_A$ is $n$.
- Fact 2: Eigenvalues are the roots of the characteristic polynomial.
- Fact 3: Number of roots is $n$, counting multiplicities (including complex roots).
Theorem: $A$ is diagonalizable if all roots of $\chi_A$ are real with multiplicity $1$.
Why?
Find a basis for ${\bf R}^n$ that consists of the eigenvectors corresponding to these eigenvalues.
Suppose $P$ is the transition matrix for these bases. Then $D=P^{-1}AP$ is a diagonal matrix.
In this case, $D$ and $A$ are called similar, $D \sim A$.
Theorem: Similarity is an equivalence relation on ${\bf M}(n,n)$.
Proof: The axioms:
1. $A \sim A$, just choose $P=I_n$.
2. $A \sim B \longleftrightarrow A = P^{-1}BP$, how?
- $B \sim A \longleftrightarrow B=Q^{-1}AQ$.
These are equivalent to:
- $A = P^{-1}BP \longleftrightarrow B=Q^{-1}AQ.$
Find $Q$.
Choose: $Q = P^{-1}$.
$$\begin{array}{} B &= (P^{-1})^{-1}AP^{-1} = PAP^{-1} \\ P^{-1}B &= AP^{-1} \\ P^{-1}BP &= A \\ P^{-1}BP = A, \end{array}$$
as required.
3. $A \sim B \sim C \rightarrow A \sim C$, how?
We need this
- $C = P^{-1}BP,$
- $B=Q^{-1}AQ,$
to imply that
- $C=R^{-1}AR$.
We need to find $R$.
Substitute:
$$\begin{array}{} C &= P^{-1}(Q^{-1}AQ)P \\ &= P^{-1}Q^{-1}AQP \end{array}$$ Compare to $C$ above...
Choose $R=QP$, finish (exercise). $\blacksquare$
Theorem: The characteristic polynomial is preserved under similarity:
$$\chi_{P^{-1}AP} = \chi_A.$$
Proof: Recall: $$\chi_{P^{-1}AP}(\lambda) = \det (\lambda I - P^{-1}AP).$$ We need to get rid of $P$ here.
Idea: distribute $P$, then take $P$ out, use this
Fact: the determinant is preserved under similarity: $$\det (P^{-1}AP)=\det A.$$ Then $$\chi_{P^{-1}AP}(\lambda) =$$ $$={\rm det}(\lambda P^{-1}IP - P^{-1}AP)$$ factor $P^{-1}$ and $P$ out... $$={\rm det}(P^{-1} \lambda IP - P^{-1}AP)$$ use similarity... $$={\rm det}[P^{-1} (\lambda I - A)P]$$ or use the multiplicative property of determinants instead... $$={\rm det}(\lambda I - A)$$ $$=\chi_A(\lambda).$$ $\blacksquare$
Another characteristic of matrices preserved under similarity is the trace:
$${\rm tr \hspace{3pt}}A = \displaystyle\sum_{k=1}^n a_{k_k},$$
the sum of the diagonal elements of $A$.
Theorem: The trace is linear operator: ${\rm tr \hspace{3pt}} \colon {\bf M}(n,n) \rightarrow {\bf R}$. (just like the determinant)
Proof: $$\begin{array}{} {\rm tr}(A+B) &= \sum_{k=1}^n (a_{k_k} + b_{k_k}) \\ &= \sum_{k=1}^n a_{k_k} + \sum_{k=1}^n b_{k_k} \\ &= {\rm tr \hspace{3pt}}A + {\rm tr \hspace{3pt}}B. \end{array}$$
Same, ${\rm tr}(rA)$. $\blacksquare$
Theorem: ${\rm tr}(AB) = {\rm tr}(BA)$.
Proof: $C=AB$, $C^1 = BA$. Recall
$$c_{pq} = \sum_{i=1}^n a_{pi} b_{iq}.$$
Then
$${\rm tr}(AB) = \sum_{k=1}^n c_{k_k} = \sum_{k=1}^n \sum_{i=1}^n a_{ki} + b_{ik}$$
$${\rm tr}(BA) = \sum_{k=1}^n c^1_{k_k} = \sum_{k=1}^n \sum_{i=1}^n b_{ki}a_{ik} = \sum_{k=1}^n \sum_{i=1}^n a_{ik}b_{ki}$$
These two sums don't seem to match.
What do we do? Compare them closely:
$${\rm tr}(AB) = \sum_{ki} a_{ki} b_{ik}$$
$${\rm tr}(BA) = \sum_{ki} a_{ik} b_{ki}$$
They are equal, why?
Recall that changing (as in $\displaystyle\int_0^1 x^2 dx = \displaystyle\int_0^1 t^2 dt$) names of variables is ok, when the outcome is independent of its.
Rename $k$ and $i$. How about $k \rightarrow i$ and $i \rightarrow k$.
$${\rm tr}(BA) = \sum_{ik}a_{ki}b_{ik} = {\rm tr}(AB).$$ $\blacksquare$
The idea comes from: sum of all entries in a table, rows first or columns first, same.
Theorem: The trace is preserved under similarity:
$${\rm tr}(P^{-1}AP) = {\rm tr}(A).$$
Proof: Use the last theorem
$${\rm tr}(P^{-1}AP) = {\rm tr}(PP^{-1}A) = {\rm tr}(IQ) = {\rm tr}(A).$$
$\blacksquare$
Exercise: Find all matrices similar to $I$.