This site is being phased out.
Gradient
How do we understand the derivative of functions of several variables?
Fix a point $a$. Then
- The directional derivative $\nabla_e f(a)$ is a number for each $e$, $||e||=1$, there are infinitely many.
- The partial derivative $\frac{\partial f}{\partial x_k}(a)$ is a number, the gradient, and depends on the choice of coordinates.
- The total derivative $f'(a)$ is a linear map from ${\bf R}^n$ in ${\bf R}$ and is given by a matrix of dimension $1 \times n$ if coordinates are chosen.
Theorem. Suppose $a \in {\rm int \hspace{3pt}}D(f)$, and $f$ is differentiable at $a$. Then
- $\nabla_e f(a) = f'(a)(e)$ ($f'(a)$ a linear map applied to $e$)
- $f′(a) = {\rm grad \hspace{3pt}}f(a)^T$ (transposed)
- $f′(a)(x) = < {\rm grad \hspace{3pt}}f(a)^T, x >.$
Proof 1. Given $e$ with $||e|| = 1$, we plug in the definition of $f'(a)$:
$$\displaystyle\lim_{x \rightarrow a} \frac{f(x) - T(x)}{|| x - a ||} = 0.$$
Take $x = a + te, t > 0$. Then $x \rightarrow a$ if $t \rightarrow 0$. Then
$$\displaystyle\lim_{t \rightarrow 0} \frac{f( a + te ) - T( a + te)}{t} = 0,$$
because $f$ is differentiable at $a$.
Recall $T$ is an affine function, i.e.
$$T(x) = f(a) + f'(a)( x - a ).$$
Substituting the above term into the limit, we obtain
$$\displaystyle\lim_{t \rightarrow 0} \frac{f( a + te ) - f(a) - f'(a)( te )}{t} = 0,$$
or
$$\displaystyle\lim_{t \rightarrow 0} \left[ \frac{ f( a + te ) - f(a) }{t} - \frac{f'(a)( te )}{t} \right] = 0.$$
Note that the first term is the directional derivative. Further, $f'(a)$ is a linear map, so we can take out t. Hence
$$\displaystyle\lim_{t \rightarrow 0} \left[ \frac{f( a + te ) - f(a)}{t} - f'(a)( e ) \right] = 0.$$
Taking out $f'(a)( e )$ (which we can do because it is independent of $t$) yields
$$\displaystyle\lim_{t \rightarrow 0} \frac{f( a + te ) - f(a)}{t} = f'(a)( e ).$$
Prove 2 and 3 as an exercise.
Algebraic Properties of $f'$
where $f: {\bf R}^n \rightarrow {\bf R}$, $f(x)$ a number.
1. Sum Rule
$$( f + g )' (a) = f'(a) + g'(a)$$
2. Product Rule $$( f \cdot g )' (a) = f'(a) g(a) + g'(a) f(a)$$
(Note that $f'(a), g'(a)$ are vectors, $f(a), g(a)$ are scalars)
3. Quotient Rule $$\left( \frac{f}{g} \right)' (a) = \frac{f'(a) g(a) - g'(a) f(a)}{g(a)^2}, g(a) \neq 0.$$
Chain Rule Let
$$(a) f: {\bf R}^n \rightarrow {\bf R}, {\rm \hspace{3pt} and \hspace{3pt}} g: {\bf R} \rightarrow {\bf R}, {\rm \hspace{3pt} or}$$
$$(b) f: {\bf R} \rightarrow {\bf R}^n, {\rm \hspace{3pt} and \hspace{3pt}} g: {\bf R}^n \rightarrow {\bf R}.$$
Then
$$( g \circ f )' (a) = g'( f(a) ) \cdot f'(a).$$
"Image"
Example. 5
Let
$$f: {\bf R} \rightarrow {\bf R}^2,$$
$$f(t) = ( t {\rm cos \hspace{3pt}} t, t^2 {\rm sin \hspace{3pt}} t )$$
$$g: {\bf R}^2 \rightarrow {\bf R},$$
$$g( x_1, x_2 ) = x_1^2 + x_2^2.$$
Further $t = \frac{\pi}{2}, ( x_1, x_2 ) = ( 0, \frac{\pi^2}{2^2} ) = f( \frac{\pi}{2} ).$
Then
$$\frac{\partial g}{\partial x_1} = 2x_1 {\rm \hspace{3pt} and \hspace{3pt}} \frac{\partial g}{\partial x_2} = 2x_2$$
$$\rightarrow g'( f \left( \frac{\pi}{2} \right) ) = \left[ 0, \frac{\pi^2}{2} \right],$$
$$f'(t) = ( {\rm cos \hspace{3pt}} t - t {\rm sin \hspace{3pt}} t, 2t {\rm sin \hspace{3pt}}t + t^2 {\rm cos \hspace{3pt}} t ),$$
$$\begin{array}{} f' \left( \frac{\pi}{2} \right) &= \left( \frac{\pi}{2}, \pi \right) \\ &= | \frac{\pi}{2} | \\ &| \pi | \end{array}$$
Exercise. With Chain Rule: $$\begin{array}{} \frac{d}{dt} ( g \circ f )\left( \frac{\pi}{2} \right) &= g'( f \left( \frac{\pi}{2} \right) ) \cdot f' \left( \frac{\pi}{2} \right) \\ &= < g'( f \left( \frac{\pi}{2} \right) ), f′ \left( \frac{\pi}{2} \right) > \\ &= < \left[ 0, \frac{\pi}{2} \right], \left[ \frac{2 \pi}{2} , \pi \right] > \\ &= \frac{\pi^2}{2} \end{array}$$
(Verify)
Chain Rule
$$\frac{d}{dx} f( g(x) ) = f'( g(x) ) \cdot g'(x) {\rm \hspace{3pt} with \hspace{3pt} Calc 1}$$
$$\frac{d}{dx} ( f \circ g )(a) = f'(b) \cdot g'(a), b = g(a).$$
The derivative of the composition is the product of the derivatives, and the derivative of the composition is the composition of the derivatives. (The same because they are linear)
Let ${\rm dim \hspace{3pt}} = 1, f'(b) = 3$ (not a number, it's a linear map), $g'(a) = 5$.
Then
$$\frac{d}{dx}( f \circ g )(a) = 3 \cdot 5 = 15.$$
Let $f'(b)(u) = 3u$ linear with respect to $u, g'(a)(v) = 5v$ linear with respect to $v$.
"Image"
Let ${\rm dim \hspace{3pt}} = n.$
Fastest growth - what is the direction?
"Image"
Maximize the directional derivatives: Find $e$ with $|| e || = 1$ such that
$$\nabla_e f(a) \geq \nabla_u f(a)$$
for all other $u, || u || = 1$.
We maximize by changing $e$:
$$\begin{array}{} \nabla_e f(a) &= < f'(a), e > \\ &= || f'(a) || \cdot || e || {\rm cos \hspace{3pt}} \varphi, {\rm \hspace{3pt} where \hspace{3pt}} \varphi {\rm \hspace{3pt} is \hspace{3pt} the \hspace{3pt} angle \hspace{3pt} between \hspace{3pt}} e {\rm \hspace{3pt} and \hspace{3pt}} f'(a), \\ &= || f'(a) || \cdot {\rm cos \hspace{3pt}} \varphi, \end{array}$$
where $f'(a)$ is fixed and ${\rm cos \hspace{3pt}} \varphi$ varies.
We maximize ${\rm cos \hspace{3pt}} \varphi$, the largest value is $1 = {\rm cos \hspace{3pt}} 0$. So we choose
$$e = \frac{f'(a)}{|| f'(a) ||} = \nabla \frac{f(a)}{|| \nabla f(a) ||},$$
i.e. the direction is that of the gradient.
Following the gradient will lead us to a local maximum. Also,
$$\max_{||e||=1} \nabla_e f(a) = || \nabla f(a) ||.$$