Differentiation

.title[
# Differentiation
]
.author[
### <a href="https://jmclip.github.io/MACSS_math_camp/">MACS 33000</a> <br /> University of Chicago
]

---

# Learning objectives

* Define the slope of a line
* Summarize tangent lines, rates of change, and derivatives
* Define derivative rules for common functions
* Apply the product, quotient, and chain rules for differentiation
* Summarize the exponential function and natural logarithms
* Identify properties of derivatives helpful to statistical methods

---

# What is calculus?

* Calculus
* Differential calculus
* Integral calculus
* Underlies most statistical and computational methods
* Crucial for **optimization**

## Potential uses

* Given data, what is the most likely value of a parameter(s)?
* Game theory: given another player's strategy, what is the action that maximizes utility?

---

# How functions change

* Derivatives
* A special limit
* Cover three broad ideas 
    1. Geometric interpretation/intuition
    1. Formulas/algebra derivatives
    1. Famous theorems

---

# The tangent as a limit

---

# The tangent as a limit

]

$$
`\begin{aligned}
P &= (x_0, y_0) \\
Q &= (x_1, y_1) \\
\text{slope of } L &= \frac{y_1 - y_0}{x_1 - x_0}
\end{aligned}`
$$

]

---

# The tangent as a limit

]

$$
`\begin{aligned}
P &= (x_0, y_0) \\
Q &= (x_1, y_1) \\
\text{slope of } L &= \frac{y_1 - y_0}{x_1 - x_0}
\end{aligned}`
$$

### Substitute values

$$
`\begin{aligned}
h &= x_1 - x_0 \\
x_1 &= x_0 + h \\
y_0 &= f(x_0) \\
y_1 &= f(x_0 + h) \\
\text{slope of } L &= \frac{f(x_0 + h) - f(x_0)}{h}
\end{aligned}`
$$

]

---

# Derivative

Suppose `$f:\Re \rightarrow \Re$`. Measure rate of change at a point `$x_{0}$` with a function `$R(x)$`,

`$$R(x) = \frac{f(x) - f(x_{0}) }{ x- x_{0} }$$`

* `$R(x)$` defines the rate of change
* A derivative will examine what happens with a small perturbation at `$x_{0}$`

---

# Derivative

* Let `$f:\Re \rightarrow \Re$`. If the limit

`$$\begin{aligned} \lim_{x\rightarrow x_{0}} R(x) & = \frac{f(x) - f(x_{0}) }{x - x_{0}} \\ & = f^{'}(x_{0}) \end{aligned}$$`

exists then we say that `$f$` is **differentiable** at `$x_{0}$`
* If `$f^{'}(x_{0})$` exists for all `$x \in \text{Domain}$`, then we say that `$f$` is differentiable

---

# Derivative

* Let `$f$` be a function whose domain includes an open interval containing the point `$x$`
* The derivative of `$f$` at `$x$` is given by

`$$\frac{d}{dx}f(x) =\lim\limits_{h\to 0} \frac{f(x+h)-f(x)}{(x+h)-x} = \lim\limits_{h\to 0} \frac{f(x+h)-f(x)}{h}$$`

## Derivative notation

* Leibniz notation: `$\frac{d}{dx}(f(x))$`
* Prime or Lagrange notation: `$f'(x)$`

---

# Example derivatives

<div class="figure" style="text-align: center">
<img src="03-differentiation_files/figure-html/derivsimple-1.png" alt="The Derivative as a Slope" width="864" />
<p class="caption">The Derivative as a Slope</p>
</div>

* If `$f'(x)$` exists at a point `$x_0$`, then `$f$` is said to be **differentiable** at `$x_0$`
* Implies continuity of `$f(x)$` at `$x_0$`

---

# Rates of change in a function

* Consider `$y = f(x)$`
* As `$x$` changes from `$x_0$` to `$x_0 + h$`, the value of the function changes from `$f(x_0)$` to `$f(x_0 + h)$`
* Change in `$x$` is `$h$`
* Change in `$f(x)$` is `$f(x_0 + h) - f(x_0)$`
* Rate of change of `$f(x)$` is defined to be

`$$\frac{f(x_0 + h) - f(x_0)}{h}$$`

* Same thing as a derivative

---

# Rates of change in a function

* Rate of change `$\leadsto$` return on vote share for dollars invested
* Instantaneous rate of change `$\leadsto$` increase in vote share in response to infinitesimally small increase in spending
* A type of **limit**

---

# Examples of derivatives

* `$f(x) = x^2$` and consider `$x_{0} = 1$`

$$
`\begin{aligned}
\lim_{x\rightarrow 1}R(x) & = \lim_{x\rightarrow 1} \frac{x^2 - 1^2}{x - 1}  \\
 	& = \lim_{x\rightarrow 1} \frac{(x- 1)(x + 1) }{ x- 1}   \\
	& =  \lim_{x\rightarrow 1} x + 1  \\
	& = 2
\end{aligned}`
$$

---

# Examples of derivatives

* `$f(x) = |x|$` and consider `$x_{0} = 0$`

`$$\lim_{x\rightarrow 0} R(x) = \lim_{x\rightarrow 0} \frac{ |x| } {x}$$`

* `$\lim_{x \rightarrow 0^{-}} R(x) = -1$`
* `$\lim_{x \rightarrow 0^{+}} R(x) = 1$`
* Not differentiable at `$0$`

---

# Continuity and derivatives

* `$f(x) = |x|$` is **continuous** but not differentiable
* Change is **too abrupt**
* Suggests differentiability is a stronger condition

Let `$f:\Re \rightarrow \Re$` be differentiable at `$x_{0}$`. Then `$f$` is continuous at `$x_{0}$`.

---

# What goes wrong?

$$
`\begin{aligned}
f(x)  & = x^{2} \text{ for all  } x \in \Re \setminus 0 \\
f(x) & = 1000  \text{ for  } x = 0
\end{aligned}`
$$

* `$f'(0)$`

`$$\begin{aligned} \lim_{x \rightarrow 0 } R(x) & = \lim_{x \rightarrow 0 } \frac{f(x+h)-h}{ h   } \\ &= \lim_{x \rightarrow 0 }\frac{h^2}{h}\\ &= \lim_{x \rightarrow 0 } h \end{aligned}$$`

* `$\lim_{x \rightarrow 0 } h \neq 1000$`, so the limit doesn't exist

---

# Calculating derivatives

* Rarely will we take a limit to calculate derivative
* Rather, rely on **rules** and properties of derivatives
* **Important**: do not forget core intuition

## Strategy

* Algebra theorems
* Some specific derivatives
* Work on problems

---

# Derivative rules: WRITE THESE DOWN

Suppose `$a$` is some constant, `$f(x)$` and `$g(x)$` are functions:

$$
`\begin{aligned}
f(x) &= x & \quad f^{'}(x) &= 1 \\
f(x) &= a x^{k} & \quad f^{'}(x) &= (a) (k) x ^{k-1} \\
f(x) &= e^{x } & \quad f^{'} (x) &= e^{x} \\
f(x) &= \sin(x) & \quad f^{'} (x) &= \cos (x) \\
f(x) &= \cos(x) & \quad f^{'} (x) &= - \sin(x) \\
\end{aligned}`
$$

---

# Derivative rules

##### Constant rule `$\left[k f(x)\right]' = k f'(x)$`

##### Sum rule `$\left[f(x)\pm g(x)\right]' = f'(x)\pm g'(x)$`

##### Product rule `$\left[f(x)g(x)\right]' = f'(x)g(x)+f(x)g'(x)$`

##### Quotient rule `$\frac{f(x)}{g(x)}' = \frac{f'(x)g(x)-f(x)g'(x)}{[g(x)]^2}, g(x)\neq 0$`

##### Power rule `$\left[x^k\right]' = k x^{k-1}$`

---

# Derivatives with a friend

Differentiate the following functions:

1. `$f(x)= x^3 + 5 x^2  + 4 x$`, <span style="color: gray;"> at `$x_{0} = 2$`</span>
1. `$f(x) = \sin(x) x^3$`, <span style="color: gray;"> at `$x_{0} = 2$`</span>
1. `$h(x) = \dfrac{e^{x}}{x^3}$`, <span style="color: gray;"> at `$x_{0} = 2$` </span>
1. `$h(x) = \log (x) x^3$`, <span style="color: gray;"> at `$x_{0} = e$` </span>

---

# Composite functions

* Differentiating functions of functions

`$$x^2 + 1^2 \quad \text{vs.} \quad (x^2 + 1)^2$$`

* Composite functions

`$$f \circ g=f[g(x)]$$`

* Range of `$g$` must be contained (at least in part) within the domain of `$f$`
* Domain of `$f \circ g$` consists of all the points in the domain of `$g$` for which `$g(x)$` is in the domain of `$f$`

---

# Composite functions

$$
`\begin{aligned}
f(x) &= \log x \in (0, \infty) \\
g(x) &= x^2 \in (-\infty, \infty)
\end{aligned}`
$$

`$$f[g(x)] = \log x^2 \in (-\infty, \infty) - \{0\}$$`

`$$g[f(x)] = [\log x]^2 \in (0, \infty)$$`

---

# Chain rule

* `$y = f[g(x)]$`
* Derivative of `$y$` with respect to `$x$` is

`$$\frac{d}{dx} \{ f[g(x)] \} = f'[g(x)] g'(x)$$`

* Derivative of the "outside" times the derivative of the "inside"
The chain rule can be thought of as the derivative of the "outside" times 
* "Outside" evaluated at the value of the inside function

---

# Chain rule

$$
`\begin{aligned}
h(x) &= e^{2x} \\
g(x) &= e^{x} \\
f(x) &= 2x
\end{aligned}`
$$

`$$h(x) = g(f(x)) = g(2x) = e^{2x}$$`

`$$h^{'}(x) = g^{'}(f(x))f^{'}(x) = e^{2x}2$$`

---

# Chain rule

$$
`\begin{aligned}
h(x) &= \log(\cos(x) ) \\
g(x) &= \log(x) \\
f(x) &= \cos(x)
\end{aligned}`
$$

`$$h(x) = g(f(x)) = g( \cos(x)) = \log(\cos(x))$$`

`$$h^{'}(x) = g^{'}(f(x))f^{'}(x) = \frac{-1}{\cos(x)} \sin(x) = -\tan (x)$$`

---

# Generalized power rule

* If `$f(x)=[g(x)]^p$` for any rational number `$p$`

`$$f^\prime(x) =p[g(x)]^{p-1}g^\prime(x)$$`

---

# Exponential function

---

# Derivative of exponential function

`$$\frac{d}{dx}(e^x) = e^x$$`

* Why?

$$
`\begin{aligned}
\frac{d}{dx}f(x) & = \lim\limits_{h\to 0} \frac{f(x+h)-f(x)}{h} \\
&= \lim\limits_{h\to 0} \frac{e^{x + h} - e^x}{h} \\
&= \lim\limits_{h\to 0} \frac{e^x e^h - e^x}{h} \\
&= \lim\limits_{h\to 0} \frac{e^x(e^h - 1)}{h} \\
&= e^x \lim\limits_{h\to 0} \frac{e^h - 1}{h}
\end{aligned}`
$$

---

# Derivative of exponential function

---

# Derivative of exponential function

---

# Derivative of exponential function

<div class="figure" style="text-align: center">
<img src="03-differentiation_files/figure-html/fig-derivexponent-1.png" alt="Derivative of the Exponential Function" width="864" />
<p class="caption">Derivative of the Exponential Function</p>
</div>

---

# Derivative of the natural logarithm

* Logarithm to base `$e$` of `$x$`, where `$e$` is defined as **Euler's number**

`$$y = \log_e (x) \iff x = e^y$$`

* Relationship between `$e^x$` and `$\log_e(x)$`

`$$\begin{aligned} e^{\log(x)} &= x \, \text{for every positive number} \, x \\ \log(e^y) &= y \, \text{for every real number} \, y \\ \end{aligned}$$`

---

# Exponential function and natural logarithm

<div class="figure" style="text-align: center">
<img src="03-differentiation_files/figure-html/exp-log-1.png" alt="Exponential function and natural logarithm" width="864" />
<p class="caption">Exponential function and natural logarithm</p>
</div>

---

# Derivative of a natural logarithm
    
  $$(f^{-1})'(x) = \frac{1}{f'(f^{-1}(x))} $$
    
Let's find the derivative of `$log(x)$`. We already know the derivative of its inverse, `$e^x$` is `$e^x$`. So, we begin with `$f(x) = e^x$` (and `$f'(x) = e^x$`) and want to know `$(f^{-1})'(x)$`.
  
We know `$f'(f^{-1}(x)) = f'(log(x)) = e^{log(x)} = x$`
  
--

Therefore,
  `$$\begin{aligned} (f^{-1})'(x) &= \frac{1}{f'(f^{-1}(x))} \\
   &= \frac{1}{(e^{log(x)})} \\
  &= \frac{1}{x} \end{aligned}$$`

---

# Derivative of a natural logarithm

<div class="figure" style="text-align: center">
<img src="03-differentiation_files/figure-html/fig-derivlog-1.png" alt="Derivative of the Natural Log" width="864" />
<p class="caption">Derivative of the Natural Log</p>
</div>

---

# Relevance of `$e^x$` and `$\log(x)$`

* Growth over time (e.g. compounding interest)
* Elasticity models
* Variable transformation

---

# Derivatives and properties of functions

* Often used to **optimize** a function (tomorrow)
* But also reveal **average rates of change**
* Or crucial properties of functions

---

# Rolle's theorem

> Suppose `$f:[a, b] \rightarrow \Re$`.  Suppose `$f$` has a relative maxima or minima on `$(a,b)$` and call that `$c \in (a, b)$`.  Then `$f'(c) = 0$`.

---

# Rolle's theorem

> Suppose `$f:[a, b] \rightarrow \Re$`.  Suppose `$f$` has a relative maxima or minima on `$(a,b)$` and call that `$c \in (a, b)$`.  Then `$f'(c) = 0$`.

* Consider (without loss of generalization) a relative maximum `$c$`
* Consider the left-hand and right-hand limits

`$$\begin{aligned} \lim_{x \rightarrow c^{-}} \frac{f(x) - f(c) }{x - c } & \geq 0  \\ \lim_{x \rightarrow c^{+}} \frac{f(x) - f(c) } {x - c }  & \leq 0 \end{aligned}$$`

* But we also know that

`$$\begin{aligned} \lim_{x \rightarrow c^{-}} \frac{f(x) - f(c ) }{x - c } & = f^{'}(c)  \\ \lim_{x \rightarrow c^{+}} \frac{f(x) - f(c) } {x - c }  &  =  f^{'}(c) \end{aligned}$$`

---

# Rolle's theorem

* The only way, then, that we can have something `$\geq 0$` equal to `$\leq 0$` is if `$f^{'}(c) = 0$` :
`$$\lim_{x \rightarrow c^{-}} \frac{f(x) - f(c) }{x -c}  = \lim_{x \rightarrow c^{+}} \frac{f(x) - f(c) } {x - c}$$`

---

# Mean value theorem

* If `$f:[a,b] \rightarrow \Re$` is continuous on `$[a,b]$` and differentiable on `$(a,b)$`, then there is a `$c \in (a,b)$` such that

`$$f^{'}(c) = \frac{f(b) - f(a) } { b - a}$$`

* Rolle's theorem is a special case of the MVT, where `$f'(c) = 0$`

---

# Mean value theorem

---

# Applications of the mean value theorem

* This will come up in a formal theory article. You'll at least know where to look
* It allows us to say lots of powerful stuff about functions
* Especially useful for approximating derivatives (see next set of slides)

---

# Applications of the mean value theorem

Suppose that `$f:[a,b] \rightarrow \Re$` is continuous on `$[a,b]$` and differentiable on `$(a,b).$` Then,

1. If `$f^{'}(x) \neq 0$` for all `$x \in (a,b)$` then `$f$` is 1-1
1. If `$f^{'}(x) = 0$` then `$f(x)$` is constant 
1. If `$f^{'}(x)> 0$` for all `$x \in (a,b)$` then then `$f$` is strictly increasing
1. If `$f^{'}(x)<0$` for all `$x \in (a,b)$` then `$f$` is strictly decreasing

---

# If `$f^{'}(x) \neq 0$` for all `$x \in (a,b)$` then `$f$` is 1-1

---

# If `$f^{'}(x) \neq 0$` for all `$x \in (a,b)$` then `$f$` is 1-1

* Suppose that `$f$` is not 1-1. Then there is `$x, y \in (a,b)$` such that `$f(x) = f(y)$`. Then,

`$$f'(c) = \frac{f(x) - f(y)}{x- y} = \frac{0}{x -y}  = 0$$`

* This means `$f' \neq 0$` for all `$x$`
* **Contradiction**

---

# If `$f^{'}(x) = 0$` then `$f(x)$` is constant

* By way of contradiction, suppose that there is `$x, y \in (a,b)$` such that `$f(x) \neq f(y)$`. But then,

`$$f'(c) = \frac{f(x) - f(y) } {x - y} \neq 0$$`

---

# If `$f^{'}(x)> 0$` for all `$x \in (a,b)$` then `$f$` is strictly `$\uparrow$`

* By way of contradiction, suppose that there is `$x, y \in (a,b)$` with `$y<x$` but `$f(y)>f(x)$`. But then,

`$$f'(c) = \frac{f(x) - f(y) }{x - y } < 0,$$` which contradicts `$f^{'}(x)> 0$`
* Proof for strictly decreasing is the reverse of this

---

# Cauchy mean value theorem

* Suppose `$f$` and `$g$` are differentiable functions and `$a$` and `$b$` are real numbers such that `$a < b$`
* Suppose also that `$g'(x) \neq 0$` for all `$x$` such that `$a < x < b$`
* There exists a real number `$c$` such that `$a < c < b$` and

`$$\frac{f'(c)}{g'(c)} = \frac{f(b) - f(a)}{g(b) - g(a)}$$`

* The classical mean value theorem is the special case where `$g(x) = x$` for all `$x$`

---

# Implications and a little help from Cauchy

Suppose we want to find

`$$\lim_{x \rightarrow a} \frac{f(x)}{g(x)}$$`

where `$f$` and `$g$` are continuous functions

* Suppose `$g(a) \neq 0$`, then we can do this:

`$$\lim_{x \rightarrow a} \frac{f(x)}{g(x)} = \frac{f(a)}{g(a)}$$`

* BUT if `$f(a) = g(a) = 0$`... OH NO! What do we do with an indeterminate form (e.g. `$\frac{0}{0}$`)?

---

# New bestie: L'Hôpital's Rule

* Suppose that `$f(a) = g(a) = 0$` and `$g'(x) \neq 0$` if `$x$` is close but not equal to `$a$` (also applies for `$\frac{\pm\infty}{\infty}$`).
* Then

`$$\lim_{x \rightarrow a} \frac{f(x)}{g(x)} = \lim_{x \rightarrow a} \frac{f'(x)}{g'(x)}$$`

$$
`\begin{aligned}
\frac{f(x) - f(a)}{g(x) - g(a)} &= \frac{f'(c)}{g'(c)} \\
\frac{f(x) - 0}{g(x) - 0} &= \frac{f'(c)}{g'(c)} \\
\frac{f(x)}{g(x)} &= \frac{f'(c)}{g'(c)}
\end{aligned}`
$$

---

# Example of L'Hôpital's Rule

]

$$
`\begin{aligned}
f'(x) &= \frac{1}{3} (1 + x)^{-2/3} \\
f'(0) &= \frac{1}{3} (1)^{-2/3} = \frac{1}{3} (1) = \frac{1}{3} \\
g'(x) &= 1 - 2x \\
g'(0) &= 1 - 2(0) = 1
\end{aligned}`
$$

`$$\lim_{x \rightarrow a} \frac{f(x)}{g(x)} = \lim_{x \rightarrow a} \frac{f'(x)}{g'(x)} = \frac{1/3}{1} = \frac{1}{3}$$`

]

---

# Example of L'Hôpital's Rule

]

$$
`\begin{aligned}
f(x) &= x - \log(1 + x) \\
f'(x) &= 1 - \frac{1}{1 + x} \\
g(x) &= x^2 \\
g'(x) &= 2x
\end{aligned}`
$$

]

---

# Simplify the expression

$$
`\begin{aligned}
L &= \lim_{x \rightarrow 0} \frac{1 - \frac{1}{1 + x}}{2x} \\
&= \lim_{x \rightarrow 0} \frac{1}{2x} - \frac{\frac{1}{1 + x}}{2x} = \lim_{x \rightarrow 0} \frac{1}{2x} - \frac{1}{2x(1 + x)} \\
&= \lim_{x \rightarrow 0} \frac{1(1 + x)}{2x(1 + x)} - \frac{1}{2x(1 + x)} \\
&= \lim_{x \rightarrow 0} \frac{1(1 + x) - 1}{2x(1 + x)} \\
&= \lim_{x \rightarrow 0} \frac{1 + x - 1}{2x(1 + x)} \\
&= \lim_{x \rightarrow 0} \frac{x}{2x(1 + x)} = \lim_{x \rightarrow 0} \frac{1}{2(1 + x)} \\
&= \lim_{x \rightarrow 0} \frac{1}{2(1 + 0)} = \frac{1}{2}
\end{aligned}`
$$

---

# Iterative application

`$$\lim_{x \rightarrow a} \frac{f(x)}{g(x)} = \lim_{x \rightarrow a} \frac{f'(x)}{g'(x)} = \lim_{x \rightarrow a} \frac{f''(x)}{g''(x)} = \ldots$$`

$$
`\begin{aligned}
f''(x) &= \frac{1}{(1 +x)^{2}} \\
g''(x) &= 2 \\
\lim_{x \rightarrow 0} \frac{f''(x)}{g''(x)} &= \frac{(1 + x)^{-2}}{2} = \frac{1^{-2}}{2} = \frac{1}{2}
\end{aligned}`
$$

---

# RECAP

---
RECAP:

* Good chunk of calc 1
* Rate of change
* Intuition for derivative
* Mathematical foundation for derivative
* Rules for derivatives
* Making magic: MVT + implications

---

## Helpful formulas

---

# Derivative rules: WRITE THESE DOWN

Suppose `$a$` is some constant, `$f(x)$` and `$g(x)$` are functions:

---

# Derivative rules

##### Constant rule `$\left[k f(x)\right]' = k f'(x)$`

##### Sum rule `$\left[f(x)\pm g(x)\right]' = f'(x)\pm g'(x)$`

##### Product rule `$\left[f(x)g(x)\right]' = f'(x)g(x)+f(x)g'(x)$`

##### Quotient rule `$\frac{f(x)}{g(x)}' = \frac{f'(x)g(x)-f(x)g'(x)}{[g(x)]^2}, g(x)\neq 0$`

##### Power rule `$\left[x^k\right]' = k x^{k-1}$`