General random variables

class: center, middle, inverse, title-slide

.title[
# General random variables
]
.author[
### <a href="https://jmclip.github.io/MACSS_math_camp/">MACS 33000</a> <br /> University of Chicago
]

---

`$$\newcommand{\E}{\mathrm{E}} \newcommand{\Var}{\mathrm{Var}} \newcommand{\Cov}{\mathrm{Cov}}$$`
# Misc

* Donuts: YES
* Homework moving along
* Career services TOMORROW
  * related: get a linkedin page and add me!

---
# Survey Results

* Learning good--I would like people to feel like they're learning a lot
  * part is because it's review for people, part because it's fast?
* Supportive learning: 90% 
  * my goal is 100% reach out to me and/or TAs if you want to talk more
* Valued as student: 90% 
  * ditto above -- would like everyone to feel valued

---
# Survey:

* Shorter, more regular
* Yes, pre-test
* Even split on 9 vs 9:30am start time
* 60/40 on IMSI vs Kent
* Split on pacing and examples
* GREAT IDEA: pm TA slots on zoom!
* Unhappiness re: early problem sets
  * going to trim them and move some to additional practice

---

# Learning objectives

* Define a continuous random variable
* Identify continuous random variable distributions relevant to social science
* Define expected value and variance of continuous random variables
* Relate continuous random variables to discrete random variables
* Define cumulative distribution functions (CDFs) for continuous random variables and compare to discrete random variables
* Estimate probability of events using probability density functions (PDFs) and cumulative distribution functions (CDFs)

---

# Continuous random variables

* Random variables that are not discrete
    * Approval ratings
    * GDP
    * Wait time between wars: `$X(t) = t$` for all `$t$`
    * Proportion of vote received: `$X(v) = v$` for all `$v$`
* Many analogues to discrete probability distributions
* We need calculus to answer questions about probability

---

# Probability density function

What is the area under the curve under `$f(x)$` between `$.5$` and `$2$`?

`$$\int_{1/2}^{2} f(x)\,dx = F(2) - F(1/2)$$`

---

# Continuous random variable

`$X$` is a **continuous random variable** if there exists a nonnegative function defined for all `$x \in \Re$` having the property for any (measurable) set of real numbers `$B$`,

`$$\Pr(X \in B) = \int_{B} f_X(x)\,dx$$`

---

# Continuous random variable

The probability that the value of `$X$` falls within an interval is

`$$\Pr (a \leq X \leq b) = \int_a^b f_X(x) \,dx$$`

--
Compare discrete and continuous variables. Suppose we have a probability function `$f(x)= 1/3$` where `$0 < x \leq 3$`. What is `$p(X=x)$`?

* For any single value `$a$`, `$\Pr (X = a) = \int_a^a f_X(x) \,dx = 0$`

### Requirements to be a PDF

* Non-negative
* Normalization property

`$$\int_{-\infty}^{\infty} f_X(x) \,dx = \Pr (-\infty \leq X \leq \infty) = 1$$`

---

# Uniform random variable

`$$X \sim \text{Uniform}(0,1)$$`

`$$f_X(x) = \left\{    \begin{array}{ll}        c & \quad \text{if } 0 \leq x \leq 1 \\        0 & \quad \text{otherwise}    \end{array}\right.$$`

`$$1 = \int_{-\infty}^{\infty} f_X(x)\,dx = \int_0^1 c \,dx = c \int_0^1 \,dx = c$$`

---

# Uniform random variable

---

# Uniform random variable

`$$\begin{aligned}
\Pr(X \in [0.2, 0.5]) & = \int_{0.2}^{0.5} 1 \,dx \\
& = X |^{0.5}_{0.2} \\
& = 0.5  - 0.2  \\
& = 0.3 
\end{aligned}$$`

`$$\begin{aligned}
\Pr(X \in [0.5, 0.5]) & = \int_{0.5}^{0.5} 1\,dx \\
& = X|^{0.5}_{0.5} \\
& = 0.5 - 0.5 \\
& = 0 
\end{aligned}$$`

`$$\begin{aligned}
\Pr(X \in \{[0, 0.2]\cup[0.5, 1]\}) & = \int_{0}^{0.2} 1\,dx + \int_{0.5}^{1} 1\,dx \\
 & = X_{0}^{0.2} + X_{0.5}^{1} \\
 & = 0.2 - 0 + 1 - 0.5 \\
 & = 0.7 
\end{aligned}$$`

---

# Uniform random variable

`$$f_X(x) = \left\{    \begin{array}{ll}        \frac{1}{b-a} & \quad \text{if } a \leq x \leq b \\        0 & \quad \text{otherwise}    \end{array}\right.$$`

---

# Continuous random variables

* `$\Pr(X = a) = 0$`
* `$\Pr(X \in (-\infty, \infty) ) = 1$` 
* If `$F$` is antiderivative of `$f$`, then `$\Pr(X \in [c,d]) = F(d) - F(c)$`

---
class: middle

# Having a moment

---
# Moments: what they are and what they tell us
Moments help us understand the shape of the distribution

1. Shape around the center (mean): `$E[X]$`
1. Width (variance): `$E[X^2]$`
1. Balance of width (skew): `$E[X^3]$`
1. Tail-y-ness (kurtosis): `$E[X^4]$`

---
class: middle, inverse

# Great expectations: first moment

* Summarizing a distribution
* What we can expect FROM THE DISTRIBUTION re: center, ON AVERAGE
* This will give us our baseline expectation for what we *should* be observing *if* our assumptions are true

---

# Expectation

`$$\E[X] = \int_{-\infty}^{\infty} x f_X(x) \,dx$$`

* Integration instead of summation

* Expected value rule

`$$\E[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x) \,dx$$`

* `$n$`th moment = `$\E[X^n]$`

* `$\Var(X) = E[(X-\mu)^2]=E[(X - \E[X])^2] = \E[X^2] - (\E[X])^2$`

---

# Uniform random variable: Mean
Start with our pdf: `$f(x) = \frac{1}{b-a}$` (defined over a to b).

`$$\E[X] = \int_{-\infty}^{\infty} x f_X(x) \,dx$$`
--
`$$= \int_a^b x * \frac{1}{b-a} \,dx$$`
--
`$$= \frac{1}{b-a} * \frac{1}{2}x^2 \Big|_a^b$$`
--
`$$= \frac{1}{b-a} * \frac{b^2 - a^2}{2}= \frac{a+b}{2}$$`
---
# Uniform random variable: Mean
Start with our pdf: `$f(x) = \frac{1}{b-a}$` (defined over a to b). `$$\mu = \frac{a+b}{2}$$`

SO WHAT?????

This is what we can anticipate as the mean of our distribution. 
--
On average. 
--
Over time. 
--
In the long run.

---

# Uniform random variable: Variance
Start with our pdf: `$f(x) = \frac{1}{b-a}$` (defined over a to b). We need to recall two things: the formula for `$V(X) \text{ and } E(X)$`

`$$\E[X^2] = \int_a^b x^2 * f(x) \,dx$$`
--
`$$\int_a^b x^2 * \frac{1}{b-a} \,dx$$`
--
`$$= \frac{1}{b-a} \int_a^b x^2 \,dx = \frac{1}{b-a} * \frac{1}{3}x^3 \Big|_a^b$$`
--

`$$= \frac{b^3 - a^3}{3(b-a)} = \frac{a^2 + ab + b^2}{3}$$`

---
# Uniform random variable: Variance Cont'd

So, we can just plug these values into our formula:

`$$\Var(X) = \E[X^2] - (\E[X])^2 = \frac{a^2 + ab + b^2}{3} - \left( \frac{a+b}{2} \right)^2 = \frac{(b-a)^2}{12}$$`

---

# Exponential random variable

`$$f_X(x) = \left\{    \begin{array}{ll}        \lambda e^{-\lambda x} & \quad \text{if } x \geq 0 \\        0 & \quad \text{otherwise}    \end{array}\right.$$`

* `$\lambda > 0$`

* NOTE: you will sometimes see this with `$\lambda$` replaced with `$\frac{1}{\beta}$`.

* SUPER FUN NOTE: You can find its expected value yourself using integration by parts!!

---

# Exponential random variable

`$$\E[X] = \frac{1}{\lambda}, \quad \Var(X) = \frac{1}{\lambda^2}$$`

---

# Cumulative distribution function

For a continuous random variable `$X$` define its **cumulative distribution function** (CDF) `$F_X(x)$` as,

`$$F_X(x) = \Pr(X \leq x) = \int_{-\infty} ^{x} f_X(t) \,dt$$`

---

# Uniform distribution

Suppose `$X \sim \text{Uniform}(0,1)$`

`$$\begin{aligned} F_X(x) & = \Pr(X\leq x)  \\& = 0 \text{, if }x< 0 \\ & = 1 \text{, if }x >1 \\ & = x \text{, if } x \in [0,1]\end{aligned}$$`

---

# Properties of CDFs

* `$F_X$` is monotonically nondecreasing
* `$F_X(x)$` tends to `$0$` as `$x \rightarrow -\infty$`, and to `$1$` as `$x \rightarrow \infty$`
* `$F_X(x)$` is a continuous function of `$x$`
* If `$X$` is continuous, the PDF and CDF can be obtained from each other by integration or differentiation

`$$F_X(x) = \int_{-\infty}^x f_X(t) \,dt, \quad f_X(x) = \frac{dF_X}{dx} (x)$$`

---

# Normal distribution

Suppose `$X$` is a random variable with `$X \in \Re$`

`$$f(x) = \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$$`

`$$X \sim \text{Normal}(\mu, \sigma^2)$$`

---

# Normal distribution

---

# Expected value/variance

* `$Z$` is a standard normal distribution if

`$$Z \sim \text{Normal}(0,1)$$`

* CDF of `$Z$`

`$$F_{Z}(x) = \frac{1}{\sqrt{2\pi} }\int_{-\infty}^{x} \exp(-z^2/2) \,dz$$`

---

# Expected value/variance

Suppose `$Z \sim \text{Normal}(0,1)$`

* `$Y = 2Z + 6$`
* `$Y \sim \text{Normal}(6, 4)$`

---

# Expected value/variance

If `$Z \sim N(0,1)$`, then `$Y = aZ + b$` is

`$$Y \sim \text{Normal} (b, a^2)$$`

Assume we know

$$
`\begin{aligned}
\E[Z]  & = 0 \\
\Var(Z) & = 1 
\end{aligned}`
$$

For `$Y \sim \text{Normal}(\mu, \sigma^2)$`

`$$\begin{aligned} \E[Y] & = \E[\sigma Z + \mu] \\& = \sigma \E[Z] + \mu \\& = \mu \\\Var(Y) & = \Var(\sigma Z + \mu) \\ & = E((\sigma Z + \mu)^2) - (E(\sigma Z + \mu))^2\\&=\sigma^2 \Var(Z) + \Var(\mu) \\ & = \sigma^2 + 0 \\ & = \sigma^2 \end{aligned}$$`

---

# Standard normal distribution

If `$X$` is a normal random variable with mean `$\mu$` and variance `$\sigma^2$`, and if `$a \neq 0, b$` are scalars, then the random variable

`$$Y = aX + b$$`

is also normal, with mean and variance

`$$\E[Y] = a\mu + b, \quad \Var(Y) = a^2 \sigma^2$$`

### Why rely on the standard normal distribution

* Normal distribution is commonly used in statistical analysis
* Ease of standardization
* Saves time on the calculus

---

# Support for President Biden

Suppose we are interested in modeling presidential approval

* `$Y$`: proportion of population who "approves job president is doing"
* Individual responses are independent and identically distributed
* Average of those individual binary responses
* `$N\rightarrow \infty$`
* By Central Limit Theorm, `$Y$` is normally distributed

$$
`\begin{aligned}
Y & \sim \text{Normal}(\mu, \sigma^2) \\
f_Y(y) & = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(y-\mu)^2}{2\sigma^2} \right)
\end{aligned}`
$$

---

# Central limit theorem

---

# Back to Biden

* Suppose `$\mu = 0.39$` and `$\sigma^2 = 0.0025$`
* `$\Pr(Y\geq 0.45)$`

`$$\begin{aligned} \Pr(Y \geq 0.45)  & = 1 - \Pr(Y \leq 0.45 ) \\& = 1 - \Pr(0.05 Z + 0.39 \leq  0.45) \\& = 1 - \Pr(Z \leq \frac{0.45-0.39 }{0.05} ) \\& = 1 - \frac{1}{\sqrt{2\pi} } \int_{-\infty}^{6/5} \exp(-z^2/2) \,dz \\& = 1 - F_{Z} (\frac{6}{5} ) \\& = 0.1150697\end{aligned}$$`

---
# Gamma distribution

We use the gamma distribution when we're looking at something that takes on positive values and has is skewed (long right tail).

Example: waiting time between events

---

# Gamma distribution

Suppose `$\alpha>0$`. Define `$\Gamma(\alpha)$` as

$$
`\begin{aligned}
\Gamma(\alpha) &= \int_{0}^{\infty} y^{\alpha- 1} e^{-y} \,dy \\
&= (\alpha- 1)! \, \forall \alpha \in \{1, 2, 3, \ldots\}
\end{aligned}`
$$

* `$\Gamma(\frac{1}{2}) = \sqrt{\pi}$`

---

# Gamma distribution

Suppose we have `$\Gamma(\alpha)$`

$$
`\begin{aligned}
\frac{\Gamma(\alpha)}{\Gamma(\alpha)} & = \frac{\int_{0}^{\infty} y^{\alpha-1} e^{-y} dy}{\Gamma(\alpha)} \\
1 & = \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} y^{\alpha-1} e^{-y} \,dy 
\end{aligned}`
$$

Set `$X = Y/\beta$`

$$
`\begin{aligned}
F(x) = \Pr(X \leq x) & = \Pr(Y/\beta \leq x ) \\
& = \Pr(Y \leq x \beta ) \\
& = F_{Y} (x \beta) \\
\frac{\partial F_{Y} (x \beta) }{\partial x} & = f_{Y} (x \beta) \beta
\end{aligned}`
$$

`$$f(x|\alpha, \beta) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} x^{\alpha - 1} e^{-x\beta}$$`

---

# Gamma distribution

Suppose `$X$` is a continuous random variable, with `$X \geq 0$`. `$X$` is a Gamma random variable if

`$$f(x|\alpha, \beta) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} x^{\alpha - 1} e^{-x\beta}$$`

if `$x\geq 0$` and `$0$` otherwise

`$$X \sim \text{Gamma}(\alpha, \beta)$$`

---

# Gamma distribution

Suppose `$X \sim \text{Gamma}(\alpha, \beta)$`

$$
`\begin{aligned}
\E[X] & = \frac{\alpha}{\beta} \\
\Var(X) & = \frac{\alpha}{\beta^2}
\end{aligned}`
$$

Suppose `$\alpha = 1$` and `$\beta = \lambda$`. If

$$
`\begin{aligned}
X & \sim \text{Gamma}(1, \lambda) \\
f(x|1, \lambda ) & = \lambda e^{- x \lambda}
\end{aligned}`
$$

`$$X \sim \text{Exponential}(\lambda)$$`

---

# Properties of Gamma distributions

Suppose we have a sequence of independent random variables, with

`$$X_{i} \sim \text{Gamma}(\alpha_{i}, \beta)$$`

Then

`$$Y = \sum_{i=1}^{N} X_{i}$$`

`$$Y \sim \text{Gamma}(\sum_{i=1}^{N} \alpha_{i} , \beta)$$`

---

# Gamma distribution

---

# Gamma distribution

---

# Importance of the Gamma distribution

* Exponential and `$\chi^2$` distributions are special cases of the gamma distribution
* Commonly used in Bayesian statistics (conjugate prior)

---

# `$\chi^2$` distribution

Suppose `$Z \sim \text{Normal}(0,1)$`. Consider `$X = Z^2$`

$$
`\begin{aligned}
F_{X}(x)   & =  \Pr(X \leq x) \\
& = \Pr(Z^2 \leq x ) \\
& = \Pr(-\sqrt{x} \leq Z \leq \sqrt{x}) \\
& = \frac{1}{\sqrt{2\pi}} \int_{-\sqrt{x}}^{\sqrt{x} } e^{-\frac{z^2}{2}} \,dz\\
& = F_{Z} (\sqrt{x}) - F_{Z} (-\sqrt{x})
\end{aligned}`
$$

---

# `$\chi^2$` distribution

$$
`\begin{aligned}
\frac{\partial F_{X}(x) }{\partial x }  & = f_{Z} (\sqrt{x}) \frac{1}{2\sqrt{x}} + f_{Z}(-\sqrt{x}) \frac{1}{2\sqrt{x}} \\
& = \frac{1}{\sqrt{x}}\frac{1}{2 \sqrt{2\pi}} ( 2e^{-\frac{x}{2}}) \\
& = \frac{1}{\sqrt{x}}\frac{1}{\sqrt{2\pi}} ( e^{-\frac{x}{2}}) \\
& = \frac{(\frac{1}{2})^{1/2}}{\Gamma(\frac{1}{2})}\left(x^{1/2 - 1} e^{-\frac{x}{2}}\right)
\end{aligned}`
$$

---

# `$\chi^2$` distribution

`$$f_X(x) = \frac{(\frac{1}{2})^{1/2}}{\Gamma(\frac{1}{2})}\left(x^{1/2 - 1} e^{-\frac{x}{2}}\right)$$`

`$$X \sim \text{Gamma}(1/2, 1/2)$$`

If `$X = \sum_{i=1}^{N} Z^2$`, `$X \sim \text{Gamma}(n/2, 1/2)$`

---

# `$\chi^2$` distribution

Suppose `$X$` is a continuous random variable with `$X\geq 0$`, with PDF

`$$f(x) = \frac{1}{2^{n/2} \Gamma(n/2) } x^{n/2 - 1} e^{-x/2}$$`

`$X$` is a `$\chi^2$` distribution with `$n$` degrees of freedom

`$$X \sim \chi^{2}(n)$$`

---

# `$\chi^2$` distribution

---

# `$\chi^2$` properties

Suppose `$X \sim \chi^2(n)$`

$$
`\begin{aligned}
\E[X] & = \E\left[\sum_{i=1}^{N} Z_{i}^2\right] \\
 & = \sum_{i=1}^{N} \E[Z_{i}^{2} ] \\
\Var(Z_{i} ) & = \E[Z_{i}^2] - \E[Z_{i}]^2\\
1 & = \E[Z_{i}^2]- 0 \\
\E[X] & = n 
\end{aligned}`
$$

---

# `$\chi^2$` properties

$$
`\begin{aligned}
\Var(X) & = \sum_{i=1}^{N} \Var(Z_{i}^2) \\
& = \sum_{i=1}^{N} \left(\E[Z_{i}^{4} ] - \E[Z_{i}]^{2}  \right) \\
& = \sum_{i=1}^{N} \left(3 - 1\right ) = 2n 
\end{aligned}`
$$

---

# Student's `$t$` distribution

Suppose `$Z \sim \text{Normal}(0, 1)$` and `$U \sim \chi^2(n)$`.  Define the random variable `$Y$` as,

`$$Y = \frac{Z}{\sqrt{\frac{U}{n}}}$$`

If `$Z$` and `$U$` are independent then `$Y \sim t(n)$`, with PDF

`$$t(n) = \frac{\Gamma(\frac{n+1}{2})}{\sqrt{\pi n } \Gamma(\frac{n}{2})}\left(1 + \frac{x^2}{n}\right)^{-\frac{n+1}{2}}$$`

---

# Differences from the Normal distribution

* Normal distribution always has the same shape
* The shape of the student's `$t$`-distribution changes depending on the sample size
* Low `$n$`
* As `$n \uparrow$`, the confidence bounds shrink
* As `$n \rightarrow \infty$`, student's `$t$`-distribution takes on the same shape as the normal distribution

---

# Differences from the Normal distribution

---

# Recap:

* Continuous vs discrete
* Integrals!
* Distributions and their formal properties
* Different types of distributions:
  * Uniform
  * Exponential
  * Gamma
  * Normal
  * `$\chi^2$` 
  * Student's t