class: center, middle, inverse, title-slide .title[ # Multivariate distributions ] .author[ ###
MACS 33000
University of Chicago ] --- `$$\newcommand{\E}{\mathrm{E}} \newcommand{\Var}{\mathrm{Var}} \newcommand{\Cov}{\mathrm{Cov}} \newcommand{\Cor}{\mathrm{Cor}}$$` # Misc * Calcs spoken for * Link for practice questions open on github / pset * Meetings with preceptors! --- # Acronym recap: * Discrete: pmf and CMF * Continuous: pdf and CDF * The lowercase reflects that we're associating it with particular values (e.g. f) * Uppercase is integrating (e.g. F) * Rules of probability: * `\(0 \leq p(x) \leq 1\)` * Total probability must sum to 1! * Expected values: tells us what we can anticipate getting from our function: IN THE LONG RUN * multiply your value times its probability * Moments: tell us about the function and multiply `\(x^n\)` by the probability for the `\(n^{th}\)` moment --- # Learning objectives * Define a joint probability density function * Condition pdfs on other random variables * Identify independence between two random variables * Define covariance and correlation * Examine sums of random variables * Define the multivariate normal distribution --- # Multivariate distribution `\(X\)` and `\(Y\)` are **jointly continuous** if, for all `\(x\in\Re\)` and `\(y\in \Re\)`, there exists a function `\(f(x,y)\)` such that `\(C \subset \Re^{2}\)`, `$$\Pr\{(X, Y) \in C \} = \iint_{(x,y)\in C} f(x,y)\, dx\, dy$$` -- What is `\(C \subset R^{2}\)`? -- * `\(R^{2} = R \underbrace{\times}_{\text{Cartesian Product}} R\)` * `\(C\)` is a subset of the 2-d plane --- ## `\(C = \{x, y: x \in [0,1] , y\in [0,1] \}\)` <img src="12-multivariate-pdf_files/figure-html/a-2d-rect-1.png" width="864" style="display: block; margin: auto;" /> --- # `\(C = \{x, y: x^2 + y^2 \leq 1 \}\)` <img src="12-multivariate-pdf_files/figure-html/a-2d-circle-1.png" width="864" style="display: block; margin: auto;" /> --- # `\(C = \{ x, y: x> y, x,y\in(0,2)\}\)` <img src="12-multivariate-pdf_files/figure-html/a-2d-triangle-1.png" width="864" style="display: block; margin: auto;" /> --- # Multivariate distribution: role of integrals `$$\begin{aligned} C &= \{ x, y: x \in A, y \in B \} \\ \Pr\{(X,Y) \in C \} &= \int_{B} \int_{A} f(x,y) \, dx\, dy \end{aligned}$$` --- # Examples of joint pdfs Consider a function `\(f:\Re \times \Re \rightarrow \Re\)`: * Input: an `\(x\)` value and a `\(y\)` value. * Output: a number from the real line * `\(f(x,y) = a\)` --- # Examples of joint pdfs: normal `\(f(x,y) = \frac{1}{2\pi}exp[\frac{(x^2+y^2)}{-2}]\)`
--- # Examples of joint pdfs: normal <img src="12-multivariate-pdf_files/figure-html/multivar-contour-1.png" width="864" style="display: block; margin: auto;" /> --- # Examples of joint pdfs: uniform `\(f(x,y) = 1\)` if `\(x \in [0,1], y \in [0,1]\)`, `\(f(x,y) = 0\)`
--- # Examples of joint pdfs: uniform `\(f(x,y) = 1\)` if `\(x \in [0,1], y \in [0,1]\)`, `\(f(x,y) = 0\)` <img src="12-multivariate-pdf_files/figure-html/joint-pdf-uniform-2d-1.png" width="864" style="display: block; margin: auto;" /> --- # Examples of joint pdfs: exponential `\(f(x,y) = \frac{2 \exp(-2x)}{\sqrt{2\pi}}\exp\left(-\frac{(y)^2}{2}\right)\)` if `\(x \in [0,\infty), y \in \Re\)`, `\(f(x,y) = 0\)` otherwise
--- # Examples of joint pdfs `\(f(x,y) = \frac{2 \exp(-2x)}{\sqrt{2\pi}}\exp\left(-\frac{(y)^2}{2}\right)\)` if `\(x \in [0,\infty), y \in \Re\)`, `\(f(x,y) = 0\)` otherwise <img src="12-multivariate-pdf_files/figure-html/joint-pdf-exp-2d-1.png" width="864" style="display: block; margin: auto;" /> --- # Examples of joint pdfs: plane `\(f(x,y) = x + y\)`, if `\(x \in [0,1], y \in [0,1]\)`
--- # Examples of joint pdfs: plane `\(f(x,y) = x + y\)`, if `\(x \in [0,1], y \in [0,1]\)` <img src="12-multivariate-pdf_files/figure-html/joint-pdf-plus-2d-1.png" width="864" style="display: block; margin: auto;" /> --- class: middle # Evaluating probability -- ### IT WORKS THE SAME-*ish* --- # Multivariate CDF For jointly continuous random variables `\(X\)` and `\(Y\)` define, `\(F(b,a)\)` as `$$F(b,a) = \Pr\{ X \leq b , Y \leq a\} = \int_{-\infty}^{a} \int_{-\infty}^{b} f(x,y) \, dx\, dy$$` --- # Multivariate CDF <img src="12-multivariate-pdf_files/figure-html/multi-cdf-1.png" width="864" style="display: block; margin: auto;" /> --- # Multivariate CDF <img src="12-multivariate-pdf_files/figure-html/multi-cdf-1-1.png" width="864" style="display: block; margin: auto;" /> --- # Multivariate CDF <img src="12-multivariate-pdf_files/figure-html/multi-cdf-2-1.png" width="864" style="display: block; margin: auto;" /> --- # Multivariate CDF <img src="12-multivariate-pdf_files/figure-html/multi-cdf-3-1.png" width="864" style="display: block; margin: auto;" /> --- # Multivariate CDF <img src="12-multivariate-pdf_files/figure-html/multi-cdf-4-1.png" width="864" style="display: block; margin: auto;" /> --- class: middle # MARGINAL PROBABILITIES! -- Remember how we talked about `\(f_X\)`? .... -- Now we can see why/how that would be worthwhile. (how one variable changes relative to another vs on its own) --- # Marginalization Define `\(f_{X}(x)\)` as the pdf for `\(X\)`, `$$f_{X}(x) = \int_{-\infty}^{\infty} f(x,y) \, dy$$` -- Similarly, define `\(f_{Y}(y)\)` as the pdf for `\(Y\)`, `$$f_{Y}(y) = \int_{-\infty}^{\infty} f(x,y)\, dx$$` -- `\(X\)` and `\(Y\)` are continuous random variables with joint pdf `\(f(x,y)\)`. Define the conditional probability function `\(f(x|y)\)` as `$$f(x|y) = \frac{f(x, y) }{f_{Y}(y) }$$` --- # Marginalization in action `\(f(x,y) = -x^2 - y^2\)`, if `\(x \in [-2,2], y \in [-2,2]\)`
--- # Marginalization `$$f(x, y) = -x^2 - y^2$$` .pull-left[ <img src="12-multivariate-pdf_files/figure-html/conditional-pdf-1.png" width="432" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="12-multivariate-pdf_files/figure-html/conditional-pdf-examples-1.gif" style="display: block; margin: auto;" /> ] --- # Why does marginalization work? ### Discrete case Consider jointly distributed discrete random variables, `\(X\)` and `\(Y\)` with joint PMF `$$\Pr(X =x, Y = y) = p(x, y)$$` -- Suppose that the distribution allocates its mass at `\(x_{1}, x_{2}, \ldots, x_{M}\)` and `\(y_{1}, y_{2}, \ldots, y_{N}\)`. Define the conditional mass function `\(\Pr(X= x| Y= y)\)` as, `$$\begin{aligned} \Pr(X=x|Y=y) &\equiv p(x|y) \\ & = \frac{p(x,y)}{p(y)} \end{aligned}$$` -- `$$p(x,y) = p(x|y)p(y)$$` --- # Marginalization (discrete): So, we have: `$$p(x,y) = p(x|y)p(y)$$` -- Marginalizing **over** `\(y\)` to get `\(p(x)\)` is then, `$$p(x_{j}) = \sum_{i=1}^{N} p(x_{j} |y_{i})p(y_{i} )$$` --- # Why does marginalization work? | | `\(Y = 0\)` | `\(Y = 1\)` | | |-------|-------------|-------------|------------| | `\(X = 0\)` | `\(p(0,0)\)` | `\(p(0, 1)\)` | `\(p_{X}(0)\)` | | `\(X = 1\)` | `\(p(1,0)\)` | `\(p(1,1)\)` | `\(p_{X}(1)\)` | | | `\(p_{Y} (0)\)` | `\(p_{Y} (1)\)` | | --- # Why does marginalization work? | | `\(Y = 0\)` | `\(Y = 1\)` | | |-------|-------------|-------------|------------| | `\(X = 0\)` | `\(0.01\)` | `\(0.05\)` | `\(p_{X}(0)\)`? | | `\(X = 1\)` | `\(0.25\)` | `\(0.69\)` | `\(p_{X}(1)\)`? | | | `\(0.26\)` | `\(0.74\)` | | -- `$$p_{X}(0) = \Pr(0|y = 0) *\Pr(y= 0) + \Pr(0|y=1)*\Pr(y=1)$$` -- `$$= \frac{0.01}{0.26} * 0.26 + \frac{0.05}{0.74} * 0.74 \\ = 0.06$$` -- `$$\begin{aligned} p_{X}(1) & = \Pr(1|y = 0) \Pr(y= 0) + \Pr(1|y=1) \Pr(y=1) \\ & = \frac{0.25}{0.26} * 0.26 + \frac{0.69}{0.74} * 0.74 \\ & = 0.94 \end{aligned}$$` --- # Why does marginalization work? | | `\(Y = 0\)` | `\(Y = 1\)` | | |-------|-------------|-------------|------------| | `\(X = 0\)` | `\(0.01\)` | `\(0.05\)` | `\(0.06\)`| | `\(X = 1\)` | `\(0.25\)` | `\(0.69\)` | `\(0.94\)` | | | `\(0.26\)` | `\(0.74\)` | | --- # Why does marginalization work? ### Continuous case For **jointly distributed continuous** random variables `\(X\)` and `\(Y\)` define, `$$f_{X|Y}(x|y) = \frac{f(x,y)}{f_{Y}(y) }$$` -- `$$f_{X}(x) = \int_{-\infty }^{\infty} f_{X|Y}(x|y)f_{Y}(y) \, dy$$` -- * Think of `\(f_{X|Y}(x|y)\)` as the pdf for `\(X\)` at a value of `\(Y\)` * Average over those pdfs to get the final pdf for `\(X\)` * Weighted average --- # A simple(ish) example Suppose `\(X\)` and `\(Y\)` are jointly continuous and that $$ `\begin{aligned} f(x,y) & = x + y \text{ , if } x \in [0,1], y \in [0,1] \\ & = 0 \text{ , otherwise } \end{aligned}` $$ <img src="12-multivariate-pdf_files/figure-html/joint-pdf-plus-dup-1.png" width="864" style="display: block; margin: auto;" /> --- # A simple(-ish) example We want `\(f_{X}(x)\)`. We can find this one of two ways: by either taking the derivative of the original function with respect to y OR by using independence. Option 1: `\(\int_0^1f(x,y) dy = \int_0^1 (x+ y) dy = xy + \frac{y^2}{2} |_0^1 = x + 1/2\)` -- Option 2: we know that `\(f(x|y) = \frac{f(x,y)}{f_Y(y)}\)` -- Suppose you are given: `\(f_{Y}(y) = 1/2 + y\)` -- Then `$$f(x|y) = \frac{x + y}{1/2 + y}$$` `$$f(x) = \int_{0}^{1} f(x|y)f(y)\, dy = 1/2 + x$$` --- # A simple(-ish) example Here are all the possible values for f(x), based on what y might be: <img src="12-multivariate-pdf_files/figure-html/joint-pdf-plus-margin-1.png" width="864" style="display: block; margin: auto;" /> --- # A simple(-ish) example Here's our 'weighted average' for f(x) across all y: <img src="12-multivariate-pdf_files/figure-html/joint-pdf-plus-cond-1.png" width="864" style="display: block; margin: auto;" /> --- # More complex example Suppose `\(X\)` and `\(Y\)` are jointly distributed with pdf `$$f(x,y) = 2 \exp(-x) \exp(-2y), \forall \, x>0, y>0)$$` -- Note: equivalent to : `$$f(x,y) = 2 e^{(-x)} e^{(-2y)}, \forall \, x>0, y>0$$` --- # Verify this is a pdf (e version) `$$\int_{0}^{\infty} \int_{0}^{\infty} f(x, y) = 2\int_{0}^{\infty} \int_{0}^{\infty} e^{(-x)} e^{(-2y)} \, dx\, dy$$` -- `$$2 \int_{0}^{\infty}e^{(-2y)} \, dy \int_{0}^{\infty} e^{(-x)} \, dx$$` -- `$$= 2 (-\frac{1}{2} e^{(-2y)}|_{0}^{\infty} ) ( - e^{(-x)}|_{0}^{\infty} )$$` -- `$$= 2\left[ (-\frac{1}{2}(\lim_{y\rightarrow\infty} e^{(-2y)} - 1))(- (\lim_{x\rightarrow \infty} e^{(-x)} - 1) ) \right]$$` -- `$$= 2 \left[ -\frac{1}{2} (-1) * -1 (-1) \right] = 1$$` --- # Verify this is a pdf (exp version) `$$\int_{0}^{\infty} \int_{0}^{\infty} f(x, y) = 2\int_{0}^{\infty} \int_{0}^{\infty} \exp(-x) \exp(-2y) \, dx\, dy$$` -- `$$2\int_{0}^{\infty}\exp(-2y) \, dy \int_{0}^{\infty} \exp(-x) \, dx$$` -- `$$= 2 (-\frac{1}{2} \exp(-2y)|_{0}^{\infty} ) ( - \exp(-x)|_{0}^{\infty} )$$` -- `$$= 2\left[ (-\frac{1}{2}(\lim_{y\rightarrow\infty} \exp(-2y) - 1))(- (\lim_{x\rightarrow \infty} \exp(-x) - 1) ) \right]$$` -- `$$= 2 \left[ -\frac{1}{2} (-1) * -1 (-1) \right]= 1$$` --- # Calculate CDF `$$F(x,y) \equiv \Pr\{X \leq b, Y \leq a\} = 2 \int_{0}^{a} \int_{0}^{b} \exp(-x) \exp(-2y) \, dx\, dy$$` -- `$$= 2 (\int_{0}^{a} \exp(-2y) \, dy) (\int_{0}^{b} \exp(-x) \, dx)$$` -- $$ = 2 \left[-\frac{1}{2} (\exp(-2a) -1 )\right]\left[ - (\exp(-b) - 1) \right]$$ -- $$ = \left[1 - \exp(-2a) \right] \left[ 1- \exp(-b) \right]$$ -- NOW WHAT? -- Plug in values for a and b, respectively. --- # Calculate `\(f_{X}(x)\)` and `\(f_{Y}(y)\)` `$$\begin{aligned}f_{X}(x) & = \int_{0}^{\infty} 2\exp(-x) \exp(-2y) \, dy \\& = 2 \exp(-x) \int_{0}^{\infty} \exp(-2y) \, dy \\& = 2 \exp(-x) \left[ -\frac{1}{2}(0 - 1) \right] \\& = \exp(-x)\end{aligned}$$` -- `$$\begin{aligned}f_{Y}(y) & = \int_{0}^{\infty} 2 \exp(-x) \exp(-2y) \, dx \\& = 2 \exp(-2y) \int_{0}^{\infty} \exp(-x) \, dx \\& = 2 \exp(-2y) \left[-(0 -1) \right] \\& = 2 \exp(-2y)\end{aligned}$$` --- # Conditional distribution Two random variables `\(X\)` and `\(Y\)` are independent if for any two sets of real numbers `\(A\)` and `\(B\)`, `$$\Pr\{ X \in A , Y \in B \} = \Pr\{X \in A\} \Pr\{Y \in B\}$$` -- `$$f(x,y) = f_{X}(x) f_{Y}(y)$$` -- If `\(X\)` and `\(Y\)` are not independent, we will say they are **dependent**. -- If `\(X\)` and `\(Y\)` are independent, then `$$\begin{aligned}f_{X|Y} (x|y) & = \frac{f(x,y)}{f_{Y}(y)} \\& = \frac{f_{X}(x)f_{Y}(y)}{f_{Y}(y) } \\& = f_{X}(x) \end{aligned}$$` --- # A (simple) example of dependence Suppose `\(X\)` and `\(Y\)` are jointly continuous and that `$$\begin{eqnarray}f(x,y) & = & x + y \text{ , if } x \in [0,1], y \in [0,1] \\& = & 0 \text{ , otherwise } \end{eqnarray}$$` -- Are they dependent? --- # A (simple) example of dependence `$$f_{X}(x) = \int_{0}^{1} \left(x + y \right) \, dy$$` -- `$$xy + \frac{y^2}{2} |^{1}_{0} = x + \frac{1}{2}$$` -- `$$f_{Y}(y) = \int_{0}^{1} \left(x + y \right) \, dx = \frac{x^2}{2} + xy |^{1}_{0} = \frac{1}{2} + y$$` -- `$$\begin{eqnarray}f_{X}(x) f_{Y}(y) & = & (\frac{1}{2} + x) (\frac{1}{2} + y) \\& = & \frac{1}{4} + \frac{x + y}{2} + xy \neq (x+y)\end{eqnarray}$$` --- # Dependence: deep thoughts `$$\begin{eqnarray}f_{X}(x) f_{Y}(y) & = & (\frac{1}{2} + x) (\frac{1}{2} + y) \\& = & \frac{1}{4} + \frac{x + y}{2} + xy \neq (x+y)\end{eqnarray}$$` We can think about this as getting additional INFORMATION from y when we know x and vice versa. --- class: middle # But what to expect? --- # Expectation For jointly continuous random variables `\(X\)` and `\(Y\)`, -- `$$\E[X] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x f(x,y) \, dx\, dy$$` -- `$$E[Y] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} y f(x,y) \, dx\, dy$$` -- `$$E[XY] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x y f(x,y) \, dx\, dy$$` -- Suppose `\(g:\Re^{2} \rightarrow \Re\)`. Then `$$\E[g(X, Y)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x, y) f(x,y) \, dx\, dy$$` --- # Expectation ex with g(x) `$$\E[g(X, Y)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x, y) f(x,y) \, dx\, dy$$` Suppose we have the following functions: `\(f(x,y) = xy\)` where `\(0 \leq x \leq 1\)` and `\(0 \leq y \leq 2\)`. 1. Is this a valid pdf? why or why not? 2. Suppose you earn revenue equal to `\(x^3+2\)`. What is your expected utility? --- # Expectation ex with g(x): pdf Suppose we have the following functions: `\(f(x,y) = xy\)` where `\(0 \leq x \leq 1\)` and `\(0 \leq y \leq 2\)`. `$$\int_0^2\int_0^1 xy\, \,dx\,dy$$` -- `$$\int_0^2\frac{x^2y}{2}|_0^1 \, dy= \int_0^2 \frac{y}{2} \, dy$$` `$$=\frac{y^2}{4}|_0^2 = 1$$` -- So, yes it appears to satisfy our requirements! --- # Expectation ex with g(x): expectation > Suppose you earn revenue equal to `\(x^3+2\)`. What is your expected utility? Well, our function is: `\(\int_0^2\int_0^1 xy\, \,dx\,dy\)`, so our expected utility will be: `$$\int_0^2\int_0^1 (x^3+2)*xy\, \,dx\,dy$$` We know we can do this because: `$$\E[g(X, Y)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x, y) f(x,y) \, dx\, dy$$` --- # Expected value: Math party `$$\int_0^2\int_0^1 (x^3+2)*xy\, \,dx\,dy = \int_0^2\int_0^1 (x^4+2x)y\, \,dx\,dy$$` -- `$$=\int_0^2 y \,dy* \int_0^1 (x^4+2x) \,dx= \frac{y^2}{2}|_0^2*(\frac{x^5}{5}+x^2 )|_0^1$$` -- `$$=2*(1^5/5+1)=2.4$$` -- NOW WHAT? -- This is our expected value! --- class: center, middle, inverse # Examples! --- # Example land: Let's look at what we have here: `\(\int_{1}^{2} \int_{0}^{0.5} (x^2 - \frac{8y}{3}+\frac{1}{3}) \text{ } dy \text{ }dx\)` <img src="12-multivariate-pdf_files/figure-html/multi-cdf-int-1.png" width="864" style="display: block; margin: auto;" /> --- # Example land: Let's look at what we have here: `\(\int_{1}^{2} \int_{0}^{0.5} (x^2 - \frac{8y}{3}+\frac{1}{3}) \text{ } dy \text{ }dx\)` So we will effectively add up all the chunks of our function in one direction, then we can add up along the other. I have this set up for doing y first. I leave doing it in the x direction as an exercise. -- `$$\begin{align}F(x,y) &= \int_{1}^{2} \int_{0}^{0.5} (x^2 - \frac{8y}{3}+\frac{1}{3}) \text{ } dy \text{ }dx \\ &=\int_{1}^{2}(x^2y - \frac{4y^2}{3}+\frac{y}{3}) \text{ } |_0^{0.5} \text{ }dx \\ &= \int_{1}^{2}(0.5x^2 - \frac{1}{3}+\frac{1}{6}) \text{ } \text{ }dx \\ &= \int_{1}^{2}(0.5x^2 - \frac{1}{6}) \text{ } \text{ }dx \\ &= (\frac{x^3}{6} - \frac{x}{6}) \text{ }|_1^2 = (\frac{8}{6}- \frac{2}{6})-(\frac{1}{6}-\frac{1}{6}) \text{ } \\ &= 1 \end{align}$$` -- PHEW! It's a pdf (also need to verify we don't have negative values for f(x)) --- # Example land: Let's look at a slice of it: `\(\int_{1}^{1.5} \int_{0}^{0.5} (x^2 - \frac{8y}{3}+\frac{1}{3}) \text{ } dy \text{ }dx\)` `$$\begin{align}F(x,y) &= \int_{1}^{1.5} \int_{0}^{0.5} (x^2 - \frac{8y}{3}+\frac{1}{3}) \text{ } dy \text{ }dx \\ &=\int_{1}^{1.5}(x^2y - \frac{4y^2}{3}+\frac{y}{3}) \text{ } |_0^0.5 \text{ }dx \\ &= \int_{1}^{1.5}(0.5x^2 - \frac{1}{3}+\frac{1}{6}) \text{ } \text{ }dx \\ &= \int_{1}^{1.5}(0.5x^2 - \frac{1}{6}) \text{ } \text{ }dx \\ &= (\frac{x^3}{6} - \frac{x}{6}) \text{ }|_1^{1.5} \\ &= (\frac{1.5^3}{6}-\frac{3}{12})-(\frac{1}{6}- \frac{1}{6}) \text{ } \\ &= 0.3125 \end{align}$$` -- So....what does this tell us?? --- class: middle # And now, something quite a bit different* *(but also not!)* --- # Covariance and correlation We are interested in how our variables vary with one another. For jointly continuous random variables `\(X\)` and `\(Y\)` we define the covariance of `\(X\)` and `\(Y\)` as, `$$Cov(X,Y) = E[(X-E[X])(Y- E[Y])]$$` -- `$$=E\left[ XY - E[X] Y - E[Y] X + E[X]E[Y] \right]$$` -- `$$=E[XY] - 2E[X]E[Y] + E[E[X]E[Y]]$$` -- `$$=E[XY] - E[X]E[Y]$$` -- Define the correlation of `\(X\)` and `\(Y\)` as, `$$\Cor(X,Y) = \frac{\Cov(X,Y) }{\sqrt{\Var(X) \Var(Y) } }$$` --- # Variance is covariance with itself `$$\begin{eqnarray}\Cov(X,X) & = & E[X X] - E[X]E[X] \\& = & E[X^2] - E[X]^2\end{eqnarray}$$` --- # Correlation measures linear relationship Like covariance, bigger is more, but now we're standardizing relative to variance. Suppose `\(X = Y\)`: `$$\begin{eqnarray}\Cor(X,Y) & = & \frac{\Cov(X,Y)}{\sqrt{\Var(X)\Var(Y)} } \\& = & \frac{\Var(X)}{\Var(X)} \\& = & 1\end{eqnarray}$$` -- Suppose `\(X = -Y\)`: `$$\begin{eqnarray}\Cor(X,Y) & = & \frac{\Cov(X,Y)}{\sqrt{\Var(X)\Var(Y)} } \\& = & \frac{- \Var(X)}{\Var(X)} \\& = & -1 \end{eqnarray}$$` --- # Correlation is between -1 and 1 `$$|\Cor(X,Y)| \leq 1$$` --- # Sums of random variables Suppose we have a sequence of random variables `\(X_{i}\)` , `\(i = 1, 2, \ldots, N\)` Suppose that they have joint pdf `$$f(\boldsymbol{x}) = f(x_{1}, x_{2}, \ldots, x_{n})$$` -- * `\(\E[\sum_{i=1}^{N}X_{i} ] = \sum_{i=1}^{N} \E[X_{i}]\)` -- * `\(\Var(\sum_{i=1}^{N} X_{i} ) = \sum_{i=1}^{N} \Var(X_{i} ) + 2 \sum_{i<j} \Cov(X_{i}, X_{j})\)` --- # Expected value of sums of RVs Suppose we have a sequence of random variables `\(X_{i}\)` , `\(i = 1, 2, \ldots, N\)` Suppose that they have joint pdf, `$$f(\boldsymbol{x}) = f(x_{1}, x_{2}, \ldots, x_{n})$$` Then `$$\E[\sum_{i=1}^{N} X_{i} ] = \sum_{i=1}^{N} \E[X_{i} ]$$` -- $$ `\begin{eqnarray} \E[\sum_{i=1}^{N} X_{i} ] & = & \E[X_{1} + X_{2} + \ldots + X_{N}] \\ & = & \int_{-\infty}^{\infty} \cdot \cdot \cdot \iint_{-\infty}^{\infty} (x_{1} + x_{2} + \ldots + x_{N}) f(x_{1}, x_{2}, \ldots, x_{N}) \, dx_{1}\, dx_{2}\ldots \, dx_{N} \\ & = & \int_{-\infty}^{\infty}x_{1} f_{X_{1}}(x_{1}) \, dx_{1} + \int_{-\infty}^{\infty}x_{2} f_{X_{2}}(x_{2}) \, dx_{2} + \ldots + \int_{-\infty}^{\infty}x_{N} f_{X_{N}}(x_{N}) \, dx_{N} \\ & = & \E[X_{1} ] + \E[X_{2}] + \ldots + \E[X_{N}] \end{eqnarray}` $$ --- # Variance of sums of RVs Suppose `\(X_{i}\)` is a sequence of random variables. Then $$ `\begin{eqnarray} \Var(\sum_{i=1}^{N} X_{i} ) & = & \sum_{i=1}^{N} \Var(X_{i} ) + 2 \sum_{i<j} \Cov(X_{i}, X_{j} ) \end{eqnarray}` $$ -- Consider two random variables, `\(X_{1}\)` and `\(X_{2}\)`. Then, $$ `\begin{eqnarray} \Var(X_{1} + X_{2} ) & = & \E[(X_{1} + X_{2})^2] - \left(\E[X_{1}] + \E[X_{2}] \right)^2 \\ & = & \E[X_{1}^2] + 2 \E[X_{1}X_{2}] + \E[X_{2}^2] \\ && - (\E[X_{1}])^2 - 2 \E[X_{1}] \E[X_{2}] - 2 \E[X_{2}]^2 \\ & = & \underbrace{\E[X_{1}^2] - (\E[X_{1}])^2}_{\Var(X_{1}) } + \underbrace{\E[X_{2}^2] - \E[X_{2}]^{2}}_{\Var(X_{2})} \\ && + 2 \underbrace{(\E[X_{1} X_{2} ] - \E[X_{1}] \E[X_{2} ] )}_{\Cov(X_{1}, X_{2} ) } \\ & = & \Var(X_{1} ) + \Var(X_{2} ) + 2 \Cov(X_{1}, X_{2}) \end{eqnarray}` $$ --- class: middle # Oh good, matrices! -- ### AKA: What happens when you bring multiple variables to the party? -- These are for information -- they haven't featured into my work, but for some people they have been relevant --- # Multivariate normal distribution Suppose `\(\boldsymbol{X} = (X_{1}, X_{2}, \ldots, X_{N})\)` is a vector of random variables. If `\(\boldsymbol{X}\)` has pdf `$$f(\boldsymbol{x}) = (2 \pi)^{-N/2} \text{det}\left(\boldsymbol{\Sigma}\right)^{-1/2} \exp\left(-\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu})^{'}\boldsymbol{\Sigma}^{-1} (\boldsymbol{x} - \boldsymbol{\mu} ) \right)$$` where `\(det(\mathbf{\Sigma})\)` is the determinant of the covariance matrix, known as generalized variance -- `\(\boldsymbol{X}\)` is a **Multivariate Normal** Distribution, `$$\boldsymbol{X} \sim \text{Multivariate Normal} (\boldsymbol{\mu}, \boldsymbol{\Sigma})$$` -- * Regularly used for likelihood, Bayesian, and other parametric inferences --- # Bivariate example Consider the (bivariate) **special case** where `\(\boldsymbol{\mu} = (0, 0)\)` and `$$\boldsymbol{\Sigma} = \begin{pmatrix} 1 & 0 \\0 & 1 \\\end{pmatrix}$$` -- `$$\begin{eqnarray}f(x_{1}, x_{2} ) & = & (2\pi)^{-2/2} 1^{-1/2} \exp\left(-\frac{1}{2}\left( (\boldsymbol{x} - \boldsymbol{0} ) ^{'} \begin{pmatrix} 1 & 0 \\0 & 1 \\\end{pmatrix}(\boldsymbol{x} - \boldsymbol{0} ) \right) \right) \\& = & \frac{1}{2\pi} \exp\left(-\frac{1}{2} (x_{1}^{2} + x_{2} ^ 2 ) \right) \\& = & \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{x_{1}^{2}}{2} \right) \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{x_{2}^{2}}{2} \right)\end{eqnarray}$$` -- `\(\leadsto\)` product of univariate standard normally distributed random variables --- # Standard multivariate normal Suppose `\(\boldsymbol{Z} = (Z_{1}, Z_{2}, \ldots, Z_{N})\)` is `$$\boldsymbol{Z} \sim \text{Multivariate Normal}(\boldsymbol{0}, \boldsymbol{I}_{N} )$$` Then we will call `\(\boldsymbol{Z}\)` the standard multivariate normal --- # Properties of the multivariate normal Suppose `\(\boldsymbol{X} = (X_{1}, X_{2}, \ldots, X_{N} )\)` $$ `\begin{eqnarray} \E[\boldsymbol{X} ] & = & \boldsymbol{\mu} \\ \Cov(\boldsymbol{X} ) & = & \boldsymbol{\Sigma} \end{eqnarray}` $$ So that, $$ `\begin{eqnarray} \boldsymbol{\Sigma} & = & \begin{pmatrix} \Var(X_{1}) & \Cov(X_{1}, X_{2}) & \ldots & \Cov(X_{1}, X_{N}) \\ \Cov(X_{2}, X_{1}) & \Var(X_{2}) & \ldots & \Cov(X_{2}, X_{N} ) \\ \vdots & \vdots & \ddots & \vdots \\ \Cov(X_{N}, X_{1} ) & \Cov(X_{N}, X_{2} ) & \ldots & \Var(X_{N} ) \\ \end{pmatrix} \end{eqnarray}` $$ --- # Independence and multivariate normal Suppose `\(X\)` and `\(Y\)` are independent. Then `$$\Cov(X, Y) = 0$$` -- `$$\Cov(X, Y) = \E[XY] - \E[X]\E[Y]$$` -- $$ `\begin{eqnarray} \E[XY] & = & \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x y f(x,y)\, dx\, dy \\ & =& \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x y f_{X}(x) f_{Y}(y)\, dx\, dy \\ & = & \int_{-\infty}^{\infty} x f_{X}(x) \, dx \int_{-\infty}^{\infty} y f_{Y}(y) \, dy \\ & = & \E[X] \E[Y] \end{eqnarray}` $$ -- Then `\(\Cov(X,Y) = 0\)`. --- # Independence and multivariate normal **Zero covariance does not generally imply independence!** -- * Suppose `\(X \in \{-1, 1\}\)` with `\(\Pr(X = 1) = \Pr(X = -1) = 1/2\)` * Suppose `\(Y \in \{-1, 0,1\}\)` with `\(Y = 0\)` if `\(X = -1\)` and `\(\Pr(Y = 1) = \Pr(Y= -1)\)` if `\(X = 1\)` -- $$ `\begin{eqnarray} \E[XY] & = & \sum_{i \in \{-1, 1\} } \sum_{j \in \{-1, 0, 1\}} i j \Pr(X = i, Y = j) \\ & = & -1 * 0 * \Pr(X = -1, Y = 0) + 1 * 1 * \Pr(X = 1, Y = 1) \\ && - 1 * 1 * \Pr(X = 1, Y = -1) \\ &= & 0 + \Pr(X = 1, Y = 1) - \Pr(X = 1, Y = -1 ) \\ & = & 0.25 - 0.25 = 0 \\ \E[X] & = & 0 \\ \E[Y] & = & 0 \end{eqnarray}` $$ --- # Independence, matrices, covariance: why might we care? * we are looking for information * duplication/dependence in variables can be problematic for predicting relationships (recall: determinants and eigenvalues) --- # Recap! * Joint pdf: same concept, integrate over variables * Condition pdfs on other random variables and marginal probability * Covariance and correlation: linear relationship * Matrices: can keep going with many dimensions!