Matrix algebra

class: center, middle, inverse, title-slide

.title[
# Matrix algebra
]
.author[
### <a href="https://jmclip.github.io/MACSS_math_camp/">MACS 33000</a> University of Chicago
]

---

# Misc updates

* Grading up soon 
  * PLEASE TAG ALL QUESTIONS
* Check github for updates

---

# Learning objectives

* Visualize vectors in multiple dimensions
* Demonstrate applicability of linear algebra to text analysis and cosine similarity
* Perform basic algebraic operations on vectors and matricies

---
class: center, middle, inverse

# Vectors

---

# Inner product: vector operations
Multiplying two vectors produces a scalar (single number) output.

$$
`\begin{aligned}
\mathbf{u} \cdot \mathbf{v} &=  u_{1} v_{1} + u_{2}v_{2} + \ldots + u_{n} v_{n}  \\
														& =  \sum_{i=1}^{N} u_{i} v_{i} 
\end{aligned}`
$$

* Aka **dot product**
* Results in a scalar value

---

# Inner product

* `$\mathbf{u} = (1, 2, 3)$` and `$\mathbf{v} = (2, 3, 1)$`

`$$\begin{aligned} \mathbf{u} \cdot \mathbf{v} & =  1 \times 2 +  2 \times 3 +  3 \times 1 \\ 				& = 2+ 6 + 3 \\				& = 11				\end{aligned}$$`

---

# Calculating vector length
The length of a vector, the vector norm, can be thought of as the distance of the vector from the origin.

---

# Vector norm

$$
`\begin{aligned}
\| \mathbf{v}\| & = (\mathbf{v} \cdot \mathbf{v} )^{1/2} \\
						   & = (v_{1}^2 + v_{2}^{2} + v_{3}^{2} + \ldots + v_{n}^{2} )^{1/2}
\end{aligned}`
$$

* Vector norm of a three-dimensional vector `$\mathbf{x} = (1,1,1)$`:

`$$\begin{aligned}\| \mathbf{x}\| & = (\mathbf{x} \cdot \mathbf{x} )^{1/2} \\						   & = (x_{1}^2 + x_{2}^{2} + x_{3}^{2})^{1/2} \\						   & = (1 + 1 + 1)^{1/2} \\						   &= \sqrt{3}\end{aligned}$$`

---

# Text analysis

```
 a abandoned abc ability able about above abroad absorbed absorbing abstract
43         0   0       0    0    10     0      0        0         0        1
```

`$$(43,0,0,0,0,10,\dots)$$`

---

# Text analysis

$$
`\begin{aligned}
\text{Doc1} & = (1, 1, 3, \ldots, 5) \\
\text{Doc2} & = (2, 0, 0, \ldots, 1) \\
\textbf{Doc1}, \textbf{Doc2} & \in \mathbb{R}^{M}
\end{aligned}`
$$

---

# Inner product

$$
`\begin{aligned}
\textbf{Doc1} \cdot \textbf{Doc2}  &  =  (1, 1, 3, \ldots, 5) \cdot (2, 0, 0, \ldots, 1)  \\
 & =  1 \times 2 + 1 \times 0 + 3 \times 0 + \ldots + 5 \times 1 \\
& = 7 
\end{aligned}`
$$

---

# Length

$$
`\begin{aligned}
\| \textbf{Doc1} \| & \equiv  \sqrt{ \textbf{Doc1} \cdot \textbf{Doc1} } \\
 & =  \sqrt{(1, 1, 3, \ldots , 5) (1, 1, 3, \ldots, 5)' } \\
  & =  \sqrt{1^{2} +1^{2} + 3^{2} + 5^{2} } \\
   & =   6
\end{aligned}`
$$

---

# Cosine similarity

$$
`\begin{aligned}
\cos (\theta) & \equiv  \left(\frac{\textbf{Doc1} \cdot \textbf{Doc2}}{\| \textbf{Doc1}\| \|\textbf{Doc2} \|} \right) \\
 & = \frac{7} { 6 \times  2.24} \\
  & = 0.52
\end{aligned}`
$$

---

# Measuring similarity

* Usefulness
* Desirable properties
    * The **maximum** should be the document with itself
    * The **minimum** should be documents which have no words in common (orthogonal to one another)
    * Increasing when more of the same words are used
    * Normalize for document length
    
* We can look at the similarity and also calculate the angle between them (`$\theta$`)

---

# Similarity: Using the inner product

`$$(2,1) \cdot (1,4) = 6$$`

---

# Length dependence

`$$(4,2) \cdot (1,4) = 12$$`

---

# Cosine similarity
Allows us to look at the angle between them instead

---

# Cosine similarity

$$
`\begin{aligned}
(4,2) \cdot (1,4) &= 12 \\
\mathbf{a} \cdot \mathbf{b} &= \|\mathbf{a} \| \times \|\mathbf{b} \| \times \cos(\theta) \\
\frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a} \| \times \|\mathbf{b} \|}  &= \cos(\theta)
\end{aligned}`
$$

---

# Cosine similarity

$$
`\begin{aligned}
\cos (\theta) & \equiv  \left(\frac{\textbf{Doc1} \cdot \textbf{Doc2}}{\| \textbf{Doc1}\| \|\textbf{Doc2} \|} \right) \\
 & = \frac{(2, 1) \cdot (1, 4)} {\| (2,1)\| \| (1,4) \|} \\
 & = \frac{6} {(\sqrt{2^2 + 1^2}) (\sqrt{1^2 + 4^2})} \\
 & = \frac{6} {(\sqrt{5}) (\sqrt{17})} \\
  & \approx 0.65
\end{aligned}`
$$

---

# Cosine similarity

$$
`\begin{aligned}
\cos (\theta) & \equiv  \left(\frac{\textbf{Doc1} \cdot \textbf{Doc2}}{\| \textbf{Doc1}\| \|\textbf{Doc2} \|} \right) \\
 & = \frac{(4, 2) \cdot (1, 4)} {\| (4,2)\| \| (1,4) \|} \\
 & = \frac{12} {(\sqrt{4^2 + 2^2}) (\sqrt{1^2 + 4^2})} \\
 & = \frac{12} {(\sqrt{20}) (\sqrt{17})} \\
  & \approx 0.65
\end{aligned}`
$$

---

# Cosine similarity

$$
`\begin{aligned}
\cos (\theta) & \equiv  \left(\frac{\textbf{Doc3} \cdot \textbf{Doc2}}{\| \textbf{Doc3}\| \|\textbf{Doc2} \|} \right) \\
 & = \frac{(1,2) \cdot (1, 4)} {\| (1,2)\| \| (1,4) \|} \\
 & = \frac{9} {(\sqrt{1^2 + 2^2}) (\sqrt{1^2 + 4^2})} \\
 & = \frac{9} {(\sqrt{5}) (\sqrt{17})} \\
  & \approx 0.976
\end{aligned}`
$$
--

Interpretation: closer to 1, more similar

---

# Exploring similarity:

.pull-left[
<img src="05-matrix-algebra_files/figure-html/cosine-simd1-1.png" alt="" width="85%" style="display: block; margin: auto;" />
]
.pull-right[
<img src="05-matrix-algebra_files/figure-html/cosine-simb-1.png" alt="" width="85%" style="display: block; margin: auto;" />

]

We can see how the similarity of 0.65 has a larger `$\theta$` and a similarity of 0.97 has a smaller `$\theta$`. We can also use the inverse of cos(arccos) to find our respective angles: approx 0.86 radians (approx 50 degrees) vs 0.25 radians (approx 15 degrees).

---
class: center, middle, inverse
# The Matrix, Part I
<img src="https://media.tenor.com/ang0VzOwbdAAAAAC/the-matrix-reloaded-matrix.gif" alt="" width="85%" style="display: block; margin: auto;" />
---

# Matricies

* **Rectangular** arrangement (array) of numbers defined by two **axes**
    1. Rows
    1. Columns

`$$\mathbf{A} = \begin{bmatrix}
a_{11} & a_{12} & \ldots & a_{1n} \\
a_{21} & a_{22} & \ldots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \ldots & a_{mn} \\
\end{bmatrix}$$`

---

# Example matricies: Fun facts

* Dimensionality matters
* Diagonal from upper left to lower right matters 
  * Sometimes the opposite diagonal is referred to by a name, but this varies
* Matrices referred to by their overall properties (elements) and/or behavior of the diagonal 
* Special names for *special* matrices.

---
# Special matrices

* **Diagonal matrix** (only numbers on diagonal; zero elsewhere)
    * **Identity** (only 1s on diagonal; zero elsewhere -- special case of diagonal -- called `$I$`)
  * **Square** (n rows = n columns)
  * **Triangular matrix** (half of matrix has zeros -- see below for lower/upper distinction)
    * **lower triangular** (numbers in lower half (diagonal down) and zeros above diagonal)
    * **upper triangular** (numbers in upper half (diagonal up) and zeros below diagonal)
  * **Zero matrix** (all zeros!)
  * **Matrix** of only ones (sometimes given letter J)
  * **Transpose** / transposition (mathematical operation on a matrix) (superscript with T)
  * **Inverse** (this is a mathematical operation) (superscipt with `$-1$`)

---

# Example matricies

Consider the following:

`$$\mathbf{X} = \begin{bmatrix}1 & 2 & 3 & 4 \\
2 & 1 & 4 & 3\\
\end{bmatrix}$$`

`$$\mathbf{Y} = \left[ \begin{array}{rr} 1 & 2 \\3 & 2 \\1 & 4\end{array} \right]\\$$`

`$$\mathbf{Z} = \left[ \begin{array}{rrr}1 & 2 & 3 \\5 & 1 & 4 \\6 & 1 & 7\end{array} \right]$$`

---

# Matrix oparations: addition

* `$\mathbf{X}$` and `$\mathbf{Y}$` are `$m \times n$` matrices

`$$\begin{aligned} \mathbf{X} + \mathbf{Y} & =  \begin{pmatrix} x_{11} & x_{12} & \ldots & x_{1n} \\ x_{21} & x_{22} & \ldots & x_{2n} \\\vdots & \vdots & \ddots & \vdots \\ x_{m1} & x_{m2} & \ldots & x_{mn} \\ \end{pmatrix} + \begin{pmatrix}  y_{11} & y_{12} & \ldots & y_{1n} \\ y_{21} & y_{22} & \ldots & y_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ y_{m1} & y_{m2} & \ldots & y_{mn} \\ \end{pmatrix} \\ & = \begin{pmatrix}  x_{11} + y_{11} & x_{12} + y_{12} & \ldots & x_{1n} + y_{1n} \\ x_{21} + y_{21} & x_{22} + y_{22} & \ldots & x_{2n} + y_{2n} \\ \vdots & \vdots & \ddots & \vdots\\ x_{m1} + y_{m1} & x_{m2} + y_{m2} & \ldots & x_{mn} + y_{mn} \\ \end{pmatrix} \end{aligned}$$`

---

# Scalar Multiplication
You can just multiply the constant (scalar) through the matrix.

* `$\mathbf{X}$` is an `$m \times n$` matrix and `$k \in \mathbb{R}$`

`$$k \mathbf{X} = \begin{pmatrix} 
k x_{11} & k x_{12} & \ldots &  k x_{1n} \\
k x_{21} & k x_{22} & \ldots & k x_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
k x_{m1} & k x_{m2} & \ldots & k x_{mn} \\
\end{pmatrix}$$`

---

# Matrix transposition

Transposing a matrix means *flipping* it in a sense -- swapping across the diagonal.

`$$\mathbf{X} = \begin{pmatrix} x_{11} & x_{12} & \ldots & x_{1n} \\ x_{21} & x_{22} & \ldots & x_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ x_{m1} & x_{m2} & \ldots & x_{mn} \\ \end{pmatrix}$$`

`$$\mathbf{X}' = \begin{pmatrix}  x_{11} & x_{21} & \ldots & x_{m1} \\ x_{12} & x_{22} & \ldots & x_{m2} \\ \vdots & \vdots & \ddots & \vdots \\ x_{1n} & x_{2n} & \ldots & x_{mn}\end{pmatrix}\\$$`
* Transpose has the prime or a small upper T

---
 
# Matrix multiplication
 
* Step 1: have two (or more matrices to multiply)
* Step 2: check dimensions (rows vs columns)
* Step 3: **ORDER MATTERS** the number of COLUMNS of the FIRST matrix MUST MATCH the number of ROWS of the second matrix (YES, I KNOW!! )
* Step n: final output will have the number of ROWS of the FIRST and the number of COLUMNS of the second matrix. 
(steps continued on next slide -- just setting up logic here!)

`$$\mathbf{X} = \begin{pmatrix} 1 \\ 10\end{pmatrix}$$`

`$$\quad \mathbf{Y} = \begin{pmatrix} 5 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

Note our two matrices above. In what order do we multiply them? What dimensions will our final matrix have?

--
Well, `$X$` is `$2\times 1$`, `$Y \text{ is } 2 \times 2$`. Since the number of cols of X do not match the number of rows of Y, we can't multiply them. BUT! We **CAN** multiply `$Y$` and `$X$`! This is because we will have `$2 \times 2 \text{ and } 2 \times 1$`, producing a final matrix of `$2 \times 1$`.

---

# Matrix multiplication: Color-coded example
To multiply, you go ACROSS the ROW of the first matrix and multiply pairwise DOWN THE COLUMN of the second matrix. You then add these together. For example, if you have matrix A times matrix B, it would be element `$a_{1,1}*b_{1,1} + ... + a_{1,n}*b_{n,1}$`. This sum goes into the first cell in the resulting matrix. This sounds confusing (because it is), so let's see an example.

--

`$$\quad \mathbf{Y} = \begin{pmatrix} 5 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

`$$\mathbf{X} = \begin{pmatrix} 1 \\ 10\end{pmatrix}$$`

Here, we have a very simple example -- so, we'll take the row (in this case, the entirety) of X and multiply the pairs (each element with its corresponding partner) of X and Y. So, `$5*$` `$1$` + `$2*$` `$10$` . This will give us `$5+ 20 = 25$` as the first cell in our resulting matrix.

---
# Matrix multiplication: Color-coded example, 2
To multiply, you go ACROSS the ROW of the first matrix and multiply pairwise DOWN THE COLUMN of the second matrix. You then add these together. For example, if you have matrix A times matrix B, it would be element `$a_{1,1}*b_{1,1} + ... + a_{1,n}*b_{n,1}$`. This sum goes into the first cell in the resulting matrix. This sounds confusing (because it is), so let's see an example.

--

`$$\quad \mathbf{Y} = \begin{pmatrix} 5 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

`$$\mathbf{X} = \begin{pmatrix} 1 \\ 10\end{pmatrix}$$`

Similarly, we'll multiply the remaining row of our matrix Y by the column in X to produce the following: `$3*$` `$1$` + `$4*$` `$+$` `$10$` . This will give us `$3+ 40 = 43$` as the second cell in our resulting matrix.

---

`$$\quad \mathbf{Y} = \begin{pmatrix} 5 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

`$$\mathbf{X} = \begin{pmatrix} 1 \\ 10\end{pmatrix}$$`

**This produces**

`$$\mathbf{Z} = \begin{pmatrix} 25 \\ 43 \\ \end{pmatrix}$$`

---

# Matrix multiplication

Try this example and check with a neighbor: `$\mathbf{XY}$`

`$$\mathbf{X} = \begin{pmatrix} 1 & 1 \\ 1& 1 \\ \end{pmatrix} , \quad \mathbf{Y} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

`$$\begin{aligned} \mathbf{A} & = \mathbf{X} \mathbf{Y} \\& = \begin{pmatrix}1 & 1 \\ 1 & 1 \\\end{pmatrix} \begin{pmatrix}1 & 2 \\3 & 4 \\\end{pmatrix} \\&= \begin{pmatrix}1 * 1 + 1 * 3 & 1 * 2 + 1 * 4 \\1 * 1 + 1 * 3 & 1 * 2 + 1 * 4\\\end{pmatrix} \\&= \begin{pmatrix}4 & 6 \\4 & 6\end{pmatrix}\end{aligned}$$`
--
For extra fun, try $$ \mathbf{Y} \mathbf{X}$$ and see how this answer is different!!
---

# Matrix multiplication: Identity Matrix

Try this example and check with a neighbor: `$\mathbf{IY}$` (let's see why `$I$` is called the **identity matrix**)

`$$\mathbf{I} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ \end{pmatrix} , \quad \mathbf{Y} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ \end{pmatrix}$$`

`$$\begin{aligned} \mathbf{A} & = \mathbf{I} \mathbf{Y} \\& = \begin{pmatrix}1 & 0 \\ 0 & 1 \\\end{pmatrix} \begin{pmatrix}1 & 2 \\3 & 4 \\\end{pmatrix} 
\\&= \begin{pmatrix}1 * 1 + 0 \times 3 & 1 \times 2 + 0 \times 4 \\
0 \times 1 + 1 \times 3 & 0 \times 2 + 1 \times 4\\\end{pmatrix} 
\\&= \begin{pmatrix}1 & 2 \\3 & 4\end{pmatrix}\end{aligned}$$`

IT'S LIKE MAGIC! (*actually it's just the matrix version of multiplying by 1*)

---
# Let's try this out on our own:

Suppose we have a `$3 \times 4$` matrix, **A** and a `$4 \times 1$` matrix **v**: 
`$$\begin{aligned} \mathbf{A} & = 
\begin{pmatrix}   2 & 3 & 4 & 5 \\
	1 & 5 & 1 & 2\\
	3 & 5 & 3 & 4 \\\end{pmatrix} \\
	\mathbf{v} & = \begin{pmatrix}3 \\ 3 \\ 4 \\10
 \\\end{pmatrix} \end{aligned}$$`

* What is `$\boldsymbol{A} \boldsymbol{v}$`?

* What is `$\boldsymbol{A}^{'} \boldsymbol{v}$`?

* What is `$\boldsymbol{A} \boldsymbol{A}^{'}$`?

* What is `$\boldsymbol{A}^{'} \boldsymbol{A}$`?

---

## Answers:

* What is `$\boldsymbol{A} \boldsymbol{v}$`?
`$$\small{\begin{aligned} \mathbf{A}*\mathbf{v}  & = 
\begin{pmatrix}   2 & 3 & 4 & 5 \\
	1 & 5 & 1 & 2\\
	3 & 5 & 3 & 4 \\\end{pmatrix}  \begin{pmatrix}3 \\ 3 \\ 4 \\10
 \\\end{pmatrix} = \begin{pmatrix}2*3 + 3*3+4*4+5*10 \\ 1*3+5*3+1*4+2*10 \\ 3*3+5*3+3*4+4*10 \end{pmatrix} \\
 &= \begin{pmatrix} 81 \\ 42 \\ 76 \\ \end{pmatrix}\end{aligned}}$$`

* What is `$\boldsymbol{A}^{'} \boldsymbol{v}$`? **Not possible: `$\boldsymbol{A}^{'}$` has dimensions `$4\times 3$` and `$\boldsymbol{v}$` has dimensions `$4 \times 1$` **

---
## Answers:

* What is `$\boldsymbol{A} \boldsymbol{A}^{'}$`?

`$\mathbf{A}*\mathbf{v} =$`
`$$\begin{aligned} 
  & = 
\begin{pmatrix}   
  2 & 3 & 4 & 5 \\
	1 & 5 & 1 & 2\\
	3 & 5 & 3 & 4 \end{pmatrix}  
	\begin{pmatrix}
	2 & 1 & 3 \\
	3 & 5 & 5 \\
	4 & 1 & 3 \\
	5 & 2 & 4 \\
	\end{pmatrix} \\
	&= 
	\begin{pmatrix} 
	 2^2 + 3^2 + 4^2 + 5^2    & 2* 1 + 3* 5 + 4*1 + 5*2     & 2*3 + 3* 5 + 4*3 + 5*4 \\
	1*2 + 5*3 + 1* 4 + 2* 5     & 1^2 + 5^2 + 1^1 + 2^2       & 1*3 + 5*5 + 1*3 + 2*4 \\
	3*2 + 5*3 + 3* 4 + 4*5    & 3*1 + 5*5 + 3*1 + 4*2       & 3^2 + 5^ 2 + 3^ 2 + 4^2  \\ 
	\end{pmatrix} \\
	& = \begin{pmatrix}
	54 & 31 & 53 \\
	31 & 31 & 39 \\
	53 & 39 & 59
	\end{pmatrix}
	\end{aligned}$$`

---
## Answers:

* What is `$\boldsymbol{A}^{'}\boldsymbol{A}$`?
`$$\begin{aligned} 
  & = \begin{pmatrix}
	2 & 1 & 3 \\
	3 & 5 & 5 \\
	4 & 1 & 3 \\
	5 & 2 & 4 \\
	\end{pmatrix} \begin{pmatrix}   
  2 & 3 & 4 & 5 \\
	1 & 5 & 1 & 2\\
	3 & 5 & 3 & 4 \end{pmatrix}  
	& = \begin{pmatrix}   
  14 & 26 & 18 & 24 \\
	26 & 59 & 32 & 45\\
	18 & 32 & 26 & 34 \\
	24 & 45 & 34 & 45 \end{pmatrix}  \\
	\end{aligned}$$`

---

# So why did we do all that?

We just computed `$\mathbf{A}\mathbf{A}'$` and `$\mathbf{A}'\mathbf{A}$` and
you were probably wondering why.

In practice, our matrix is a **data matrix**. Call it `$\mathbf{X}$`:

* Each **row** is an observation (a person, a document, a country-year)
* Each **column** is a variable (a feature, a predictor)
* So `$\mathbf{X}$` is `$n \times p$`: `$n$` observations, `$p$` variables

And `$\mathbf{y}$` is our outcome, one value per observation: `$n \times 1$`.

---

# Checking our dimensions

`$$\underset{(p \times n)}{\mathbf{X}'} \; \underset{(n \times p)}{\mathbf{X}} \; = \; \underset{(p \times p)}{\mathbf{X}'\mathbf{X}}$$`

Square! Symmetric! One row and column per variable. (Notice this is why we
always write `$\mathbf{X}'\mathbf{X}$` and not `$\mathbf{X}\mathbf{X}'$` --
the other order gives us `$n \times n$`, one row per *observation*, which is
enormous and not what we want.)

`$$\underset{(p \times n)}{\mathbf{X}'} \; \underset{(n \times 1)}{\mathbf{y}} \; = \; \underset{(p \times 1)}{\mathbf{X}'\mathbf{y}}$$`

One number per variable.

---

# Least squares

Put those two pieces together with the inverse we just learned:

`$$\boldsymbol{\hat\beta} = (\mathbf{X}'\mathbf{X})^{-1} \mathbf{X}'\mathbf{y}$$`

`$$\underset{(p \times p)}{(\mathbf{X}'\mathbf{X})^{-1}} \; \underset{(p \times 1)}{\mathbf{X}'\mathbf{y}} \; = \; \underset{(p \times 1)}{\boldsymbol{\hat\beta}}$$`

**One coefficient per variable.** Exactly what a regression gives you.

* This is the formula behind `lm()` in R
* Every single operation in it is something we did today: transpose,
  multiply, invert
* And when `$\mathbf{X}'\mathbf{X}$` is singular -- perfectly correlated
  predictors, or more variables than observations (`$p > n$`) -- there is
  no inverse, and there is no unique answer. Your software will complain,
  and now you know why.
  
---

class: center, middle

# Uses of matrices: how you might see them in your life

---
class: center, middle, inverse

# Machine learning

---

# Everything is a tensor

You will hear the word **tensor** constantly. A tensor is just an array of numbers, organized by how many axes it
has:

* **Scalar** -- a single number (0 axes)
* **Vector** -- a list of numbers (1 axis)
* **Matrix** -- rows and columns (2 axes)
* **3D tensor and up** -- stack matrices together (3+ axes)

An image is a 3D tensor: height `$\times$` width `$\times$` color channel. A
batch of images is 4D.

**If you can do it with a matrix, you can do it with a tensor.** Addition,
multiplication, transposition -- all the rules generalize. You already
know the hard part.

---

# Neural networks

<img src="https://upload.wikimedia.org/wikipedia/commons/9/99/Neural_network_example.svg" alt="" width="55%" style="display: block; margin: auto;" />
---

# What a neural network layer actually is

`$$\mathbf{Y} = \text{activation}(\mathbf{W}\mathbf{X} + \mathbf{B})$$`

* `$\mathbf{X}$` -- your input data
* `$\mathbf{W}$` -- a matrix of **weights** (this is what gets learned)
* `$\mathbf{B}$` -- a vector of **biases** (added on, like an intercept)
* `$\mathbf{Y}$` -- the output, which becomes the input to the next layer

That `$\mathbf{W}\mathbf{X}$` is our matrix multiplication -- each entry of
the output is a **dot product** of one row of `$\mathbf{W}$` with
`$\mathbf{X}$`. It's the same operation we used for cosine similarity, just
stacked up and repeated.

Compare it to `$\boldsymbol{\hat\beta}$` from a few slides ago. A neural
network layer is a regression, run many times, with a twist at the end.

---

# Uses for neural networks

* Self-driving cars
* Voice activated assistants
* Automatic machine translation
* Image recognition
* Detection of diseases

---
# Social networks

<img src="https://macs40700.netlify.app/slides/13-visualize-text-network/index_files/figure-html/unnamed-chunk-67-1.png" alt="" width="85%" style="display: block; margin: auto;" />
---

# Uses for social networks

* Connections
* Groupings (e.g. cliques)
* Density
* 'Spread' (ideas, disease, etc.)

---

# Algebraic properties: Recap!

* Matricies must be **conformable** (right dimensions)
* Matricies have weird rules: 
  * Associative property: `$(\mathbf{XY})\mathbf{Z} = \mathbf{X}(\mathbf{YZ})$`
  * Additive distributive property: `$(\mathbf{X} + \mathbf{Y})\mathbf{Z} = \mathbf{XZ} + \mathbf{YZ}$`
  * Zero property: `$\mathbf{X0} = 0$`
  * Identity matrix: `$\mathbf{IX} = \mathbf{X}$`
  * Order matters: `$\mathbf{XY} \neq \mathbf{YX}$` (in general)
    * Different from scalar multiplication: `$xy = yx$`
* Order of matrix: sometimes you'll see subscripts referring to a matrix's order -- this is particularly true for square matrices. E.g:
  * `$\mathbf{I_2}$` is an identity matrix (and therefore a square matrix) with two rows and two columns
  * `$\mathbf{J}_{2\times3}$` is a matrix (likely full of ones) with two rows and three columns
* Operations on matrix: transposing means flipping / reflecting across the diagonal

---
# Example matricies: Fun facts

* Dimensionality matters
* Diagonal from upper left to lower right matters 
* Matrices referred to by their overall properties (elements) and/or behavior of the diagonal 
* Special names for *special* matrices:
    * **Diagonal matrix** (only numbers on diagonal; zero elsewhere)
    * **Identity** (only 1s on diagonal; zero elsewhere -- special case of diagonal -- called `$I$`)
  * **Square** (n rows = n columns)
  * **Triangular matrix** (half of matrix has zeros -- see below for lower/upper distinction)
    * **lower triangular** (numbers in lower half (diagonal down) and zeros above diagonal)
    * **upper triangular** (numbers in upper half (diagonal up) and zeros below diagonal)
  * **Zero matrix** (all zeros!)
  * **Matrix** of only ones (sometimes given letter J)
  * **Transpose** / transposition (mathematical operation on a matrix) (superscript with T)
  * **Inverse** (this is a mathematical operation) (superscipt with `$-1$`)

---
class: center, middle, inverse
## Let’s get to know each other a bit more – Name, pronouns, subfield/research area, where you are currently, something fun/interesting about you and/or your hobbies

<!--

# Linear algebra roots

* Tensor
* Scalars (0D tensors)
* Vectors (1D tensors)
* Matricies (2D tensors)
* 3D tensors and higher-dimensional tensors

# Tensor operations

* Generalizations of matrix operations
* Tensor addition
* Tensor multiplication
* **If you can do it with a matrix, you can do it with a tensor**

# Linear algebra notation

`$$\mathbf{Y} = \text{activation}(\mathbf{W} \cdot \mathbf{X} + \mathbf{B})$$`

* `$\mathbf{X}$`
* `$\mathbf{Y}$`
* `$\mathbf{W}, \mathbf{B}$`
* `$\text{activation}()$`
    * Rectified Linear Units (RELU)
        
        `$$R(z) = \max(0, z)$$`
            
    * Sigmoid function (aka logistic regression)
        
        `$$S(z) = \frac{1}{1 + e^{-z}}$$`
        
  -->