12.2 Multilayer Neural Networks

DATA 442/642 · Week 12 · PDF · Slides

Hidden layers and a composition of linear maps and nonlinearities.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

Figures and notes follow Géron, Hands-On Machine Learning; Deisenroth, Faisal, and Ong, Mathematics for Machine Learning; Theodoridis; and 3blue1brown.

1. A multi-layer net

A multi-layer perceptron stacks dense layers. Below: pixels of a digit flatten to an input vector, then hidden layers, then class scores.

2. What the layers are doing

Think of a deep net as multistage distillation: each layer filters the previous representation; what remains is more useful for the task.

3. Linear-algebra notation

Weights, activations, and the next layer line up as matrix–vector products. The picture is the notation you will write.

4. Mathematics of a layer

Linear map. Neuron $l$ does not see raw $\mathit{x}$ after the first layer. It sees the previous activations ${\mathit{a}}^{(l-1)}$:

where ${\mathit{z}}^{(l)}$ is the pre-activation, ${\mathit{W}}^{(l)}$ the weight matrix, ${\mathit{b}}^{(l)}$ the bias.

Activation. Elementwise nonlinearity:

Without $\sigma$, stacked layers collapse to one linear map. The nonlinearity is what a hidden layer is for.

5. Activation functions

Common choices: sigmoid, tanh, ReLU, softmax. Applied elementwise after the linear map (softmax on the output for mutually exclusive classes).

Practice

1. Write the forward pass of a two-layer net in one line of matrix products.

2. If you drop $\sigma$ from every hidden layer, what class of functions can the net represent?

3. In ${\mathit{z}}^{(l)}={\mathit{W}}^{(l)}{\mathit{a}}^{(l-1)}+{\mathit{b}}^{(l)}$, what is ${\mathit{a}}^{(0)}$?