1.3 Derivatives for Optimization

DATA 442/642 · Week 1 · PDF · Slides

Gradients point uphill. We walk the other way.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

The figures follow Chollet, Deep Learning with Python, and Géron.

1. A derivative is a slope

Take a smooth map $J(x)=y$ that sends a number to a number. You can draw it as a curve in the plane.

Because $J$ is continuous, a small change in $x$ can only make a small change in $y$. Increase $x$ by a small ${\epsilon }_x$; $y$ moves by a small ${\epsilon }_y$.

x makes a small step in $y$" src="files/graphics/derivative-epsilon.png" />

Because the curve has no kinks, when ${\epsilon }_x$ is small enough around a point $p$ you may treat $J$ as a line of slope $a$:

The number $a=J^{\prime }(p)$ is the derivative at $p$.

• $a<0$: a small increase in $x$ decreases $J$.
• $a>0$: a small increase in $x$ increases $J$.
• $|a|$ (the magnitude) is how fast that happens.

To minimize $J$, walk against the sign of $a$. That one sentence is gradient descent in one dimension. Being able to differentiate is the main tool this course uses to find those $x$.

2. Continuity is not enough

Continuity says a small input change cannot jump the output. Smoothness says the graph has no corners, so a linear approximation exists. The algorithms in Module 1 assume at least that much. ReLU networks are allowed later; they are smooth almost everywhere. You still write a gradient.

3. Higher dimensions: the gradient

A map from one scalar to one scalar is a curve. A map $J(x,y)=z$ from a pair of scalars to a scalar is a surface in 3-space.

The analogue of $a$ is the gradient $\nabla J$: a vector of partial derivatives. It points to the direction of steepest increase. Descent uses $-\nabla J$.

Week 1 note 1.4 writes the gradient and the Hessian in the $\mathit{\theta }$ notation of the rest of the course. Week 2 uses both.

4. Practice

1. At a point where $J^{\prime }(p)=0$, what does a small step in $x$ do to $J$, to first order? Why is that not enough to call $p$ a minimum?

2. Why does a kink (a corner in the graph) break the “local line of slope $a$” story?

3. If $\nabla J$ points uphill, what vector do you add to $\mathit{\theta }$ in a descent step, up to a step-size $\alpha$?