1.4 Basic Vector Calculus

Gradients, Jacobians, and Hessians in the notation this course uses.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

Notation follows Deisenroth, Faisal, and Ong, Mathematics for Machine Learning.

1. Gradient

The gradient is the derivative for several variables. Vary one coordinate of $θ$ , hold the others. Those partial derivatives, stacked, are the gradient.

For $J (θ) : ℝ^{l} \to ℝ$ ,

\frac{\partial J}{\partial θ_{i}}, i = 1, \dots, l .

This course writes the collection as a row

\nabla J = [\frac{\partial J}{\partial θ_{1}} \frac{\partial J}{\partial θ_{2}} \dots \frac{\partial J}{\partial θ_{l}}] \in ℝ^{1 \times l} .

Some books write a column. Be consistent inside one derivation. Descent updates look like $θ \leftarrow θ - α (\nabla J)^{⊤}$ if $\nabla J$ is a row.

2. Chain rule

Let $J : ℝ^{2} \to ℝ$ depend on $θ_{1} (t)$ and $θ_{2} (t)$ . The derivative of $J$ with respect to $t$ is

\frac{d J}{d t} = [\frac{\partial J}{\partial θ_{1}} \frac{\partial J}{\partial θ_{2}}] [\begin{matrix} \partial θ_{1} / \partial t \\ \partial θ_{2} / \partial t \end{matrix}] = \frac{\partial J}{\partial θ_{1}} \frac{d θ_{1}}{d t} + \frac{\partial J}{\partial θ_{2}} \frac{d θ_{2}}{d t} .

That is a gradient times a Jacobian. Back-propagation (Module 6) is this rule applied to a long composition. You do not need the neural-net story yet. You do need to be willing to write the product.

3. Hessian

If $J (θ_{1}, θ_{2})$ is twice differentiable, the Hessian is

\nabla^{2} J = [\begin{matrix} \partial^{2} J / \partial θ_{1}^{2} & \partial^{2} J / \partial θ_{1} \partial θ_{2} \\ \partial^{2} J / \partial θ_{2} \partial θ_{1} & \partial^{2} J / \partial θ_{2}^{2} \end{matrix}] .

The mixed partials match (Schwarz), so the Hessian is symmetric. It measures curvature of $J$ around a point.

At a critical point $\nabla J = 0$ :

$\nabla^{2} J$ positive definite $\Rightarrow$ local minimum;
negative definite $\Rightarrow$ local maximum;
indefinite $\Rightarrow$ saddle.

Newton’s method (Week 2) solves a linear system with this matrix every step. Eigenvalues of $\nabla^{2} J$ are why a long thin valley makes steepest descent zigzag.

4. Practice

If a book prints $\nabla J$ as a column, how does the descent update $θ \leftarrow θ - α \nabla J$ change compared with the row convention in these slides?
In the chain rule, which factor is “how $J$ depends on $θ$ ” and which is “how $θ$ depends on $t$ ”?
Why does a symmetric Hessian with one positive and one negative eigenvalue make a critical point a saddle, not a min?

Ahmad Mousavi

1. Gradient

2. Chain rule

3. Hessian

4. Practice