4.4 Practical Considerations

Tuning, scaling, and what to watch for in sparse models.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

The figures follow Deisenroth, Faisal, and Ong, Mathematics for Machine Learning, and Theodoridis, Machine Learning: A Bayesian and Optimization Perspective.

1. Paths under $ℓ_{1}$ and $ℓ_{2}$

Gradient paths with <math xmlns= ℓ1 and $ℓ_{2}$ penalties" src="files/graphics/4.4-sparsity-practical/Optimization.png" />

Left. Contours of unregularized MSE plus an $ℓ_{1}$ penalty with $λ = 0.5$ . The global min sits on the $θ_{2} = 0$ axis. Gradient descent hits $θ_{2} = 0$ first, then walks along the axis to the min.

Right. The same MSE plus an $ℓ_{2}$ penalty.

On the LASSO cost the path bounces around the min after it reaches an axis. Shrink the learning rate during training so the steps get smaller and the walk can converge.

2. Elastic Net

Elastic Net puts both penalties on a linear model: $ℓ_{1}$ (LASSO) and $ℓ_{2}$ (ridge).

LASSO: sparsity and feature selection.
Ridge: multicollinearity and more stable coefficients.

J (θ) = \frac{1}{2 N} ‖ 𝐲 - 𝐗 θ ‖_{2}^{2} + λ_{1} ‖ θ ‖_{1} + λ_{2} ‖ θ ‖_{2}^{2} .

3. Advantages and disadvantages

Advantages	Disadvantages
High-dimensional data with multicollinearity	Two penalty weights to tune
Feature selection with correlated predictors	Less intuitive than LASSO or ridge alone
A compromise between LASSO and ridge	Higher computational cost; less sparse than LASSO; sensitive to feature correlations

4. Practice

Why does gradient descent on the LASSO cost bounce near an axis, and what do you change so it can settle?
What two penalties does Elastic Net mix, and what does each one contribute?
Name one advantage and one disadvantage of Elastic Net relative to LASSO.

Ahmad Mousavi

1. Paths under ℓ1 and ℓ2

2. Elastic Net

3. Advantages and disadvantages

4. Practice

1. Paths under $ℓ_{1}$ and $ℓ_{2}$