7.2 Canonical Correlation Analysis

Find paired directions that co-vary across two views.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

Figures and notes follow Deisenroth, Faisal, and Ong, Mathematics for Machine Learning; Theodoridis; and Géron.

1. Canonical correlation analysis (CCA)

PCA reduces dimension for one data set. Often you have several related sets from different sources.

Canonical correlation analysis (CCA) processes two sets together. The correlation of two random vectors depends on the coordinate system you write them in.

CCA looks for a pair of linear maps, one per set, so that the transformed variables are maximally correlated.

Two objects to keep straight:

Canonical variables (variates)
Canonical correlation

2. The math

Two random vectors $𝐱 \in ℝ^{p}$ and $𝐲 \in ℝ^{q}$ , observations $x_{n}, y_{n}$ for $n = 1, \dots, N$ .

As in PCA, start with one pair of directions, $u_{x, 1}$ and $u_{y, 1}$ , that maximize the correlation of the projections.

Equivalently, the constrained problem

& max_{u_{x}, u_{y}} & & u_{x}^{⊤} {\hat{Σ}}_{x, y} u_{y} & subject to & & u_{x}^{⊤} {\hat{Σ}}_{x, x} u_{x} = 1, u_{y}^{⊤} {\hat{Σ}}_{y, y} u_{y} = 1,

where $Σ_{x, y}$ is the cross-covariance.

Compared with PCA: two directions, not one; the constraints use a weighted $Σ$ norm, not the Euclidean norm; the objective is correlation of the two projections, not variance of one.

Lagrangian:

ℒ (u_{x}, u_{y}, λ_{x}, λ_{y}) = u_{x}^{⊤} {\hat{Σ}}_{x, y} u_{y} - \frac{λ_{x}}{2} (u_{x}^{⊤} {\hat{Σ}}_{x, x} u_{x} - 1) - \frac{λ_{y}}{2} (u_{y}^{⊤} {\hat{Σ}}_{y, y} u_{y} - 1) .

Gradients in $u_{x}$ and $u_{y}$ set to zero give $λ_{x} = λ_{y} = λ$ and

{\hat{Σ}}_{x, y} u_{y} = λ {\hat{Σ}}_{x, x} u_{x}, {\hat{Σ}}_{y, x} u_{x} = λ {\hat{Σ}}_{y, y} u_{y} .

Solve the second for $u_{y}$ (assuming ${\hat{Σ}}_{y, y}$ invertible) and substitute:

{\hat{Σ}}_{x, y} {\hat{Σ}}_{y, y}^{- 1} {\hat{Σ}}_{y, x} u_{x} = λ^{2} {\hat{Σ}}_{x, x} u_{x}

and

u_{y} = \frac{1}{λ} {\hat{Σ}}_{y, y}^{- 1} {\hat{Σ}}_{y, x} u_{x} .

If ${\hat{Σ}}_{x, x}$ is invertible too, this is an ordinary eigenproblem:

({\hat{Σ}}_{x, x}^{- 1} {\hat{Σ}}_{x, y} {\hat{Σ}}_{y, y}^{- 1} {\hat{Σ}}_{y, x}) u_{x} = λ^{2} u_{x} .

So $u_{x, 1}$ is an eigenvector of that product. Take the eigenvector for the largest eigenvalue $λ^{2}$ to maximize the correlation.

$u_{x, 1}$ and $u_{y, 1}$ are the normalized canonical correlation basis vectors. The eigenvalue $λ^{2}$ is the squared canonical correlation. The projections $u_{x, 1}^{⊤} 𝐱$ and $u_{y, 1}^{⊤} 𝐲$ are the canonical variates.

3. What to read off in practice

Two questions after you fit CCA:

Canonical correlation. How much is each canonical variable from set 1 correlated with its counterpart in set 2?
Canonical variates. Which original variables does each variate represent?

Practice

CCA needs two views of the same objects. Name a pair.
What is CCA maximizing that PCA is not, and what replaces the unit-Euclidean constraint?
After you compute $u_{x, 1}$ , how do you get $u_{y, 1}$ ?

Ahmad Mousavi

1. Canonical correlation analysis (CCA)

2. The math

3. What to read off in practice

Practice