13.1 Math for Topic Modeling

DATA 441/641 · Week 13 · PDF · Slides

Documents as mixtures, and the linear algebra underneath.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

1. Topics are a matrix problem

Topic modeling finds themes in a large collection without document labels. A news archive is not 10,000 unrelated files. It is a mix of elections, sports, markets, and weather, and each article is a mix of those themes.

The starting object is a document-term matrix (DTM): rows are documents, columns are terms, cells are counts or TF–IDF. Everything this module does is an operation on that matrix, or on a close cousin of it.

Linear algebra is not decoration here. SVD, NMF, PCA, and the inference inside LDA are all matrix factorizations or eigenvalue problems.

2. Why the linear algebra shows up

You do not need a full linear-algebra course. You need vectors, matrix multiply, inverse and transpose, and the eigenvalue equation.

3. Vectors and matrices

A vector $\mathit{x}\in {ℝ}^n$ is a list of $n$ numbers. Three operations you will use:

• Scalar: $\alpha \mathit{x}=[\alpha x_1,\dots ,\alpha x_n]$
• Dot product: $\mathit{x}·\mathit{y}={\mathit{x}}^{\top }\mathit{y}={\sum }_i{x}_i{y}_i$
• Euclidean norm: $\Vert \mathit{x}{\Vert }_2=\sqrt{{\sum }_i{x}_i^2}$

A matrix $\mathbf{A}\in {ℝ}^{m\times n}$ is the DTM shape. Pointwise $+$, $-$, and scalar multiply are entrywise. Matrix–vector multiply $\mathbf{A}\mathit{x}$ is a new vector whose $i$th entry is the dot product of row $i$ with $\mathit{x}$. Matrix–matrix multiply $\mathbf{A}\mathbf{B}$ is defined only when the inner dimensions match. $\mathbf{A}\mathbf{B}$ is not $\mathbf{B}\mathbf{A}$.

The identity ${\mathbf{I}}_n$ has ones on the diagonal. The inverse satisfies $\mathbf{A}{\mathbf{A}}^{-1}=\mathbf{I}$ when it exists. The transpose ${\mathbf{A}}^{\top }$ swaps rows and columns. Symmetric matrices ($\mathbf{A}={\mathbf{A}}^{\top }$) are the ones whose eigenvectors we trust first.

4. Eigenvalues, in one picture

For a square matrix $\mathbf{A}$,

$\mathit{v}$ is an eigenvector: a direction that $\mathbf{A}$ does not rotate. $\lambda$ is the eigenvalue: how much that direction is stretched. $\lambda >1$ stretches, $0<\lambda <1$ shrinks, $\lambda <0$ flips.

In a term–term or document–document covariance, a large $\lambda$ is a direction where the corpus actually varies. Those directions are the candidates for "topics" or for the axes PCA will keep.

5. PCA as a recipe

PCA turns correlated features into a new orthogonal basis, ordered by variance.

1. Center. Subtract the column mean: ${\mathbf{X}}_{\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{d}}=\mathbf{X}-\overline{\mathbf{X}}$.
2. Covariance. $\mathbf{C}=\frac{1}{n-1}{\mathbf{X}}_{\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{d}}^{\top }{\mathbf{X}}_{\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{d}}$.
3. Eigendecompose. $\mathbf{C}\mathit{v}=\lambda \mathit{v}$. Large $\lambda$ = a principal component that explains more variance.
4. Keep the top $k$. Columns of ${\mathbf{V}}_k$ are those eigenvectors.
5. Project. ${\mathbf{X}}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{u}\mathrm{c}\mathrm{e}\mathrm{d}}={\mathbf{X}}_{\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{d}}{\mathbf{V}}_k$.

Each document is now a vector in $k$ dimensions. You can cluster those vectors (k-means, DBSCAN) or pass them to LDA or NMF. Words and documents that load on the same component are a first draft of a topic.

PCA is a tool, not the whole topic model. Note 13.2 is SVD / LSA, which is the factorization people actually run on a DTM. Week 14 is LDA and NMF: same matrix picture, different constraints (probabilities, non-negativity). Learn the multiply and the eigen-equation well enough to read those notes. You will not invert a 20,000-column matrix by hand.

6. Practice

1. A DTM is 5,000 documents by 20,000 terms. What are $m$ and $n$ if we write $\mathbf{A}\in {ℝ}^{m\times n}$ as documents × terms?

2. Why is $\mathbf{A}\mathbf{B}$ a different size from $\mathbf{B}\mathbf{A}$ in the 3×2 times 2×3 example?

3. In PCA, what does a small eigenvalue tell you about that direction, and why might you drop it before clustering?

If you can compute $\mathbf{A}\mathit{v}$ and compare it to $\lambda \mathit{v}$ on a 2×2 example, you are ready for SVD.