14.3 Variational Autoencoders

A probabilistic latent space and the reparameterization trick.

These notes walk through the lecture slides. The original deck is unchanged; use (slides) on the course hub if you want that PDF.

Figures and notes follow Géron, Hands-On Machine Learning; Deisenroth, Faisal, and Ong, Mathematics for Machine Learning; and Theodoridis, Machine Learning: A Bayesian and Optimization Perspective.

1. Variational autoencoders

A variational autoencoder (VAE) is a probabilistic autoencoder: after training, the output is still partly random. A denoising autoencoder uses noise only while it trains.

It is also generative: you can draw new points that look as if they came from the training set.

VAE: encoder to <math xmlns= μ,σ, sample a coding, decode" src="files/graphics/14.3-vaes/var_auto.png" />

The figure builds the coding as $μ$ plus Gaussian noise times $σ$ . The random draw is an input, not a node you cannot differentiate.

2. Applications

Use	Sketch
Image generation	Art, avatars, synthetic frames.
Anomaly detection	Odd transactions, factory defects, unusual medical images.
Data compression	A short latent vector for storage or transmission.
Drug discovery	New molecules with target properties.
NLP	Coherent text: chat, translation, copy.

3. The procedure

The encoder does not emit one coding. It emits a mean $μ$ and a standard deviation $σ$ . The coding is then drawn from a Gaussian with that mean and that width. The decoder reconstructs from the draw, as usual.

Training pushes the cloud of codings toward a Gaussian blob in latent space. After that, generation is cheap: sample a coding from the Gaussian, decode it.

Input to encoder, Gaussian over <math xmlns= z, sample, decoder reconstruction" src="files/graphics/14.3-vaes/VAEs.png" />

4. The cost function

Two pieces.

Reconstruction loss. Make $\hat{x}$ look like $x$ (cross-entropy, or the negative log-likelihood of $x$ under a Gaussian).
Latent loss. Make the coding distribution look like a simple Gaussian. That term is the KL divergence between the actual coding law and the target Gaussian.

5. ELBO

The training objective is the variational lower bound, or evidence lower bound (ELBO). It is a tractable stand-in for the true log-likelihood of the data.

Write $x$ for the observation, $z$ for the latent, $θ$ for decoder parameters, $ϕ$ for encoder parameters. The ELBO is

ℒ (θ, ϕ; x) = 𝔼_{q (z | x)} [\log p (x | z)] - D_{KL} (q (z | x) ∥ p (z)),

where $p (x | z)$ is the likelihood, $q (z | x)$ is the approximate posterior, and $p (z)$ is the prior.

Maximize it in both parameter blocks:

{max}_{θ, ϕ} ℒ (θ, ϕ; x) .

SGD or Adam is the usual optimizer. Reconstruction plus a Gaussian prior on $z$ is what you are buying: a latent space you can sample.

6. Gaussian assumption

If the decoder likelihood is Gaussian, the reconstruction term is a weighted squared error plus constants:

𝔼_{q (z | x)} [\log p (x | z)] = - \frac{1}{2} \sum_{i = 1}^{N} (\frac{(x_{i} - {\hat{x}}_{i})^{2}}{σ_{i}^{2}} + \log (σ_{i}^{2}) + \log (2 π)) .

Here $N$ is the dimension of $x$ , and $σ_{i}^{2}$ is the variance on coordinate $i$ .

The KL between a diagonal Gaussian posterior and a standard Gaussian prior is closed form:

D_{KL} (q (z | x) ∥ p (z)) = - \frac{1}{2} \sum_{i = 1}^{N} (1 + \log (σ_{i}^{2}) - μ_{i}^{2} - σ_{i}^{2}),

with $μ_{i}$ the mean of latent coordinate $i$ .

Put the two together:

ℒ (θ, ϕ; x) & = - \frac{1}{2} \sum_{i = 1}^{N} (\frac{(x_{i} - {\hat{x}}_{i})^{2}}{σ_{i}^{2}} + \log (σ_{i}^{2}) + \log (2 π)) & - \frac{1}{2} \sum_{i = 1}^{N} (1 + \log (σ_{i}^{2}) - μ_{i}^{2} - σ_{i}^{2}) .

That is the ELBO when both the approximate posterior and the prior are Gaussian.

Practice

What problem does the reparameterization trick solve?
Name the two terms in the VAE cost (the ELBO).
After training, how do you generate a new sample without feeding in a real $x$ ?

Ahmad Mousavi