Diffusion: Why ELBO?

1 minute read

Published:

Why is the Evidence Lower Bound (ELBO) so important for training generative models such as Variational Autoencoders (VAEs) and Diffusion Models?

The problem is latent variables. In a VAE, the observed data is generated through a latent variable. To maximize the likelihood of the observed data, we would ideally need to integrate over all possible values of this latent variable. But this integral is generally intractable.

Diffusion models have a similar problem, though their latent structure is different. Instead of a single latent variable, we have a sequence of latent noisy states:

\[x_T \rightarrow x_{T-1} \rightarrow \cdots \rightarrow x_1 \rightarrow x_0\]

where \(x_0\) is the observed data, and \(x_T\) follows the terminal distribution, chosen to be approximately standard Gaussian: \(x_T \sim \mathcal{N}(0, I)\).

To compute the likelihood of \(x_0\), we need to marginalize over all possible intermediate states \(x_1, \ldots, x_T\):

\[-logp_\theta(x_0) = -log\int p_\theta(x_{0:T})dx_{1:T}\]

In other words, over all possible latent trajectories connecting \(x_T\) to \(x_0\) (golden paths). Of course, trajectories must follow a specific Markov factorization for both

complete forward process (red path):

\[q(x_{1:T} | x_0) := \prod_{t=1}^T q(x_t | x_{t-1}) \qquad q(x_t|x_{t-1}) := \mathcal{N}(x_t; \sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)\]

and complete reverse process (golden paths):

\[p_\theta(x_{0:T}) := p(x_T)\prod_{t=1}^T p_\theta(x_{t-1}|x_t) \qquad p_\theta(x_{t-1} | x_t) := \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))\]

With many diffusion steps, this high-dimensional marginalization becomes intractable. The likelihood requires integrating over the high-dimensional continuous latent trajectory.


So, instead of directly maximizing the exact likelihood, we find and optimize a tractable lower bound on it, hence the Evidence Lower Bound, or ELBO:

\[logp_\theta(x_0) \geq \mathbb{E}_{q(x_{1:T} | x_0)}\left[log \frac{p_\theta(x_{0:T})}{q(x_{1:T} | x_0)}\right]\]
  • The exact likelihood is difficult to compute because we need to marginalize over latent variables or latent trajectories.
  • The ELBO gives us a tractable objective that we can optimize.