Optimizing Diffusion Models in High-Dimensional Latent Spaces
This research explores how to make diffusion models, which generate images, work better when they use complex, high-dimensional internal representations from other AI models. It finds that by focusing on reconstructing the original image accurately, and then changing how the diffusion model learns in this altered space, we can significantly improve image generation quality.
Reading in Plain English mode — toggle below to switch
This research explores how to make diffusion models, which generate images, work better when they use complex, high-dimensional internal representations from other AI models. It finds that by focusing on reconstructing the original image accurately, and then changing how the diffusion model learns in this altered space, we can significantly improve image generation quality.
Imagine you're trying to draw a detailed map of a mountain range (generating an image). If your initial survey tools (pretrained encoders) only capture the major peaks but miss the intricate valleys and streams, your map will be incomplete. This paper suggests refining your survey tools to capture all details, which surprisingly makes the landscape 'simpler' to map effectively, allowing you to draw much better maps by focusing on the actual terrain rather than random noise.
How it works: Refining Latent Spaces for Diffusion
- 11. Start with a pretrained visual encoder that converts images into a compressed 'latent' representation. These encoders often discard fine details.
- 22. Finetune this encoder specifically to reconstruct the original image as faithfully as possible. This recovers lost details but, counterintuitively, reduces the effective complexity (dimensionality) of the latent space.
- 33. When using diffusion models (specifically flow matching) in this refined latent space, standard 'velocity prediction' methods become inefficient because they try to learn irrelevant noise directions.
- 44. Instead, switch to 'clean data parameterization' ($\boldsymbol{x}_{0}$-prediction), which directly predicts the original image from the noisy latent. This focuses the learning on the meaningful signal within the latent space.
- 55. This change in prediction strategy, combined with the refined latent space, consistently leads to better text-to-image generation performance.
x₀-prediction
Instead of predicting the 'velocity' or change needed to denoise a latent representation, $x_0$-prediction directly predicts the original, clean data point $x_0$ from a noisy version. This simplifies the learning task by focusing on the underlying signal.
In text-to-image generation, if you ask a model to create 'a photo of a cat wearing a tiny hat,' using the optimized latent space and $x_0$-prediction would result in a much sharper, more coherent image of the cat and hat, with fewer artifacts or blurry details, compared to models using standard methods with unoptimized latent spaces.
Finetuning visual encoders for faithful reconstruction, which reduces the effective dimensionality of the latent space, combined with using $\boldsymbol{x}_{0}$-prediction in flow matching, significantly improves text-to-image generation by focusing learning on the signal manifold.