NEPA: Adaptive Conditioning for Diffusion Transformers
This paper introduces Next-Embedding Predictive Autoregression (NEPA), a method that improves image generation in diffusion transformers by dynamically predicting and adapting the conditioning signal at each denoising step. Unlike traditional methods that use a static condition, NEPA allows the generator to respond more effectively to the evolving noisy state of the image during synthesis.
Reading in Plain English mode — toggle below to switch
This paper introduces Next-Embedding Predictive Autoregression (NEPA), a method that improves image generation in diffusion transformers by dynamically predicting and adapting the conditioning signal at each denoising step. Unlike traditional methods that use a static condition, NEPA allows the generator to respond more effectively to the evolving noisy state of the image during synthesis.
Imagine a sculptor refining a clay model. Instead of having a fixed blueprint from the start, NEPA is like a sculptor who continuously re-evaluates and updates their plan based on the current state of the clay, leading to a more precise and detailed final sculpture. The 'blueprint' (conditioning) adapts as the 'sculpture' (image) takes shape.
How NEPA Enhances Image Generation
- 11. **Traditional Diffusion:** In standard diffusion transformers, a fixed embedding (e.g., class label or text prompt) guides the image generation process from start to finish.
- 22. **The NEPA Idea:** NEPA proposes that instead of a static embedding, a dynamically predicted embedding, which changes with each denoising step, could provide a more effective conditioning signal.
- 33. **Predicting Next Embeddings:** A dedicated Transformer model (the NEPA model) is trained to predict the 'next' continuous embeddings. These 'next' embeddings represent the cleaner image state that follows the current noisy one.
- 44. **Multi-Embedding Prediction:** NEPA predicts multiple embeddings simultaneously: the condition embedding, and the embeddings corresponding to the noisy and clean image states.
- 55. **Adaptive Conditioning:** During image generation, the DiT generator is conditioned on these newly predicted embeddings at every denoising step. This means the conditioning signal continuously adapts to the current noisy state of the image being generated.
- 66. **Improved Efficiency:** This adaptive conditioning allows NEPA-DiT-XL to achieve competitive performance (e.g., FID 1.32) with significantly less training compute compared to prior methods like REPA.
E_pᵣₑ_d = f_N_E_P_A(E_c_oₙ_d, Eₙ_oᵢ_s_y)
$E_{pred}$ represents the predicted next embeddings, $f_{NEPA}$ is the NEPA Transformer model, $E_{cond}$ is the initial conditioning embedding, and $E_{noisy}$ is the embedding of the current noisy image.
In class-conditional ImageNet generation, NEPA allows the model to refine the image based on a continuously updated understanding of what a 'cat' or 'dog' should look like at different stages of denoising, rather than relying on a single, static 'cat' or 'dog' embedding throughout the entire process.
Dynamically predicting and adapting the conditioning signal at each denoising step significantly improves the quality and efficiency of image generation in diffusion transformers.