Almost every image, video, and audio generator you have used in the last few years runs on the same idea: learn to turn static into a picture, one small step at a time. That idea is barely five years old, and the papers that built it are readable — once someone strips away the notation and tells you what is actually going on.
That is what this section is for. One paper at a time, I rewrite the paper the way I wish it had been explained to me:
- the question it was trying to answer, and why the previous answers were unsatisfying
- the one idea that makes the paper work, in words before symbols
- pictures, because most of these papers are much easier to see than to read
- code you can run, small enough to read in one sitting
- what it got wrong, and which paper fixed it
You do not need a measure-theory background. If you are comfortable with “a neural network takes an input and produces an output” and you can squint at a normal distribution without flinching, you have enough.
Start here
If you read nothing else, read the first one. Every other paper on this list is a modification of it.
- Denoising Diffusion Probabilistic Models Ho, Jain & Abbeel · 2020 Explained The paper that made diffusion work. Destroy an image with noise, train a network to undo one step of the damage, then run it backwards from pure static.
The rest of the map
These are the papers I am working through next, roughly in the order that makes them easiest to understand. Each builds on the one above it.
- Denoising Diffusion Implicit Models (DDIM) Song, Meng & Ermon · 2020 Next up Sampling in 50 steps instead of 1000, without retraining anything. Also the paper that makes diffusion models deterministic enough to interpolate between images.
- Improved Denoising Diffusion Probabilistic Models Nichol & Dhariwal · 2021 Planned The engineering pass: the cosine noise schedule, learning the variance instead of fixing it, and why the original schedule wastes steps.
- Score-Based Generative Modeling through SDEs Song et al. · 2021 Planned The view from above: DDPM and score matching are the same algorithm, and both are a differential equation you can solve with any solver you like.
- Diffusion Models Beat GANs on Image Synthesis Dhariwal & Nichol · 2021 Planned Classifier guidance, and the moment diffusion overtook GANs on ImageNet.
- Classifier-Free Diffusion Guidance Ho & Salimans · 2022 Planned Two forward passes and a subtraction, and suddenly prompts are obeyed. The single most consequential trick in text-to-image generation.
- High-Resolution Image Synthesis with Latent Diffusion Models Rombach et al. · 2022 Planned Stable Diffusion. Do the diffusion in a compressed latent space instead of on pixels, and the whole thing fits on a consumer GPU.
- Scalable Diffusion Models with Transformers (DiT) Peebles & Xie · 2022 Planned Replace the U-Net with a transformer and the scaling laws come along for free. The backbone behind Sora and most modern video models.
- Flow Matching / Rectified Flow Lipman et al. · Liu et al. · 2022 Planned A cleaner reformulation that quietly replaced diffusion training in most 2024-and-later systems. Straight paths instead of a noisy random walk.
Missing a paper you think belongs here, or spotted something I got wrong? Open an issue — corrections are genuinely welcome.