[{"content":" Paper Denoising Diffusion Probabilistic Models Authors Jonathan Ho, Ajay Jain, Pieter Abbeel — UC Berkeley Published June 2020 · NeurIPS 2020 Code hojonathanho/diffusion (original TensorFlow release) In one sentence If you slowly destroy an image with noise, a network can learn to undo one small step of that destruction — and running those small steps backwards from pure static produces a brand-new image. The problem nobody had solved cleanly Generating a realistic image is hard for a reason that is easy to state and brutal in practice: the space of possible images is unimaginably large, and almost all of it is garbage. A 256×256 colour photo is roughly 200,000 numbers. Pick those numbers at random a trillion times and you will never once land on something that looks like a dog.\nSo the job of a generative model is to learn where the tiny, tangled, valid region of that space is, and then produce points inside it.\nBy 2020 there were three main answers, and each came with a tax:\nGANs produced beautiful images, but training them is a fight between two networks that can collapse without warning. They also tend to quietly ignore parts of the data — you get great dogs and no cats. VAEs trained reliably but produced blurry output, because they are effectively rewarded for hedging. Normalising flows gave you exact probabilities, but only if you cripple the architecture so that every layer stays invertible. DDPM\u0026rsquo;s contribution is that it sidesteps all three taxes. It trains with plain mean-squared error, it is about as stable as training an image classifier, it has no second network to fight with, and the samples are sharp.\nThe catch — and there is one — is that generating a single image takes a thousand forward passes. We will get to that.\nThe idea, in the form of an analogy Here is the whole paper as a thought experiment.\nTake a photograph. Add a tiny amount of static to it — so little that you can barely tell. Do it again. And again. A thousand times. By the end you are holding pure television snow; every trace of the original photo is gone.\nNow ask a much smaller question. Not \u0026ldquo;can you draw a photo?\u0026rdquo; but:\n\u0026ldquo;Here is a slightly noisy image. Can you tell me what noise I just added?\u0026rdquo;\nThat question is easy. It is a supervised learning problem with a free answer key, because you were the one who added the noise. You have infinite training data: take any image, add noise you generated yourself, and you know exactly what the right answer is.\nAnd here is the payoff. If a network can reliably answer that easy question, you can start from pure static — which costs nothing, it\u0026rsquo;s just a random number generator — and repeatedly ask \u0026ldquo;what noise is in here?\u0026rdquo;, subtract a bit of it, and ask again. A thousand small subtractions later, an image that never existed emerges.\nFORWARD PROCESS \u0026#8212; a fixed recipe, nothing is learned each step stirs in a little Gaussian noise x0 the real photo x100 still obvious x300 barely there x600 a hint of structure x1000 pure static REVERSE PROCESS \u0026#8212; this is the part we train a neural network guesses the noise that was added, and subtracts a little of it Top: the forward process. It is a fixed recipe with no learned parameters — you could implement it in five lines. Bottom: the reverse process. This is the only part that is a neural network, and it only ever has to undo one small step. The key move Learning to generate an image from nothing is very hard. Learning to make an image slightly less noisy is very easy. DDPM turns the first problem into a thousand copies of the second one.\nThe forward process is a recipe, not a model Let\u0026rsquo;s make \u0026ldquo;add a little noise\u0026rdquo; precise, because there is one detail that determines whether the whole thing works.\nAt each step ttt you shrink the image slightly toward zero and add a splash of Gaussian noise:\nxt=1−βt xt−1 + βt ϵ,ϵ∼N(0,I) x_t = \\sqrt{1-\\beta_t}\\; x_{t-1} \\;+\\; \\sqrt{\\beta_t}\\; \\epsilon, \\qquad \\epsilon \\sim \\mathcal{N}(0, I) xt​=1−βt​​xt−1​+βt​​ϵ,ϵ∼N(0,I)The numbers β1,…,βT\\beta_1, \\dots, \\beta_Tβ1​,…,βT​ are the noise schedule — a fixed list you choose in advance. DDPM uses T=1000T = 1000T=1000 steps with βt\\beta_tβt​ rising linearly from 0.00010.00010.0001 to 0.020.020.02. Nothing here is learned. It is a recipe.\nWhy shrink the image by 1−βt\\sqrt{1-\\beta_t}1−βt​​ instead of just adding noise on top? Because otherwise the pixel values would grow without bound as you keep adding. The shrink-and-add pairing is chosen so that the total variance stays fixed at 1 forever. The image doesn\u0026rsquo;t get louder, it gets replaced.\nThe shortcut that makes training possible If you had to actually run a thousand sequential steps to build one training example, this would be hopeless. You don\u0026rsquo;t. Because each step is Gaussian, a thousand of them compose into a single Gaussian, and you can jump straight to any timestep in one line:\nxt=αˉt x0 + 1−αˉt ϵ,αˉt=∏s=1t(1−βs) x_t = \\sqrt{\\bar{\\alpha}_t}\\; x_0 \\;+\\; \\sqrt{1-\\bar{\\alpha}_t}\\; \\epsilon, \\qquad \\bar{\\alpha}_t = \\prod_{s=1}^{t} (1 - \\beta_s) xt​=αˉt​​x0​+1−αˉt​​ϵ,αˉt​=s=1∏t​(1−βs​)Read that as a crossfade. The quantity αˉt\\sqrt{\\bar\\alpha_t}αˉt​​ is how much of the original image survives, and 1−αˉt\\sqrt{1-\\bar\\alpha_t}1−αˉt​​ is how much noise has taken its place. They always sum in quadrature to one — as one fades in, the other fades out.\nThis single formula is what makes training cheap. Want a training example at t=700t = 700t=700? One multiply, one add. No loop.\n1.0 0.5 0.0 fraction of the mix 0 200 400 600 800 1000 timestep t \u0026#8594; t \u0026#8776; 255: half signal, half noise and it is downhill from there noise mixed in original image left The crossfade over time under DDPM\u0026rsquo;s linear schedule. By tapprox255t \\\\approx 255tapprox255 the image is half signal and half noise; by t=600t = 600t=600 only about 16% of the original is left. Notice how much of the schedule is spent in the near-pure-noise region on the right — later papers (Nichol \u0026amp; Dhariwal) pointed out that this is mostly wasted, and fixed it with a cosine schedule. What the network actually has to do Now the learned half. The network sees a noisy image xtx_txt​ and the timestep ttt, and it outputs a guess of the noise that is in there. That guess is written ϵθ(xt,t)\\epsilon_\\theta(x_t, t)ϵθ​(xt​,t).\nThree things about this are worth pausing on.\nIt predicts the noise, not the image. You could equally well train it to output the clean image directly — the two are algebraically interchangeable, since rearranging the crossfade formula turns one into the other. Ho and colleagues tried both and found that predicting the noise trains noticeably better. The intuition people usually give: the noise is a fixed-scale, zero-mean target no matter what ttt is, while the clean image has structure that varies enormously across noise levels. The easier target wins.\nIt is told which timestep it is looking at. One single network handles all 1000 noise levels. An image at t=50t = 50t=50 needs a delicate touch; an image at t=900t = 900t=900 needs bold strokes. Without knowing ttt, the network would have to guess how much damage to undo. So ttt is converted into a vector — the same sinusoidal embedding as in Transformers — and injected into every block.\nIt is secretly pointing uphill. There is a deeper reading, which I will mention once and then leave alone: predicting the noise in xtx_txt​ is mathematically the same as estimating the direction in which the image becomes more probable under the data distribution. That direction has a name, the score, and it is why this line of work and the score-matching line of work turned out to be the same algorithm in different clothes. That connection is its own paper, and it is on the reading list.\nxt 32\u0026#178; 16\u0026#178; 8\u0026#178; 4\u0026#178; self-attention 8\u0026#178; 16\u0026#178; 32\u0026#178; \u0026#949;\u0026#770; skip connections \u0026#8212; fine detail is handed straight across shrink the picture, widen the channels rebuild it at full resolution t \u0026#8594; sinusoidal embedding \u0026#8594; small MLP the same network handles every noise level, so it has to be told which one it is looking at The architecture: a U-Net. The encoder shrinks the picture while widening the channels, letting the middle of the network reason about global layout; the decoder rebuilds full resolution; the skip connections hand fine detail straight across so it does not have to survive the squeeze. Nothing here is diffusion-specific — it is a segmentation architecture from 2015, borrowed wholesale. Training: five lines and a random number generator This is the part that surprises people who expect something exotic. One training step:\nGrab a real image x0x_0x0​ from your dataset. Pick a timestep ttt uniformly at random from 1 to 1000. Draw fresh noise ϵ∼N(0,I)\\epsilon \\sim \\mathcal{N}(0, I)ϵ∼N(0,I). Build the noisy image xtx_txt​ with the crossfade formula. Ask the network to name the noise, and punish it by mean-squared error. a real image x0 from your dataset fresh noise \u0026#949; drawn from N(0, I) + blend, using the recipe for t xt \u0026#8212; the noisy version \u0026#949;\u0026#952;(xt, t) a U-Net \u0026#8220;which noise is this?\u0026#8221; t ~ Uniform{1, \u0026#8230;, T} \u0026#949;\u0026#770; \u0026#8212; the guess \u0026#8214; \u0026#949; \u0026#8722; \u0026#949;\u0026#770; \u0026#8214;\u0026#178; plain mean-squared error \u0026#8212; that is the whole loss the noise we actually used \u0026#8212; the answer key One training step, end to end. Everything on the left is free — the noise is something you generated, so it doubles as the answer key. There is no adversary, no second network, no sampling loop during training. The entire loss function is this:\nL=Ex0,t,ϵ[ ∥ ϵ−ϵθ(xt, t) ∥2 ] L = \\mathbb{E}_{x_0, t, \\epsilon} \\Big[\\; \\big\\lVert\\, \\epsilon - \\epsilon_\\theta(x_t,\\, t) \\,\\big\\rVert^2 \\;\\Big] L=Ex0​,t,ϵ​[​ϵ−ϵθ​(xt​,t)​2]That is it. That is the whole objective. In code:\nt = torch.randint(0, T, (batch_size,), device=device) # random timesteps eps = torch.randn_like(x0) # the answer key x_t = abar[t].sqrt() * x0 + (1 - abar[t]).sqrt() * eps # the crossfade loss = F.mse_loss(model(x_t, t), eps) # that\u0026#39;s the loss Notice step 2: the timestep is random, not sequential. Each gradient step teaches the network about one randomly chosen noise level. Over a training run it sees all of them millions of times, and because the weights are shared, skill at one noise level transfers to its neighbours.\nWhere the mean-squared error actually comes from (optional) The paper does not begin with MSE — it begins with a variational bound, the same machinery as a VAE, treating the whole noising chain as a 1000-layer latent variable model. Squeezing that bound down gives a sum of KL divergences, one per timestep, each comparing two Gaussians.\nThe KL divergence between two Gaussians with the same variance is just the squared distance between their means. So each term collapses into a squared error. Grind through the algebra and you land at a weighted version of the loss above:\nLweighted=∑twt ∥ϵ−ϵθ(xt,t)∥2 L_{\\text{weighted}} = \\sum_t w_t \\, \\big\\lVert \\epsilon - \\epsilon_\\theta(x_t, t) \\big\\rVert^2 Lweighted​=t∑​wt​​ϵ−ϵθ​(xt​,t)​2with weights wtw_twt​ that heavily emphasise the very small timesteps. The paper\u0026rsquo;s most useful practical finding is that throwing the weights away and setting every wt=1w_t = 1wt​=1 works better. Uniform weighting stops the model obsessing over imperceptible high-frequency detail at t≈0t \\approx 0t≈0 and makes it spend capacity on the noisy steps where the large-scale structure of the image is decided.\nSo the principled derivation gets you to a weighted MSE, and then an empirical hack gets you to plain MSE. The simple thing is the thing that works.\nSampling: a thousand small corrections Training is one step. Generating is the loop.\nStart with xTx_TxT​ drawn straight from a random number generator — pure static, no model involved. Then for t=Tt = Tt=T down to 111:\nxt−1=1αt(xt−βt1−αˉt ϵθ(xt,t)) + σtz x_{t-1} = \\frac{1}{\\sqrt{\\alpha_t}} \\left( x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\, \\epsilon_\\theta(x_t, t) \\right) \\;+\\; \\sigma_t z xt−1​=αt​​1​(xt​−1−αˉt​​βt​​ϵθ​(xt​,t))+σt​zIgnore the constants and read the shape of it: take the current image, subtract a scaled portion of the predicted noise, then add a little fresh noise back.\nxT ~ N(0, I) free, from a random number generator \u0026#949;\u0026#952;(xt, t) guess all the noise sitting in this image subtract a slice not all of it \u0026#8212; just one step\u0026#8217;s worth add a pinch back fresh noise, smaller than what you removed t \u0026#8592; t \u0026#8722; 1, and around again \u0026#8212; one thousand times in the original paper at t = 1 x0 a brand-new image The sampling loop. The network is called once per step — a thousand times for one image. This is why DDPM sampling is slow, and why the next three years of research were largely about making this loop shorter. Why on earth do we add noise back in? This is the step that makes everyone stop and re-read, so it is worth answering properly.\nThe network\u0026rsquo;s output is a best guess, and a best guess is an average. At t=900t = 900t=900, when the image is nearly pure static, there are countless different pictures that could plausibly be hiding underneath. The network cannot know which, so it predicts something in the middle of all of them — which, averaged over countless pictures, is a grey blur.\nIf you followed that average deterministically at every step, you would slide straight down to the blurriest, most average-looking output the model can produce. Every sample would look the same, and that same thing would look like fog. This is exactly the failure mode that makes VAE samples blurry.\nInjecting a small amount of fresh randomness at each step breaks the tie. It says: of the many images consistent with what we have so far, commit — arbitrarily — a little further in this direction. Do that a thousand times and each early arbitrary nudge gets amplified and made coherent by the steps that follow. The result is a sharp, specific image rather than the average of all possible images.\nA useful way to hold it The predicted noise tells you which way to walk. The injected noise decides which valid image you end up at. Remove the second and every walk ends in the same grey place.\nOn the final step (t=1t = 1t=1) the noise is skipped, because you want to hand back a clean image rather than a slightly speckled one.\nWhat it achieved DDPM hit an FID of 3.17 on unconditional CIFAR-10, which beat every GAN of the era on that benchmark and was, at the time, a genuinely startling result for a non-adversarial model. It also produced convincing 256×256 faces on LSUN.\nThe number matters less than what it signalled. A simple, stable, MSE-trained model had just walked past years of adversarial-training research. Everything that followed — Stable Diffusion, Midjourney, DALL·E 2 and 3, Imagen, Sora — descends from this paper.\nWhat it got wrong, and who fixed it Being honest about the limitations is the most useful part of reading a paper, because the limitations are the roadmap for everything that came next.\nIt is agonisingly slow. One thousand sequential forward passes per image, and they cannot be parallelised — each depends on the last. Generating a single CIFAR-10 image took minutes. → Fixed by DDIM, which reaches comparable quality in 20–50 steps with no retraining, and later by distillation methods that get to single digits.\nThe noise schedule is wasteful. As the crossfade plot shows, the linear schedule reaches near-total noise well before t=1000t = 1000t=1000, so a large fraction of the steps teach the network almost nothing. → Fixed by Improved DDPM with a cosine schedule.\nIt works on raw pixels. Every one of those thousand forward passes runs at full image resolution, which makes high-resolution generation ruinously expensive. → Fixed by Latent Diffusion: compress the image first, diffuse in the small latent space, decode at the end. This is the change that put Stable Diffusion on consumer GPUs.\nThere is no way to tell it what to draw. DDPM is purely unconditional. You get an image from the training distribution; you cannot ask for a specific one. → Fixed by classifier guidance and then, much more cleanly, classifier-free guidance.\nEvery one of these is on the reading list.\nA complete implementation you can run Small enough to read in one sitting, real enough to produce recognisable MNIST digits in a few minutes on a single GPU. The diffusion logic is about fifteen lines; the rest is the U-Net.\nimport math import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.data import DataLoader from torchvision import datasets, transforms device = \u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34; # ---------------------------------------------------------------- the schedule T = 1000 betas = torch.linspace(1e-4, 0.02, T, device=device) alphas = 1.0 - betas abar = torch.cumprod(alphas, dim=0) # \\bar{alpha}_t def add_noise(x0, t, eps): \u0026#34;\u0026#34;\u0026#34;The crossfade: jump straight to timestep t in one operation.\u0026#34;\u0026#34;\u0026#34; a = abar[t].view(-1, 1, 1, 1) return a.sqrt() * x0 + (1 - a).sqrt() * eps # ------------------------------------------------------------------ the model class TimeEmbedding(nn.Module): \u0026#34;\u0026#34;\u0026#34;Turn the integer t into a vector the network can condition on.\u0026#34;\u0026#34;\u0026#34; def __init__(self, dim): super().__init__() self.dim = dim self.mlp = nn.Sequential( nn.Linear(dim, dim * 4), nn.SiLU(), nn.Linear(dim * 4, dim * 4) ) def forward(self, t): half = self.dim // 2 freqs = torch.exp( -math.log(10000) * torch.arange(half, device=t.device) / half ) ang = t[:, None].float() * freqs[None] return self.mlp(torch.cat([ang.sin(), ang.cos()], dim=-1)) class Block(nn.Module): \u0026#34;\u0026#34;\u0026#34;Two convolutions, with the timestep added in between.\u0026#34;\u0026#34;\u0026#34; def __init__(self, cin, cout, tdim): super().__init__() self.conv1 = nn.Conv2d(cin, cout, 3, padding=1) self.conv2 = nn.Conv2d(cout, cout, 3, padding=1) self.norm1 = nn.GroupNorm(8, cout) self.norm2 = nn.GroupNorm(8, cout) self.temb = nn.Linear(tdim, cout) self.skip = nn.Conv2d(cin, cout, 1) if cin != cout else nn.Identity() def forward(self, x, t): h = F.silu(self.norm1(self.conv1(x))) h = h + self.temb(t)[:, :, None, None] h = F.silu(self.norm2(self.conv2(h))) return h + self.skip(x) class UNet(nn.Module): def __init__(self, ch=64, tdim=64): super().__init__() self.time = TimeEmbedding(tdim) td = tdim * 4 self.inp = nn.Conv2d(1, ch, 3, padding=1) self.d1 = Block(ch, ch, td) self.d2 = Block(ch, ch * 2, td) self.mid = Block(ch * 2, ch * 2, td) self.u2 = Block(ch * 4, ch, td) self.u1 = Block(ch * 2, ch, td) self.out = nn.Conv2d(ch, 1, 3, padding=1) self.down = nn.AvgPool2d(2) def forward(self, x, t): te = self.time(t) h0 = self.inp(x) h1 = self.d1(h0, te) # 28 x 28 h2 = self.d2(self.down(h1), te) # 14 x 14 m = self.mid(self.down(h2), te) # 7 x 7 u2 = self.u2(torch.cat([F.interpolate(m, scale_factor=2), h2], 1), te) u1 = self.u1(torch.cat([F.interpolate(u2, scale_factor=2), h1], 1), te) return self.out(u1) # ---------------------------------------------------------------- the training tf = transforms.Compose( [transforms.ToTensor(), transforms.Normalize([0.5], [0.5])] # -\u0026gt; [-1, 1] ) loader = DataLoader( datasets.MNIST(\u0026#34;./data\u0026#34;, train=True, download=True, transform=tf), batch_size=128, shuffle=True, drop_last=True, ) model = UNet().to(device) opt = torch.optim.Adam(model.parameters(), lr=2e-4) for epoch in range(20): for x0, _ in loader: x0 = x0.to(device) t = torch.randint(0, T, (x0.size(0),), device=device) eps = torch.randn_like(x0) loss = F.mse_loss(model(add_noise(x0, t, eps), t), eps) opt.zero_grad() loss.backward() opt.step() print(f\u0026#34;epoch {epoch} loss {loss.item():.4f}\u0026#34;) # ---------------------------------------------------------------- the sampling @torch.no_grad() def sample(n=16): x = torch.randn(n, 1, 28, 28, device=device) for t in reversed(range(T)): tt = torch.full((n,), t, device=device, dtype=torch.long) eps_hat = model(x, tt) # subtract a slice of the predicted noise x = (x - betas[t] / (1 - abar[t]).sqrt() * eps_hat) / alphas[t].sqrt() # add a pinch back, except on the very last step if t \u0026gt; 0: x = x + betas[t].sqrt() * torch.randn_like(x) return x.clamp(-1, 1) images = sample() Two things to watch while it runs. The loss drops fast and then looks stuck around 0.03–0.05 — that is normal and is not a bug. The loss is averaged over random timesteps, and the high-noise ones are genuinely close to unpredictable, so there is an irreducible floor. Judge progress by looking at samples, not at the number.\nThings that confused me A few questions I had to work through, in case you have the same ones.\nIf the network can predict all the noise in xtx_txt​, why not subtract all of it and finish in one step? You can, and the code is a one-liner. The result is blurry, for the reason discussed above: at high ttt the network\u0026rsquo;s prediction is an average over every image that could be hiding in there. The thousand-step loop works because each step only asks the network to be right about a small correction, and each step\u0026rsquo;s injected randomness commits to a specific answer that later steps can build on. (This one-shot estimate is not useless, though — DDIM uses exactly this quantity, just more carefully.)\nWhy random timesteps in training but sequential ones in sampling? They are different jobs. Training needs an unbiased sample of the loss across all noise levels, and random ttt gives that at one forward pass per example. Sampling needs an actual trajectory, and the only way down the chain is one rung at a time.\nDoes the network memorise 1000 separate denoisers? No, and that is the point. Denoising at t=500t = 500t=500 and t=501t = 501t=501 are nearly identical tasks. Weight sharing plus the timestep embedding lets the network learn one smooth family of denoisers indexed by ttt, which is enormously more sample-efficient than 1000 independent models.\nIs ϵ\\epsilonϵ during sampling the same ϵ\\epsilonϵ from training? No. During training, ϵ\\epsilonϵ is a specific noise sample you generated and kept as the answer key. During sampling there is no answer key — the zzz added at each step is fresh randomness whose only job is to keep the walk from collapsing to the mean.\nWhere to go next The paper itself: arXiv:2006.11239. It is short. With the intuition above in place, the two algorithm boxes on page 4 read almost like pseudocode you already know. Lilian Weng\u0026rsquo;s survey is the standard reference for the full derivations. Next on this site: DDIM, which takes the same trained network and gets comparable samples in 50 steps instead of 1000. See the reading list for what is coming after that. Found an error, or an explanation that did not land? Tell me — I would rather fix it than leave it wrong.\n","permalink":"https://woletee.github.io/diffusion-papers-/diffusion/ddpm/","summary":"The 2020 paper that started the diffusion era. Wreck an image with noise, teach a network to guess the noise, then run the wrecking process backwards. Explained with pictures, almost no maths, and code you can run.","title":"DDPM: How Adding Noise Taught Machines to Draw"},{"content":"Paper Attention Is All You Need\nThe problem Explain the problem using your own words.\nMain idea Describe the important concepts, diagrams, and equations.\nPython implementation import numpy as np def softmax(values): exp_values = np.exp(values - np.max(values)) return exp_values / exp_values.sum() print(softmax(np.array([1.0, 2.0, 3.0]))) What I learned Summarize the practical lessons and link to your complete GitHub project.\n","permalink":"https://woletee.github.io/diffusion-papers-/understanding-transformers/","summary":"\u003ch2 id=\"paper\"\u003ePaper\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"https://arxiv.org/abs/1706.03762\"\u003eAttention Is All You Need\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"the-problem\"\u003eThe problem\u003c/h2\u003e\n\u003cp\u003eExplain the problem using your own words.\u003c/p\u003e\n\u003ch2 id=\"main-idea\"\u003eMain idea\u003c/h2\u003e\n\u003cp\u003eDescribe the important concepts, diagrams, and equations.\u003c/p\u003e\n\u003ch2 id=\"python-implementation\"\u003ePython implementation\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"nn\"\u003enumpy\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"nn\"\u003enp\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edef\u003c/span\u003e \u003cspan class=\"nf\"\u003esoftmax\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003evalues\u003c/span\u003e\u003cspan class=\"p\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eexp_values\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003enp\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eexp\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003evalues\u003c/span\u003e \u003cspan class=\"o\"\u003e-\u003c/span\u003e \u003cspan class=\"n\"\u003enp\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003emax\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003evalues\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eexp_values\u003c/span\u003e \u003cspan class=\"o\"\u003e/\u003c/span\u003e \u003cspan class=\"n\"\u003eexp_values\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003esum\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003esoftmax\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enp\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003earray\u003c/span\u003e\u003cspan class=\"p\"\u003e([\u003c/span\u003e\u003cspan class=\"mf\"\u003e1.0\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mf\"\u003e2.0\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mf\"\u003e3.0\u003c/span\u003e\u003cspan class=\"p\"\u003e])))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"what-i-learned\"\u003eWhat I learned\u003c/h2\u003e\n\u003cp\u003eSummarize the practical lessons and link to your complete GitHub project.\u003c/p\u003e","title":"Understanding Transformers From a Research Paper"},{"content":"I read research papers and write them up the way I wish they had been explained to me the first time: the question before the notation, a picture wherever a picture helps, and code short enough to read in one sitting.\nThe main project here is a growing library of diffusion model papers — starting with the 2020 paper that made the whole approach work, and following the chain of ideas forward to the systems behind today\u0026rsquo;s image and video generators.\nTwo rules I try to hold myself to:\nNo hand-waving. If I cannot explain why a step is there, I say so rather than papering over it. Say what the paper got wrong. The limitations are the interesting part — they are what the next paper is about. Corrections, arguments, and requests for papers to cover are all welcome: open an issue.\nElsewhere: GitHub.\n","permalink":"https://woletee.github.io/diffusion-papers-/about/","summary":"About this site","title":"About"}]