Diffusion Models for Image Generation

Learning to remove noise

A diffusion model generates images by learning to reverse a gradual noising process. In the forward process, you take a real image and add a little Gaussian noise over many steps (for example 1,000) until only noise remains; this needs no learning, and a closed formula jumps straight to any step t. In the reverse process, a neural network, traditionally a U-Net and increasingly a transformer, is trained to predict the noise that was added at step t, given the noisy image and t. The training loss is simply the mean squared error between predicted and actual noise. To generate, you start from pure noise and repeatedly subtract the predicted noise, step by step, until an image appears. Samplers such as DDIM and newer solvers reach good quality in tens of steps instead of a thousand. Generation is slower than a single forward pass because the network runs once per step.

The diffusion round trip

Training noises real images and teaches the network to predict that noise; generation runs the denoiser repeatedly from pure noise, guided by the text prompt.

How much signal survives the forward process

With a linear schedule, the clean image dominates early, and by the last step almost nothing of it is left.

import numpy as np

T = 1000
betas = np.linspace(1e-4, 0.02, T)       # linear noise schedule
alpha_bar = np.cumprod(1 - betas)        # how much signal survives to step t

rng = np.random.default_rng(0)
x0 = np.ones(4)                          # a tiny "clean image"
for t in [0, 250, 500, 999]:
    eps = rng.standard_normal(4)
    xt = np.sqrt(alpha_bar[t]) * x0 + np.sqrt(1 - alpha_bar[t]) * eps
    print(f"t={t:4d} signal={np.sqrt(alpha_bar[t]):.3f} noise={np.sqrt(1 - alpha_bar[t]):.3f}")

Output:

t=   0 signal=1.000 noise=0.010
t= 250 signal=0.722 noise=0.692
t= 500 signal=0.279 noise=0.960
t= 999 signal=0.006 noise=1.000

Quick check: During diffusion training, what does the network usually learn to predict?

• The caption of the image
• The noise that was added at step t
• The next pixel in raster order
• The class label of the image