Text Conditioning, Latent Diffusion and Guidance

Steering the denoiser with words

A text-to-image model needs the prompt to influence every denoising step. The prompt is encoded by a text encoder (CLIP's text tower, T5, or similar), and the denoiser reads those embeddings through cross-attention. Running diffusion on full-resolution pixels is expensive, so latent diffusion (the idea behind Stable Diffusion) first trains a VAE that compresses images into a much smaller latent tensor; diffusion happens in that latent space and the VAE decoder converts the final latent back into pixels. Classifier-free guidance (CFG) strengthens prompt adherence: during training the prompt is sometimes dropped, so the same network learns both conditional and unconditional predictions. At generation time it predicts noise both ways and moves further in the direction of the prompt by a guidance scale w. Higher w follows the prompt more literally but can oversaturate images and reduce variety; very low w ignores the prompt.

Classifier-free guidance arithmetic

w = 1 returns the conditional prediction unchanged. Larger w extrapolates past it, away from the unconditional prediction.

import numpy as np

eps_uncond = np.array([0.10, -0.20, 0.05])   # noise predicted with an empty prompt
eps_cond = np.array([0.30, -0.10, 0.00])     # noise predicted with the real prompt

for w in [1.0, 3.0, 7.5]:
    eps = eps_uncond + w * (eps_cond - eps_uncond)
    print(w, eps.round(3))

Output:

1.0 [ 0.3 -0.1  0. ]
3.0 [ 0.7  0.1 -0.1]
7.5 [ 1.6    0.55  -0.325]

Generating with diffusers

A typical diffusers call. It downloads a large checkpoint and needs a GPU, so no output is shown.

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
).to("cuda")

image = pipe(
    prompt="a watercolour painting of a Kerala houseboat at sunset",
    negative_prompt="blurry, low quality",
    num_inference_steps=30,
    guidance_scale=7.0,
).images[0]
image.save("houseboat.png")

Seeds make experiments fair

Pass a torch.Generator with a fixed seed when comparing prompts or guidance scales. Otherwise every run starts from different noise, and you cannot tell whether your change or the random start caused the difference.

Quick check: Why does latent diffusion run the denoising in a VAE's latent space?

• Latents are text, which is easier to generate
• It removes the need for a text encoder
• The latent is much smaller than the pixel image, so each step is cheaper
• It guarantees photorealistic output