The dynamic computation graph

Define-by-run graphs

Autograd builds the computation graph while your code runs, a style called define-by-run. Every forward pass creates a fresh graph, so ordinary Python control flow works: an if that depends on the data, or a loop whose length changes per input, simply produces a different graph that step.

Tensors you create directly with requires_grad=True, and the parameters of layers, are leaf tensors. Results of operations on them are non-leaf tensors, and each carries a grad_fn attribute such as MulBackward0 or AddBackward0 that knows how to send gradients back to its inputs.

By default, only leaves get .grad filled in; intermediate gradients are used and thrown away to save memory. Call .retain_grad() on an intermediate if you want to inspect it. After backward() finishes, the graph's saved buffers are freed, so calling backward() through the same graph again raises an error unless the first call used retain_graph=True, which you rarely need.

Tracing a backward pass

Follow d = a * b + a. The forward pass builds two graph nodes; the backward pass sends gradients through both paths into a and adds them.

import torch

a = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(4.0, requires_grad=True)
c = a * b
d = c + a
d.backward()
print(a.grad.item(), b.grad.item())

1. Line 3: `a` is created by you with requires_grad, so it is a leaf. Its `.grad` starts as None.
2. Line 4: `b` is a second leaf. Nothing has been computed yet, so the graph is empty.
3. Line 5: Multiplying records a MulBackward0 node. It saves a and b because the derivative of a*b needs them.
4. Line 6: Adding records AddBackward0. Note that `a` now feeds the graph through two paths: directly and through c.
5. Line 7: Backward starts with dd/dd = 1. Addition passes 1 to both c and a. Multiplication sends 1*b = 4 to a and 1*a = 3 to b. The two paths into a add up: 1 + 4 = 5.
6. Line 8: Only leaves have `.grad`. The printed values match the calculus: dd/da = b + 1 = 5 and dd/db = a = 3.

Why dynamic graphs help you debug

Because the graph is ordinary Python execution, you can put print(x.shape) or breakpoint() inside forward and inspect real values. If a tensor unexpectedly has grad_fn=None when you expected gradients, something upstream was detached or computed under no_grad.

Quick check: By default, which tensors have their `.grad` attribute populated after `loss.backward()`?

• Every tensor in the graph, including intermediates
• Only the loss tensor itself
• Only tensors created inside `torch.no_grad()`
• Leaf tensors that have `requires_grad=True`