# The dynamic computation graph — PyTorch

Source: https://www.skillbyai.com/en/pytorch/pt-computation-graph

> Explain leaf and non-leaf tensors, grad_fn and define-by-run graphs, and trace a backward pass through a small example.

## Define-by-run graphs

Autograd builds the computation graph **while your code runs**, a style called define-by-run. Every forward pass creates a fresh graph, so ordinary Python control flow works: an `if` that depends on the data, or a loop whose length changes per input, simply produces a different graph that step.

Tensors you create directly with `requires_grad=True`, and the parameters of layers, are **leaf** tensors. Results of operations on them are **non-leaf** tensors, and each carries a `grad_fn` attribute such as `MulBackward0` or `AddBackward0` that knows how to send gradients back to its inputs.

By default, only leaves get `.grad` filled in; intermediate gradients are used and thrown away to save memory. Call `.retain_grad()` on an intermediate if you want to inspect it. After `backward()` finishes, the graph's saved buffers are freed, so calling `backward()` through the same graph again raises an error unless the first call used `retain_graph=True`, which you rarely need.

## Tracing a backward pass

Follow `d = a * b + a`. The forward pass builds two graph nodes; the backward pass sends gradients through both paths into `a` and adds them.

```python
import torch

a = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(4.0, requires_grad=True)
c = a * b
d = c + a
d.backward()
print(a.grad.item(), b.grad.item())
```

- Line 3: `a` is created by you with requires_grad, so it is a leaf. Its `.grad` starts as None.
- Line 4: `b` is a second leaf. Nothing has been computed yet, so the graph is empty.
- Line 5: Multiplying records a MulBackward0 node. It saves a and b because the derivative of a*b needs them.
- Line 6: Adding records AddBackward0. Note that `a` now feeds the graph through two paths: directly and through c.
- Line 7: Backward starts with dd/dd = 1. Addition passes 1 to both c and a. Multiplication sends 1*b = 4 to a and 1*a = 3 to b. The two paths into a add up: 1 + 4 = 5.
- Line 8: Only leaves have `.grad`. The printed values match the calculus: dd/da = b + 1 = 5 and dd/db = a = 3.

## Why dynamic graphs help you debug

Because the graph is ordinary Python execution, you can put `print(x.shape)` or `breakpoint()` inside `forward` and inspect real values. If a tensor unexpectedly has `grad_fn=None` when you expected gradients, something upstream was detached or computed under `no_grad`.

**Quiz:** By default, which tensors have their `.grad` attribute populated after `loss.backward()`?

- [ ] Every tensor in the graph, including intermediates
- [ ] Only the loss tensor itself
- [ ] Only tensors created inside `torch.no_grad()`
- [x] Leaf tensors that have `requires_grad=True`

*Answer:* Leaf tensors that have `requires_grad=True`. Intermediate gradients are discarded to save memory unless you call `retain_grad()`. Leaves with requires_grad, such as model parameters, keep theirs in `.grad`.