requires_grad, backward and .grad

How autograd computes gradients

Set requires_grad=True on a tensor and PyTorch starts recording every operation that uses it. When you call .backward() on a scalar result, autograd walks those recorded operations in reverse, applies the chain rule, and stores d(result)/d(tensor) in the .grad attribute of each leaf tensor that has requires_grad=True. This is reverse-mode automatic differentiation: one backward pass gives the gradient of one scalar with respect to every parameter, whether there are ten or ten million. That is exactly the shape of the training problem, where one loss depends on many weights.

You rarely set requires_grad yourself: the weights inside nn.Linear, nn.Conv2d and other layers have it on by default, while input data does not.

Two rules trip up beginners. backward() needs a scalar; for a non-scalar output you must reduce it (for example with .mean()) or pass a gradient argument. And gradients accumulate: each backward() call adds to .grad instead of replacing it, which is why training loops clear gradients every step.

Forward records, backward replays

Leaf tensors with requires_grad take part in forward operations, which autograd records as a graph. The forward pass ends in a scalar loss. Calling backward replays the graph in reverse and fills the .grad of each leaf.

Your first gradient

For y = x^2 + 3x, calculus gives dy/dx = 2x + 3, which is 7 at x = 2. Autograd reaches the same number without you writing the derivative.

import torch

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x     # y = 4 + 6 = 10
y.backward()           # computes dy/dx and stores it in x.grad
print(y.item())        # 10.0
print(x.grad.item())   # 7.0, which is 2 * 2 + 3

Check a gradient numerically

A central difference approximates the derivative without any calculus. Comparing it with an analytic or autograd gradient is a standard way to test custom gradient code.

# Check the gradient numerically: (f(x+h) - f(x-h)) / 2h
def f(x):
    return x ** 2 + 3 * x

x, h = 2.0, 1e-5
numeric = (f(x + h) - f(x - h)) / (2 * h)
analytic = 2 * x + 3
print(round(numeric, 6), analytic)

Output:

7.0 7.0

त्वरित जाँच: Starting from the same leaf `x` (with `requires_grad=True`), you run the forward pass twice, building a new graph each time, and call `backward()` on each result without clearing `.grad` in between. What is in `x.grad`?

• Only the gradient from the second call
• The sum of the gradients from both calls
• None, because the second call resets it
• An error is raised on the second `backward()` call