The training loop idea, without PyTorch

Learning is repeated small corrections

In supervised learning, a model is a function f(x; w) with adjustable parameters w. Given an input x it makes a prediction. A loss function turns the gap between prediction and the correct answer into one number: lower is better.

The gradient of the loss with respect to a parameter says how the loss changes if that parameter increases slightly. Moving each parameter a small step against its gradient lowers the loss. That is gradient descent: w = w - lr * grad, where the learning rate lr sets the step size. Too large and the loss jumps around or explodes; too small and training crawls.

In practice you compute the loss on a mini-batch of examples, update, and repeat. One full pass over the training data is an epoch. PyTorch maps these ideas directly: DataLoader gives batches, an nn.Module is f, a loss module scores predictions, loss.backward() computes gradients, and optimizer.step() applies the update.

Gradient descent in plain Python

This fits y = w * x to data generated with w = 2. The gradient is derived by hand here; in PyTorch, autograd will replace that line and an optimizer will replace the update.

# Fit y = w * x to data generated with w = 2, using plain Python.
xs = [1.0, 2.0, 3.0, 4.0]
ys = [2.0, 4.0, 6.0, 8.0]
w, lr = 0.0, 0.01

for epoch in range(1, 51):
    # dL/dw for mean squared error: mean(2 * (w*x - y) * x)
    grad = sum(2 * (w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
    w -= lr * grad
    if epoch % 10 == 0:
        loss = sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
        print(f"epoch {epoch}: w={w:.4f} loss={loss:.6f}")

Output:

epoch 10: w=1.6063 loss=1.162786
epoch 20: w=1.9225 loss=0.045069
epoch 30: w=1.9847 loss=0.001747
epoch 40: w=1.9970 loss=0.000068
epoch 50: w=1.9994 loss=0.000003

Adjusting sugar in chai

You taste the chai (compute the loss), notice it is not sweet enough (the gradient tells you the direction), and add a little sugar (the update). The spoon size is the learning rate: a big spoon overshoots and makes it too sweet, a tiny spoon takes forever. You keep tasting and adjusting until it is right.

Everything else is automation

Keep this loop in your head for the whole course. Autograd automates the grad = ... line for any model, torch.optim automates w -= lr * grad with smarter rules, and DataLoader automates feeding batches. When something breaks, ask which of these four steps went wrong.

Quick check: In gradient descent, what does the learning rate control?

• The size of each parameter update step
• How many examples are in one batch
• How many epochs training runs for
• The direction in which parameters move