Why Hidden Layers Matter: The XOR Problem

Linear models have limits

A single linear layer can only separate classes with a straight line (or flat plane). The classic XOR problem (output 1 when exactly one input is 1) cannot be separated that way. Adding a hidden layer with a non-linear activation lets the network bend its decision boundary and solve it. More layers and units let networks represent increasingly complex functions; in practice depth makes it efficient to build complex features from simpler ones.

Linear model versus one hidden layer on XOR, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. The linear model stalls at loss 0.693 (no better than guessing) and gets two of four cases wrong. A network with one hidden layer of 8 ReLU units reaches loss 0.000 and predicts 0, 1, 1, 0 exactly.

import torch
torch.manual_seed(0); torch.set_num_threads(1)
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])          # XOR
def train(model, steps=2000):
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(steps):
        opt.zero_grad()
        loss = torch.nn.functional.binary_cross_entropy_with_logits(model(X), y)
        loss.backward(); opt.step()
    pred = (model(X) > 0).float()
    return loss.item(), pred.flatten().tolist()
linear = torch.nn.Linear(2, 1)
mlp = torch.nn.Sequential(torch.nn.Linear(2, 8), torch.nn.ReLU(), torch.nn.Linear(8, 1))
for name, m in [("linear model", linear), ("one hidden layer", mlp)]:
    loss, pred = train(m)
    print(f"{name:<17} loss {loss:.3f} predictions {pred} (target [0, 1, 1, 0])")

Output:

linear model      loss 0.693 predictions [1.0, 0.0, 1.0, 0.0] (target [0, 1, 1, 0])
one hidden layer  loss 0.000 predictions [0.0, 1.0, 1.0, 0.0] (target [0, 1, 1, 0])

Width and depth are hyperparameters

Start small, then grow layers or units only when validation results show the model is underfitting.

त्वरित जाँच: Why can't a single linear layer solve XOR?

• XOR needs images
• XOR has too much data
• Linear layers have no weights
• XOR's classes cannot be separated by a straight line