# Tensors and Shapes — Deep Learning & Neural Networks

Source: https://www.skillbyai.com/en/deep-learning/w-tensors

> Most bugs are shape bugs.

## Multi-dimensional arrays with a convention

A **tensor** is a multi-dimensional array with a data type and a device (CPU or GPU). Deep learning code follows shape conventions: images as (batch, channels, height, width), sequences as (batch, length, features) when `batch_first=True`, and fully connected layers expect (batch, features). Matrix multiplication needs the inner dimensions to match; **broadcasting** stretches smaller tensors (like a bias vector) across a batch. Printing shapes at each step is the fastest way to understand and debug a model.

## Tensors, modules, loops

Every PyTorch project uses the same building blocks: tensors with shapes, modules with parameters, and a training loop.

![Three ideas: tensors, modules and loops, batches and devices.](assets/figures/deep-learning/section-3-map.svg) — Figure 3.1 — Tensors, modules and loops.

## Shapes, flattening and broadcasting, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. A batch of 32 RGB 28x28 images has 75,264 numbers. Flattening gives 32 rows of 2,352 values; multiplying a (32, 784) batch by a (784, 10) weight matrix gives (32, 10), and adding a bias of length 10 broadcasts across the batch.

```python
import torch
batch = torch.zeros(32, 3, 28, 28)          # 32 images, 3 channels, 28x28 pixels
print("shape:", tuple(batch.shape), "| dtype:", batch.dtype, "| elements:", batch.numel())
flat = batch.view(32, -1)
print("flattened per image:", tuple(flat.shape))
a = torch.randn(32, 784); W = torch.randn(784, 10)
print("matrix multiply (32x784) @ (784x10) ->", tuple((a @ W).shape))
print("broadcast add of a bias (10,) ->", tuple((a @ W + torch.zeros(10)).shape))
```

Output:

```
shape: (32, 3, 28, 28) | dtype: torch.float32 | elements: 75264
flattened per image: (32, 2352)
matrix multiply (32x784) @ (784x10) -> (32, 10)
broadcast add of a bias (10,) -> (32, 10)
```

## Assert shapes in forward

Add a few assert x.shape[1:] == (...) checks while developing; they turn silent shape mistakes into clear errors.

**Quiz:** What shape convention do PyTorch image layers expect?

- [ ] (channels, batch)
- [ ] (height, width) only
- [x] (batch, channels, height, width)
- [ ] (pixels, labels)

*Answer:* (batch, channels, height, width). NCHW is the default layout.