# Images as Tensors — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-images-as-tensors

> Convert an image into a normalised tensor and explain HWC vs CHW layouts and per-channel normalisation.

## Pixels, channels and layout

A colour image is a grid of pixels, and each pixel holds three numbers for red, green and blue, usually integers from 0 to 255. Libraries such as Pillow and NumPy store this as **HWC**: height, width, channels. PyTorch models expect **CHW** (channels first), plus a batch dimension in front, giving NCHW. Before an image reaches an encoder it is typically resized to the model's input size, converted to floats in 0 to 1, and **normalised** per channel by subtracting a mean and dividing by a standard deviation. The values 0.485, 0.456, 0.406 and 0.229, 0.224, 0.225 are the ImageNet statistics many vision models were trained with; CLIP uses its own values. Using the wrong statistics does not crash anything, which is exactly why it is dangerous: the model silently sees inputs from a different distribution and accuracy drops. Always use the preprocessing that ships with the checkpoint.

## Three signals, one destination

Images, audio and video all end up as tensors, but each takes a different route: pixels are normalised, waveforms become spectrograms, and video is sampled into frames.

![Three connected stages: raw signal, preprocessing, model-ready tensor.](assets/figures/multimodal-ai/section-2-map.svg) — Figure 2.1: raw signal (pixels, waveform, frames) -> preprocessing (resize, normalise, spectrogram, sampling) -> model-ready tensor.

## From PIL image to normalised CHW tensor

A 4x3 orange image becomes a (3, 4, 3) HWC array, then a (3, 3, 4) CHW float tensor after normalisation.

```python
import numpy as np
from PIL import Image

img = Image.new("RGB", (4, 3), color=(255, 128, 0))  # width 4, height 3
arr = np.asarray(img)
print(arr.shape, arr.dtype)            # (height, width, channels)

chw = arr.transpose(2, 0, 1).astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32).reshape(3, 1, 1)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32).reshape(3, 1, 1)
x = (chw - mean) / std
print(x.shape)                         # (channels, height, width)
print(x[:, 0, 0].round(3))             # R, G, B of the top-left pixel
```

Output:

```
(3, 4, 3) uint8
(3, 3, 4)
[ 2.249  0.205 -1.804]
```

## Width-height order trap

Pillow's `Image.new("RGB", (4, 3))` takes **(width, height)**, but the NumPy array has shape **(height, width, 3)**. Mixing the two orders is one of the most common bugs when cropping or resizing for a model.

**Quiz:** A PyTorch vision model expects input of shape (1, 3, 224, 224). What does the 3 represent?

- [ ] The batch size
- [x] The colour channels (R, G, B)
- [ ] The number of patches
- [ ] The image height in hundreds of pixels

*Answer:* The colour channels (R, G, B). In NCHW layout the second dimension is channels; an RGB image has three.