Images as Tensors

Pixels, channels and layout

A colour image is a grid of pixels, and each pixel holds three numbers for red, green and blue, usually integers from 0 to 255. Libraries such as Pillow and NumPy store this as HWC: height, width, channels. PyTorch models expect CHW (channels first), plus a batch dimension in front, giving NCHW. Before an image reaches an encoder it is typically resized to the model's input size, converted to floats in 0 to 1, and normalised per channel by subtracting a mean and dividing by a standard deviation. The values 0.485, 0.456, 0.406 and 0.229, 0.224, 0.225 are the ImageNet statistics many vision models were trained with; CLIP uses its own values. Using the wrong statistics does not crash anything, which is exactly why it is dangerous: the model silently sees inputs from a different distribution and accuracy drops. Always use the preprocessing that ships with the checkpoint.

Three signals, one destination

Images, audio and video all end up as tensors, but each takes a different route: pixels are normalised, waveforms become spectrograms, and video is sampled into frames.

From PIL image to normalised CHW tensor

A 4x3 orange image becomes a (3, 4, 3) HWC array, then a (3, 3, 4) CHW float tensor after normalisation.

import numpy as np
from PIL import Image

img = Image.new("RGB", (4, 3), color=(255, 128, 0))  # width 4, height 3
arr = np.asarray(img)
print(arr.shape, arr.dtype)            # (height, width, channels)

chw = arr.transpose(2, 0, 1).astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32).reshape(3, 1, 1)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32).reshape(3, 1, 1)
x = (chw - mean) / std
print(x.shape)                         # (channels, height, width)
print(x[:, 0, 0].round(3))             # R, G, B of the top-left pixel

Output:

(3, 4, 3) uint8
(3, 3, 4)
[ 2.249  0.205 -1.804]

Width-height order trap

Pillow's Image.new("RGB", (4, 3)) takes (width, height), but the NumPy array has shape (height, width, 3). Mixing the two orders is one of the most common bugs when cropping or resizing for a model.

त्वरित जाँच: A PyTorch vision model expects input of shape (1, 3, 224, 224). What does the 3 represent?

• The batch size
• The colour channels (R, G, B)
• The number of patches
• The image height in hundreds of pixels