# Patch Embeddings Step by Step — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-vit-patches

> Compute the number of patches for an image size and build patch embeddings with NumPy.

## The patch arithmetic

With an image of size H x W and a patch size P, a ViT produces (H / P) x (W / P) patches. A 224 x 224 image with 16-pixel patches gives 14 x 14 = 196 patches; with 14-pixel patches it gives 16 x 16 = 256. Each patch has P x P x 3 numbers, which are flattened and multiplied by a learned matrix (the **patch embedding**, equivalent to a convolution with kernel size and stride P) to produce a vector of the model width, for example 768. Many ViTs add a special **[CLS]** token whose final state summarises the whole image, and every token gets a **position embedding** because attention itself has no notion of order. The number of patches grows with the square of the resolution: doubling height and width gives four times as many tokens, and self-attention cost grows roughly with the square of the token count. That is the root of most cost problems with high-resolution images.

## Tracing the patch embedding

Follow the shapes. Each step changes what the array means, not just its size.

```python
import numpy as np

rng = np.random.default_rng(0)
img = rng.random((224, 224, 3), dtype=np.float32)
P = 16                                   # patch size
h, w, c = img.shape

patches = (img.reshape(h // P, P, w // P, P, c)
              .transpose(0, 2, 1, 3, 4)
              .reshape(-1, P * P * c))
print(patches.shape)                     # (num_patches, pixels_per_patch)

W = rng.standard_normal((P * P * c, 768), dtype=np.float32) * 0.02
tokens = patches @ W                     # linear patch embedding
cls = np.zeros((1, 768), dtype=np.float32)
seq = np.concatenate([cls, tokens])
print(seq.shape)                         # [CLS] + 196 patch tokens
```

- Line 4: A random 224x224 RGB image stands in for a real photo, stored height-width-channels.
- Line 5: The patch size decides the token count: 224 / 16 = 14 patches per side.
- Line 8: reshape splits each axis into (14 blocks, 16 pixels): rows and columns of patches appear.
- Line 9: transpose brings the two block axes together so each patch's pixels are contiguous.
- Line 10: Flatten: 196 patches, each a vector of 16*16*3 = 768 raw pixel values.
- Line 14: A learned linear map projects each patch to the model width; here also 768.
- Line 16: Prepending a [CLS] token gives the final sequence the transformer receives.

## Run it

The same code with its real output: 196 patch vectors, then 197 tokens including [CLS].

```python
import numpy as np

rng = np.random.default_rng(0)
img = rng.random((224, 224, 3), dtype=np.float32)
P = 16                                   # patch size
h, w, c = img.shape

patches = (img.reshape(h // P, P, w // P, P, c)
              .transpose(0, 2, 1, 3, 4)
              .reshape(-1, P * P * c))
print(patches.shape)                     # (num_patches, pixels_per_patch)

W = rng.standard_normal((P * P * c, 768), dtype=np.float32) * 0.02
tokens = patches @ W                     # linear patch embedding
cls = np.zeros((1, 768), dtype=np.float32)
seq = np.concatenate([cls, tokens])
print(seq.shape)                         # [CLS] + 196 patch tokens
```

Output:

```
(196, 768)
(197, 768)
```

## Patch size 14 vs 16

Model names often encode this: `vit-base-patch16-224` means 16-pixel patches at 224 resolution; `clip-vit-large-patch14-336` means 14-pixel patches at 336, giving 24 x 24 = 576 patches. Read the name before estimating token counts.

**Quiz:** A ViT uses 14-pixel patches on a 336 x 336 image. How many patch tokens does it produce (excluding [CLS])?

- [x] 576
- [ ] 256
- [ ] 196
- [ ] 1,024

*Answer:* 576. 336 / 14 = 24 patches per side, and 24 x 24 = 576.