Image Tokens, Resolution and Cost

Where the money and latency go

For a VLM, cost and latency are driven largely by tokens, and images can be the biggest part of a prompt. With a ViT encoder the patch count grows with the square of the side length, so a 448-pixel image has four times the tokens of a 224-pixel one. Models handle high resolution in different ways. Some resize everything to a fixed size, which is cheap but blurs small text. Some tile large images into several crops plus a low-resolution overview, so cost grows with the number of tiles. Others support dynamic resolution, producing a token count proportional to the image area, and then merge neighbouring patch tokens (for example 2 x 2 into one) to cut the count by four. Hosted APIs publish their own image-token rules and sometimes a low-detail mode. The practical lever is yours: send the smallest resolution and crop that still contains what the question needs.

Three levers on multimodal cost

Most savings come from shrinking what you send, reusing what you computed, and choosing a smaller or quantised model where quality allows.

Patch tokens by resolution

With 14-pixel patches, doubling the side length quadruples the tokens; a 2x2 merge divides them by four.

def image_tokens(width, height, patch=14, merge=1):
    """Patch tokens a ViT-style encoder produces; merge=2 pools 2x2 patches."""
    cols, rows = width // patch, height // patch
    return (cols // merge) * (rows // merge)

for size in [224, 336, 448, 896]:
    print(size, "->", image_tokens(size, size), "tokens,",
          image_tokens(size, size, merge=2), "after 2x2 merge")

Output:

224 -> 256 tokens, 64 after 2x2 merge
336 -> 576 tokens, 144 after 2x2 merge
448 -> 1024 tokens, 256 after 2x2 merge
896 -> 4096 tokens, 1024 after 2x2 merge

Cap the long side before sending

thumbnail shrinks in place and keeps the aspect ratio, so a 4032x3024 photo becomes 1024x768.

from PIL import Image

img = Image.new("RGB", (4032, 3024))           # a typical phone photo size
img.thumbnail((1024, 1024))                    # keeps aspect ratio, in place
print(img.size)

Output:

(1024, 768)

त्वरित जाँच: An image's width and height are both doubled before it reaches a fixed-patch ViT. Roughly how does the patch token count change?

• It stays the same
• It doubles
• It halves
• It quadruples