Sending Images to a Model API

Messages with multiple content parts

Hosted multimodal APIs extend the familiar chat format: instead of a single string, a message's content becomes a list of parts, each with a type such as text or image. An image part can usually reference a public URL the provider fetches, or carry the bytes inline as base64, often wrapped in a data URL such as data:image/png;base64,.... The exact field names differ between providers, so check the documentation of the one you use; the OpenAI-style image_url part shown here is also accepted by many OpenAI-compatible servers, including local ones. Some APIs offer a detail or resolution setting that trades accuracy for token cost. Base64 inflates the payload by about a third, and providers enforce limits on image size, count and format, so resize images on your side before sending. For repeated use of the same large file, some providers offer a file upload endpoint so you send the bytes once.

A multimodal request end to end

Your code prepares the image, wraps it with the text in a message, sends it to an API or local server, and parses the reply.

Building an image + text message

The image is encoded in memory as base64 and placed next to the question. Printing the part types confirms the structure before any network call.

import base64, io, json
from PIL import Image

img = Image.new("RGB", (64, 64), "white")      # stand-in for a real photo
buf = io.BytesIO()
img.save(buf, format="PNG")
b64 = base64.b64encode(buf.getvalue()).decode("ascii")

message = {
    "role": "user",
    "content": [
        {"type": "text", "text": "What is written on this receipt?"},
        {"type": "image_url",
         "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ],
}
print([part["type"] for part in message["content"]])
print(message["content"][1]["image_url"]["url"][:22] + "...")
print(len(json.dumps(message)) > len(buf.getvalue()))   # base64 grows the payload

Output:

['text', 'image_url']
data:image/png;base64,...
True

Ask for structure, then validate it

For extraction tasks (bills, ID cards, forms) ask the model for JSON with named fields and validate the result with a schema library such as Pydantic. Treat a parse failure as a retry or a human-review case, never as data.

Quick check: Why should you resize a 12-megapixel phone photo before sending it to a vision API?

• APIs only accept square images
• Base64 cannot encode large files
• Resizing adds EXIF metadata the model needs
• Smaller images upload faster, fit size limits and usually cost fewer image tokens