Serving Fast and Cheap

Engineering levers in production

Beyond resizing inputs, four techniques do most of the work. Cache embeddings: images in a catalogue or document store rarely change, so compute their embeddings once at ingest and store them, instead of re-encoding on every query. Batch encoder calls: GPUs are far more efficient processing many images together than one at a time, so offline ingestion should always batch, and serving frameworks such as vLLM batch concurrent requests automatically. Quantise weights to 8-bit or 4-bit to cut memory and often speed up inference, accepting a small quality loss that you must measure on your eval set, since fine text reading can suffer more than general captioning. Route requests: a cheap path (OCR, a small VLM, a CLIP classifier) handles easy cases and only hard or low-confidence cases go to a large model. Finally, stream generated text so users see the first words quickly, which matters more to perceived speed than total time.

A busy dhaba kitchen

A dhaba does not knead fresh dough for every single roti order; it preps in batches (batching), keeps chopped onions ready (caching), and the junior cook handles simple orders while the head cook takes the complicated ones (routing).

Measure before optimising

Log per-stage timings: image download, preprocessing, encoding, prefill, generation. Teams often quantise the LLM while the real bottleneck is a slow image download or an unbatched encoder.

Quick check: Your product search embeds the same 2 million catalogue images on every user query. What is the biggest fix?

• Precompute and store image embeddings once; embed only the query at search time
• Use a larger image encoder
• Increase the generation temperature
• Convert images to grayscale per query