# Serving Fast and Cheap — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-optimization

> Apply caching, batching, quantisation and model routing to cut multimodal latency and cost.

## Engineering levers in production

Beyond resizing inputs, four techniques do most of the work. **Cache embeddings**: images in a catalogue or document store rarely change, so compute their embeddings once at ingest and store them, instead of re-encoding on every query. **Batch** encoder calls: GPUs are far more efficient processing many images together than one at a time, so offline ingestion should always batch, and serving frameworks such as vLLM batch concurrent requests automatically. **Quantise** weights to 8-bit or 4-bit to cut memory and often speed up inference, accepting a small quality loss that you must measure on your eval set, since fine text reading can suffer more than general captioning. **Route** requests: a cheap path (OCR, a small VLM, a CLIP classifier) handles easy cases and only hard or low-confidence cases go to a large model. Finally, **stream** generated text so users see the first words quickly, which matters more to perceived speed than total time.

## A busy dhaba kitchen

A dhaba does not knead fresh dough for every single roti order; it preps in batches (batching), keeps chopped onions ready (caching), and the junior cook handles simple orders while the head cook takes the complicated ones (routing).

## Measure before optimising

Log per-stage timings: image download, preprocessing, encoding, prefill, generation. Teams often quantise the LLM while the real bottleneck is a slow image download or an unbatched encoder.

**Quiz:** Your product search embeds the same 2 million catalogue images on every user query. What is the biggest fix?

- [x] Precompute and store image embeddings once; embed only the query at search time
- [ ] Use a larger image encoder
- [ ] Increase the generation temperature
- [ ] Convert images to grayscale per query

*Answer:* Precompute and store image embeddings once; embed only the query at search time. Catalogue embeddings are static; compute them at ingest and only encode the query text online.