# Visual Hallucination and Grounding — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-hallucination

> Recognise object and text hallucination in VLMs and reduce it with grounding, prompting and verification.

## When the model sees what is not there

A VLM **hallucinates** when it states something the image does not support: an object that is absent, a wrong count, a misread number, or a confident description of a blurry region. One cause is the language model's strong prior: a kitchen photo "usually" has a fridge, so the model may mention one. Another is resolution: small text is downsampled away before the model ever sees it, yet the model still produces a plausible-looking value. Counting, spatial relations (left of, behind) and reading charts are known weak spots. Mitigations work in layers. Send higher resolution or crops for detail questions. Ask for **grounded** output, such as bounding boxes or quoted text, which some models support and which you can check. Allow and encourage "not visible" as an answer. Verify with deterministic tools where possible: OCR for text, a detector for counts, arithmetic for totals.

## Defence in depth for multimodal apps

Safe multimodal systems layer controls: clean inputs, isolate untrusted content, verify outputs and keep a human in the loop for high-stakes cases.

![Four connected stages: sanitise input, isolate untrusted content, verify output, human review.](assets/figures/multimodal-ai/section-10-map.svg) — Figure 10.1: sanitise input (strip metadata, resize) -> isolate untrusted content (images as data) -> verify output (rules, OCR, detectors) -> human review for high-stakes cases.

## Give the model a way out

Prompts like "If the value is not clearly readable, answer `null`" usually reduce invented values in extraction tasks. Without an allowed escape, the model tends to fill the field anyway.

**Quiz:** A VLM reports the wrong amount from a small, blurry total on a receipt. What is the best first fix?

- [ ] Raise the temperature
- [ ] Ask the model to be more confident
- [x] Send a higher-resolution crop of the total region and allow a 'not readable' answer
- [ ] Switch to a text-only LLM

*Answer:* Send a higher-resolution crop of the total region and allow a 'not readable' answer. Detail errors usually come from downsampling; a crop restores resolution and an escape option prevents guessing.