# What Multimodal AI Means — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-what-is-multimodal

> Define modality, tell unimodal and multimodal systems apart, and name the common multimodal tasks.

## Modalities and multimodal models

A **modality** is a kind of signal: text, images, audio, video, sensor readings, tables. A **unimodal** model takes one kind of input and produces one kind of output, such as a text classifier or an image classifier. A **multimodal** model reads, relates or produces more than one modality. The key word is *relates*: a system that runs speech-to-text and then hands the text to a chatbot is a pipeline of unimodal parts, while a model that looks at a photo and answers a question about it must connect pixels to words inside one network. Common tasks include image captioning (image to text), visual question answering (image plus question to answer), text-to-image generation, speech recognition (audio to text), text-to-speech, cross-modal retrieval (find photos with a sentence) and document understanding (scanned pages to structured data). The hard part in all of them is the same: different signals arrive in very different shapes, and the model must map them into representations it can compare.

## From raw signals to one answer

Every multimodal system follows the same broad path: raw inputs are encoded per modality, aligned into a shared representation, and then a head or a language model produces the output.

![Four connected stages: raw inputs, per-modality encoders, shared representation, output.](assets/figures/multimodal-ai/section-1-map.svg) — Figure 1.1: raw inputs (image, audio, text) -> per-modality encoders -> shared representation -> output (text, label or image).

## A doctor reading a case

A doctor at a clinic does not diagnose from the X-ray alone or from your description alone. She looks at the scan, listens to you describe the pain, reads the lab report, and connects all three. A multimodal model aims for the same thing: evidence from several senses combined into one judgement.

## Pipeline is not the same as multimodal

Interviewers often ask whether "Whisper + an LLM" is a multimodal model. Strictly, it is a multimodal *system* built from unimodal parts. Information lost at the hand-off (tone of voice, a pause, background sound) never reaches the LLM. A natively multimodal model can use those cues because it sees the signal itself.

**Quiz:** Which task is a genuinely cross-modal problem for a single model?

- [ ] Classifying a tweet as positive or negative
- [x] Answering "how many people are wearing helmets?" about a photo
- [ ] Translating Hindi text into English text
- [ ] Detecting spam in an email body

*Answer:* Answering "how many people are wearing helmets?" about a photo. Visual question answering needs the model to connect a text question with pixel content. The other three are text-only tasks.