What Multimodal AI Means

Modalities and multimodal models

A modality is a kind of signal: text, images, audio, video, sensor readings, tables. A unimodal model takes one kind of input and produces one kind of output, such as a text classifier or an image classifier. A multimodal model reads, relates or produces more than one modality. The key word is relates: a system that runs speech-to-text and then hands the text to a chatbot is a pipeline of unimodal parts, while a model that looks at a photo and answers a question about it must connect pixels to words inside one network. Common tasks include image captioning (image to text), visual question answering (image plus question to answer), text-to-image generation, speech recognition (audio to text), text-to-speech, cross-modal retrieval (find photos with a sentence) and document understanding (scanned pages to structured data). The hard part in all of them is the same: different signals arrive in very different shapes, and the model must map them into representations it can compare.

From raw signals to one answer

Every multimodal system follows the same broad path: raw inputs are encoded per modality, aligned into a shared representation, and then a head or a language model produces the output.

A doctor reading a case

A doctor at a clinic does not diagnose from the X-ray alone or from your description alone. She looks at the scan, listens to you describe the pain, reads the lab report, and connects all three. A multimodal model aims for the same thing: evidence from several senses combined into one judgement.

Pipeline is not the same as multimodal

Interviewers often ask whether "Whisper + an LLM" is a multimodal model. Strictly, it is a multimodal system built from unimodal parts. Information lost at the hand-off (tone of voice, a pause, background sound) never reaches the LLM. A natively multimodal model can use those cues because it sees the signal itself.

त्वरित जाँच: Which task is a genuinely cross-modal problem for a single model?

• Classifying a tweet as positive or negative
• Answering "how many people are wearing helmets?" about a photo
• Translating Hindi text into English text
• Detecting spam in an email body