# Video Understanding Applications — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-video-understanding

> Build a video question-answering flow from frame sampling, transcripts and timestamps.

## Answering questions about a recording

Suppose you want to answer "when does the lecturer derive the formula?" over a two-hour class recording. A practical pipeline has four steps. **Segment** the video into chunks of a fixed length or at scene changes. For each chunk, **transcribe** the audio with timestamps and **sample** a few frames. **Describe and index**: caption the frames with a VLM or embed them directly, and store frame descriptions, transcript text and timestamps together. At question time, **retrieve** the relevant chunks and send their frames plus transcript to a VLM, asking it to answer and cite timestamps. Models with very long context can accept many frames at once, but cost and latency grow with every frame, so retrieval-first designs remain common for long video. Evaluate on questions whose answers depend on the visuals, the audio and both, because a system strong on transcripts can look good while ignoring the screen entirely.

## Timestamps are your citations

Always carry timestamps through every stage. An answer like "at 01:12:40 the board shows the derivation" lets the user jump there and verify, which builds far more trust than an unreferenced summary.

## Frames per chunk with ffmpeg

Extract one frame every 5 seconds as JPEGs, a common first step before captioning or embedding. Needs ffmpeg and a video file, so no output is shown.

```bash
mkdir -p frames
ffmpeg -i lecture.mp4 -vf fps=1/5 -q:v 3 frames/frame_%05d.jpg
```

**Quiz:** A video QA system answers transcript questions well but fails on "what colour is the chart on screen?". What is the most likely cause?

- [ ] The audio sample rate is too high
- [x] Too few or no frames are being sent to the model for that segment
- [ ] The transcript has timestamps
- [ ] The model's temperature is zero

*Answer:* Too few or no frames are being sent to the model for that segment. Visual questions need frames; a transcript-heavy pipeline with sparse or missing frames cannot see the screen.