Video Understanding Applications

Answering questions about a recording

Suppose you want to answer "when does the lecturer derive the formula?" over a two-hour class recording. A practical pipeline has four steps. Segment the video into chunks of a fixed length or at scene changes. For each chunk, transcribe the audio with timestamps and sample a few frames. Describe and index: caption the frames with a VLM or embed them directly, and store frame descriptions, transcript text and timestamps together. At question time, retrieve the relevant chunks and send their frames plus transcript to a VLM, asking it to answer and cite timestamps. Models with very long context can accept many frames at once, but cost and latency grow with every frame, so retrieval-first designs remain common for long video. Evaluate on questions whose answers depend on the visuals, the audio and both, because a system strong on transcripts can look good while ignoring the screen entirely.

Timestamps are your citations

Always carry timestamps through every stage. An answer like "at 01:12:40 the board shows the derivation" lets the user jump there and verify, which builds far more trust than an unreferenced summary.

Frames per chunk with ffmpeg

Extract one frame every 5 seconds as JPEGs, a common first step before captioning or embedding. Needs ffmpeg and a video file, so no output is shown.

mkdir -p frames
ffmpeg -i lecture.mp4 -vf fps=1/5 -q:v 3 frames/frame_%05d.jpg

Quick check: A video QA system answers transcript questions well but fails on "what colour is the chart on screen?". What is the most likely cause?

• The audio sample rate is too high
• Too few or no frames are being sent to the model for that segment
• The transcript has timestamps
• The model's temperature is zero