Video, Documents and Other Inputs

Video is many images plus time

A video is a sequence of frames, often 24 to 60 per second, usually with an audio track. Feeding every frame to a model is wasteful because neighbouring frames are nearly identical, and it quickly exceeds any context budget. Most systems therefore sample frames: a fixed number per clip, one every N seconds, or only at scene changes. Each sampled frame is encoded like an image, and timestamps or positional embeddings tell the model the order. Sampling is a trade-off: sparse sampling misses short events such as a ball crossing a line, dense sampling costs tokens and time. Documents are another common input. A scanned invoice can be passed as an image of each page, as text extracted by OCR, or as both. Images keep layout, tables and handwriting; extracted text is cheaper and searchable. Many production systems send both.

Sampling one frame every two seconds

A 10-second clip at 30 fps has 300 frames. Sampling every 2 seconds keeps 5.

def sample_frames(total_frames, fps, every_sec):
    step = max(1, round(fps * every_sec))
    return list(range(0, total_frames, step))

idx = sample_frames(total_frames=300, fps=30, every_sec=2)
print(idx)
print(len(idx), "frames instead of 300")

Output:

[0, 60, 120, 180, 240]
5 frames instead of 300

Do not forget the audio track

Many video questions ("what did the speaker say about pricing?") are answered by the audio, not the frames. Transcribe the audio with timestamps and align it with the sampled frames instead of sending frames alone.

त्वरित जाँच: Why do video understanding systems sample frames instead of using all of them?

• Adjacent frames are highly redundant and every frame costs tokens and compute
• Models cannot read more than one image
• Frames must be converted to audio first
• Sampling improves the video resolution