What do you actually verify when a model says it "watched" your video?

I've been building video tooling for a few months and the thing that still gets me is how confidently a model will describe a video it barely saw.

Most pipelines hand it a frame every few seconds plus a transcript. Then you ask about pacing, or camera work, or whether the speaker sounded unsure, and it answers. Fluently. The answer just isn't grounded in anything that was actually passed in.

So I've started treating "what did the model receive" as a separate question from "what did the model say". Different problem, different fix, and only one of them is visible in the output.

Curious how others handle this:

- do you sample frames on a timer, or on scene changes? timers are cheap but they miss hard cuts

- do you ever inspect what actually reached the model, or do you trust the answer if it reads well?

- for anyone running this at volume, what breaks first for you, the cost or the accuracy?

Not fishing for tool recommendations. I'd genuinely like to know where other people draw the line between "good enough context" and "the model is guessing".