Short version: an AI agent can watch a video, but it doesn't watch the way you do. It samples a handful of frames per second and reads the transcript, so the exact frame where you clicked — and the pixel you clicked — is usually gone before the model ever sees it. Fixing the bug needs capture-time data, not a bigger model.
You recorded a 90-second repro. The dropdown opens in the wrong place, you click "Save," nothing happens. You drop the link into Claude Code or Cursor and ask it to find the bug. The agent comes back with something plausible-sounding and completely wrong. It's not being lazy. It literally never saw the click.
What actually happens when you hand an agent a video
A video is not a thing an LLM ingests whole. Multimodal models turn a clip into two streams: a set of sampled still frames, and a transcript of the audio. The model reasons over those, not over the pixels-in-motion you watched.
Frame sampling is where the click dies. Most video pipelines sample at roughly 1 frame per second, because frames are expensive. Anthropic's own vision docs put a single image at hundreds of visual tokens; a minute of video at even 1 fps is 60 images before you've added a word of prompt. Research on scaling video understanding to 16 frames per second exists precisely because low-frame-rate sampling is the default and it throws away fast events. A survey of frame-selection methods frames the whole problem the same way: you can't afford every frame, so you pick a few and hope the important one is in the set.
A mouse click takes about 100 milliseconds. At 1 fps, the odds that a sampled frame lands on the frame where your cursor is pressing "Save" are close to zero. The agent sees the frame before the click and the frame after. It infers a click happened. It does not know where.
Audio tells you what was said, not where you pointed
The transcript is the other half of what the model gets, and it's genuinely useful — for the parts of the recording that are narrated. "Now I click Save and it just spins" is a real signal.
But most bug repros aren't narrated like a tutorial. People point. They say "this button here," "that one," "it's this thing at the top." The transcript captures "this button here" verbatim and the referent is gone. Deixis — pointing words like this, here, that — only resolves against a shared visual frame, and the agent doesn't have your frame. It has a word.
Even a perfectly narrated recording hits a wall the moment the important thing is silent. A layout that's 4px off, a hover state that flickers, a focus ring on the wrong element — nobody says those out loud. They're purely visual, they happen between sampled frames, and they never reach the model. This is the core of the LLM video context limitations problem: the two channels an agent gets, frames and audio, are exactly the two channels a UI bug is best at hiding in.
"The button" is meaningless without coordinates
Here's the part people underestimate. Even if the agent could see every frame, "the Save button" is not an address it can act on.
An AI coding agent fixes bugs by editing code. To connect "the click that failed" to a line of code, it needs to know which element was clicked — ideally its position, and better still its identity. A pixel coordinate (842, 310) in a 1440-wide frame is a lead: it tells the agent where to look in the DOM, which component renders there, which handler is bound. A vague "the button in the corner" is not.
This is why "just use a smarter model" doesn't fix it. The information was never captured. You cannot recover, at inference time, a coordinate that the recorder never wrote down. No amount of model scale reconstructs a measurement that doesn't exist in the file. The gap is upstream of the model entirely.
The fix isn't a bigger model — it's capture-time data
The whole problem inverts once you stop treating a recording as pixels to be re-analyzed later and start treating it as an instrument that writes down what happened while it happens.
At capture time, the recorder is inside the event loop. It knows the millisecond the mouse went down. It knows the cursor's exact position. On a native app it can ask the OS what element is under the pointer. None of that has to be inferred from a downsampled JPEG after the fact — it's ground truth, recorded once, at the source. Inference-time video analysis is trying to reverse-engineer a measurement; capture-time data just takes the measurement.
That distinction is the entire thesis:
| Inference-time video analysis | Capture-time data | |
|---|---|---|
| Where the click is | Guessed from ~1 sampled frame/sec | Recorded coordinate + timestamp |
| What was pointed at | "this button here" (unresolved) | Frame + pointer position at that moment |
| Element identity | Not available | AX role + label (Mac native capture) |
| Cost to recover later | Impossible if never captured | Already written to the file |
What Clipy records at capture time
This is the design Clipy ships today, and it's the reason an agent can act on a Clipy recording instead of narrating it back to you.
Every clipy.online/video/<id> link is also a fetchable machine-readable document. Append .md (or ?format=md / .json) and you get, per the for-agents spec:
- The full timestamped transcript — what was said, aligned to the timeline.
- Key moments with extracted frame images — the recorder pulls the frames that matter (clicks, scene changes) so the model sees the moment, not a 1-fps miss.
- Recorded click coordinates + timestamps —
x/yas fractions of the frame at the moment of the interaction, plus a "move this → to here" motion target when there's a drag. This is the coordinate the agent needs to map a click to a component. - On the Mac app, OS-level element semantics — through the macOS Accessibility layer, each click records the element's AX role (
AXButton,AXLink, …) and its visible label. That's not a pixel guess; it's the operating system naming the thing you clicked. No browser extension can reach that.
To be precise about the boundary: the Chrome extension records click coordinates and key-moment frames, but it does not capture console logs, network requests, or a device-metadata dump — the OS-level element role/label depth is the Mac app's. We'd rather say what's real than imply parity we don't ship.
Wiring an agent to it takes one of two paths, both live. Install the Clipy agent skill once and any Clipy link you paste is fetched as .md automatically — no MCP, no config. Or add the MCP server for private recordings and library search:
npx -y @clipy/mcp
Then the prompt you already wanted to write actually works:
Here's the repro: https://clipy.online/video/abc123
Read the recording. The click at the failing moment — which component
renders at those coordinates, and why does the handler no-op?
The agent fetches the transcript, the key-moment frame, and the click coordinate, and now has an address to reason from. Same recording. The difference is that the data it needs was written down at capture time instead of guessed at inference time.
FAQ
Can an AI agent watch a video and find a bug?
It can process a video — as sampled frames plus a transcript — but it usually can't find a UI bug from that alone. The exact frame of a click, and the pixel that was clicked, fall between the ~1 fps sample points. An agent needs recorded click coordinates and timestamps, not just the raw clip.
Why can't a bigger LLM just analyze the whole video?
Because the missing information was never captured. Frame sampling and token budgets mean the model sees a fraction of frames, and a coordinate the recorder never wrote down can't be reconstructed at inference time. Model scale doesn't recover a measurement that isn't in the file.
What data does an AI agent actually need from a screen recording?
The transcript (what was said), the specific frames where interactions happen (what was shown), and the click coordinates with timestamps (where you pointed). On native macOS captures, the element's accessibility role and label make it unambiguous which control was clicked.
How does Clipy give agents this context?
Every Clipy link doubles as a fetchable .md/.json document with the summary, transcript, key-moment frames, and recorded click coordinates. Install the Clipy skill or run npx -y @clipy/mcp, and agents like Claude Code or Cursor read the recording directly.
Does the Clipy Chrome extension capture console and network logs?
No. The extension records click coordinates and key-moment frames but does not capture console output, network requests, or device metadata. The deepest element-level context (AX role and label per click) comes from the Mac native app.