An agent-readable screen recording is a recording published in a format an AI agent can actually read: a structured document with the full transcript, extracted keyframe images, click coordinates, and timestamps — not just an opaque video file. A raw MP4 gives an agent pixels it can't watch; an agent-readable recording gives it text and images it can reason over.
Disclosure: Clipy is our product. The examples below use our implementation, but the format idea is open — the point is the contract between a recording and an agent, not the vendor.
The problem: agents don't watch video
Hand a coding agent a link to a 6-minute screen recording of a bug and nothing happens. Claude, GPT, Gemini, and every agent framework built on top of them accept text and still images as input — not hour-long video streams. Claude's vision support, for example, takes images, not MP4 files. The agent literally cannot open the recording you sent it.
So the human on the receiving end does the translation manually: watches the video, scrubs back and forth, transcribes the important bits, screenshots the moment where the button was clicked, and pastes all of that into the agent as context. That's the job an agent-readable recording removes. Instead of a file only a human can consume, you publish the recording as data the agent consumes directly.
This is a different thing from the two products that currently rank for "agent screen recording." One is contact-center software that records employees' screens for QA monitoring. The other is a wave of tools that use an agent to generate narrated product demos. Neither is what we mean here. An agent-readable recording is about the output format of a normal screen recording — making an ordinary "here's the bug I hit" capture legible to a machine.
The spec: what an agent actually needs from a recording
Think of it as a contract. A recording is agent-readable when it exposes these four things in a structured, fetchable document:
| What the agent needs | Why | What a raw MP4 gives instead |
|---|---|---|
| Full transcript, timestamped | The agent reads what was said and when | Nothing — audio is encoded, not text |
| Keyframe images at the moments that matter | Vision models reason over stills, not video | You'd have to extract frames yourself |
| Click coordinates + a pointer target | Resolves "when I click this" to an actual location | A cursor blob in pixels, unlabeled |
| Timestamps tying it all together | Lets the agent line up "at 2:14 they said X and clicked Y" | Only a playback clock |
The load-bearing insight is the third row. A transcript alone can't resolve deixis — "click this button," "drag it over here," "the error shows up right there." Those words point at the screen, and without the pixel the pointer is meaningless. An agent-readable recording captures what was said (caption), what was shown (the actual frame as an image), and where the pointer was (click coordinates, expressed as x/y fractions of the frame so they survive any resolution). Put together, the agent can reconstruct the interaction the way a human viewer would.
That bundle — summary, action items, key moments with frames and click coordinates, and the full transcript — is exactly what Clipy's agent recording format publishes for every recording.
Transcript ≠ agent-readable
It's worth being precise, because a lot of tools claim to be "AI-ready" when they only mean "we auto-transcribe."
A transcript is necessary but not sufficient. It captures speech and drops everything visual — which UI element was clicked, what the error message actually said on screen, which of three identical buttons the user meant. For a talking-head video the transcript is most of the value. For a screen recording, where the whole point is what's happening on the display, the transcript is maybe half of it. The other half lives in the frames and the pointer data. An agent-readable recording is the union: transcript plus keyframes plus coordinates plus the timeline that binds them.
The two live integration patterns
There are two ways to hand a recording to an agent, and they solve different access problems. Clipy ships both today.
Pattern A — convention: any link is fetchable as .md, no MCP
Every public Clipy watch URL has a machine-readable twin. Append .md (or .json, or ?format=md) to the link and instead of the video page you get the structured document — summary, action items, key moments with frame images and coordinates, and the transcript.
https://clipy.online/video/<id> → the human watch page
https://clipy.online/video/<id>.md → the agent-readable document
No API key, no server, no auth for public links. This is a pure convention: the URL you'd share with a teammate is already the URL an agent fetches. To make an agent reliably reach for the .md twin, install the one-file Clipy agent skill:
mkdir -p ~/.claude/skills/watch-clipy-recording
curl -fsSL -o ~/.claude/skills/watch-clipy-recording/SKILL.md \
https://clipy.online/skills/watch-clipy-recording/SKILL.md
After that, you paste a normal Clipy link into Claude Code and the skill teaches the agent to fetch the .md document on its own. A skill is just a small instruction file the agent loads — there's no server to run and nothing to authenticate for public recordings.
Pattern B — MCP server: authenticated, private, searchable
The convention covers public links. When an agent needs your private recordings, your whole library, or search across recordings, you add the Clipy MCP server. MCP (the Model Context Protocol) is the open standard agents use to call external tools, so this drops into any MCP-capable client:
claude mcp add --scope user clipy \
--env CLIPY_API_KEY=clipy_sk_live_xxx \
-- npx -y @clipy/mcp
The API key scopes the agent to your account. From there the agent gets tools to search your recordings, fetch a transcript or summary, list your library, and wait for a just-finished recording to finish processing before reading it. There's also a v1 REST API if you'd rather call it directly.
The rule of thumb: install the skill first; add MCP only when the agent needs authenticated library access. Public bug link to a teammate's agent → skill and the .md convention. Agent that needs to search across your private recordings → MCP.
What click semantics add (and where the Mac app goes deeper)
Click coordinates get you the location of a pointer. On the Mac, Clipy's native app reads one level higher.
Because the Mac app captures with ScreenCaptureKit and taps the macOS Accessibility layer, at each click it records not just the x/y coordinate but the AX role and visible label of the element under the cursor — e.g. AXButton "Save", AXLink "Download invoice." So the recording can tell an agent "the user clicked the button labeled Save" rather than "the user clicked at (0.62, 0.48)." That's the difference between a pixel and a semantic event, and it's genuinely beyond what a browser extension can reach, because the OS accessibility tree isn't exposed to page-level capture. The Mac app also captures cross-application (whole screen or a specific window, not just a browser tab), system audio plus mic, and a cursor-movement + click sidecar with millisecond timings.
None of this requires the agent to run object detection on frames. The structure is already in the recording document.
Try it in 60 seconds
You don't need to change your recording habits to test this. Record anything with Clipy's free screen recorder, grab the share link, and hand it to an agent:
Read this recording and tell me what broke and how to reproduce it:
https://clipy.online/video/<your-id>
With the skill installed, the agent fetches the .md twin, reads the transcript and key-moment frames, and answers from the actual content — the summary, the steps shown, the element that was clicked — instead of asking you to describe the video it can't watch. For a full walkthrough of wiring this into a coding agent, see Claude Code screen recording.
Why agent-readable screen recording is a category, not a feature
Screen recording was designed for one consumer: a human watching later. The moment the "later" consumer is an agent — reviewing a bug, writing a repro, drafting an SOP, checking whether a fix matches what the user actually did — the video file is the wrong artifact. It has all the information locked in a format the reader can't parse.
An agent-readable screen recording unlocks it: same recording, published once, served to humans as a video and to agents as a document. Record it once. Humans watch it, agents read it.
FAQ
What is an agent-readable screen recording?
It's a screen recording published in a format an AI agent can read directly — a structured document containing the full timestamped transcript, extracted keyframe images, click coordinates, and (on the Mac app) the accessibility role and label of clicked elements. The agent reasons over that text and those images instead of trying to watch the video.
Can AI agents watch video files?
No. Claude, GPT, Gemini, and the agent frameworks built on them accept text and still images, not video streams. That's why a raw MP4 is useless as agent context and why the recording has to be translated into text plus frames first — either by a human doing it manually or by an agent-readable format doing it automatically.
Do I need an MCP server to give a recording to an agent?
Not for public links. A public Clipy recording has a .md twin — append .md to the share URL — and the one-file agent skill teaches the agent to fetch it, with no server and no API key. You only add the MCP server when the agent needs authenticated access to your private recordings or to search your library.
How is this different from just auto-transcribing a recording?
A transcript captures speech and drops everything visual — which button was clicked, what the on-screen error said, which element "this" refers to. For a screen recording that's only half the information. An agent-readable recording adds keyframe images, click coordinates, and the timeline that ties speech to what was on screen, so the agent can reconstruct the actual interaction.
Is an agent-readable recording the same as accessibility captions?
No. Accessibility captions make audio readable for humans who can't hear it. An agent-readable recording makes the whole recording — speech, visuals, and pointer actions — legible to a machine, so an AI agent can act on it. On the Mac, Clipy does reuse the OS accessibility layer to label clicked elements, but the output is agent context, not human captions.