Inside the Markdown Twin: Anatomy of an Agent-Readable Recording

The markdown twin is the plain-text document a Clipy share link returns to an AI agent: summary, key moments, click targets, cursor paths, and transcript, in markdown. This page opens that document and reads it section by section, so you can see exactly what an agent gets and why each part earns its place.

Updated: July 2026

If you want the higher-level "what is this and who is it for" framing, start with the agent-readable recording pillar. This page is the internals. An agent-readable recording is a screen recording whose share link doubles as a structured text document (summary, key moments, click targets, and transcript) that AI agents can read instead of watching video. The rest of this page dissects that document.

What does a markdown twin look like?

Here is a representative example of the markdown twin for the recurring Export CSV bug: a payouts dashboard, the Export CSV button stuck in a loading state, and a console TypeError from exportWorker.ts:142. This is an illustrative example, not literal product output. Field names follow the AREC spec.

# Recording: Export CSV button stuck loading

## Metadata
- source_url: https://clipy.online/video/3kelcef8wo8h
- duration: 00:41
- created: 2026-07-22T14:03:00Z
- spec_version: 0.1

## Summary
On the payouts dashboard, clicking Export CSV leaves the button in a
loading state and never downloads a file. The browser console shows a
TypeError from exportWorker.ts line 142.

## Key moments
- timestamp: 00:06
  frame: frames/0006.jpg
  click_target: { element: "Export CSV button", x: 812, y: 274 }
  description: User clicks Export CSV on the payouts dashboard.
- timestamp: 00:09
  frame: frames/0009.jpg
  click_target: null
  description: Button enters a spinner state and stays there.
- timestamp: 00:14
  frame: frames/0014.jpg
  click_target: { element: "DevTools console", x: 640, y: 393 }
  description: Console logs "TypeError: cannot read properties of undefined
  (reading 'rows')" at exportWorker.ts:142.

## Cursor paths
- from: 00:04 { x: 470, y: 240 }
  to:   00:06 { x: 812, y: 274 }
- from: 00:10 { x: 812, y: 274 }
  to:   00:14 { x: 640, y: 393 }

## Transcript
- 00:02 "I'm on the payouts dashboard and I'll try to export."
- 00:06 "Clicking Export CSV now."
- 00:10 "It just spins, nothing downloads."
- 00:13 "Opening the console. There's the error, exportWorker line 142."

An agent fetches this by appending .md to the watch URL. Nothing about it needs a decoder, an SDK, or a video pipeline on the agent's side. It is text a language model already reads. Below, each section is pulled apart: what it contains, what an agent does with it, and what would be lost without it.

Metadata: what does the document header hold?

The metadata block is the document header, and it answers where, how long, and when. It carries four fields: source_url (the canonical share link), duration (total length), created (an ISO 8601 timestamp), and spec_version (the AREC format version this document follows).

What an agent does with it. The agent cites source_url when it references the recording, so its output points back at the exact clip a human can open. It reads duration to reason about pacing and where in a 41-second timeline a moment at 00:14 sits. It reads spec_version to know which parsing rules apply, so a document written to a newer version does not silently confuse it.

What is lost without it. Provenance and self-consistency. An agent that cannot cite source_url produces claims a reviewer cannot trace back to a source. Without duration a timestamp of 00:14 is unanchored. Without spec_version, forward compatibility breaks, because the agent has no signal for which field rules to apply.

Summary: what is the summary for?

The summary is a short natural-language description of what the recording shows, written for a first read. In the Export CSV example it states the surface (payouts dashboard), the symptom (Export CSV stuck loading, no file), and the likely cause (a TypeError from exportWorker.ts:142) in three sentences.

What an agent does with it. It triages. The agent reads the summary before parsing anything else and decides how deeply to go: whether the transcript is worth reading in full, which key moment to jump to, whether the bug is even in its area. It is the cheapest way for an agent to get the gist of a recording in one read.

What is lost without it. The fast path. Strip the summary and the agent has to reconstruct intent from raw key moments and transcript lines every time, which is slower and more error-prone. The summary is the layer that lets an agent skim before it commits to a deep parse.

Key moments: what makes a moment machine-actionable?

Key moments are the ordered list of important instants in the recording, and each one is built to be acted on, not just read. Every key moment carries four fields: timestamp, frame, click_target, and description. Together they turn "something happened" into "this happened here, this is what it looked like, this is what was clicked, this is what it means."

The frame reference. frame points at an extracted still image captured at that instant, for example frames/0006.jpg. The agent can look at the exact pixels that mattered at 00:06 without decoding a single video frame. This is the part that lets a vision-capable agent inspect UI state (a spinner, an error toast, a specific value in a table) directly, while a text-only agent still reads everything else and skips the image.

The click target. click_target records the UI element interacted with as a description plus x/y coordinates: { element: "Export CSV button", x: 812, y: 274 }. The element name says what was clicked. The coordinates say where. When no click happened at a moment (the button just sat spinning at 00:09), the target is null, which is itself information: the state changed without user input.

Why coordinates matter to an agent, specifically. An element name alone is ambiguous. A page can have two buttons labeled "Export," a toolbar and a modal both showing "Save," a link that reads the same in three places. The x/y pair disambiguates which one, and it pins the interaction to a concrete point in the viewport. That precision is what makes a flow reproducible: an agent driving a browser, writing an end-to-end test, or reasoning about layout can map "Export CSV at 812, 274" to an exact position instead of guessing among lookalikes. Coordinates also let the agent correlate a click with the frame and the console error that followed it, so it can say the failure started at the Export CSV click and not somewhere else. For an agent that goes on to reproduce the flow, as in the Claude Code fix loop, that grounding is the difference between a repeatable step and a hopeful one.

What an agent does with the whole moment. It builds a timeline of cause and effect. Click Export CSV at 00:06 (target known, coordinates known), button spins at 00:09 (no input), console throws at 00:14. That chain is a bug repro the agent can open a PR against without watching the video.

What is lost without key moments. The actionable spine of the document. Drop them and you are left with a summary and a wall of transcript: an agent knows roughly what happened but not the exact instants, the pixels, or the precise element and position that made it reproducible. Key moments are where an agent-readable recording stops being a description and becomes something an agent can act on.

Cursor paths: what do cursor paths add?

Cursor paths are the movement of the cursor between key moments, recorded as timestamped coordinates. In the AREC format this section is optional: a consumer may use it to reconstruct motion, and a consumer that ignores it still has a valid read.

What it contains. A sequence of from/to coordinate pairs with timestamps, tracing how the pointer traveled. In the example, the cursor moves toward the Export CSV button before the 00:06 click, then travels to the console region before 00:14.

What an agent does with it. It reads intent from motion. Hesitation, a path that overshoots and corrects, time spent hovering before a click: these are signals about where a user struggled or searched. An agent building a UX summary or a step-by-step guide can use the path to describe not just the clicks but the navigation between them.

What is lost without it. Nuance, not correctness. Because cursor paths are optional, a document without them is still complete and still actionable, the key moments carry the load. What you lose is the connective motion between moments, useful for usability analysis and richer walkthroughs but not required to reproduce the bug.

Transcript: why does the transcript matter?

The transcript is the timestamped narration of the recording, the words spoken while the screen was captured. In the Export CSV example it runs from "I'm on the payouts dashboard" through "There's the error, exportWorker line 142," each line pinned to a timestamp.

What an agent does with it. It recovers intent that no pixel carries. Frames show a spinner; the transcript says "it just spins, nothing downloads," which tells the agent the spinner is the failure and not a normal loading state. Timestamps let the agent align a spoken line with the key moment and frame at the same offset, so narration, image, and click reinforce each other.

What is lost without it. The reasoning track. Without the transcript, an agent sees what happened but not the person's read of it: which state was expected, which was wrong, what they were trying to accomplish. Spoken context is where "here's the bug" and "this is the part I care about" live, and those cues sharpen every other section.

This structure is an open format

None of the above is Clipy-proprietary shape. The section names, the required fields, and the rule that a reader must ignore what it does not recognize are all defined in the AREC spec (AREC is Agent Recording). Any recorder can emit a conformant markdown twin and any agent can consume one, which is what makes an agent-readable recording a format rather than a feature. If you want to see this document arrive in an agent's context in practice, the next page shows how to give agents video context step by step.

FAQ

What is a markdown twin? The markdown twin is the plain-text version of a Clipy recording that an AI agent reads instead of watching the video. It holds the summary, key moments, click targets, cursor paths, and transcript as markdown, fetched by appending .md to a watch URL.

Which sections are required and which are optional? Metadata, summary, key moments, and transcript are required in the AREC format. Cursor paths are optional: a document is complete and actionable without them. Console events, network events, and a signature block are reserved for future versions and do not ship today.

Why do key moments include x/y coordinates? Because an element name can be ambiguous when a page has several lookalike controls. The x/y pair pins a click to an exact point in the viewport, which disambiguates the target and makes the flow reproducible for an agent driving a browser or writing a test.

Can a text-only agent use the frames? A text-only agent skips the image and still reads every other field: summary, click targets, descriptions, cursor paths, and transcript. A vision-capable agent can additionally inspect the extracted still to check exact UI state. The document works for both.

Do I need an API key to read a markdown twin? No for public share links, which resolve to the markdown twin on their own. A personal key from clipy.online/settings/api-keys is only needed for private-library access and search across transcripts through the Clipy MCP server.

Record once. Paste the link. Your agent does the rest. clipy.online.