Accessibility tree vs screenshots for Playwright AI testing: which is the agent's source of truth?

Token figures from public benchmarks of Playwright MCP. The 4x ratio is what shows up on your bill at the end of the month.

Two paradigms, one runner

A Playwright AI agent has, in principle, two ways to perceive a page. It can read a structured representation of what is on the page, or it can look at the rendered pixels. These are not equivalent. They are not even the same kind of thing. One is a tree of facts, the other is an image to be interpreted. The choice of which one feeds the model on every step decides almost everything about how the agent behaves: its latency, its determinism, its cost, the quality of its failure logs.

Toggle between the two below. Same login flow, two different substrates.

What this looks like in code

The shape of the test loop is not subtle. With a screenshot-first agent, the model is the locator. With a tree-first agent, the accessibility tree is the locator and the model just picks which ref to act on. Toggle between the two:

// Vision-first agent: input is a 500 KB JPEG.
// The model "sees" the page and decides what to click.
//
// What the model sees: pixels.
// What the test code says: nothing structural.
// What the failure log shows: (x: 482, y: 311), good luck.

const screenshot = await page.screenshot();
const action = await visionModel.decide({
  image: screenshot,
  goal: "sign in with the email and password",
});
await page.mouse.click(action.x, action.y);

The right column is what Assrt does, and what Microsoft's Playwright MCP does by default. The left column is what vision-first tools do, and it is what most teams pattern-match to when they hear “AI testing.” The pattern match is wrong: AI testing does not require pixels.

The anchor fact: a JPEG quality of 50

The most direct evidence that the tree is the substrate and the screenshot is an artifact lives in the open-source Assrt code. In assrt-mcp/src/core/browser.ts, the screenshot tool is called like this:

async screenshot(): Promise<string | null> {
  const result = await this.callTool(
    "browser_take_screenshot",
    { type: "jpeg", quality: 50 },
  );
  // ...
}

That is a deliberate downgrade. If you were betting your test decisions on the screenshot, you would push the format to PNG and the quality to 95. You would never throw half the bytes away on purpose. Assrt does throw them away on purpose, because the screenshot is not how the agent decides what to click. It is a thumbnail for the run viewer, a frame on the failure overlay, something a human glances at while scrolling a CI report. Quality 50 is plenty for that.

The other anchor fact: a 120,000 character ceiling

The flip side of trusting the tree is dealing with its weight. A dense admin dashboard or a Wikipedia article serializes to an accessibility tree that will overflow any agent's context window. Assrt's answer is a hard cap, also in assrt-mcp/src/core/browser.ts:

/** Max characters for a snapshot before truncation
 *  (roughly ~30k tokens). */
private static readonly SNAPSHOT_MAX_CHARS = 120_000;

When the cap is hit, the snapshot is truncated at the last clean newline and a marker is appended so the agent knows the view was partial. Practically this means tests should target the section of the page they care about (a dialog, a form, a table) rather than crawling everything. That is a useful constraint, not a limitation. Tests that try to operate on entire pages are usually testing the wrong thing.

The third anchor fact: screenshot polling, removed

Earlier versions of the agent polled a screenshot every 1.5 seconds while a scenario ran, to keep a live preview flowing into the run viewer. The current version does not. From assrt-mcp/src/core/agent.ts:

// Screenshot polling removed: screenshots are now
// captured only after visual actions (navigate, click,
// type, select, scroll, press_key) to avoid duplicate
// /redundant captures that waste tokens and time.

The agent now snapshots the accessibility tree before each decision, and grabs a screenshot only when something visible actually changed. Two consequences. The token budget on a long scenario drops sharply. And a human reading the run sees a frame for every meaningful state change, not 60 frames of a static loading spinner.

The agent loop, in five tools

Putting all three anchor facts together, here is the loop the agent actually runs. Notice that snapshot is called twice (once before, once after), and screenshot is called once, only after the click changed the page state.

Head to head

What lives in an aria snapshot

The point of the tree is not that it is small (although it is). The point is that everything in it is the kind of thing a test wants to target, and almost nothing in it is coupled to the markup that designers rewrite every sprint.

What it does not contain: CSS classes, inline styles, image pixel data, JavaScript event handlers, or markup that exists purely for layout. That omission is the point. If a thing is not in the accessibility tree, your tests cannot target it, and your users with assistive tech cannot use it either. Both gaps are the same gap.

When screenshots earn their keep

The honest position is not that screenshots are bad. They are the wrong substrate for behavioral test decisions, and the right substrate for a small set of cases.

Treat the two as different runners. Behavioral tests run against the accessibility tree. Visual regression runs against pixels and lives in its own pipeline. Mixing the substrates is how teams end up with flaky pixel diffs every time the design system tweaks a token, and missed regressions every time someone removes a button's name.

What Assrt gives you, in one paragraph

Assrt is the open-source, MIT-licensed implementation of the tree-first pattern described above. It runs as a Node CLI plus a local MCP server. Under the hood it spawns @playwright/mcp as a child process, so every click, type, and snapshot is a real Playwright primitive. The agent reads the accessibility tree, picks a ref, calls real Playwright. The generated tests are standard Playwright spec files you commit to git. No rented YAML, no proprietary cloud editor, no per-seat fee.

npx @m13v/assrt discover https://your-app.com

For a deeper guide on why the tree is the right target, read the long form on accessibility tree web testing. If you want a side-by-side against a competing vision-first product, see Playwright vs AI browser automation.

Frequently asked questions