AI for software testing is three perception channels, not one

The sentence every other guide skips past

Read any introduction to this topic and you will see a variant of the same claim: modern tools let an AI agent read your application, decide what to do, and verify it worked. That sentence is true but it hides three very different operations behind one verb. What does "read your application" actually mean, at the wire level, when an LLM is driving a browser?

In Assrt the answer is three separate channels, each governed by a rule you can audit. The accessibility tree feeds the model on every turn. Screenshots are captured conditionally, and the condition is a named deny-list. External APIs are reachable through a tool that returns status code and body text, not pixels. Together these are the full perception surface of the agent. Individually, each exists because the others miss something.

Below is each channel with the exact source-level rule that governs it, and then the diagram of how all three combine in a single turn of the agent loop.

The three channels converging on one model

Channel one: the accessibility tree

The primary channel is the text tree the model receives from the snapshot tool. Playwright MCP walks the DOM and emits each interactive element with a role, a name, and a reference ID the model can quote back when it wants to click or type. The tree is text, which is cheap; it is structured, which is deterministic; and it uses stable ref IDs, which removes the need for the model to guess pixel coordinates or generate XPath selectors. Below is what the model actually reads on a typical turn.

The system prompt given to the model is explicit about how to use this tree: always call snapshot first, find the target element in the returned tree, cite its ref value in the next click or type_text call, and pass a human-readable description for logging. If a ref goes stale because the page changed, the next failure message nudges the model to re-snapshot and pick a fresh ref rather than retry with the old one.

Channel two: screenshots, conditional on which tool just ran

A screenshot on every turn would blow up both token cost and latency. Assrt's rule, defined as a literal list of tool names in agent.ts, is that only six tools trigger a screenshot after they run. The remaining eleven tools suppress it. The model still has full access to the accessibility tree on the following snapshot call, so suppressing the screenshot is not a loss of signal when the UI did not change.

This is the anchor fact of this guide, and you can verify it directly. Open /Users/.../assrt-mcp/src/core/agent.ts and search for the expression that starts with ![ near line 1024. The eleven-name array is the perception policy.

The shape of this rule matters beyond cost control. It encodes a design claim: when the agent calls a non-visual tool, the state of the page is unchanged, and sending another JPEG to the model would waste tokens on a duplicate frame. A screenshot after assert or http_request or wait_for_stable would be identical to the one taken after the last visual action. The deny-list is a lossless compression of the perception stream.

Channel three: HTTP requests to everything the browser cannot see

The browser is opaque to its own side effects. When a form submission sends a Telegram message, emails a webhook, or writes to a Slack channel, the browser only sees the confirmation screen. Any guide that stops at screenshot plus DOM assertion misses half the system under test.

Assrt exposes an http_request tool. The model passes a URL, method, headers, and optional body; the tool fires a 30-second-timeout fetch with a Content-Type: application/json default header and returns the status line plus up to 4000 characters of response text. The model then asserts against that payload the same way it asserts against a visible heading.

In a typical test this tool fires once or twice per scenario, not on every turn. It is the least common channel but the most decisive when what you care about is server-side state. Writers of test plans do not need to request it explicitly; the model decides to use it when the step description mentions an external service and the browser alone cannot confirm the outcome.

How the three combine in a single turn

Across one turn of the agent loop, the three channels are interleaved deterministically. The model snapshots (channel one), decides a tool call, executes it, optionally receives a screenshot (channel two) or an HTTP response body (channel three), and passes the combined context into its next decision. The loop does not terminate on a step count; it runs until the model emits complete_scenario or stops producing tool calls.

What this looks like next to the standard "AI sees your app" framing

Most introductions to this topic collapse the three channels into a single line. That framing is fine as a tagline and broken as a model of the system. Side by side, the difference between an auditable perception policy and a black box looks like this.

The numbers that define the policy

Every number below is countable in the source. The tool total comes from the length of the TOOLS array. The visual and non-visual splits come from the deny-list condition at line 1024.

Reading the policy yourself

This page is written as a derivation of the source, not an abstract summary. The claims above are each anchored to a specific section of the agent file. If you want to verify or extend them, these are the landmarks.