AI agent browser automation reliability is not one prompt. It is five recovery primitives, named and open.

It is the probability that the agent finishes a real end-to-end scenario without human intervention, given that the page will re-render, the DOM will mutate, the model API will occasionally 529, OTP inputs will be split across six fields, and at least one tool call will throw. A reliable agent does not just get lucky on the happy path; it has a named response for each of those failure classes. Assrt ships five: fresh-tree injection on tool throw (agent.ts:932), a MutationObserver stability primitive (agent.ts:878), a retry-vs-fatal matrix keyed on error-message regex (agent.ts:642), a DataTransfer paste recipe pinned in the system prompt (agent.ts:234), and a sliding-window message pruning rule that preserves tool_use / tool_result adjacency (agent.ts:976). Reliability is the sum of those five, not the quality of one prompt.

The switch statement that dispatches every tool call is wrapped in a try/catch. On throw (any cause: the ref is gone, the element moved, the selector ambiguous), the catch at agent.ts:927-937 calls this.browser.snapshot() one more time, slices the fresh accessibility tree to 2000 chars, and formats a single string: "Error: {msg}\n\nThe action \"{toolCall.name}\" failed. Current page accessibility tree:\n{tree}\n\nPlease call snapshot and try a different approach." That string becomes the tool_result the model sees on the next turn. In practice the model then re-picks a ref from the fresh tree instead of retrying the stale one. No explicit retry counter, no hand-tuned selector healing; the recovery mechanism is: tell the model exactly what is on screen right now and let it re-plan.

Because network idle and load events tell you the network is quiet, not that the page is visually ready. A streaming LLM response, an optimistic UI that swaps once the real data arrives, or a virtualized list that keeps mounting rows can all complete their first render long before the visible content settles. wait_for_stable (agent.ts:872-925) sidesteps this by measuring the one thing that actually matters: mutations against document.body. It injects the observer, polls window.__assrt_mutations every 500ms, and only unblocks after stable_seconds (default 2) of true silence. The observer is disconnected and the globals are deleted when done, so a scenario that calls wait_for_stable fifty times does not leak state. This is the single biggest reason Assrt's AI agent browser automation reliability holds up against modern SPAs; fixed sleeps and network-idle both lie.

At agent.ts:642-660 there is a two-regex matrix around the provider call. /529|429|503|overloaded|rate/i means retryable: the agent sleeps (attempt + 1) * 5000 ms (so 5s, 10s, 15s across four total attempts) and emits a reasoning event so the run log shows the backoff. /tool_use|tool_result|invalid_request/i means fatal: these errors are almost always a message-shape problem that a retry cannot fix, so the scenario ends cleanly with scenarioPassed = false and summary = "API error: ...". Anything else throws back up to the outer try/catch which logs and moves to the next scenario. The practical effect: your CI job does not hang for 30 minutes waiting on an overloaded API, and it does not flip a real functional failure into a "flaky retry" ticket either.

A typical six-box OTP input (six separate <input maxlength="1">) is one of the worst shapes for an AI agent to type into because per-field typing requires tracking focus, digit index, and autosubmit timing, all of which drift by one if the scenario runs against a slightly slower page. agent.ts:234-237 pins the exact recipe in the system prompt: find an input with maxlength="1", take its parentElement, construct a DataTransfer with the full code, and dispatch a single ClipboardEvent of type "paste" on the parent. Browsers fan the digits across all six inputs in one event. The prompt explicitly tells the model not to modify the expression. The result is that OTP handling is deterministic: one tool call, one event, six fields filled.

Every turn appends both the assistant's response and the user-side tool_result block. In a 40-step scenario with screenshots after each visual action, that grows fast. The sliding window at agent.ts:976-996 keeps the first user message (which contains the scenario plan) and the most recent turns. The trick is the cut point: naively slicing by index can split an assistant block that contains a tool_use from the user block that contains its tool_result, which fails the next API call with invalid_request. The code walks forward from the initial cut index until it reaches an assistant or model message (the start of a new turn), and cuts there. No tool_use is ever orphaned. Long scenarios stay stable without tripping the retry-vs-fatal matrix.

The five recovery primitives are provider-agnostic because they operate below the model layer. Fresh-tree-on-throw, MutationObserver stability, and the DataTransfer paste recipe are browser-side. The retry matrix regexes on error strings, and 529 / 429 / 503 are standard HTTP so the same matcher fires for Gemini's rate limits. The sliding-window rule cuts at assistant/model boundaries (model is the Gemini role name, assistant is the Anthropic name), so both shapes work. What does change per provider is the tool-schema translation: the same TOOLS array is remapped into Gemini function declarations at agent.ts:278-301. The rest of the reliability stack, the part that makes AI agent browser automation actually survive in CI, is the same code path regardless of which API you point it at.

Yes. A signup flow against localhost that uses a disposable inbox does it in one pass. The agent navigates, snapshot returns a tree, and the first click either lands clean or throws (primitive 1). Form submission triggers React re-renders that wait_for_stable handles (primitive 2). The LLM is doing serious planning and may occasionally 529 (primitive 3). The verification code lands in a six-input OTP box (primitive 4). If it is a long multi-case run with screenshots, the message array eventually hits the sliding-window cut (primitive 5). The terminal snippet on this page shows exactly this flow; every line is the kind of log Assrt emits. The whole thing runs on your machine, against your localhost, with your scenario.md on your disk. Nothing is mocked, nothing is cloud-tied.

Because a smarter model does not change the failure classes; it just reshuffles which ones it hits more often. Stale refs come from the page, not the model. DOM churn is the page. 529s are the provider's rate-limit behavior, not the model's IQ. Six-field OTPs are a design choice by whatever site you are testing against. Message-history corruption is a protocol constraint. A reliability story that hinges on a model upgrade evaporates the moment the next model version ships with a different prompt style or tool-call shape. A reliability story built from named, versioned, file-and-line-numbered primitives lives longer than any single model generation. You can literally git blame every one of Assrt's.

Yes. The file referenced throughout this page (/Users/matthewdi/assrt/src/core/agent.ts, mirrored at /Users/matthewdi/assrt-mcp/src/core/agent.ts) is plain TypeScript, checked into the repo. Every line number cited is an actual line number you can open and read. There is no closed core, no proprietary backend that owns "the reliability logic." The optional hosted app at app.assrt.ai stores run artifacts for sharing; the test engine itself does not depend on it and can run the whole stack against localhost forever. Compare to closed-SaaS agent testing tools that charge $7,500 a month and lock your generated tests inside their dashboard; here, the tests are markdown on your disk and the engine is yours to fork.