Testing AI without flaky assertions

Two reasons, both structural. First, streaming: an LLM chat reply arrives as tokens over two to five seconds, so a selector-based wait triggers on the first token and your test asserts against a half-finished sentence. Second, non-determinism: the same prompt returns a different paraphrase on every run, so `expect(text).toBe("…")` fails even when the answer is correct. The two problems have two fixes: an adaptive wait that measures DOM quiet instead of elapsed time, and a semantic assertion that judges meaning instead of comparing strings. Assrt builds both into the same Haiku 4.5 agent loop so you do not need a second model acting as judge.

assrt-mcp/src/core/agent.ts, lines 956 to 1009. It is the `case "wait_for_stable"` block inside the main tool switch. The tool itself is declared earlier at lines 186 to 195, with two parameters: `timeout_seconds` (default 30, clamped to 60) and `stable_seconds` (default 2, clamped to 10). At runtime, the agent injects `window.__assrt_mutations = 0` and a `new MutationObserver` that increments that counter on every DOM change, then polls the counter every 500 ms. A quiet period is declared when the counter has not changed for stable_seconds. When the page goes quiet, the observer is disconnected and both globals are deleted. You can clone assrt-mcp and grep for `wait_for_stable` to see the whole thing on one screen.

The `assert` tool, declared at agent.ts:133-144 and implemented at 893-904, takes three fields: `description` (what you claim is true), `passed` (boolean), and `evidence` (the model's English sentence describing what it saw). The Haiku 4.5 driver reads the page snapshot, decides whether the claim holds, and writes evidence like "the assistant response describes the refund timeline as 5 to 7 business days, which matches the documented policy." That sentence is stored alongside the pass/fail bit in the run's events.json. When you say `Assert the bot explained the refund policy`, the model decides, and you read its rationale afterward instead of debugging a brittle regex.

No. The driver and the judge are the same model in this design. Haiku 4.5 (pinned at agent.ts:9 as `DEFAULT_ANTHROPIC_MODEL = "claude-haiku-4-5-20251001"`) performs the click, reads the snapshot, writes the evidence, and sets the passed boolean in one tool call. The alternative architecture, where a driver model records the conversation and a separate judge model scores it, doubles your LLM cost and adds a second round of hallucination. Collapsing the two roles keeps latency low (one Haiku request per assertion) and makes the full audit trail visible in events.json.

A string of the form `Page stabilized after 3.4s (412 total mutations)`, written to the step log at agent.ts:998. If it times out instead, the string is `Timed out after 30s (page still changing, 1874 mutations)`. Both results are stored in the step record that lands in events.json under the run directory. That counter, the total mutation count, is the single most useful diagnostic for testing AI UIs: a chat response that keeps mutating past the timeout is almost always one of three things — a typing indicator that never stops, a retry loop firing on a 5xx, or a streaming endpoint with no end-of-stream signal.

The three temp-email tools (create_temp_email, wait_for_verification_code, check_email_inbox, declared at agent.ts:114-131) run independently of the stability wait. A typical AI product signup looks like: create a throwaway address, type it into the signup form, click continue, call wait_for_stable because the magic-link email takes a variable amount of time to land, then wait_for_verification_code with its own 60 s poll, then assert. The stable wait protects you when the signup UI streams a welcome message; the verification wait is a separate poll against the disposable inbox. Either can run inside a single `#Case` block without you having to juggle promises.

Stored. At agent.ts:897 the handler pushes `{description, passed, evidence}` into the `assertions` array for the current scenario, and that array is serialized into the ScenarioResult that lands in /tmp/assrt/results/latest.json and /tmp/assrt/results/<runId>.json after the run. The record is durable: you can cat the JSON a week later and still see the exact sentence the model wrote about the page. For testing AI features, that sentence is often more useful than the screenshot — it captures the model's understanding of what the assistant said, which is the thing you actually wanted to verify.

The wait block hits its timeout_seconds ceiling and returns the `Timed out after ...` string. Control passes back to the driver model, which then calls snapshot and decides what to do. Usually it calls assert with passed=false and evidence like "the response was still streaming after 30 s, which is outside the product's documented 10 s SLA." That failure mode is a legitimate bug in the AI product — the whole point of the primitive is to turn a fuzzy "it feels slow" complaint into a numeric fact (mutation count over elapsed seconds) that you can attach to the bug report.

Yes, the primitive is 54 lines of TypeScript and has no Assrt-specific dependencies inside the evaluate() block. You can copy the MutationObserver expression from agent.ts into a Playwright test and call it from your own runner. What Assrt adds on top is the loop: an LLM driver that decides when to call wait_for_stable, what to assert about afterwards, and what to write into the evidence field. Rebuilding that loop yourself is the bulk of the work. But the browser-side primitive is literally copy-paste, and nothing prevents you from using only that piece.

Category tools like mabl and Testim ship a similar pattern behind a proprietary DSL: you describe the assertion in English, they route it through an internal judge model, and you pay per-test execution. The resulting artifacts (videos, assertions, model outputs) live behind their dashboard. Assrt gives you the same pattern in open source: the tool list is 18 entries in agent.ts, the MutationObserver is a copy-pasteable block, the evidence field is a plain string in JSON on disk, and the cost is the Anthropic tokens for one Haiku 4.5 call per assertion. For an eight-case suite, you are looking at fractions of a cent per run versus an annualized contract.

A complete `#Case` for testing a RAG chatbot looks like this: `1. Navigate to /chat. 2. Type "What's your refund policy?" into the message input. 3. Click Send. 4. Call wait_for_stable with stable_seconds=3 to ride out the stream. 5. Assert the assistant response describes the refund policy, and provide the response text as evidence. 6. Assert the response cites at least one source URL visible in the message footer.` Six plaintext lines, one `#Case` header, zero TypeScript. The Haiku driver decides which of the 18 tools to call for each line. The run artifacts land under /tmp/assrt/<runId>/ and include the WebM recording so you can watch the stream replay at 5x.

Safer than a fixed wait, bounded by design. The outer timeout (default 30 s, max 60 s) always fires even if the DOM never stabilizes, so a hung app cannot wedge the runner. After the timer or the stability condition returns, the cleanup block at lines 990-994 unconditionally disconnects the observer and deletes `window.__assrt_mutations` and `window.__assrt_observer`, so a subsequent navigation does not inherit a stale observer. A chat UI that genuinely streams forever (say, a broken backend that keeps emitting keep-alive tokens) will cost you exactly one timeout's worth of time and leave no residue.