End to end testing for AI generated code, without selectors that break the next regeneration

Because the AI is allowed to rewrite the implementation any time. A normal test suite assumes the markup is stable across runs: today the Send button is a `<button class='btn-primary'>`, tomorrow it is still a `<button class='btn-primary'>`, and any selector you wrote keeps working. AI codegen breaks that contract. Cursor regenerates the form and decides to use Tailwind utility classes instead of a custom class; Claude Code switches the wrapper from a `<div>` to a `<form>` with a different aria-label; Lovable rewrites the entire page in a different shadcn variant. The user-visible behavior is the same: there is still a Send button, still labeled Send, still triggers a network call. But every selector and test id in your suite is now wrong. Flaky tests are timing problems. AI churn tests are contract problems, and the only way to make them survive is to bind the test to the user-visible contract (the accessible label, the role, the visible text) instead of the implementation.

Two design choices in the agent. First, the scenario file is plain Markdown that says what a user would do, not what the implementation looks like. A real scenario reads `Click Send` not `await page.locator('.btn-primary[type=submit]').click()`. Second, the agent resolves the click at runtime by calling its `snapshot` tool, which returns the page's accessibility tree with `[ref=eN]` IDs (agent.ts:27-30). The agent reads the tree, finds the element labeled Send, picks the ref ID it just received (e.g. `e22`), and calls `browser.click(element, ref)` with that ref. Tomorrow the AI regenerates the component, the className changes, the DOM rearranges, the new render still labels the Send button Send. Snapshot reads the new tree, the new ref is `e34`, the agent uses `e34`. The plan file is unchanged. The test passes.

Six bullets per case in a plain text file. A real one: `#Case 1: New chat, type a message, send it, verify it appears.` followed by `Navigate to /chat. Click 'New chat'. Type 'hello' into the message input. Click Send. Wait for the message to appear in the conversation. Assert the latest assistant turn is non empty.` That is the entire test. No imports, no fixtures, no selectors. Run it with `npx @assrt-ai/assrt run --url http://localhost:3000 --plan path/to/case.md`. The agent fills in the rest each run: which ref to click, when to wait, when to take a screenshot, when to write the result. The contract you maintain is `#Case` plus user intent.

True, and the accessibility tree is exactly how you disambiguate. The tree includes role, accessible name, and the surrounding hierarchy for every element. The agent's tool description for click (agent.ts:32-42) takes a human readable element description plus the ref. So you write `Click the Send button in the message composer` and the agent reads the tree, finds two buttons named Send, sees that one is in a region named Composer and the other is in a region named Settings, and picks the right ref. The disambiguation is in the description string, not in a CSS selector. If your UI has two truly indistinguishable buttons, that is a real accessibility bug for screen reader users; fix the bug, the test follows.

The surface is exactly 18 fixed tools, declared as a typed array at agent.ts:16-196. They are: navigate, snapshot, click, type_text, select_option, scroll, press_key, wait, screenshot, evaluate, create_temp_email, wait_for_verification_code, check_email_inbox, assert, complete_scenario, suggest_improvement, http_request, wait_for_stable. The agent cannot invent new tools. The Anthropic SDK rejects any tool name not in the schema. So while the agent can decide which of the 18 to call and what arguments to pass, it cannot generate, eval, or escape into arbitrary Playwright. `evaluate` does run JavaScript inside the browser (the `page.evaluate` analogue), but that is the only escape hatch and it returns a string the agent then has to act on. Most scenarios never call it.

If your UI is hand maintained and the selectors are stable, you should not switch. Playwright is great for that case. The reason this exists is that AI generated UIs are not that case. Every time Cursor regenerates a screen, half your selectors lie. The maintenance cost of a 60 line .spec.ts grows linearly with the number of regenerations. A six bullet #Case file is bound to user intent, so its maintenance cost is closer to zero across the same churn. You can also keep both: write Playwright for the slow moving pages, write #Case files for the AI churned ones, run them in the same CI pipeline. The two outputs are interoperable because the underlying engine is real Playwright through @playwright/mcp. Nothing is locked in.

Real Playwright through the official @playwright/mcp package. The Assrt agent is a thin layer that holds the agent loop, the scenario parser, and the test report writer; the browser primitives (navigate, click, type, snapshot) are forwarded to a Playwright MCP server that the agent spawns over stdio. browser.ts:116 shows `callTool('browser_snapshot')`, browser.ts:561 shows `callTool('browser_snapshot')` returning the tree. Anything Playwright can do is reachable. The video recording is the same WebM Playwright produces. The Chrome instance is the same. The accessibility tree is Playwright's. No custom rendering, no proprietary protocol.

The agent caps the snapshot at 120,000 characters before it gets sent to the LLM (browser.ts:523, the SNAPSHOT_MAX_CHARS constant). A typical app page is 5k to 20k characters of tree. A long marketing page or a Wikipedia article hits the cap and the agent gets a truncation marker telling it which refs were preserved. In practice this only matters for two cases: very long lists (tables, feeds) where you want to scroll then re-snapshot, and pages that dump huge inline SVG. The cap exists because the model context budget is the real constraint, not the browser. If your AI generated app routinely produces pages too large for 120k of tree, that is also a UX bug worth fixing.

Three pieces. First, your dev server runs against the PR branch. Second, the CI job calls `npx @assrt-ai/assrt run --url http://localhost:3000 --plan tests/case.md --json` and gates the merge on the JSON exit. Third, on failure the run uploads the WebM video and the JSON report as CI artifacts so the human reviewer (or the next agent loop) sees exactly what happened. Because the plan never names a selector, you can run the same plan against `main` and the PR branch with no changes and trust that any failure is a real regression in user visible behavior, not a selector that drifted. The MCP server form (`assrt_test`) lets a coding agent run the same checks itself before opening the PR, which is the closed loop case.

Five steps. Install the MCP into your editor (`npx @assrt-ai/assrt setup`). Generate the feature with Cursor or Claude Code. Write a four line #Case file describing the user flow in plain English. Run `assrt_test` against `http://localhost:3000`. Watch the recorded video to confirm the agent did what you expected. Total time on a small feature is under five minutes. The plan file goes into the repo. The next time the AI regenerates the same component, you re run the same plan; if it fails you have a real regression, if it passes you have proof the new code preserves the user contract.