AI generated Playwright E2E tests: two shapes, only one survives the next UI regen

Yes, with a caveat that decides whether they keep passing. Two shapes are possible. One is one-shot codegen: feed a model your app, ask for `.spec.ts`, paste it into your repo. This works for the first run. It fails the moment the underlying app changes any className, aria label, or DOM nesting that the model hard coded into a locator, which on an AI generated UI is roughly every day. The other shape is plan-then-resolve: the model writes a short plain English plan describing user intent, and a runtime agent re-derives the Playwright calls each run from a fresh accessibility tree. Assrt is the second shape. Both are 'AI generated Playwright E2E tests'; only the second survives a regen.

Six bullets in plain English. A real one: `#Case 1: New user signs up and reaches the dashboard.` then `Navigate to /signup. Fill the email input with a fresh address. Fill the password input. Click Create account. Wait for the dashboard heading. Assert the URL contains /dashboard.` That is the entire test file. No imports, no fixtures, no selectors. The model that wrote it never had to guess what the Send button's className is. At run time the Assrt agent calls its `snapshot` tool, reads the accessibility tree, sees a button with the accessible name Create account, gets back ref `e7`, and calls `browser.click('Create account', 'e7')`. The Playwright code exists, it just exists for one run.

From a fixed 18-tool surface declared at src/core/agent.ts lines 16 to 198 in the assrt-mcp repo. The tools are: navigate, snapshot, click, type_text, select_option, scroll, press_key, wait, screenshot, evaluate, create_temp_email, wait_for_verification_code, check_email_inbox, assert, complete_scenario, suggest_improvement, http_request, wait_for_stable. The default model is `claude-haiku-4-5-20251001` (agent.ts line 9). The model cannot invent a 19th tool; the Anthropic SDK rejects any tool name not in the typed array. So the agent gets to decide which Playwright primitive fits the next step and which ref to pass to it, but it cannot generate freeform Playwright code that you then have to trust or review.

Real Playwright through the official `@playwright/mcp` package. The Assrt agent spawns it as a stdio subprocess; the resolver is at src/core/browser.ts line 284 (`require_.resolve('@playwright/mcp/package.json')`). Every browser primitive the agent calls forwards to that subprocess. The snapshot tool calls `browser_snapshot` (browser.ts line 590). The Chrome instance is Playwright Chromium. The video output is the WebM Playwright records. Nothing in the path is custom rendering or a proprietary protocol. Anything Playwright can do is reachable.

Because the contract those tests bind to does not survive. A normal test suite assumes today's `button.btn-primary[type=submit]` is still there tomorrow. AI codegen breaks that. Cursor regenerates the form and decides Tailwind utility classes are cleaner; Claude Code switches a `<div>` to a `<form>`; Lovable rewrites the whole page in a different shadcn variant. The user visible behavior is identical: there is still a Send button, still labeled Send, still triggers a network call. Every selector in your suite is now wrong. The one-shot `.spec.ts` flavor was great for slow-moving hand-maintained UIs; it is a tax on AI-generated ones.

No, you gain a different one. Each step writes three things to disk: a zero-padded PNG of the post-action page, an entry in `/tmp/assrt/results/latest.json` with the exact tool call and arguments, and a WebM recording of the whole run. You also get the resolved ref the agent clicked, which is more informative than a CSS selector because it tells you what the page actually exposed to assistive tech at that instant. The Playwright trace viewer still works because the run is real Playwright. The plan is what you commit; the resolved Playwright calls per run are diffable artifacts.

Yes, the 18-tool surface includes `create_temp_email`, `wait_for_verification_code`, and `check_email_inbox` for exactly that case. The agent provisions a disposable inbox, types its address into your signup form, then polls the inbox for the code and types it back in. The plan bullet looks like `Fill email with a fresh disposable address. Fill password. Click Sign up. Wait for verification code. Enter the code.` The agent does the rest. You do not write the inbox wiring; you do not hard code a test email address that goes stale.

Playwright's recorder watches you click, then emits `.spec.ts` with `getByRole`, `getByText`, and the like. The output is good Playwright but the same brittleness category as any one-shot codegen: it freezes a specific path through a specific render of the page. If the page changes, you re-record. The plan-then-resolve shape skips the freeze. The recorder is great when you already know the flow and the UI is stable. The plan format is great when the UI churns or when you want a non-Playwright-fluent teammate (or a coding agent) to maintain the suite by editing a Markdown file.

Same place you run any Playwright E2E suite. The Assrt CLI is `npx @m13v/assrt discover https://your-app.com` for a fresh plan, then `npx @m13v/assrt run --url $URL --plan path/to/case.md --json` per CI job. The JSON exit gates the merge. On failure, the WebM video and the latest.json report upload as CI artifacts. Because the plan never names a selector, the same plan runs against `main` and the PR branch with no diff; any failure is a real regression in user-visible behavior, not a selector that drifted. The MCP form (`assrt_test`) lets a coding agent run the same checks before opening the PR.

Three screenshots taken at different scroll depths, the page's accessibility tree truncated to about 8000 characters, and a system prompt that lives at src/mcp/server.ts line 219. The prompt opens with `You are a Senior QA Engineer generating test cases for an AI browser agent.` It enumerates the agent's capabilities and explicit limits (cannot resize the browser, cannot inspect CSS, cannot test network errors) so the generated cases stay inside what the runtime agent can actually execute. The output is `#Case` Markdown, which the executor parses one case at a time. The whole pipeline is open source, MIT, and the prompts are not minified or obfuscated; they are 18-line literals you can read in the repo.