AI-generated Playwright tests review: watch the run, not the .spec.ts

A .spec.ts file can be syntactically perfect and still not verify the feature. The model can invent a selector like getByTestId('submit-btn-v2') that matches nothing, write an assertion against a string that never appears, or take a happy path that skips the actual bug. Reading the code tells you the test compiles; only running it against a real browser tells you the test verifies. That is the gap: reviewing code confirms syntax, reviewing a run confirms behavior.

First, /tmp/assrt/scenario.md, the plan in plain Markdown with `#Case N:` blocks. Second, the accessibility-tree ref the agent clicked (ref='e5', ref='e12', etc.) pulled from a `browser_snapshot` call on the live page. Third, the per-step PNG at /tmp/assrt/<runId>/screenshots/NN_stepN_action.png and the standalone WebM at /tmp/assrt/<runId>/video/recording.webm. Reading those three answers the question 'did this test actually verify the feature' in about a minute.

Playwright MCP's `browser_snapshot` tool returns the page as an accessibility tree and tags every focusable element with an opaque id like `e5`. When the agent clicks, it passes `ref='e5'` back to Playwright, which looks up the element by that id on the live page. There is no CSS selector to hallucinate. If the snapshot did not contain the element, the agent cannot click it; if the snapshot says it is a button, it is a button. As a reviewer, you can open the run's trace, see `click ref='e5' element='Submit order button'`, and know the agent clicked a real, identified element rather than guessing a selector.

Four paths. `/tmp/assrt/scenario.md` holds the #Case plan. `/tmp/assrt/results/latest.json` holds the full run report as JSON (plan snapshot, per-case timing, assertion evidence). `/tmp/assrt/<runId>/screenshots/` holds the PNGs, named `00_step1_init.png`, `01_step2_click.png`, and so on (zero-padded so shell sort stays correct). `/tmp/assrt/<runId>/video/recording.webm` is the standalone recording. You can `cat`, `jq`, `open`, and `ffmpeg` every one. There is no dashboard required to do the review.

Playwright codegen emits brittle selectors like page.locator('div:nth-child(3) > button') that look fine in a pull request but fail the week you ship a new div. Reviewing it means guessing which selectors will drift. Assrt never commits CSS selectors to the plan; the plan names elements in English ('click the Add to cart button') and the runtime pairs each click with the accessibility ref at run time. You review the Markdown for intent and the video for behavior, not a file of selectors that will rot.

Hallucinated APIs are a real pain point with code-generating tools. Assrt sidesteps that class of bug because the agent does not emit Playwright code at all. It calls a fixed list of 14 tools defined in `/Users/matthewdi/assrt-mcp/src/core/agent.ts` lines 14 to 196: navigate, snapshot, click, type_text, select_option, scroll, press_key, wait, screenshot, evaluate, create_temp_email, wait_for_verification_code, assert, complete_scenario, plus http_request and wait_for_stable. Every call goes through Playwright MCP's tool schema. If the model tries to invent a method name, the MCP server rejects the call. You cannot hallucinate a Playwright API when the surface is a tool schema.

Call `assrt_diagnose`. The prompt at /Users/matthewdi/assrt-mcp/src/mcp/server.ts line 240 feeds the failing case, the evidence, and the URL to a senior-QA-engineer persona that returns a root-cause verdict: bug in the app, flaw in the test, or environment issue, plus a corrected `#Case` you can paste back into scenario.md. Review workflow: watch the video to see what actually happened, then run diagnose to classify the failure. Most drift-type failures are the test; most regressions are the app.

Rough benchmark: the bundled video player auto-plays at 5x speed with keyboard shortcuts (Space to pause, 1/2/3/5/10 digit keys to change speed, arrows to seek 5s). A 30-second test run plays back in 6 seconds. Add 30 seconds to skim the #Case plan and 30 seconds to check the assertion evidence in results/latest.json. A reviewer can clear five tests per minute this way, which is faster than reviewing five .spec.ts files by eye and much more honest about whether they actually pass.

Yes. The scenario is a plain Markdown file with `#Case N:` blocks. Commit `/tmp/assrt/scenario.md` to your repo, or copy each block into a `tests/scenarios/` directory. `git diff` shows exactly what changed between runs in human-readable text, not a DSL. The runtime watches the file with fs.watch (scenario-files.ts line 97); save in any editor and the cloud copy re-syncs with a 1-second debounce. Reviewer behavior is the same workflow they already use for any other Markdown file in the repo.

Two answers. One: Assrt does not replace an existing Playwright suite; it runs alongside. If you have 400 .spec.ts files, keep them. Use Assrt for the tests a human would rather describe in English than write in code, and for the flows the AI keeps failing to author correctly. Two: the agent connects to a real Chrome instance via Playwright's `--extension` mode, so the runtime is the same Playwright you already use. You are not adopting a parallel testing stack; you are writing Markdown that drives it.