Sentence to Playwright test generator: a runtime, not a compiler

A sentence to Playwright test generator takes plain English input and produces a runnable Playwright test. Most tools in this category are compilers: you describe a flow in English, the tool emits a .spec.ts file with locators and expect chains, and that file is what runs. Assrt is not a compiler. The tool stores your sentences in a Markdown file at /tmp/assrt/scenario.md, splits them into named #Case blocks via a regex (parseScenarios at /Users/matthewdi/assrt-mcp/src/core/agent.ts:620), then hands each block to an LLM along with a fresh accessibility snapshot. The LLM reads the sentence, looks at the live page tree, and chooses one of 17 typed tools (snapshot, click, type_text, scroll, assert, and so on, defined at agent.ts:16-196) for every step. There is no .spec.ts on disk. The translation from sentence to Playwright tool call happens once per run, against the page that exists right now.

The dispatch loop is at /Users/matthewdi/assrt-mcp/src/core/agent.ts lines 693-747. Each iteration sends the current message stack to Claude with system: SYSTEM_PROMPT (line 198 in the same file) and tools: TOOLS, gets back zero or more tool_use blocks, and runs them. The tools are concrete wrappers around @playwright/mcp primitives: case 'click' on line 784 calls this.browser.click(element, ref); case 'type_text' on line 791 calls this.browser.type(element, text, ref); case 'snapshot' on line 778 calls this.browser.snapshot(). The model is the only translator. Your sentence 'Click the Sign Up button' is never compiled into a string like page.getByRole('button', { name: 'Sign Up' }).click(). It becomes a tool_use with name='click' and input={ element: 'Sign Up button', ref: 'e17' }, where 'e17' came from the snapshot the model just read.

scenario.md, the same Markdown file the generator wrote. The on-disk layout is defined at /Users/matthewdi/assrt-mcp/src/core/scenario-files.ts: scenario.md is the plan, scenario.json is metadata (UUID and name), results/latest.json is the most recent run report. You can grep scenario.md, diff it on a pull request, render it on GitHub, or paste half of it into a different project. There is no generated TypeScript next to it that needs to stay in sync. When CI runs your suite, the runner reads the same file, gets a fresh accessibility snapshot from the live app, and re-decides every tool call. So the artifact you commit and the artifact CI runs are literally one file of plain English.

navigate, snapshot, click, type_text, select_option, scroll, press_key, wait, screenshot, evaluate, create_temp_email, wait_for_verification_code, check_email_inbox, assert, complete_scenario, suggest_improvement, http_request, wait_for_stable. Full schemas are at /Users/matthewdi/assrt-mcp/src/core/agent.ts lines 16-196. Three of those (create_temp_email, wait_for_verification_code, check_email_inbox) wire a disposable inbox so signup flows can complete an OTP loop without a human. wait_for_stable injects a MutationObserver into the page (agent.ts:956 onward) and waits until DOM mutations stop, which is what an English sentence like 'Wait for the search results to load' becomes at runtime. http_request lets the agent verify external side effects (a webhook fired, a Telegram bot got the message). The toolset is small enough to read in one sitting and big enough that almost any user-facing sentence has a tool that fits.

No, and this is the load-bearing reason a runtime beats a compiler for this category. A compiled test contains a frozen reference: page.locator('button[data-testid=signup-v3]'). When the build pipeline ships a new design system and that data-testid becomes signup-v4, the compiled test breaks and your CI goes red. Self-healing tools paper over this by guessing the right new selector after the failure. With Assrt, your sentence 'Click the Sign Up button' contains no selector. At run time the agent calls snapshot() on the new page, gets the new accessibility tree (which still has a button with the visible text 'Sign Up'), and dispatches click with the new ref. There is nothing to heal because nothing was ever bound. The only way the sentence breaks is if the visible affordance the sentence describes is genuinely gone from the page, in which case the test should fail because the user-facing flow is gone.

Codegen is the opposite shape. You drive a real browser by hand, Codegen records your clicks and keystrokes, and it emits TypeScript with locators inferred from your actions. The input is not sentences, it is a recording. The output is a .spec.ts file that lives in your repo and that you maintain forever. Both Codegen and Assrt write tests so you do not have to, but they pick different inputs and different artifacts. Use Codegen when you have a flow you can demonstrate but cannot articulate, and you want a TypeScript spec to commit. Use Assrt when you can describe a flow in English and you want the description itself to be the durable test, with no compiled artifact in between.

On the first sentence of a scenario it gets one user message containing the initial page screenshot (base64 JPEG), the full accessibility tree from the first snapshot call, the scenario name, the full sentence body, and (optionally) a Pass Criteria block and any test variables. The exact assembly is at agent.ts:679. After that, it gets back tool_use blocks, the runner executes them, and the results (snapshot text, navigation outcome, click confirmation, screenshot bytes) come back as tool_result messages on the next loop turn. The model is never asked to memorize selectors between turns; every snapshot is a fresh reading of the live page. The reason this is cheap enough to run in CI is that snapshots are accessibility text, not screenshots: a 41-node accessibility tree fits in a few hundred tokens, while a screenshot is JPEG-compressed and only emitted after visual actions to keep the round trip lean.

Three rules, all enforced by the PLAN_SYSTEM_PROMPT at /Users/matthewdi/assrt-mcp/src/mcp/server.ts:219-236 even when you write the sentences yourself. First, name the affordance by what a user sees, not what the developer named it: 'Click the Sign Up button' beats 'Click the primary CTA in the hero'. Second, verify observable things: 'Assert: the URL path is /dashboard' or 'Assert: the heading reads Welcome' both work; 'Assert: the button has the right CSS color' will fail because the agent has no CSS introspection tool. Third, keep cases short: 3-5 sentences per #Case is the sweet spot, because each sentence costs a model round trip plus a snapshot. A single 20-step #Case spends more wall-clock and is harder to diagnose when it fails. The same prompt also caps you at 5-8 cases per scenario for the same reason. None of this is enforced by a parser; the prompt sets the norm and the model executes accordingly.

Edit freely. The MCP server's instructions explicitly call this out: 'Read /tmp/assrt/scenario.md to see the current plan; Edit /tmp/assrt/scenario.md to modify test cases; changes auto-sync to cloud storage.' Each test run is auto-saved with a UUID (visible in scenario.json), and you can re-run the same scenario later with assrt_test({url, scenarioId: '...'}). The common workflow is: generate a draft from a URL, open scenario.md in your editor, delete the cases you do not care about, rewrite a sentence that the model phrased awkwardly, add a new case the generator missed (often something behind auth that the URL-only generator could not see). Save, run. The runner picks up the new file by mtime.