A test automation guide for the era when the test is a plan, not a script

Test automation is the practice of turning 'did this feature work' into a check that a machine can repeat without you watching. In the scripted era, that check lived as code: selectors, waits, assertions, typed out by a person and re-typed every time the UI shifted. In 2026, the check is shifting to a plan: a short plain-English description of the scenario, executed by a runner that re-discovers the page on every step and heals stale selectors on its own. Assrt is an implementation of the second shape. Plans live at /tmp/assrt/scenario.md as lines like '#Case 1: log in as the trial user and confirm the dashboard loads'. A runner built on top of @playwright/mcp reads that plan, snapshots the DOM, clicks by accessibility reference, and records a video of the whole thing.

Automate the paths that would cost you the most if they silently broke. For most products that's three small things, not a suite of a hundred: sign up and log in, core happy-path conversion (checkout, send, create, whatever your primary verb is), and webhook or billing callbacks that come from a third-party and you can't easily manually retry. The reason to start there is cost of miss, not ease of automation. Everything else (admin panels, edge cases, rare flows) is a second pass. If your first automated scenario isn't 'new user signs up, gets a verification email, reaches the first save state', you are automating the wrong thing.

The runner's answer is DOM stability, not time-based waits. The Assrt agent exposes a tool called wait_for_stable (agent.ts:956 through 1009) that injects a MutationObserver into window, counts DOM mutations on a 500 ms poll, and returns as soon as the count stops changing for N consecutive seconds (default 2 s of quiet), with a hard cap of 30 s. The scenario writer never names a selector to wait for and never hard-codes a sleep. A fast page resolves in about two seconds; a streaming AI response resolves whenever the stream stops. Both paths use the same line in the plan.

The agent never stores a selector between steps. On every action, it calls snapshot() to get a fresh accessibility tree, each element tagged with an integer reference like [ref=e42]. The model says click(element='Sign in button', ref='e42'), the action fires, and the ref is thrown away. If the page re-renders and the next ref is different, the next snapshot finds the new one. There's no CSS selector that went stale, because there was no CSS selector to go stale. The self-healing is structural, not fuzzy-match. Source: agent.ts, TOOLS array lines 16 through 196, SYSTEM_PROMPT lines 208 through 227.

Because a human edits them. When a test fails halfway through, you open /tmp/assrt/scenario.md, rewrite a line, save, and the runner picks it up via fs.watch() with a one-second debounce (scenario-files.ts:97 through 103). The file also syncs back to the cloud store so the next invocation on the same scenario id sees your edit. Markdown means the diff is readable in a PR, the plan is readable without a parser, and the model can propose a corrected case using the same syntax the human uses. YAML adds nothing here: there's no nested structure beyond #Case lines and bullet steps. A DSL adds a learning cost for zero gain over plain text with a one-line header.

The runner exposes two tools for this. create_temp_email opens a disposable mailbox through mail.tm and returns the address; the agent fills your signup form with it. Then wait_for_verification_code polls that inbox for up to 120 seconds, extracts the code with a regex, and paste it into whatever OTP input the page has. For multi-field OTP layouts (six separate one-char inputs), the agent uses a hard-coded DataTransfer paste expression so React's onPaste handler distributes the digits in one render instead of losing focus between fields. See agent.ts lines 850 through 891 for the email tools and SYSTEM_PROMPT lines 234 through 236 for the paste expression.

Four things at minimum. One: a WebM video of the browser at 1600x900, recorded end to end via Playwright's devtools capability and served by a local HTTP server with Range support for seeking. Two: a structured JSON report with the list of assertions, each carrying a description, a passed boolean, and an evidence string. Three: a per-step event trace, so you can replay the agent's decisions without rewatching the video. Four: screenshots, one per visual action, indexed by step number. Assrt writes all four to /tmp/assrt/results/{runId}.json and opens the video in a player with keyboard controls at 5x playback by default. If your automation doesn't produce artifacts you can actually read later, it's a checkmark, not a test.

Three practices matter more than any specific framework choice. First, assert on outcomes, not layouts: 'the user reaches /dashboard and sees a heading that says Welcome' beats 'the third button in the second column gets clicked'. Second, use accessibility names, not CSS classes: the accessibility tree is what a screen reader sees and is the least-churn surface of a modern web app. Third, keep tests short. A plan with three to five steps is easy to rewrite when the feature moves. A 400-line Playwright spec with a nested page-object model ends up abandoned. A plan-driven runner makes the second and third practices the default, because the plan is three lines and the runner can only reach elements that are in the accessibility tree.

Both, but with different plans. In local dev the runner is an agent you talk to: 'run the signup flow, the session should stick, report back'. In CI it's a headless batch: on every pull request, replay the five smoke cases against the preview URL, fail the PR if any fail, upload the WebM artifacts to the run so a human can click through them later. The same plan file drives both. For Assrt the CI call is npx @assrt-ai/assrt run --url $PREVIEW_URL --plan-file tests/smoke.md --json --video --no-auto-open, and the output is a JSON report plus a .webm per scenario. If you want one knob instead of two, that's fine, but you pay for it either in slow local iteration or in flaky CI.

No. Unit tests verify that a pure function does what the name says; integration tests verify that two modules cooperate; end-to-end tests verify that a real browser and a real backend together produce the state a user expects. They're different instruments. Agent-executed plans are an end-to-end tool, optimized for flows that cross UI, auth, email, and billing. Unit and integration tests remain faster, cheaper, and more precise for anything inside one service. The mistake is using the wrong tool at each layer: end-to-end for a date-formatting helper is painful, unit tests for 'can a new user reach checkout in one session' is a fiction.