Assertion coverage in generated Playwright tests: the one rule that turns click logs into tests

Assertion coverage is the share of intent statements in your test plan that produce a verifiable check against the page at run time. Not source-code coverage (lines of your app executed) and not selector coverage (UI areas the agent visited). The distinction matters because most AI test generators are great at coverage of the first two and terrible at the third. They click through a flow, the page changes, they call it pass, and you never see a single expect() or assert(). A test with 12 navigations and 0 assertions is a click log, not a test. Assertion coverage is the fraction of 'I should check that X' statements in the plan that turn into an actual recorded assertion in the run report.

Three reasons compound. First, the model is rewarded by training data for producing 'working' code; tests that pass with fewer assertions look cleaner. Second, asserting on a UI state requires the model to understand what specifically to check, which is harder than knowing what to click; vague plans like 'verify it works' resolve to vague or skipped assertions. Third, when the model is unsure whether a check is possible (the element it expected is not there), the path of least resistance is to omit the check rather than to flag it as failed. Almost every AI-driven test runner inherits at least two of these failure modes. The fix has to happen at the prompt and runtime layer, not at the model layer.

With a section in the system prompt titled 'Assertion Coverage (CRITICAL — non-negotiable)' at /Users/matthewdi/assrt-mcp/src/core/agent.ts lines 256 to 262, plus a per-run reinforcement in the user prompt at line 731. The rule has five parts. Every line in the scenario steps that starts with Verify, Check, Assert, Confirm, or Ensure is a mandatory assertion. The agent must produce exactly one assert tool call per such line. Two bullets cannot be silently merged into one assert. A bullet cannot be skipped because it seems redundant or hard to check; if verifying is genuinely impossible (the element is missing), the agent must call assert with passed=false and explain what was missing as evidence. The agent cannot add extra assertions outside the scenario. The description field of each assert call must closely match the wording of the bullet, so a reviewer can match assertions to bullets one-to-one.

The assert tool has three required fields: description (what you are asserting), passed (boolean), and evidence (free-text rationale). The schema is at agent.ts lines 132 to 143. A real call for a successful 'Verify the dashboard heading appears' bullet looks like: description='Dashboard heading appears after login', passed=true, evidence='Heading element with role=heading and accessible name "Welcome back, Sarah" found at ref=e14 after wait_for_stable settled at 2.4s'. A real call for a failed bullet, where the element was missing: description='Confirm the success toast renders for 3 seconds', passed=false, evidence='No element with role=alert or text matching "saved" found in accessibility tree across 3 snapshots over 4.1s. Snapshot ref=e8 was the closest candidate but its accessible name was "Loading".' Both calls land in /tmp/assrt/results/latest.json as structured records you can grep.

Two things. First, the evidence field is required and reviewable; the agent has to write a sentence describing what it observed, and 'looks good' is a tell on review. Second, the assertions array in the run report (assertions.push at agent.ts line 949) is a structured record of every assert call with timestamps; the per-step PNG and the WebM recording let you confirm that what the agent claims it saw was actually on the page at the assert moment. The structural defense against fake passes is that the assertion was claimed against a specific accessibility-tree snapshot you can replay. The cultural defense is that the description and evidence are short prose a reviewer can read in a few seconds per bullet, not a 200-line .spec.ts they have to skim.

It is a different layer that wraps real Playwright underneath. Web-first matchers like expect(locator).toBeVisible() retry inside the matcher when the page is in a loading state but the selector is fixed. Assrt's assert tool works the other way around: the agent calls wait_for_stable first to let the DOM settle (default 30s timeout, 2s stable window), then takes a fresh accessibility-tree snapshot, then calls assert with evidence drawn from what it just saw. If you want the pixel-level toHaveScreenshot or computed-style toHaveCSS matchers, keep them in a separate spec; Assrt does not monopolise the browser. The split is: Assrt covers behavior-level assertions (element visible, text appeared, URL changed, input accepted, modal dismissed); keep classic Playwright for pixel diffs and performance budgets.

Each run writes /tmp/assrt/results/latest.json. The scenarios[].assertions array contains one entry per assert call: { description, passed, evidence }. To compute assertion coverage, parse the scenario plan from /tmp/assrt/scenario.md, count the lines that start with Verify/Check/Assert/Confirm/Ensure, and compare to assertions.length. Because the agent is required to emit exactly one assert per such line, the expected coverage is 100%. Any mismatch is a bug in the run, not a tolerated drift. The mismatch usually shows up as a complete_scenario call that the agent made before covering all the bullets, which surfaces in the JSON as a partial scenario. The pipeline treats partial coverage as a failed scenario, not as a passing one with a footnote.

No. The rule is symmetric: the agent must produce one assert per Verify-class bullet, and it must not add extra assertions outside the plan. A pure exploratory scenario with no Verify bullets produces zero asserts and one complete_scenario call with a written summary, which the reviewer can read to decide whether to add Verify bullets next time. The point is to align the assertion count with the plan's stated intent. Over-asserting is a different failure mode (it makes the test brittle to incidental UI changes) and the prompt rejects it for the same one-bullet-one-assert reason.

Yes, the prompt is a string literal in the open source TypeScript file. The repo is https://github.com/assrt-ai/assrt-mcp. The 7 lines from agent.ts:256-262 (system prompt) plus the user-prompt reinforcement at line 731 are the entire mechanism. There is no separate config, no enterprise tier feature, no plugin to install. If you fork the repo, the rule comes with it. If you write your own plan in plain English with Verify-class bullets, the rule applies. If you remove the section from the prompt, you lose the guarantee but the tool surface still works; that is your choice. The whole point of putting it in plain text in the system prompt is that you can read it, evaluate it, and change it.

QA Wolf at roughly $7.5K per month gives you human QA engineers who write Playwright with rich assertions; coverage is high because humans are in the loop. Momentic uses proprietary YAML where assertion mechanics are abstracted away, which makes coverage harder to audit because you cannot easily diff a YAML scenario against a list of assert calls. Hand-written Playwright is whatever your team's review discipline produces; coverage depends on the reviewer. Assrt's lane is teams who want AI to write the plan and execute the test but still want a verifiable coverage guarantee. The 7-line rule is the smallest thing that makes that lane real. Open source, free, MIT license, no SaaS dashboard.