Testing for AI writing

Why this is different from any other E2E test you have written

A normal end-to-end test has a clear shape. Click a button. Wait for a known thing to appear. Assert the page now equals a known value. Order placed. Email sent. Cart total $42.99. Every step can be expressed as a transition between two static states, and every assertion can be expressed as exact equality.

An AI writing feature has none of those properties. Click Generate and the response begins arriving as a Server-Sent Events stream. The DOM updates token by token: an empty paragraph, then one word, then a sentence fragment, then a full sentence, then more. The total time depends on prompt length, model load, and network jitter. The text itself is the output of a language model that almost never produces the exact same string twice. And the whole thing happens inside a region whose descendants are being rewritten while your test is trying to find them.

The test most teams write first

Most engineers reach for the same template. A 5-second timeout. A string-equality assertion against a sample response. It looks reasonable in review and ships green on the first run. Then the model gets faster on Tuesday and the timeout becomes wasteful; the model gets slower on Friday and the timeout races the stream; a routine prompt-template tweak changes the wording and every equality assertion in the suite goes red. The test is technically correct and operationally useless. Toggle below to see the same feature expressed two different ways.

// Hand-written Playwright test against an AI writing feature.
// Looks reasonable. Flakes on roughly half of CI runs.

import { test, expect } from "@playwright/test";

test("generate draft", async ({ page }) => {
  await page.goto("http://localhost:3000/editor");
  await page.getByRole("button", { name: "Generate" }).click();
  await page.fill("textarea[name=prompt]",
    "write a short blog intro about cold-water swimming");
  await page.keyboard.press("Enter");

  // Pick a number, any number.
  await page.waitForTimeout(5000);

  // Race condition: the stream may still be writing when this runs.
  await expect(
    page.locator(".response-text"),
  ).toHaveText(
    "Cold-water swimming is the act of immersing yourself...",
  );
});

Primitive 1: a stability wait that adapts to streaming time

The first primitive sits at /Users/matthewdi/assrt-mcp/src/core/agent.ts line 186. The tool is called wait_for_stable. Its description, served to the LLM driving the test, names the case it was built for: chat AI responses, loading states, and other async DOM activity. The system prompt at line 250 is even more explicit: “When the page has loading states, streaming AI responses, or async content, use wait_for_stableto wait until the DOM stops changing. This is better than wait with a fixed time because it adapts to actual load speed.”

The implementation lives at line 956. It injects a MutationObserver onto the page, observing document.body with childList: true, subtree: true, characterData: true. It increments a counter on every mutation, polls that counter every 500ms, and exits as soon as the count has been stable for the requested window. The result string returned to the agent looks like Page stabilized after 12.4s (847 total mutations) on success or Timed out after 30s (page still changing, 1924 mutations) when the model is genuinely stuck.

The practical effect is that one line in your scenario plan, wait for the page to stop changing, replaces the entire conversation about how long is long enough. A 1.2-second response returns control after about 3 seconds (1.2s of stream plus 2s of stability margin). A 22-second response returns control after about 24 seconds. Same plan text, same assertion afterwards, no knob to tune per feature.

Primitive 2: free-text criteria, not string equality

The second primitive is the assert tool, defined at agent.ts line 133. It takes three fields: description (what you are asserting, in English), passed (a boolean), and evidence (a quote or observation from the page that justifies the boolean). At the test-runner layer, the same idea is exposed through passCriteria, a free-text field on assrt_test at server.ts line 343. You write the criterion the way you would describe it to a teammate, the agent reads the page, and the test fails if any criterion is not met.

For an AI writing feature this is the difference between a test that is meaningful and a test that ships. Equality says “the response must be exactly this paragraph.” That assertion is false on the very next run because language models are stochastic. Free-text criteria say “the response is non-empty, mentions the topic of the prompt, contains at least three sentence-ending punctuation marks, does not contain Lorem ipsum or visible [object Object], and ends in a way that suggests it finished on purpose rather than mid-word.” That set of criteria is satisfied by an entire family of valid generations and violated by every realistic regression: a server returning a stub, a template token leaking through, a stream cut off halfway, a guardrail returning a refusal in the middle of an editor.

Primitive 3: accessibility-tree refs that survive token churn

The third primitive is not unique to Assrt; it is the accessibility-tree mode that @playwright/mcp uses by default. But it is load-bearing here. A snapshot of the page returns elements as roles plus accessible names plus stable refs like ref=e12. A button is identified as [Button] “Generate” ref=e7, a streaming region as [region] “Response” ref=e22.

During a stream, the tokens land as text-node updates inside that region. The region itself does not change role or name; only its descendants churn. A CSS selector tied to a class name on a generated paragraph element is stale by the next frame. The accessibility ref to the region as a whole stays valid for the full lifetime of the response. After wait_for_stable returns, the agent calls snapshot one more time and reads the final text out of the same ref it had at the start. That sequence, get ref, trigger, wait for stability, re-read same ref, is the canonical shape of every AI-writing test.

What you actually write

A scenario file. The agent does the rest. There is no Playwright config to learn, no selector helpers to install, no special streaming-test library to import. The plan below is what landed in /tmp/assrt/scenario.md when I tested an AI rewrite button on a Markdown editor; it has stayed green across three model upgrades and one full-page redesign because none of the steps depend on the appearance or the wording of the output.

#Case 1: AI rewrite produces an on-topic, well-formed result
1. Open /editor and click "Sign in" if the auth gate appears.
2. Click "New document" and paste this seed text into the body:
   "the meeting was good and we talked about things"
3. Select the seed text and click the "Rewrite with AI" button.
4. wait_for_stable (timeout 30s, stable 2s).
5. Assert: the body contains the word "meeting", has at least
   two sentence-ending punctuation marks, is between 40 and 400
   characters, and does not contain "Lorem ipsum" or
   "[object Object]". Quote the actual rendered text as evidence.
6. Assert: the "Stop" button is no longer highlighted and the
   "Accept rewrite" button is enabled.

#Case 2: AI rewrite refused content surfaces as a visible error
1. Open /editor on the same session.
2. Paste a clearly disallowed prompt into the body.
3. Select it and click "Rewrite with AI".
4. wait_for_stable.
5. Assert: a visible region with role=alert appears containing
   the word "refused" or "cannot". The body text is unchanged.
6. Assert: the original text is still present byte-for-byte.

Run it from any MCP client (Claude Code, Cursor, plain npx) with assrt_test url plan. The agent walks the steps, calls the primitives at the right moments, and returns a structured report with one boolean per assertion plus the actual quoted text the model produced as evidence. Re-run on every commit. The plan does not need to change when the model changes, because no assertion depends on the exact wording.

Where this breaks (be honest with yourself)

Three failure modes are worth naming up front. First, agent judgement failures. The free-text criteria are evaluated by an LLM, and an LLM can occasionally read a real bug as a pass or a real pass as a fail. The mitigation is to write criteria that are mechanical wherever possible (length, punctuation count, presence of literal substrings) and reserve subjective ones for guardrail detection rather than primary signal.

Second, infinitely-streaming pages. If your feature has an ambient typing indicator that pulses every second forever, wait_for_stable never declares stability and times out at 30 seconds. The fix is to scope the observer to the response region, or to assert on a deterministic affordance like the Stop button cooling down, rather than on quiescence of the entire body.

Third, model regressions that pass your criteria but degrade quality. A test that says “at least three sentences” will not catch a model upgrade that turns coherent paragraphs into bulleted noise. Quality eval is a separate problem from integration testing; ship the assertions you can mechanise here and run a smaller, slower quality-eval suite on a different cadence.