Headless chrome test flakiness is a static-script problem

Three reasons in roughly this order. First, your locator was written against the DOM at record time, and the DOM in CI is one Suspense boundary or one feature flag away from being slightly different, so the locator either resolves to nothing or resolves to the wrong node. Second, your wait is a `page.waitForTimeout(2000)` that worked locally because your machine is fast and a 2 GHz CI runner needs 2.4s for the same animation. Third, headless Chrome on CI lacks a real GPU, which makes `requestAnimationFrame` callbacks fire on a different schedule, which makes elements you assumed had finished moving still be in transit. The first two are bugs in your script. The third is a real Chromium quirk, and it is much rarer than the first two combined.

It papers over it for a class of flakes (XHR-driven content loads) and creates a different class of flake on apps with persistent connections. SaaS dashboards routinely keep an open WebSocket, a Sentry session, a PostHog autocapture queue, and an analytics heartbeat going. There is no `networkidle` on those pages: there is always a request in flight. Tests built on networkidle then time out at the worst time, which is usually right before an assertion you needed to run. A DOM-mutation-based wait is more honest because it asks the question you actually have, which is `has the part of the page I care about stopped changing`. That is what `wait_for_stable` does at /Users/matthewdi/assrt-mcp/src/core/agent.ts lines 956 to 1009.

It calls `snapshot()`, which returns the accessibility tree of the page in its current state, with each interactive element annotated with a fresh ref like `[ref=e7]`. Then it picks the element by ref. If the action fails because the ref is stale (the snapshot was taken a fraction of a second ago and the DOM moved), the runner catches the failure and re-snapshots automatically — see the catch at agent.ts:1014-1020 which builds a new accessibility tree and feeds it back to the agent. There is no element handle to keep alive across renders, because the abstraction the agent uses is not a handle, it is a label that gets re-resolved every step.

Mechanically they overlap, but the failure modes differ. `waitForFunction` lets you write any predicate and re-evaluates on a polling interval and on requestAnimationFrame. The trap is that you have to design the predicate yourself, and most teams either write `() => document.querySelector('.spinner') === null` (which fires the moment the spinner unmounts, before the next render flushes) or `() => document.body.innerText.includes('Loaded')` (which fails the first time the loaded copy is conditional). The MutationObserver-based wait at agent.ts:962-994 sidesteps this by treating absence of mutation as the signal: it counts mutations on `document.body` with `childList`, `subtree`, and `characterData` set, and considers the page stable only after `stable_seconds` of unchanged count. You do not write a predicate; you let the page tell you when it is done.

It will not, by design. If your app has a setInterval that fires every 1.5 seconds and updates a clock in the corner, `wait_for_stable` will time out at the configured maximum (default 30s, capped at 60s) because mutations never fall below the threshold. That is correct behavior: a page that never stops mutating has no stable point to assert against. The fix is in the app, not the test, and the timeout in the log is the signal that tells you which it is. The cleanup at agent.ts:990-994 disconnects the observer and deletes `window.__assrt_mutations` and `window.__assrt_observer` so the next `wait_for_stable` call gets a fresh count, even if the previous one timed out.

Mostly. Retry logic in CI exists as a band-aid for two specific kinds of flake: stale-selector failures and timing failures. If both are gone, the only flakes left are real bugs (the app actually broke under load) and infrastructure failures (the runner died, the network dropped). Both deserve a real failure, not a retry. That said, the per-call MCP timeout at /Users/matthewdi/assrt-mcp/src/core/browser.ts line 381 is set to 120 seconds (vs the SDK default of 60), specifically so a contended-CI navigation that takes 90s completes and shows up in the log as `(92000ms)` rather than dying as `Request timed out`. The point is to surface the real number, not to retry around it.

Because YAML test plans cannot be reused. The moment you switch vendors, every line of test logic is captured in their proprietary syntax and has to be rewritten. Real Playwright code, expressed as named scenarios in plain English `#Case` blocks that the agent compiles to `browser_navigate` / `browser_click` / `browser_snapshot` calls at runtime, is portable: the scenarios document intent, the agent picks the actions, and the artifacts are the actual MCP calls a Playwright engineer can read. The parser for the `#Case` format is at agent.ts:620-631 and is two regexes long.

Take any React app with a list that loads from an API. Write a Playwright test that does `await page.locator('text=ItemA').click()`. Add a `await new Promise(r => setTimeout(r, 50))` between getting the locator and clicking. Now make the list re-render asynchronously (a refetch on focus, a websocket update, anything). Run it 50 times. The locator will resolve to the old node, the new render will reparent that node, and the click will hit a detached element that does nothing. The `[ref=eN]` model in the accessibility tree avoids this because the ref is bound to the snapshot, not the DOM, and a fresh snapshot is one tool call away.

Two things, in order. First, replace every `waitForTimeout` with either `waitForFunction` (if you have a precise predicate) or with the equivalent of `wait_for_stable` (a MutationObserver-based wait, which is about 30 lines of helper code; the implementation at agent.ts:962-994 is a working reference). Second, stop reusing element handles across awaits — re-resolve the locator at the moment of each action. Both moves remove flake without changing your test runner. They are also the prerequisites for moving to an agent-driven runner later, because the agent is doing the same two things automatically.

Because the runner is one repository you can read in an afternoon: /Users/matthewdi/assrt-mcp/src for the MCP server and /Users/matthewdi/assrt for the web app. The expensive parts of test infrastructure are not the orchestrator; they are the LLM tokens, the CI minutes, and the engineer hours spent on triage. Assrt charges nothing for the orchestrator and lets you bring your own model and your own CI. Closed cloud vendors charge for the orchestrator (often 7,500 per month), make you write tests in their syntax, and own your test artifacts. When you switch off them you start over. With Assrt, the artifacts are real Playwright MCP tool calls and Markdown scenarios; switching means deleting one npm dependency.