Smoke tests for critical paths: how to know the path actually finished

A smoke test is a yes-or-no check that a critical path still works end to end in a real browser. Critical paths are the flows where failure effectively takes the product down: login, the primary CRUD flow that earns revenue, checkout or payment (including the card-declined branch), signup, role-switching, the most dangerous admin action, a report or export, the main inbound webhook, a search or filter, and a logout that clears session. Ten is usually enough. The goal is fast, deterministic feedback on every build and every 30 minutes against production, not full coverage.

Because the assertion fires before the critical path has actually finished. Checkout shows a spinner for 1.8s, then a confirmation card, then an async balance refresh. A test that sleeps 2s passes sometimes and fails sometimes. A test that waits for the confirmation text passes, but the balance assertion on the next line fires mid-update. The technique that fixes this is not a longer sleep. It is waiting for the DOM itself to stop changing, which is what Assrt does via a MutationObserver injected into the page.

The agent injects a MutationObserver onto document.body with childList, subtree, and characterData enabled, stored on window.__assrt_observer. Every 500ms it reads window.__assrt_mutations. When the count has been flat for stable_seconds (default 2, max 10), it breaks. It times out at timeout_seconds (default 30, max 60). On exit it disconnects the observer and returns a line like 'Page stabilized after 3.4s (42 total mutations)'. Source: /Users/matthewdi/assrt-mcp/src/core/agent.ts:941-994.

Fixed timeouts trade flakiness for speed: too short and the test flakes, too long and your suite is slow. waitForSelector is better, but it only tells you one element showed up, not that the page is done. A checkout confirmation card can appear while the balance is still reloading. MutationObserver stability waits for the whole critical path to finish rendering. Tests pass faster on fast runs, wait longer on slow runs, and catch the case where the page mounted a shell then bailed because nothing else mutated.

Real Playwright. Assrt drives the official @playwright/mcp server. Each action the agent takes (click, fill, navigate, wait_for_stable) gets emitted as a TypeScript Playwright line into the run trace. The artifact in /tmp/assrt/results/latest.json is copy-pasteable into a *.spec.ts file. You own it. If you stop using Assrt, your smoke suite still runs under vanilla Playwright. There is no YAML layer, no vendor DSL, no $7.5K/month cloud bill.

npx @m13v/assrt test https://your-app.com --case 'log in, see the dashboard, log out'. Write each critical path as a short markdown #Case block (one per flow). Run the whole suite on every PR against a preview deploy (block merge on failure) and every 30 minutes against production as a synthetic (page on failure). The agent figures out selectors from the accessibility tree on each run, so a UI redesign does not break the tests.

Set stable_seconds higher. wait_for_stable accepts up to 10 seconds of quiet and 60 seconds of total wait. For AI chat responses, 5 seconds of stability is usually right: tokens stream in with DOM mutations, then stop cleanly. For webhook-dependent confirmation screens, you may want to combine wait_for_stable with an explicit assertion on the confirmation text or an API poll. The mutation approach handles streaming, progressive rendering, and skeleton-to-content transitions automatically.

Service workers do not mutate the document body, so their background work is invisible to the observer, which is correct for UI smoke tests. Open websockets that push messages rendered into the DOM do register as mutations, so an SSE or WS-driven feed will keep the stability timer ticking until the feed quiets down. Background fetches that never touch the DOM do not affect it either. If you need to assert something that never renders (a background task success), use wait_for_stable for the visible flow, then a direct API check after.

Yes. networkidle waits for 500ms with no more than two in-flight requests, and Playwright's own docs now discourage it because it hangs on apps with long-lived websockets, analytics pings, or poll loops. DOM-mutation stability is network-agnostic: it watches what the user actually sees. A page can be networkidle with a half-rendered skeleton, or network-chatty with a complete UI that has stopped changing. For smoke tests of critical paths, DOM stability is the signal you want.