Find AI fallback bugs end to end without mocking the model

Because most AI fallback bugs are not in the prompt or the parser. They are in the wiring between the model response and the UI. The classic ones look like this: the model returns malformed JSON, the parser throws, the catch block sets isLoading back to false but forgets to set the error state, and the user is left looking at an empty container with no message. The unit test for the parser passes (it correctly throws). The integration test for the API route passes (it correctly returns a 500). Only an end to end test that drives the real browser through the real failure and asserts on the visible DOM catches the dead state. The Assrt agent is built for exactly that level: it talks to a real Chrome through Playwright MCP, watches DOM mutations through wait_for_stable, and asserts on the rendered text the user sees. Every other layer of testing is necessary. None of them are sufficient.

Five recurring shapes. First, the silent timeout: the model never replies, the spinner never resolves, no error message paints, the user thinks the app is alive but it is wedged. Second, the swallowed error: the parser throws, isLoading goes false, the container stays empty. Third, the fallback message that never paints because the error state is set but the conditional render guards on a different flag. Fourth, the retry loop that fires forever because the retry counter is reset on each render. Fifth, the inconsistent state: the success UI renders the partial response, the error toast also renders, both are visible at once. None of these show up in a unit test because the unit test mocks the network layer and asserts on internal state, not on what the DOM contains. The Assrt agent reads the accessibility tree of the actual rendered page and asserts on text and roles, so all five surface as scenario failures with screenshots and a video of the run.

Three options that all work without modifying the application. The first and simplest is to point the test at a development environment whose AI provider key has been revoked or rate limited on purpose, so every model call returns a 401 or 429. The second is to use the agent's http_request tool (agent.ts:172-184) to hit a fault injector you run alongside your dev server, flipping a flag like inject_timeout or inject_malformed_json before the scenario starts and clearing it after. The third is to set the environment variable AI_PROVIDER_BASE_URL to a local proxy (toxiproxy, mitmproxy, or a 50-line node script) that drops the request, returns malformed JSON, or holds the response for 90 seconds. None of these are mocks. The application code path is the one that ships. The model surface is what is broken. That distinction is the entire point: mocked tests verify the parser handles a bad response, end to end tests verify your UI is honest about it.

It injects a MutationObserver onto document.body, counts how many DOM mutations happen, and breaks out of its polling loop only after the mutation count is unchanged for stable_seconds (default 2, max 10) within timeout_seconds (default 30, max 60). The implementation is at agent.ts:956-1008. It matters for AI flows specifically because every other waiting tool waits for a thing to be true (text present, element visible, network response received). AI fallback paths violate those expectations. The text you would wait for might never arrive. The success element might never render. The network call returns 200 with malformed JSON, so a wait_for_response fires too early. wait_for_stable does not assume which UI state you will land in. It just waits for the page to stop changing, then snapshots, then asserts on what is actually there. That works whether the final state is the success UI, the error toast, the retry button, or the fallback message. It also catches the wedged case: if the page never stops changing for 60 seconds, the tool returns a timeout result with the mutation count, and the agent can mark the scenario failed with that evidence.

After wait_for_stable settles, the agent calls snapshot to read the accessibility tree of the page (the live one, not a cached representation). The tree contains roles and accessible names for every interactive element and a flat list of visible text. The agent then issues an assert call (agent.ts:133-144) with a description, a passed boolean, and an evidence string, and the runtime records that assertion against the scenario. The shape of a useful fallback assertion looks like: description='Error toast appears with an actionable retry message', passed=tree contains text matching /try again|retry|something went wrong/i, evidence='Snapshot shows alert role with text "The model is busy. Try again." and a button labeled Retry'. The agent can also call evaluate to check internal state (e.g. that a global error counter incremented) and it can call http_request to verify a backend log was written. The combination is what catches inconsistent state: the success container is gone, the error toast is present, no spinner is left over.

Plain Markdown in the same #Case format Assrt uses for every other scenario. A working example: '#Case 1: Model timeout shows the retry UI with an actionable message. - http_request POST to http://localhost:4000/inject with body {"mode":"timeout"}. - Navigate to /chat. - Type "Summarize the doc" into the message input. - Click Send. - wait_for_stable with timeout 60 and stable 3. - Assert that the page contains text matching the words try again, retry, or timed out. - Assert that the send button is enabled, not stuck disabled. - Assert that no spinner is left in the DOM.' That is all that lives in your repo. The agent fills in the rest: which selector to use, how to read the accessibility tree, when to take a screenshot, where to store the video. The scenario file is a contract; the agent figures out the implementation each run.

Every scenario runs in a real Chrome with screencast frames streaming over WebSocket and (in local mode) a video file written to disk. The video captures the full run: the model call going out, the spinner spinning, the failure path firing, the fallback UI rendering or failing to render, the assertions evaluating. The agent also captures a screenshot after every visual action and attaches it to the assertion record. When a teammate asks why a scenario failed, the answer is a 30 to 90 second video that shows them. There is no log diving, no separate APM tooling, no guess about whether the bug was a flake. The video is the evidence.

The eval harness verifies the model produces good output on a fixed input set. That is necessary, and it has nothing to do with what the user sees when the model is unavailable. You need both. The eval harness scores the happy path. The Assrt scenarios cover the unhappy paths: timeout, malformed JSON, 429, 500, content filter rejection, partial response. They are different test types serving different goals. The same agent can also run end to end happy path tests, so you do not need a second tool, but the unit of value here is specifically the failure path coverage you cannot get from offline scoring.

You can. The reason this approach exists is that the same scenario file works for every variant of the failure. A handwritten Playwright test for the timeout case is 40 to 80 lines: imports, fixtures, a helper that talks to your fault injector, a wait_for_function with a custom mutation predicate, multiple expect calls for different aspects of the fallback UI. Tomorrow you add the malformed JSON case and write a second 40 line file. Then the 429 case. Then the partial response case. The agent collapses the same surface into a six bullet Markdown file, with the wait_for_stable, http_request, and assert tools doing the work that would otherwise be helpers in your test repo. If you would rather own the Playwright code, the open source Assrt repo shows you exactly the recipes (the MutationObserver, the http_request shape, the snapshot read) you can lift into your own test suite. Either way, the artifacts that matter (which assertions, what evidence) live in one place.

Three steps. Install the Assrt MCP (npx @m13v/assrt-mcp@latest install). Stand up a one file fault injector next to your dev server that returns a 504 or malformed JSON when a query parameter is present (a 30 line node script is enough). Write a six line #Case file that flips the injector on, navigates to your AI feature, sends a request, calls wait_for_stable, and asserts the page text contains a clear error message. Run the agent against your real Chrome with the --extension flag so you can watch the full failure path play out live. If the assertion fails, you have a real bug; the video plays back the moment the spinner went stale or the fallback message never painted. If it passes, commit the scenario; it is now your regression test for that fallback path.