Test coverage during agentic coding: put the runner inside the loop, not at the end of it

Why "raise the coverage number" misses the point

Most advice you'll find on test coverage when working with coding agents reads like advice from before the agent existed. It says: write more unit tests. Set a coverage threshold in CI. Use the agent to generate the tests. Review them carefully.

That advice is fine when a human is doing the coding and CI is the only consumer of the test results. It does almost nothing for an agent in autonomous mode, because of one detail: the agent edits the code in a loop, and the loop has many turns before any commit. A typical Claude Code or Cursor agent run will make a dozen user-facing changes, fix two bugs the user did not ask about, and rename a route, all in the same tool-use loop. CI fires once at the end. By the time it fails, the agent is in a completely different mental state and the human has to reconstruct the regression by hand.

The number you actually want kept high is the number of real user flows that have passed an end-to-end run since the last edit. That number drops every time the agent touches a route, a form, an auth flow, or a payment integration, and it can only be restored by running the tests right then. Not at PR time. Right then.

Two mental models for "tests as feedback"

The shift here is small in words and large in practice. Tests stop being a thing the human runs to validate a finished change. They start being a thing the agent runs to decide what to do next.

What the loop actually looks like

Spelled out, with the file system and the MCP tool calls labelled, the agentic test loop looks like this. The interesting thing is how short it is. The whole feedback channel between the agent and the test runner is two files in /tmp/assrt and three MCP tool calls.

The agent writes its diff. The MCP tool runs the scenarios. The results land on disk. The agent reads them. If something failed, the agent calls diagnose on the same scenario and gets back a root cause it can act on. None of this requires the human to be in the chair.

The integration surface, in source

The reason this works is that the MCP server is small enough to read in one sitting. Three server.tool(...) calls register the agent-facing surface. A short string of instructions tells the agent when to call each one. Everything else (the browser driver, the scenario sync, the video player) is private to the server.

// Agent in tool-loop, no test runner
// Has to: shell out to npx playwright,
// parse stdout, ignore the trace viewer,
// fail to see the video, ignore the
// retry semantics. In practice it
// does none of this and just commits.

await Bash({
  command: "npm test",
  timeout: 120000,
});

// Output: 1 passed, 4 skipped, 0 failed
// Agent: "looks good!" Commits.
// Reality: the four skips matter.

The instructions string the MCP server hands to the connected agent is the cultural piece. It says, verbatim, what the agent should do and when:

"Proactively use Assrt after any user-facing change. Do not wait for the user to ask for testing."

"After implementing a feature or bug fix that touches UI, routes, forms, or user flows: run assrt_test against the local dev server to verify the change works end-to-end in a real browser."

"When a test fails: use assrt_diagnose to understand root cause before attempting a fix. Do not guess."

The MCP protocol delivers that string to the connected client when the agent first sees the server, so it lands directly in the agent's system context. There is no prompt engineering for the user to do; the runner ships with its own etiquette.

The two files the agent watches

The feedback channel between the test runner and the editing agent is intentionally boring: two files in /tmp/assrt. The agent already has a Read tool. It already has an Edit tool. The MCP server uses the same surface so the agent does not have to learn anything new.

That is the entire surface. No database. No proprietary trace format. No vendor server the agent has to authenticate against. When you want to leave, the directory is yours, the scenarios are plain text, and the runner under the hood is Playwright.

What "in the loop" means in practice, by edit type

You do not want the agent calling assrt_test after it edits a comment. You do want it calling after a route change, a form change, or a payment-integration touch. The following is the rule of thumb that keeps coverage useful without turning every turn into a 90-second wait.

A short, honest list of failure modes

Putting the runner inside the loop is not free. Three things to watch.

1. The agent will keep going if you let it. Once an agent learns it can test its own work, it can spend ten minutes iterating on a single failing case while you sip coffee. That is fine when the case is right. It is expensive when the case is wrong. Cap with the timeout parameter on assrt_test and review the scenario before you walk away.
2. The dev server is now part of the loop. The MCP server expects a real local URL. If the dev server crashes mid-turn, the agent sees navigation errors and may misdiagnose them as application bugs. The fix is to keep the dev server in a watched, restart-on-crash process and to instruct the agent (in your project's CLAUDE.md or equivalent) to verify the dev server is up before its first assrt_test of the session.
3. Behavioral coverage is not the same as line coverage. None of this raises the line/branch coverage number you might be reporting to a board. It raises the proportion of real flows that pass after each edit, which is the number that actually correlates with users not seeing broken pages. Decide which number you want to optimise; do not pretend they are the same.

Getting started in the agent you already use

Assrt's MCP server runs over stdio, so any client that speaks MCP can connect: Claude Code, Cursor, the open-source Continue extension, custom agents built on the Anthropic SDK. Setup is a single line in the client's MCP config. The npm package is @assrt-ai/assrt; the binary is named assrt-mcp. On first connection the SERVER_INSTRUCTIONS land in the agent's context and it knows when to call the three tools.

The repo is at github.com/assrt-ai/assrt-mcp. MIT, no telemetry by default beyond an opt-out PostHog ping. Read src/mcp/server.ts before installing if you want to see the full integration surface in one file.

Frequently asked questions