AI and feedback loops in testing

The word “loop” is doing two jobs

Read the common advice on this and you will see the same picture every time: capture outcomes, label them, retrain or fine-tune the model, validate on a held-out set, repeat. That is a real loop. It is also the slow one. It runs over days. Nothing about it tells you whether the checkout test you just kicked off is going to pass.

The loop that decides pass or fail lives inside a single test run and closes hundreds of times per minute. Every time the agent does something to the page, it looks again at what the page became before it chooses the next move. That re-looking is the loop. It is cheap, it is constant, and it is invisible in the writing because it leaves no dashboard behind.

Keeping the two straight matters because they fail differently. A stale model degrades slowly and you can measure it. A runtime loop that acts on assumptions instead of observations fails instantly and looks like a flaky test. The rest of this page is mostly about that fast loop, because it is the one you can actually do something about.

Three loops, three timescales

Here is the whole thing on one screen. Notice that the column that matters for a given test result is the first row, not the last.

Loop 1: the runtime loop, in real code

This is the loop nobody shows, so here is the actual one. Assrt drives the browser with an agent. The agent does not write a script up front and run it blind. It reads the page, picks one action, then reads the page again to see what that action did, and only then picks the next action. The exchange looks like this.

The piece that makes it a feedback loop, rather than a script, is the last leg: after every action the browser hands back a fresh result and screenshot, and that observation is pushed into the next decision before the agent moves again. In src/core/agent.ts it reads, lightly condensed, like this:

// src/core/agent.ts
const MAX_STEPS_PER_SCENARIO = Infinity;

while (!completed && stepCounter < MAX_STEPS_PER_SCENARIO) {
  // model chooses the next action from what it can see right now
  const response = await this.anthropic.messages.create({
    model: this.model, system: SYSTEM_PROMPT, tools: TOOLS, messages,
  });

  for (const toolCall of toolCalls) {
    const result = await this.execute(toolCall);   // click / type / navigate / assert
    const screenshot = await this.browser.screenshot();
    toolResults.push({
      type: "tool_result", tool_use_id: toolCall.id,
      content: [
        { type: "text",  text: result },
        { type: "image", source: screenshot },      // <- the observation
      ],
    });
  }

  // feed the *observed* page back in before the next decision
  messages.push({ role: "user", content: toolResults });
}

Two details are worth pausing on. First, MAX_STEPS_PER_SCENARIO = Infinity: the loop is not capped at a guessed number of steps. It runs until the work is actually done and the agent calls complete_scenario. Second, the screenshot and accessibility tree go back in as a tool_result on every iteration. The system prompt is blunt about why: “Call snapshot before each interaction to get fresh refs.” The agent never trusts a stale element reference, because the loop hands it a new one each turn. That is the entire trick. It is also why a small UI change does not derail the run.

Loop 2: the diagnose-and-correct loop

A red run is itself a piece of feedback, and the worst thing you can do with it is treat it as a dead end. Assrt’s diagnose step takes a failure as input and runs it through a small loop of its own, described in src/mcp/server.ts: classify the root cause, decide whether the application or the test is at fault, and, when the test was flawed, emit a corrected test in the same case format so it can go straight back into the runtime loop.

The outcome is that every failure resolves into one of two useful things: a confirmed bug worth a ticket, or a better test than the one you started with. Nothing is thrown away. This is the loop that keeps a suite from rotting, because the suite improves itself from its own failures rather than waiting for a human to babysit every red mark.

Why the fast loop is the one to care about

The slow model loop gets the attention because it sounds like the sophisticated part. In practice it is the runtime loop that earns or loses your trust in a test suite. A model that is a little out of date still produces reasonable actions. An agent that commits to a plan and then acts on a page that has moved underneath it produces a result that is confidently wrong, which is the worst possible kind of test result.

This is also the honest answer to the “does AI make tests more flaky” worry. Flakiness does not come from speed. It comes from acting on stale assumptions. A loop that re-reads the page after each action is structurally more stable than a recorded script, because it notices the modal that appeared, the row that shifted, the button that relabeled, and adjusts before any of those turn into a false failure. Tighten the observation step and you remove flakiness at the source instead of papering over it with retries.

So if you are evaluating any AI testing tool, the question to ask is not “how often do you retrain.” It is “how do you re-ground each action on the real page,” and “can I read the code that does it.”

A feedback loop you can actually inspect

Most tools that automate this hand you a loop you cannot see. The decisions happen inside a closed service, the tests are stored as a proprietary format, and you take the vendor’s word that the loop is sound. That is a strange thing to accept for the component that decides whether you ship.

Assrt is built the other way around. The agent that runs the runtime loop is open source, so the observe, act, re-observe cycle is code you can read line by line. And the output of the loop is not a black-box recording: it is standard Playwright test files you read, edit, run in your own CI, and keep if you ever walk away. The loop is inspectable at both ends, the engine and the artifact.

You can watch one run end to end with a single command:

npx @m13v/assrt discover https://your-app.com

It crawls the app, proposes scenarios so you do not start from a blank file, then drives each one through the runtime loop above and writes the Playwright tests out where you can see them.