AI test discovery and generation, in two LLM calls with different budgets

The two calls and what each one sees

The diagram below traces a single run. You hand the CLI a URL and a request to test the app. The upfront call fires once. The executor starts working through the cases it returned. Every time the agent lands on a URL it has not seen before, the in-execution discovery call fires for that URL. New cases append to the same scenario buffer the executor is reading from.

Phase 1: the upfront plan

This call runs once per invocation of the assrt_plan MCP tool. The CLI launches a real Chrome via Playwright MCP, navigates to the URL, takes a screenshot, scrolls 800 pixels, takes another, scrolls 800 more, takes a third. It concatenates the accessibility tree at each scroll position and slices the combined string to 8000 characters. All three screenshots and the sliced tree go into a single Claude Haiku call with a 4096-token budget.

The system prompt frames the model as a Senior QA Engineer and gives it six rules. The most important one for output volume is rule 6: generate 5 to 8 cases max, focused on the most important user flows visible on the page.

// /Users/matthewdi/assrt-mcp/src/mcp/server.ts, line 219
const PLAN_SYSTEM_PROMPT = `You are a Senior QA Engineer generating
test cases for an AI browser agent. The agent can: navigate URLs, click
buttons/links by text or selector, type into inputs, scroll, press keys,
and make assertions. It CANNOT: resize the browser, test network errors,
inspect CSS, or run JavaScript.

## Output Format
Generate test cases in this EXACT format:

#Case 1: [short action-oriented name]
[Step-by-step instructions the agent can execute. Be SPECIFIC about
what to click, what to type, and what to verify.]

#Case 2: [short action-oriented name]
[Step-by-step instructions...]

## CRITICAL Rules for Executable Tests
1. **Each case must be SELF-CONTAINED** ...
2. **Be specific about selectors** ...
3. **Verify observable things** ...
4. **Keep cases SHORT** — 3-5 actions max per case.
5. **Avoid testing what you can't see** ...
6. **Generate 5-8 cases max** — focused on the MOST IMPORTANT user
   flows visible on the page.`;

// later, when the tool fires:
const response = await anthropic.messages.create({
  model: model || "claude-haiku-4-5-20251001",
  max_tokens: 4096,                  // <-- the upfront budget
  system: PLAN_SYSTEM_PROMPT,
  messages: [{ role: "user", content: contentParts }],
});

The reason this call can afford 4096 tokens is that it runs exactly once. Even on a 100-page app, the upfront cost is fixed. If the model returns 7 cases averaging 60 tokens each, the response is 420 tokens, well under the cap. The cap exists for worst-case complex apps where the model wants to write longer step lists.

Phase 2: discovery during execution

Every time the agent calls navigate on a URL it has not seen, the agent code calls queueDiscoverPage(url). The URL is normalized (protocol plus pathname, trailing slash stripped), checked against three deduplication gates (already seen, queue full, skip-pattern matched), and pushed to a pending buffer. Between scenarios, the agent calls flushDiscovery(), which fires up to 3 concurrent discovery calls.

Each discovery call sees one accessibility tree (sliced to 4000 characters) and one screenshot. The system prompt is tighter: quick test cases, 3-4 actions max per case, 1-2 cases total. The model gets 1024 tokens of response budget. The output streams back as it generates so a partial set is visible mid-call.

// /Users/matthewdi/assrt-mcp/src/core/agent.ts, line 256
const DISCOVERY_SYSTEM_PROMPT = `You are a QA engineer generating quick
test cases for an AI browser agent that just landed on a new page. The
agent can click, type, scroll, and verify visible text.

## Output Format
#Case 1: [short name]
[1-2 lines: what to click/type and what to verify]

## Rules
- Generate only 1-2 cases
- Each case must be completable in 3-4 actions max
- Reference ACTUAL buttons/links/inputs visible on the page
- Do NOT generate login/signup cases
- Do NOT generate cases about CSS, responsive layout, or performance`;

const MAX_CONCURRENT_DISCOVERIES = 3;
const MAX_DISCOVERED_PAGES       = 20;
const SKIP_URL_PATTERNS = [
  /\/logout/i, /\/api\//i, /^javascript:/i,
  /^about:blank/i, /^data:/i, /^chrome/i,
];

// later, when each new URL is reached during a run:
const stream = this.anthropic.messages.stream({
  model: this.model,
  max_tokens: 1024,                  // <-- the in-execution budget
  system: DISCOVERY_SYSTEM_PROMPT,
  messages: [{ role: "user", content }],
});

The 20-page and 3-concurrent caps exist because a run that discovers 200 pages would either stall the executor waiting for generation or fan out beyond what a single API key can sustain. 20 is enough to cover the surface a typical first-pass test should reach. If the agent navigates to a 21st new URL, the discovery is silently skipped and the executor keeps moving.

The full workflow in order

The parser that lets both phases share output

The single biggest design decision here is that the two phases emit the same format. The plan call produces a long markdown string with 5-8 #Case headers; the discovery call produces a short markdown string with 1-2 #Case headers. The executor never cares which phase a case came from because the parser is identical.

// /Users/matthewdi/assrt-mcp/src/core/agent.ts, line 620
private parseScenarios(text: string): { name: string; steps: string }[] {
  const scenarioRegex = /(?:#?\s*(?:Scenario|Test|Case))\s*\d*[:.]\s*/gi;
  const parts = text.split(scenarioRegex).filter((s) => s.trim());
  if (parts.length > 1) {
    const names = text.match(scenarioRegex) || [];
    return parts.map((steps, i) => ({
      name: (names[i] || `Case ${i + 1}`)
              .replace(/^#\s*/, "")
              .replace(/[:.]\s*$/, "")
              .trim(),
      steps: steps.trim(),
    }));
  }
  return [{ name: "Test Scenario", steps: text.trim() }];
}

That regex accepts #Case 1:, Case 2., Scenario 3:, and several other shapes, because the original input could come from the plan call, the discovery call, a markdown file a human edited by hand, or a JSON payload an MCP client sent. The format is permissive on input, strict on output.

What both prompts deliberately exclude

Six categories of cases are unwelcome in either phase. The plan prompt rules them out explicitly; the discovery prompt repeats the most expensive ones. Excluding them in the prompt is cheaper than excluding them in a post-filter, because the model never spends output tokens on cases the executor cannot run.

Why the asymmetry is the interesting part

Anyone can build a single AI test generator: take a URL, give a screenshot to GPT or Claude, ask for cases. That generator either fires once (and misses every URL the user reaches after the landing page) or it fires every navigation (and burns budget on deep generation for trivial pages). The interesting design is recognizing that the cost-per-call and the value-per-call are different at the two moments, and pricing them accordingly.

Microsoft has shipped a similar idea inside the official Playwright Test Agents (Planner, Generator, Healer), and that framing is becoming standard. The structural detail that varies tool to tool is which prompts run where, what they cap output at, and what they refuse to generate. Most write-ups on this topic gloss over those details. Without them, two products that look identical on a feature matrix can produce very different runs against the same app.

The honest comparison to make is not feature-to-feature. It is prompt-to-prompt, budget-to-budget, cap-to-cap. The same way you would compare two databases by their isolation level and lock granularity, not by their landing-page bullet points.

Reading the source yourself

Three files in github.com/assrt-ai/assrt-mcp are enough to verify everything on this page:

• src/mcp/server.ts — PLAN_SYSTEM_PROMPT at line 219 and the upfront call at line 829 with max_tokens: 4096.
• src/core/agent.ts — DISCOVERY_SYSTEM_PROMPT and the three caps at lines 256-271, the queue at 555, the dispatcher at 564, the per-page LLM call at 585 with max_tokens: 1024.
• src/core/agent.ts line 620 — the parseScenarios regex that lets the same parser handle both phases.

If you want to run this locally, the install command is npx @m13v/assrt. The MCP server is registered globally on first run; thereafter you can call assrt_plan and assrt_test from Claude Code or any other MCP-aware client. Discovery is on by default during assrt_test runs and you do not have to enable it explicitly.