Claude Opus 4.7 for Playwright tests: which stage actually needs the frontier model

Two paths, both resolved at /Users/matthewdi/assrt-mcp/src/core/agent.ts:366 where the model string is picked as `model || process.env.ANTHROPIC_MODEL || DEFAULT_ANTHROPIC_MODEL`. The first is the CLI flag: `npx @assrt-ai/assrt run --url https://your-app.com --model claude-opus-4-7 --plan "#Case 1: ..."`. The second is the environment variable: `export ANTHROPIC_MODEL=claude-opus-4-7` once, and every Assrt invocation in that shell session picks Opus. The MCP server tool surface also exposes `model` as a per-call argument (server.ts:351), so an upstream agent like Claude Code can switch models per scenario without changing your shell.

The default is hardcoded as `claude-haiku-4-5-20251001` at agent.ts line 9. The reason is the shape of the inner loop, not loyalty to Anthropic's small model. The execution loop is snapshot, pick one tool call from a list of seventeen, send it, wait for the browser, repeat. The model is not generating prose, it is selecting a single tool name and a small JSON arguments object. That decision is heavily constrained by the structured tool schema (agent.ts:18..186), which is exactly the kind of work small models do well. Picking Opus here is paying frontier prices to do what amounts to multiple-choice answers.

Two specific moments. First, plan generation, the assrt_plan tool. The model gets one snapshot of the landing page and a tiny system prompt asking for one or two test cases. Opus is more likely to surface flows the tree only implies (the modal that opens after a hidden CTA, the form that validates async, the dashboard tile that becomes interactive only after data loads). Second, failure diagnosis, the assrt_diagnose tool. Here the model reads the full transcript of the failed run and is asked to reason about why. There is no narrow tool surface, no structured output schema, just a long context and a question. That is the shape of work where Opus 4.7's reasoning shows up cleanly.

Yes, in two narrow cases. The first is the very first run on a brand-new app where you do not yet have a test plan and you also do not yet trust the agent to drive your UI without supervision. Putting Opus on plan + execute + diagnose maximizes the chance that the first hour of usage produces something you want to keep. The second is when you are debugging a flaky scenario that intermittently fails on Haiku and you want to rule out model error before you blame the test. Once you have a stable plan and a stable run, swap back to Haiku for execute and keep Opus only on diagnose.

It does the second. The agent loop drives @playwright/mcp at runtime, calling navigate, snapshot, click, type_text, and the rest as MCP tools. The artifact you keep is the plan: plain Markdown with #Case blocks, stored at /tmp/assrt/scenario.md and optionally checked into your repo. There is no auto-generated *.spec.ts that Assrt writes for you. That is by design, the structure of an AI agent run is not a one-to-one mapping to a Playwright test file, and pretending otherwise produces tests that look readable but do not capture the agent's recovery logic. If you want a starting point for hand-written Playwright that mirrors a passing scenario, run with `--video --json` and use the recorded video plus the Markdown plan as the spec.

A representative example from a checkout page. The accessibility tree shows an email input, a CTA labelled "Continue", and a small disclosure link. Haiku writes one case: enter email, click Continue, assert next page. Opus writes two: the same case, plus a second case that clicks the disclosure first, asserts that the modal opens, closes it, then runs the happy path. The disclosure was implied by the tree, not stated; Haiku treated it as decorative, Opus treated it as a path. Multiply that across a dozen pages on first run and you get meaningfully more coverage out of the planning step alone.

Two signals. The CLI prints `[assrt] Model: <id>` at startup (cli.ts:501), and the agent emits a structured log line `agent.run.start` with the model field at agent.ts:383. If you run with `--json`, the resulting JSON includes a `model` field at the top level (cli.ts:580). The MCP tool response also includes the model in the metadata block (server.ts:726, server.ts:751) so an upstream agent can verify that its model override actually took effect rather than being silently dropped.

The honest answer is to read Anthropic's pricing page rather than trust a number on this site that may have moved. The shape worth knowing: Opus tokens cost an order of magnitude more than Haiku tokens, both for input and output. The execution loop sends a fresh snapshot every turn, snapshots are not small, and a typical scenario takes ten to thirty turns. That is the stage where the cost ratio compounds the most. The plan stage sends one snapshot. The diagnose stage sends a transcript that fits comfortably in Opus's context. So the cost-aware default is exactly what Assrt ships with: Haiku for execute, Opus on demand for plan and diagnose.

Not in a single CLI invocation today, but the MCP surface lets you do it across calls. assrt_plan, assrt_test, and assrt_diagnose each accept their own `model` argument (server.ts:351, server.ts:773, see also the diagnose tool registration). An upstream agent like Claude Code can call `assrt_plan(model: "claude-opus-4-7")` then `assrt_test(model: "claude-haiku-4-5-20251001")` then, on failure, `assrt_diagnose(model: "claude-opus-4-7")`. That gives you the cost-aware mixed setup automatically without you having to think about it per shell session.

It reduces the appearance of flakiness, not the underlying cause. A flaky test almost always means an unwaited async boundary, a race between a navigation and an assertion, or an auth-state assumption that breaks under parallel execution. Opus is more likely to recover gracefully when one of those conditions hits mid-scenario, which makes the run pass instead of fail. That is useful, but it also masks the bug. If your goal is a green CI, run Opus for execution. If your goal is to find and fix the actual flake source, run Haiku for execution and let the failures surface, then swap to Opus for assrt_diagnose to read the transcript and tell you what to fix.