An automation test tool that also reports the bugs you did not ask about.

The feature tables look identical because they compare the same three things: what language your tests are written in, which browsers the runner supports, and how the tool plugs into CI. Those are the 2018 axes. The 2026 axis is the architecture of the runner itself. A script-based runner (Playwright, Cypress, Selenium, and every hosted tool built on top of them) replays pre-decided clicks and asserts pre-decided outcomes. An agent-driven runner (Assrt, a handful of newer tools) reads your plan, reasons about the page in front of it, and makes the next tool call at runtime. The practical consequence shows up in what the runner produces. A script-based runner produces one output: pass or fail. An agent-driven runner can produce a second: a bug log for things it noticed that were not in the plan. That second output exists because the agent is structured enough to emit it (via a dedicated tool call) and un-scripted enough to have opinions. Pick on this axis, and your evaluation will feel different from a spec sheet.

It is a structured, orthogonal record. In Assrt, the agent has a tool named suggest_improvement (defined at assrt-mcp/src/core/agent.ts line 158) with four required fields: title (short), severity (critical, major, or minor), description (what is wrong), suggestion (how to fix it). When the agent calls it during a scenario, the runner emits an improvement_suggestion event (agent.ts line 919). In the MCP response, these events are aggregated into a dedicated improvements[] array that sits alongside the pass/fail scenarios array (mcp/server.ts line 663). In the CLI, they print as a one-line per issue: [issue] major: Stale copyright year in footer (cli.ts line 165). This is not debug output mixed into pass/fail. It is its own channel. A test scenario can pass while carrying four improvement entries; a scenario can fail while carrying none. The two signals are independent by design.

The system prompt gives the agent a tool for it, and the agent uses that tool when it sees something obvious. No extra instructions required. A typical trigger: the agent navigates to /checkout as part of a plan about signup, notices a broken 'Terms' link at the bottom of the page, calls suggest_improvement with severity=minor, and keeps going with the signup flow. It does not abandon the scenario to chase the bug. It does not rerun anything. The cost is one additional tool call and a few hundred extra tokens of context, both of which are dwarfed by the snapshot + screenshot pairs every turn already ships. The main scenario's pass/fail is unaffected because suggest_improvement does not touch the scenarioPassed variable (contrast with the assert case at agent.ts line 899, which does). From the agent's perspective, reporting a side-observation is just another tool call, like navigate or click.

Four categories show up regularly. First, copy and date drift: 'Copyright 2024' in a footer in April 2026. A script would not notice because the scenario never asserted on the year. Second, broken or stale links visible on pages the plan happened to walk through. Third, accessibility smells the model recognizes from its training: a modal that does not trap focus, a form input with no associated label, a click target under 24px. Fourth, small UX bugs: the password field clears when the user tabs to the next input, an error toast that disappears in 800ms and is unreadable, a CTA that looks disabled but is clickable. These are the 'small things you keep meaning to file a ticket for' category. A regression suite ignores them because they are not in the spec. An agent run logs them because it has a tool for it and the question 'does anything look wrong?' is adjacent to the question 'did my plan pass?'.

Hosted exploratory tools are separate products. You pay per tester or per crawl, you run them on a schedule, results live in their dashboard, and the bugs they find are on a different tab from the regression results. Assrt fuses both. You write one plan, you run it once, and you get two outputs in the same JSON report. The improvements[] array lives in the same file as the scenarios[] array (mcp/server.ts lines 654 to 664). No second product, no second subscription, no reconciliation between two dashboards. The tradeoff is depth: a dedicated exploratory crawler will click through more of your app than a targeted test plan ever will. If you need both a deep crawl and a regression pass, you run both. If you want 80% of the bug-finding value as a side effect of your existing regression runs, this architecture gives it to you for the cost of the extra tokens.

Every run writes /tmp/assrt/results/latest.json (plus /tmp/assrt/results/<runId>.json for history), defined in scenario-files.ts lines 18 to 20. The improvements array is a top-level field in that JSON: { url, passedCount, failedCount, scenarios: [...], improvements: [...] }. Each improvement is a plain object with title, severity, description, suggestion. From there, a five-line shell pipe can open Linear tickets: jq '.improvements[]' latest.json | while read -r row; do ... done. No API integration, no webhook, no proprietary output format. The file is grep-able, diff-able, and committable. For CI, assrt run --json prints the same structure to stdout so you do not even need the /tmp file. Wire it wherever you already ship test results.

The CLI is @assrt-ai/assrt on npm. Install with npx @assrt-ai/assrt setup, MIT license. The source is readable at github.com/assrt-ai. What lives in your repo after adopting it: plain Markdown plans (any text editor, any VCS), a package.json dependency on @assrt-ai/assrt, and nothing else Assrt-specific. The agent that drives the browser is Anthropic's Claude Haiku 4.5 by default (agent.ts line 9) or Google Gemini, with the provider and model configurable per run. The browser layer is @playwright/mcp, which is Microsoft's. If Assrt shuts down tomorrow, the CLI continues to work on the version you installed, your Markdown plans continue to be readable Markdown, and the only loss is the optional cloud sync for history. The agent's reasoning layer is an API key you own against a model provider you choose, not a subscription to Assrt.

Three questions. First: 'When I run one test, how many distinct artifacts do I get?' A script-based runner answers one (the report). An agent-driven runner should answer at least two (the report plus a bug log, possibly a third if it also discovers new pages). Second: 'What format is my test plan in, and can I commit it to git as a text file?' Markdown, YAML, and .spec.ts all qualify; a proprietary format that only exists inside their cloud editor does not. Third: 'What happens to my plans and my results if I cancel my account?' An acceptable answer is 'you downloaded them to disk during the trial and they still run.' An unacceptable answer is 'we export to a schema no other tool reads.' This frame replaces the usual 15-feature checkbox matrix with three architectural questions that surface whether the tool's shape matches how you actually want to work.

Yes, and it is calibrated low by design. The system prompt instructs the agent to report 'obvious' bugs, and suggest_improvement requires a severity field that forces the model to rank. In practice, on a 10-scenario run, the improvements[] array tends to land between 0 and 5 entries. A critical entry is rare (the agent reserves it for things like checkout failing or auth being broken on a public page). Minor is the usual severity, and minor entries are triaged in batch, not as tickets. False positives happen, mostly around 'this copy is confusing' style UX opinions; a .assrtignore pattern or a severity filter at the CLI level (grep) resolves them. The alternative is silence, which is what a scripted runner gives you: the bugs exist, the run just does not mention them.

You can ignore it: it never fails a scenario, and if no one reads improvements[] it sits in latest.json unread. You can also prompt it off: add 'Do not call suggest_improvement; focus only on the plan' to your scenario text, and the agent will comply. The current architecture does not expose a CLI flag to remove the tool definition from the schema (agent.ts line 158) because the tool is cheap when unused, but there is no cost to leaving it on. Most teams that adopt Assrt end up keeping the log and skimming it weekly, the same way you might skim Sentry's low-signal issues tab. It becomes a small backlog of polish items attached to your main regression suite.