The QA automation tool test nobody runs: what do you keep when you cancel?

The question the SERP refuses to ask

I read the top five results for "qa automation tool" before writing this. Tricentis, Ranorex, Testim, Opkey, and a Medium listicle. Between them they covered self-healing, codeless recording, AI test generation, CI/CD integration, reporting dashboards, and cross-browser support. Not one of them asked the exit question. If you adopt this and it does not work out, what do you walk away with? The answer for most of those tools is nothing, or a zip file of JSON that only their renderer understands.

The exit question is not an edge case. It is the first thing every careful engineering lead should ask before approving a QA automation tool budget, because a test suite is the kind of asset that accumulates slowly and then matters enormously. Six months of regression cases is a year of engineer-time saved per quarter. If the definitions of those cases live in a vendor dashboard, you are renting your own work back from the vendor.

The anchor fact: three paths

Every assrt_test run writes to three canonical locations on your local disk. The rule is defined in plain English inside the MCP server's instructions string at assrt-mcp/src/mcp/server.ts lines 294-298. You can read the source on GitHub, or you can run one test and verify all three paths exist with a single cat.

File one: scenario.md is the plan

This is the file your test suite is. It is markdown with a tiny YAML front-matter block and one or more #Case sections. Each step inside a #Case is a sentence a human would say to a careful junior tester: navigate, type, click, assert. The agent reads this file, calls the matching Playwright MCP tool for each step, and writes the result. If you edit this file before a re-run, the next run uses your edits and auto-syncs the new plan to cloud storage.

---
name: Signup flow
url: https://staging.example.com
scenarioId: 7b4e1f2a-91c3-4d6e-a8f0-2c5d9e3a6b10
---

#Case 1: A new user can sign up and reach the dashboard
1. Navigate to /signup
2. Call create_temp_email to obtain a disposable address
3. Type that address into the Email field
4. Type "correct-horse-staple" into the Password field
5. Click the Create account button
6. Wait for the "Check your email" banner
7. Call wait_for_verification_code to fetch the 6-digit code
8. Type the code into the Verification code field
9. Click Verify
10. Assert the heading "Welcome" is visible on /dashboard

Rename a step with sed. Diff two versions with git. Share one with a teammate over Slack. These are all first-class operations on a text file, and all of them are inaccessible when your plan lives in a vendor dashboard.

File two: scenario.json is the metadata

A small sibling file that records the scenario's UUID, its display name, and the URL it targets. The UUID is how you replay a scenario by ID instead of by plan text. Pass it back to assrt_test as the scenarioId argument and you re-run the exact same thing. This matters for CI: you author the plan once, commit the ID, and your pipeline replays it on every push.

File three: results/latest.json is the audit log

The third file is where proof lives. Every step the agent took, the tool name it called, the arguments, the result, pass/fail, milliseconds, and paths to the screenshot and video frame at that moment. It is large enough to be useful and small enough to commit as a CI artifact. Below is a trimmed real shape.

{
  "scenarioId": "7b4e1f2a-91c3-4d6e-a8f0-2c5d9e3a6b10",
  "ranAt": "2026-04-19T18:42:11.004Z",
  "passed": true,
  "durationMs": 11832,
  "steps": [
    { "tool": "navigate", "args": { "url": "/signup" }, "passed": true, "ms": 412 },
    { "tool": "snapshot", "passed": true, "ms": 188 },
    { "tool": "type_text", "args": { "ref": "e7", "text": "aQ9mZ2cP7r@1secmail.net" }, "passed": true, "ms": 204 },
    { "tool": "click", "args": { "ref": "e12", "description": "Create account" }, "passed": true, "ms": 176 },
    { "tool": "wait_for_stable", "passed": true, "ms": 2113 },
    { "tool": "wait_for_verification_code", "args": { "timeoutMs": 60000 }, "result": { "code": "482197" }, "passed": true, "ms": 5284 },
    { "tool": "assert", "args": { "description": "Dashboard Welcome heading visible" }, "passed": true, "ms": 92 }
  ],
  "artifacts": {
    "video": "/tmp/assrt/7b4e1f2a.../video.webm",
    "screenshots": "/tmp/assrt/7b4e1f2a.../screenshots/",
    "events": "/tmp/assrt/7b4e1f2a.../events.json"
  }
}

grep, jq, and a tiny node script are all you need to build a dashboard from these files. You do not need an API, a webhook, or a session token. A failed build is a failed JSON file, and a failed JSON file is git-blame-able.

How the three files get written

The sequence is straightforward. The MCP client calls assrt_test. assrt-mcp boots an agent that drives @playwright/mcp, which drives a real Chromium. Between steps, the agent writes the evolving plan back to scenario.md so an outside observer can read progress mid-run. When the agent calls complete_scenario, the final results land in results/latest.json.

The sequence is simple because the architecture is simple. assrt-mcp is a thin Typescript layer that owns the file I/O; Playwright MCP is the browser driver. Both are open-source. Neither is yours to get locked into.

The three MCP tools you can call from any agent

Because Assrt ships as an MCP server, any agent that speaks MCP (Claude Code, Cursor, Windsurf, a custom loop) can drive it. The surface area is three tools. That is the entire public contract for automating the suite.

The numbers that describe a single run

Concrete defaults, all readable in the source. The viewport is 1600x900 (browser.ts line 296). The navigation timeout is 0s (NAV_TIMEOUT_MS). The stability poll default is 0s. The MCP tool count that makes up the agent's universe is 0 (agent.ts lines 16-196). The files your suite reduces to is 0.

Ownership vs. the rest of the category

This is not a feature comparison. Feature comparisons are solved; everyone has "AI test generation" now. This is a portability comparison, and it looks different. A single row matters: where does the plan live?

What I would do on day one

If I were evaluating a QA automation tool from scratch, I would run the portability test first, before any feature check. Here is the shape of that test against Assrt. The whole exercise takes about five minutes and costs a few cents of Anthropic credit.

Where this argument ends

I am not claiming Assrt wins every feature race. A mature enterprise suite like Tricentis has integrations Assrt does not, and a Playwright codegen suite gives you tighter control when you need it. What Assrt uniquely offers is that the artifact your team produces over months of careful QA work is always a set of text files you own. Three of them, to be specific. The rest is interchangeable runner machinery.

If the tool you are currently evaluating cannot show you the equivalent three files, the one they would leave behind after cancellation, that is the answer to the one question the SERP refuses to ask.