The best QA automation tools, ranked by exit cost. Not feature count.

Features are what the vendor wants you to compare on. Exit cost is what you wish you had compared on two years in, when your suite is 800 scenarios deep and the procurement team is looking at next year's renewal. Every top SERP result for this keyword ranks by features, AI buzzwords, or pricing tiers. None of them answer: when the contract ends, what do you still own? The answer differs by an order of magnitude across this list. Tests stored as plain Markdown in your repo and driven by open-source runners (Assrt, raw Playwright, Selenium, Cypress) stay yours. Tests stored in Mabl's cloud, in a testRigor private workspace, in ACCELQ's Universe, or in Katalon's proprietary .ks files are the vendor's asset that you pay for access to. That is the first real decision.

A plan is plain Markdown. Each scenario is a block that starts with #Case N: and is followed by 2-5 sentences in English. The parser at assrt-mcp/src/core/agent.ts:569 is a single regex: /(?:#?\s*(?:Scenario|Test|Case))\s*\d*[:.]\s*/gi. That is the entire grammar. No schema, no compiler, no bytecode, no proprietary extension. You commit the .md files next to your source code. Runs and screenshots land in /tmp/assrt on your disk; videos are WebM. If you want them in a database, the open-source server has an optional Firestore sync; you can turn it off and nothing breaks.

It is free as in price. It is not free as in private. Per testRigor's own pricing page, the Public Open Source plan licenses your tests under The Unlicense and makes all test cases and results publicly searchable on the internet. That is fine for OSS projects. It is not fine for a private SaaS whose test plan describes internal flows, admin endpoints, or regulated user journeys. To make that suite private you move to testRigor Private at $900/mo for one suite and one parallelization; scaling up runs you into higher tiers. You are paying a subscription to keep the tests you wrote out of search engines. With Assrt you write the Markdown, commit it to your private repo, and you are done.

Best exit cost (you keep the tests): raw Playwright, raw Cypress, raw Selenium, Robot Framework, and Assrt. In all five the artifacts are text you wrote (or that a tool generated and handed you in an open format) sitting in your repo. Worst exit cost: Mabl, testRigor Private workspaces, ACCELQ Universe, Tricentis Tosca, Applitools (for the visual baselines specifically), Katalon Studio Enterprise (scripts are portable, but Test Cases in the .ks format and the TestOps cloud history are not). Middle: testRigor Public Open Source (free but public), Testsigma (open-source core but cloud features gated), LambdaTest (runner only; your tests are still yours, but your device history is not). The ranking is specific, not hand-wavey — the criterion is what the vendor explicitly keeps if you stop paying.

The MCP server and CLI are MIT-licensed and free. Running them costs zero. The only usage-based cost is the LLM API calls the agent makes while interpreting a plan: by default this is claude-haiku-4-5-20251001 with your own Anthropic API key. A typical #Case costs cents, not dollars. You can also point it at Gemini. There is an optional hosted dashboard at app.assrt.ai for sharing runs with teammates; that one is cloud and optional and your tests are not locked in it — the source of truth stays the Markdown files in your repo. Compare to testRigor Private at $900/mo minimum and AI-first enterprise platforms that quote $2k-$7.5k/mo for mid-size teams.

Complement. Raw Playwright is the gold standard for full-control E2E tests that a human engineer authors and reads. Assrt is for the layer above: the flows a product manager or a founder can describe in English but does not want to sit down and turn into a spec file. Under the hood Assrt drives real Playwright MCP calls (navigate, click, type_text, snapshot, press_key), so the runtime is the same browser automation you already trust. Many teams run both: a handful of Assrt #Case files covering critical smoke paths written by non-engineers, and a deeper Playwright suite written by QA engineers. Both are portable; both are yours.

Self-healing has two flavors. Vendor flavor: when a selector drifts, the platform's closed model rewrites the test in their cloud and you never see the diff. That is fast but you lose the review step. Assrt flavor: when an assrt_test run fails, you call assrt_diagnose which returns a four-section response, the last section of which is a literal #Case block written in the same grammar as your plan. You paste it over the failing one and re-run. The repair is a diff you can read and a text edit you can commit. That is not as flashy but it is auditable, reviewable, and survives the vendor disappearing. The diagnose contract lives in assrt-mcp/src/mcp/server.ts around line 240.

Four questions, in order. (1) In what file format are tests stored? If the answer is a vendor DSL, a proprietary extension, or a row in a cloud database, exit cost is high. (2) Where do tests execute? If the runner is open-source (Playwright, Selenium, Cypress) you can always run them yourself; if it is a proprietary engine, you cannot. (3) What licence? MIT or Apache-2 means you can fork if the vendor dies. 'The Unlicense' is fine for public tests, not private ones. A commercial EULA with a cloud-only dependency is the highest exit cost. (4) Does the vendor let you export runs, screenshots, and videos in open formats (JSON, PNG, WebM, MP4)? If not, your history is stranded. Apply these four to every tool on your shortlist before the feature comparison.

No. A number of AI-first testing platforms wrap Playwright internally and then store scenarios in their own format that you cannot export as Playwright code. The promise 'built on Playwright' is about execution, not portability. The portable tools on this list are the ones where the test source artifact (the thing you commit to git) is either raw Playwright/Cypress/Selenium code or a plain-text file (Markdown, Gherkin, Robot Framework .robot) that any compatible runner could execute. That distinction is what separates 'cheap to migrate off' from 'cheap to use this year.'

If the team has no engineers or only one engineer, start with Assrt's three-tool MCP loop: point assrt_plan at your URL, run the returned #Case blocks with assrt_test, and paste the Corrected #Case from assrt_diagnose when something fails. Zero learning curve, zero lock-in. If the team has a QA engineer and expects to scale past a few hundred scenarios, use both: Assrt for the plain-English critical-path smoke tests, raw Playwright for the deeper suite. If the team is enterprise-sized and already has a TestRail or Jira workflow the QA lead will not abandon, evaluate Katalon or ACCELQ on their own merits but go in knowing the exit cost is real.