Test automation software tools, measured by what lands on your disk.

A software tool runs on your machine, stores artifacts on your filesystem, and keeps working if the vendor's website is down. A SaaS platform runs tests in the vendor's cloud, stores traces in the vendor's database, and stops producing usable output the moment billing lapses. Most roundups mix the two under a single label because both are sold as 'tools for test automation.' The practical test: delete the vendor's dashboard URL from your browser history and cancel the account. If you still have every past test run, plan text, video, and assertion log on disk, it was software. If the test history is gone, it was a dashboard. Assrt is the first: every scenario is a Markdown file, every run is a directory under /tmp/assrt/, and the video player is a standalone HTML file that works on any localhost server.

One call to assrt_test creates /tmp/assrt/<runId>/ and writes four things into it: screenshots/NN_stepN_<action>.png (one PNG per turn, zero-padded filename so ls sorts them in order), video/recording.webm (the full Playwright video for the session), execution.log (the human-readable event stream), events.json (the same events in parsed form), and player.html (a self-contained video player with timeline markers, generated fresh each run). At the parent level, /tmp/assrt/scenario.md is the test plan, /tmp/assrt/scenario.json is its metadata, /tmp/assrt/results/latest.json is the last verdict, and /tmp/assrt/results/<runId>.json is the verdict for every historical run. Tar the /tmp/assrt directory and you have a portable artifact nobody else is holding a copy of.

It is a small detail with a real consequence. The filename convention in assrt-mcp/src/mcp/server.ts line 468 is String(screenshotIndex).padStart(2, '0') + '_step' + currentStep + '_' + currentAction + '.png'. The zero-padding makes ls and every file browser sort the screenshots in true execution order, not lexical order. In a 47-step scenario, step 10's screenshot sorts before step 2's without padding; with padding, '10_step10_click.png' comes after '02_step2_click.png'. You get a flipbook of the run by scrolling a directory, not a sorted query in a hosted viewer. Rare tool decision; most capture tools dump screenshots with a uuid or timestamp, which is precisely what breaks casual inspection.

The assrt-mcp package on npm is MIT-licensed and runs end-to-end on your machine. There is an optional cloud component at app.assrt.ai that gives each scenario a shareable URL, but the cloud is a mirror, not the source of truth. scenario-files.ts watches /tmp/assrt/scenario.md for edits and pushes them to Firestore on a 1-second debounce (lines 97-103), and buildCloudUrls in scenario-store.ts constructs the public URL deterministically so it works before the upload completes. If app.assrt.ai returns 500, writes to local scenario.md still succeed, the next assrt run still executes, the video player still plays. The cloud is off-by-default if you register as local-only (scenarioId starts with 'local-' at scenario-store.ts line 127). Delete the /tmp/assrt directory and your tests are gone; that is the authority.

Playwright, Cypress, and Selenium are libraries you program against. You write test.spec.ts, call page.locator('button.submit'), and the runner executes your code. They are software tools by every definition, and they are excellent. Assrt uses Playwright under the hood, via the Playwright MCP protocol, but the layer above is different: you do not write test.spec.ts. You write a Markdown plan with #Case blocks, and an agent turns that plan into runtime tool calls (navigate, click, type_text, assert) against the live accessibility tree. For someone who already maintains a Playwright suite, Assrt is not a replacement; it is a faster layer for the flows where selectors drift faster than humans can update them. For someone choosing their first automation tool, Assrt skips the selector layer entirely and writes plain Markdown scenarios you still own on disk.

Nothing about your tests. The package is installed from npm, the browser is launched on your machine via Playwright MCP, the LLM call goes directly from your machine to Anthropic's API using your ANTHROPIC_API_KEY, and artifacts are written to /tmp/assrt/. We get zero scenarios, zero traces, zero screenshots. If you set ASSRT_TELEMETRY=0 no run metadata ever leaves your machine. The only exception is when you explicitly pass a scenarioId that was created against app.assrt.ai; then the scenario plan (the Markdown text) is synced to that cloud record so it survives a disk wipe. That sync is a read-write mirror, not an authority: the disk copy is always canonical. If the company vanishes tomorrow, you run npx assrt from your existing node_modules and everything works.

Open /tmp/assrt/<runId>/player.html in any browser; that is a standalone HTML file generated at server.ts line 619, with a small JavaScript player that loads video/recording.webm next to it. The timeline shows step markers, so you can click to jump to the turn where an assertion failed. For the event-level view, cat /tmp/assrt/<runId>/execution.log prints the scenario start, every tool call, every reasoning step, every assertion pass or fail, and the complete_scenario summary. For machine-readable post-processing, /tmp/assrt/<runId>/events.json has the same content as structured JSON. You can grep across every past run by running grep -r 'FAIL' /tmp/assrt/; you cannot do that against a vendor dashboard.

Not that I have found, because the answer is embarrassing for most of the list. Of the 16 or so tools routinely ranked in this category, three or four produce fully local artifact trees (Playwright, Cypress, Selenium, Assrt). The rest store the canonical test log, video, and assertion history in the vendor's cloud, with a 'download as ZIP' button that bundles some subset. The download button is the shelf label; it is not the same as a tool that writes a self-contained directory every run without a cloud round-trip. If you rank the roundup by 'post-run, is the test log on my filesystem?', most entries fail silently.

Plans go in git, artifacts usually do not. The convention we use: commit /tmp/assrt/scenario.md (or move it into your repo under tests/ if you prefer), ignore the /tmp/assrt/<runId>/ directories because they contain binary artifacts that bloat a repo. If you want an audit trail, upload the video and events.json to your own object store from CI; the run ID is stable and deterministic so the paths in a CI artifact server mirror the local paths. A small assrt-ci.sh one-liner (cp -r /tmp/assrt/$RUN_ID $ARTIFACTS_DIR) is enough. The key property is that the files exist as files, not as blobs held by a third party.

npx assrt run --url https://your.app --plan '#Case 1: Load the homepage and assert the H1' --video --json. That installs assrt-mcp from npm into your npx cache, launches Playwright, navigates to the URL, writes /tmp/assrt/<runId>/screenshots/, video/recording.webm, execution.log, events.json, and player.html, and prints the JSON verdict to stdout. No account, no login, no browser-redirected auth flow. ANTHROPIC_API_KEY is the only environment variable required; everything else has a sane default. Check the directory with ls -la /tmp/assrt/ once the command returns. That is the whole surface.