Open source testing where the test file is plain Markdown and the parser is one regex

Open source testing has meant two things for most of the last decade: an MIT-licensed runner like Playwright, Cypress, or Selenium, or an MIT-licensed test manager like Kiwi TCMS or TestLink. The runner is where you spend your time and where the lock-in usually hides, because even open source runners disagree on the spec format. Playwright wants a TypeScript file, Cypress wants a TypeScript file, Selenium will take any language but wants a compiled project. An AI agent changes this because the agent does not need a spec in any programming language at all. Assrt is an example of the newer pattern: the spec is a Markdown file, the agent is the runner, and the browser underneath is still Playwright. The whole stack is open source, but the format the tests live in is human prose. That is the piece the listicles at bugbug.io, momentic.ai, and the BrowserStack guide leave out.

/tmp/assrt/scenario.md, pinned by the constant ASSRT_DIR in /Users/matthewdi/assrt-mcp/src/core/scenario-files.ts line 16. The shape is decided by a single regex in /Users/matthewdi/assrt-mcp/src/core/agent.ts line 621: /(?:#?\s*(?:Scenario|Test|Case))\s*\d*[:.]\s*/gi. Anything that matches that pattern becomes a scenario boundary, and everything between two boundaries is the body of a test. That is why you can open scenario.md in any text editor, paste plain English under each #Case, and have the runner execute it. There is no schema file, no code generation pass, no compile step. The parser is a split() call.

A chmod-less UUIDv4. scenario-store.ts line 8 comments exactly this: the UUID v4 IS the access token. When assrt_test or the CLI opens a scenario, it writes the plan to disk and calls fs.watch on it. The callback in scenario-files.ts line 97 has a 1-second debounce; when the file settles, a PATCH goes to https://app.assrt.ai/api/public/scenarios/<uuid>. Nothing checks a bearer token, nothing checks a session cookie, nothing checks a user ID. If you have the UUID, you can read and write the scenario. If you do not, there is no enumeration to walk because the ID space is 2^122. This is deliberate and unusual. Most open source testing tools either require a login (Kiwi TCMS) or keep the test entirely on disk (bare Playwright). Assrt is the middle ground: shareable by URL, private by entropy.

You can and many teams do. The case for Markdown over a .spec.ts file is specifically about what rots. A Playwright spec contains selectors and locators that are valid the moment you write them and start decaying the moment the page changes. A Markdown case says 'Click the Sign up button' and leaves the element discovery to the agent at run time. When the button moves from the navbar into a dropdown, the Playwright spec breaks and the Markdown case still passes. The tradeoff is speed and determinism: a Playwright spec runs in about 1 second per action against a pre-bound locator, and a Markdown case runs in about 4-6 seconds per action because the agent has to fetch an accessibility snapshot and pick the ref. For a smoke suite of 20 cases, that difference is real. For an auditing suite that runs once a day, it does not matter. The open source testing guide nobody writes is the one that tells you which format fits which job.

Yes. The runner is a Node CLI at /Users/matthewdi/assrt-mcp/src/cli.ts that spawns @playwright/mcp locally over stdio. The only network call that leaves your box is the one to the Anthropic API to drive the agent, and even that respects ANTHROPIC_BASE_URL, so you can point at a local proxy, a self-hosted Claude gateway, or an air-gapped endpoint. The cloud sync to app.assrt.ai is opt-in and happens only if you passed a UUID into the CLI or MCP tool; run with --plan inline and the scenario never leaves /tmp. Compared to commercial AI testing platforms that bill $7.5K/month and require sending your DOM to their servers, the bytes-out budget here is zero plus your Anthropic token spend.

The Assrt MCP server is under the license in /Users/matthewdi/assrt-mcp/LICENSE (MIT). @playwright/mcp is Apache 2.0. Playwright itself is Apache 2.0. The Anthropic SDK is MIT. There is no CLA to sign, no contributor shield, no feature held back for a paid tier. The web UI at app.assrt.ai is the hosted convenience layer on top; you can run the full open source stack without ever creating an account. When an open source testing guide tells you 'download the community edition', check whether the commercial version has features that are silently better than the community one. Assrt has only one version, and it is the one in the repo.

Open the spec file, delete everything that is locator logic or imports, leave a one-line comment per test describing what it verifies in plain English, then prefix each with #Case N:. A 40-line test.describe + test.each becomes 6 lines of Markdown. The reverse migration is also mechanical: run assrt_diagnose on a failing case to get the observed element refs, then rewrite as a Playwright spec if you need the determinism. Neither direction is a tooling cliff. That is the point of an open format.

Two decisions in the runner. First, the Markdown spec has no selectors, so there is nothing for a selector change to break. Second, wait_for_stable (agent.ts lines 956-1009) injects a MutationObserver and waits until the page goes 2 seconds with zero DOM mutations or hits a 30-second ceiling, whichever comes first. Fast pages return fast, streaming AI responses wait as long as they actually need, and the sleep(5000) anti-pattern is gone. Between intent-only specs and a real stability signal, the kind of flakiness you see on Reddit threads for AI codegen tools mostly does not happen.

User roles, permissions, multi-tenant workspaces, issue tracker integrations, pre-built report templates, an approval workflow for test cases. Kiwi TCMS is a test management system with a full web app and a SQL schema. Assrt is a runner with a hosted scenario store that supports sharing but not governance. If your team needs to assign test cases to QA analysts, tag them against a release, gate PRs on manual sign-off, and attach bug reports, Kiwi TCMS is a better match. If your team needs tests that live next to code, run in a few seconds from an MCP client, and do not require anyone to open a dashboard, Assrt is the lighter fit. They are complementary more than competitive; a large org could reasonably run both.

In practice, 5 to 8 #Case blocks per file is where the generator (see the PLAN_SYSTEM_PROMPT cap in assrt-mcp/src/mcp/server.ts) targets, and running more than about 15 in one pass risks the agent's context window on a long run. The right pattern for larger suites is many scenario.md files, one per feature area, each with its own UUID. Cloud sync handles that naturally because the UUID is the unit of scope, not the file count. The single-file ceiling is a feature budget, not a hard limit.