AI Generated Testing: Your Scenarios Should Live in a Text File

1. The file that is your test suite

When Assrt runs a scenario, the first thing it does is write that scenario to disk at/tmp/assrt/scenario.md. Two sibling files sit next to it: scenario.jsonwith metadata, and results/latest.jsonwith the most recent run output. That is the whole on-disk contract. If you want to know what is being tested, open the markdown. If you want to change what is being tested, edit the markdown.

This sounds obvious. It is not how the category works. Most AI testing tools keep scenarios as records in their own database, expose them through a UI, and treat export as a compliance checkbox. You end up with tests you cannot grep, cannot diff in a pull request, and cannot hand to a coding agent without a vendor API. The markdown file inverts all of that: humans, agents, and CI pipelines read and write the exact same artifact.

2. The entire scenario format

There is no DSL to learn. The grammar is one line:

#Case 1: user signs up with expired invite
- navigate to /invite/abc123
- click "Create account"
- expect error text "This invite has expired"

#Case 2: user signs up with valid invite
- navigate to /invite/valid
- fill email with a fresh address
- click "Create account"
- expect url to contain /onboarding

The parser that splits this into cases lives insrc/core/agent.tsas the parseScenarios function. It looks for lines beginning with#Case, strips trailing punctuation from the name, and hands each block to the runtime agent. That is the entire parser. You could rewrite it in ten lines of Python if you wanted to. The portability is not a marketing claim, it is a property of how thin the format is.

Step lines are English. There is no selector syntax, no assertion DSL, no wait helper. At run time an LLM agent reads the step, inspects the page through a Playwright MCP browser, and decides which accessibility ref to click or which value to assert against. The brittleness of a CSS selector does not live in your scenario file, because the scenario file does not contain CSS selectors.

3. What happens when you save

When a scenario is loaded, Assrt starts anfs.watch on the markdown file. The watcher is set up insrc/core/scenario-files.ts, function startWatching. Every time the file changes on disk, the watcher reads the new content and pushes it to Firestore under the same scenario id. The next time you call assrt_test, from any machine, with that id, the runner picks up your edits.

Practically, this means three different things can edit the same scenario and stay in sync: you in your editor, a coding agent like Claude Code, and the web app. The file is the boundary. Nobody has to know about anybody else. A human can rewrite a step in Vim, a model can append a new #Case via the MCP tool, and the next run will see both changes.

Local-only scenarios, whose ids begin withlocal-, skip the Firestore push. The watcher still runs, but nothing leaves your machine. That is the path you want if you are testing something sensitive and do not want any network round trip.

4. Why intent outlasts selectors

A traditional E2E test is full of selectors: data-testid="signup-button",css=form > input:nth-child(2), xpath=//button[contains(text(), 'Sign up')]. Every one of those is a coupling between the test and the current rendering of the app. When a designer moves the form or a developer renames a class, the test breaks, even though the behavior did not change.

A markdown scenario records intent: click the create account button, fill the email field, expect an error about expired invites. The agent resolves intent to concrete page elements at run time. When the DOM changes, the agent adapts. When the intent actually changes, you edit one English line, not a page object model.

This is the reason we do not emit .spec.ts files. A generated Playwright file with hardcoded selectors would be more familiar to read, and it would also inherit every selector drift problem that traditional E2E tests already have. Keeping the scenario as intent is what makes AI generated testing actually cheaper to maintain than the hand-written tests it replaces.

5. A practical workflow

Here is how this tends to play out day to day. You ask a coding agent to add a feature. It edits the code, then calls assrt_test against your local dev server with a plan it wrote. Assrt writes that plan to/tmp/assrt/scenario.md, runs it, and saves the results toresults/latest.json.

You open the markdown. Maybe the agent wrote a step that checks the wrong thing. You edit it. The watcher syncs. You re-run with the scenario id and the new plan executes. No UI round trip, no copy paste into a dashboard, no proprietary test editor.

When the scenario is stable, copy the file into your repo undertests/scenarios/, commit it, and point CI at it. The test suite is now text in your repo, reviewable in pull requests like any other code. If Assrt disappeared tomorrow, you would still have a readable, complete description of what your app is supposed to do.

6. How this compares to other AI testing tools

Most AI testing platforms fall into one of two camps. The first generates a proprietary test object, stores it in their cloud, and runs it from there. Export exists but is second class. The second generates source code, usually Playwright or Cypress files full of selectors, and hands you the maintenance burden of a traditional E2E suite.

Assrt picks a third option. The scenario is plain markdown you own. The runner is open source. The execution is an LLM agent driving a real Playwright browser through MCP. Nothing about the test artifact is vendor specific, and nothing about the execution requires a cloud account. You can self-host the whole thing.

The concrete check is simple. Ask any AI testing vendor where a single test lives on disk, in what format, and whether an editor watching that file can modify the next run. If the answer is "you would export it from our dashboard," they are building the first category. If the answer is a real file path and a real format, they are building something closer to what Assrt does.

7. FAQ