Automated AI Testing: Real Playwright Code You Own, Not Vendor YAML You Rent

1. What Automated AI Testing Actually Means

Automated AI testing is the combination of two ideas that used to live in separate tools. Automated testing means a machine runs the browser without a human clicking through it. AI testing means a language model decides what to click, what to assert, and how to recover when a selector drifts. Put them together and you get a pipeline where a developer points a CLI at a running URL and the machine produces a validated test suite in minutes.

The term is used loosely in marketing material, so it is worth being specific. Automated AI testing is not screen recorders that replay a click sequence. It is not brittle CSS selector harvesters that snapshot a DOM. It is a closed loop of plan, generate, execute, and heal in which the model reads the application like a user and writes code that another developer could read, review, and edit by hand.

The decisive question when choosing a tool is what file lands on disk. If the artifact is a proprietary YAML file that only runs inside a vendor cloud, your test investment is a rental. If the artifact is a standard .spec.ts file that runs with npx playwright test, your tests are durable assets that outlive the tool that wrote them. Assrt is built around the second answer.

2. Real Playwright Code vs Proprietary YAML

Every automated AI testing vendor claims to produce reusable artifacts. Then you read the output and realize it is a JSON or YAML file that only means something to their runtime. This is the single largest trap in the category and it is worth showing side by side.

// Assrt output: standard Playwright you own forever
import { test, expect } from '@playwright/test';

test('user completes checkout with valid card', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();

  await page.getByLabel('Full name').fill('Priya Raman');
  await page.getByLabel('Email').fill('priya@example.com');
  await page.getByLabel('Shipping address').fill('1 Market St');

  const cardFrame = page.frameLocator('iframe[title="Secure card"]');
  await cardFrame.getByPlaceholder('Card number').fill('4242424242424242');
  await cardFrame.getByPlaceholder('MM / YY').fill('12/30');
  await cardFrame.getByPlaceholder('CVC').fill('123');

  await page.getByRole('button', { name: 'Pay now' }).click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});

The Assrt output is a file you can paste into any Playwright project. The proprietary output on the right looks similar but cannot run anywhere except inside the vendor runtime. Cancel the subscription and the tests stop working. Export them and you get a dump that no open-source tool understands. This is the rent trap.

3. Architecture of an Automated AI Test Runner

Understanding the architecture helps you spot tools that take shortcuts. A serious automated AI testing system has four stages, not three. Most vendors skip the validation stage and ship tests that look plausible but break on first run. Assrt validates every file before it touches disk.

Stage 1: Crawl

A headless Chromium instance starts from your URL and follows every link, form, and route transition it can discover. It records DOM snapshots, the ARIA accessibility tree, form definitions, network traffic, and cookies. The output is a directed graph of application states with edges labeled by the actions that connect them.

Stage 2: Plan

The graph and a structured system prompt go to the language model. The model reasons about which sequences of actions form a meaningful user journey and outputs a test plan. A plan entry looks less like code and more like a design doc: flow name, preconditions, ordered steps, and the observable outcome that should be asserted.

Stage 3: Generate

Each plan entry is compiled into a Playwright spec file. The generator prefers role-based locators over CSS selectors because role-based locators survive refactors. If a form field has a visible label, the generator uses getByLabel. If a button has an accessible name, it uses getByRole. CSS selectors appear only when nothing else matches.

Stage 4: Validate

Generated files are executed against the live app in a sandboxed worker. Passing tests are written to disk. Failing tests enter a repair loop: the model receives the failure message and retries with a corrected locator or assertion. If three retries fail, the test is dropped rather than shipped broken. You never commit a red test generated by Assrt.

4. Scenario: Checkout Flow With Stripe Elements

Checkout flows are the highest-stakes tests in any product. Revenue runs through them. They also touch iframes, third-party scripts, and asynchronous state changes, which is where brittle test runners fall apart. Here is what Assrt generates for a Stripe Elements checkout on a storefront.

5. Scenario: Authenticated Multi-Tenant Dashboard

Most SaaS tests live behind a login wall and target a specific tenant. Automated AI testing has to handle both the authentication step and the tenant scoping. Assrt does this by reusing a stored Playwright storageState from a login fixture so every generated test starts already logged in.

6. Scenario: Form Validation and Error States

Forms are the place where automated AI testing shines because the state space is tedious for humans to enumerate. Empty required fields, out-of-range numbers, mismatched confirm passwords, invalid emails, the same form submitted twice. Assrt walks the combinations systematically and writes one test per meaningful failure mode.

7. Self-Hosting: Docker, CI, and Your Own LLM

Because Assrt is open-source and ships as a Node package, you can run it anywhere Node runs. That includes your laptop, your internal Kubernetes cluster, a GitHub Actions runner, a GitLab CI job, or a self-hosted LLM endpoint. There is no phone-home, no account required, and no rate limit beyond what your own LLM budget enforces.

8. Migration From Vendor Lock-In to Owned Code

If you are currently running an automated AI testing tool that emits proprietary configs, the migration path to Assrt is simpler than you would expect. You do not translate the old files. Assrt regenerates the entire suite from your running application, and because the output is standard Playwright, the new suite drops straight into your existing CI pipeline.

9. Cost Breakdown: Assrt vs QA Wolf vs mabl vs Testim

The price tags in this market vary by two orders of magnitude. QA Wolf starts at $7,500 per month. mabl and Testim sit in the mid-hundreds. Assrt is free. What you actually pay, though, is more than the license. Here is the honest first-year total cost of ownership for a 50-test automated AI testing program.

The headline is that Assrt is between 1.4 times and 21 times cheaper on year-one TCO depending on the comparison. The softer but more important number is what happens in year three. With Assrt, your tests are git-tracked TypeScript, so the marginal cost of keeping them is the marginal cost of running Playwright, which you were going to pay anyway. With proprietary vendors, every additional year is another full license payment and you still own nothing at the end of it.

// With Assrt: every file is reviewable code
// tests/ai/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('checkout completes', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();
  // ... 20 more lines of readable Playwright
});
// Git diff shows every change. Review is a code review.
// Rollback is a git revert. Ownership is total.

10. FAQ

What is the difference between automated AI testing and AI-assisted testing?

Automated AI testing runs the full loop without human intervention: crawl, plan, generate, execute, heal. AI-assisted testing keeps a human in the authoring loop and uses the model as a copilot that suggests selectors or drafts a single test at a time. Assrt supports both modes. The CLI runs end-to-end on its own; the VS Code extension offers inline suggestions while a human writes a test by hand.

Can automated AI testing handle apps that require login?

Yes. Assrt accepts a Playwright storageState file or a login script. It reuses the authenticated session for every generated test, which means generated specs start already logged in and can target any route behind the login wall. Multi-tenant apps are handled by passing a per-tenant storageState.

Does automated AI testing replace the QA team?

No, and the teams that treat it that way regret it. Automated AI testing absorbs the tedious regression work that a QA team already hated writing. The humans shift to exploratory testing, test planning, edge-case discovery, and incident review. The output of the QA team goes up, not the headcount.

How does Assrt avoid shipping broken generated tests?

Every generated spec file is executed against the live application before it is written to disk. Tests that fail enter a repair loop with up to three retries. Tests that still fail are discarded, not committed. Teams never see a red test file that Assrt generated because Assrt would have dropped it before it landed in the repo.

What happens if Assrt shuts down tomorrow?

Nothing happens to your tests. They are standard Playwright files in your git repository. You keep running them with npx playwright test forever. Assrt is open-source, so the source code is available for audit and forking. This is the whole point of generating real code instead of a proprietary format.

Which LLM does Assrt use, and can I swap it?

Assrt defaults to Claude for planning and generation but supports any OpenAI-compatible endpoint. You can point it at OpenAI, Anthropic, Azure OpenAI, a self-hosted vLLM deployment, or an internal gateway. Cost and latency are tunable because the LLM call is a standard HTTP request you control.

Can automated AI testing handle visual regressions?

Yes. Playwright ships with screenshot assertions out of the box and Assrt generates tests that use them when a page has visual state worth snapshotting. Visual diffs live next to functional tests in the same tests/ai folder and run on the same CI job.