AI test generator for medical software: the self-hosted setup that does not become a Business Associate

The compliance question hiding in your test stack

When a covered entity buys a SaaS test automation product, the contract conversation usually starts with the security questionnaire and ends with a Business Associate Agreement. That paperwork exists because the test tool, by design, ends up with access to whatever the test sees: a patient record on screen, a chart in the corner of a screenshot, a payload in a network log. Under the HHS definition of a business associate, anyone who performs a function on behalf of a covered entity that involves access to protected health information falls inside the rule. A cloud test runner that records video of your live charting workflow is performing exactly such a function.

The standard answer is to sign the BAA, complete the vendor risk review, and proceed. Some tools (like testRigor) advertise that they can act as a Business Associate. That is a legitimate path, and for some teams the simpler one. It is not the only one.

The other path is to choose an architecture where the test tool never touches PHI in the first place. The way to do that is not to add another vendor to your contract pile. It is to remove the vendor from the data path entirely.

Two architectures, one decision

The trade is not capability against capability. Both architectures run AI-driven browser tests. The trade is who gets access to what.

Where the opt-out lives, in source

The point of an open-source generator is that you can verify these claims yourself. Two specific lines in the Assrt source matter for this guide. They are short.

// Line 14
const CENTRAL_API_URL =
  process.env.ASSRT_API_URL ||
  "https://app.assrt.ai";

// Set ASSRT_API_URL to your own URL, or to an unreachable
// host, and scenario sync goes there instead. Local cache
// at ~/.assrt/scenarios is still authoritative.

The combination is what matters. The first line means the centralized scenario sync is opt-out, not opt-in. The second means a tester can run an AI agent against an authenticated healthcare workflow and leave nothing on disk between runs. Together they cover the two ways a generic test tool would normally end up with a copy of PHI: through the cloud sync, or through a stale browser profile.

What a healthcare-friendly test generation flow looks like

The flow below is what we recommend for a covered entity testing a patient-facing or charting web app. Nothing in it is novel. It is mostly about putting boundaries in the right places.

The arrows that matter are the absent ones. There is no arrow from the engineer to a vendor dashboard. No arrow from the staging app to a third-party screenshot store. The LLM provider gets text from the accessibility tree of a synthetic environment, never from a real patient record. If your model provider is itself under a BAA (most enterprise Anthropic and OpenAI plans support this), even running against environments containing PHI is on a documented footing. But the recommendation is still to run against staging.

The setup, end to end

The whole loop is six pieces of configuration. None are exotic.

Why generated Playwright code matters for medical software

Healthcare software does not get to live in a vendor dashboard. The design history file for a Software as a Medical Device submission, the IEC 62304 process documentation, and the verification evidence the FDA and notified bodies expect, all assume the artifacts are auditable things you control. A pull request with a reviewer name and a SHA is such an artifact. A test that exists only as a row in a third-party orchestrator is not.

Generated Playwright tests slot into the same review process as the rest of your TypeScript. The reviewer reads the file, sees what is asserted, signs off, merges. The git log is the audit trail. If a regulator asks how the patient consent flow is verified, the answer is a path to a file, not a screen recording of someone clicking through a SaaS UI.

The same fact lets you delete the generator without deleting your tests. Once a test is written, the AI tool that wrote it is no longer load-bearing. A team that adopts this pattern can replace the generator next year without losing a single assertion.

When to still pick a closed-source vendor

The self-hosted route is not always the right answer. It costs engineering time to wire up, requires a staging environment and synthetic data, and assumes someone on your team is comfortable owning Playwright. There are two cases where the SaaS path is more honest.

The first is when your team has no in-house QA capacity at all and needs an outsourced QA function plus AI tooling in one purchase. A managed service like QA Wolf is a different product than what this page is about. You are buying labor, not a generator.

The second is when your compliance program is already structured around BAAs with named SaaS vendors and adding one more is cheaper than introducing a self-hosted tool that needs its own security review. That is a reasonable read, and a tool that signs a BAA is a legitimate purchase. The point of this guide is that there is a third option, not that the other two are wrong.

A worked example: a charting workflow

To make this concrete, here is a small piece of generated test code against a synthetic charting flow. Nothing in this file references a real patient. The selectors come from the accessibility tree, so they survive a UI rewrite. The assertions are whatever your regulatory test plan calls for.

import { test, expect } from "@playwright/test";

test("chart save persists vital signs", async ({ page }) => {
  await page.goto(process.env.STAGING_URL!);
  await page
    .getByRole("textbox", { name: "MRN" })
    .fill("SYN-000142");
  await page
    .getByRole("button", { name: "Open chart" })
    .click();
  await page
    .getByRole("spinbutton", { name: "Heart rate" })
    .fill("78");
  await page
    .getByRole("button", { name: "Save chart" })
    .click();
  await expect(
    page.getByRole("status", { name: "Saved" }),
  ).toBeVisible();
});

The left version is a file. Your existing review process applies. The right version is content in someone else's database. The functional coverage is identical. The compliance posture is not.

Frequently asked questions