A Playwright AI test generator that handles OTP needs four tools, in a strict order. Most generators ship three of them.

The shape of the failure

Generic AI test generators are good at translating intent into syntax. They are not good at orchestrating runtime side effects that have to happen between two lines of test code. An OTP signup is exactly that kind of flow: between click("Sign up") and click("Verify"), a real email has to land somewhere readable, get parsed, and come back as a six-digit string before the test can continue. None of that fits in a single static file.

So the model does the only thing it can do from inside one file: it hardcodes a placeholder, leaves a TODO, and submits.

The four tools, in the order the agent calls them

You can read this list in the source as create_temp_email, browser_*, wait_for_verification_code, and evaluate. The order is enforced two ways: the system prompt says “FIRST call create_temp_email”, and the runtime handlers refuse to run later steps if no inbox exists. Soft and hard guards on the same rule.

What the system prompt actually pins down

A reasonable question at this point: why bake the OTP recipe into a system prompt? Could a smart model not figure it out? In practice, no. The failure modes (calling .fill() per input, hardcoding the code, treating ref attributes as DOM selectors) are stable across Sonnet, GPT-5, and Gemini. The system prompt fixes them by handing the agent the exact evaluate expression and forbidding deviation:

That last sentence is the part most generic generators do not have. A model left to its own devices treats the “dispatch a clipboard event into the parent” trick as a hint and improvises a different version every time. Pinning the expression as a literal string in the prompt is what stops the variance.

The seven-pattern regex cascade

Every email OTP is “some digits, somewhere in the body, surrounded by varying amounts of marketing copy”. There is no one regex that catches them all without false positives. The cascade tries seven patterns in order, most specific first, and stops at the first match:

// src/core/email.ts:101-109
const patterns = [
  /(?:code|Code|CODE)[:\s]+(\d{4,8})/,
  /(?:verification|Verification)[:\s]+(\d{4,8})/,
  /(?:OTP|otp)[:\s]+(\d{4,8})/,
  /(?:pin|PIN|Pin)[:\s]+(\d{4,8})/,
  /\b(\d{6})\b/, // 6-digit (most common)
  /\b(\d{4})\b/, // 4-digit
  /\b(\d{8})\b/, // 8-digit
];

The ordering matters. Without it, an email body like “Order #845710 has shipped. Your verification code is 391847.” would match the bare 6-digit pattern first and return 845710, the order number. With the cascade, verification[:\s]+(\d{4,8}) fires first and returns 391847, the actual code.

A generic LLM asked to extract a code with a regex on the spot will usually pick the bare 6-digit pattern, which is right about 80% of the time and silently wrong the other 20%. The cascade trades a few milliseconds of regex matching for that 20%, which is the difference between a test suite you trust and one you babysit.

What flows between the agent and the browser

The four tools become a single sequence at runtime. Two of them (create and poll) talk to a third-party HTTP service; two of them (drive and dispatch) go through Playwright. The diagram makes the dependency clear: nothing happens after step 1 until step 1 succeeds, and nothing happens after step 5 until step 5 returns a code.

The thing the diagram does not show, but matters, is that none of these steps share state through the test file. They share state through the agent's tool-call history. That is why the same agent can run twenty OTP scenarios in a row against twenty different inboxes without the file growing or breaking. The static .spec.ts artifact is what lands in your repo at the end, after the dust settles.

What gets committed: the file vs. the orchestration

Once the agent has run the scenario successfully, it serializes the run as a standard Playwright file you can read, edit, and check in. The orchestration contract still applies at runtime, but the file itself is the same shape a senior engineer would produce by hand if they had two free hours and the seven-pattern cascade in their head:

// What ChatGPT, Cursor, and Codex generate when you say
// "write me a Playwright test for OTP signup"
import { test, expect } from "@playwright/test";

test("signup with OTP", async ({ page }) => {
  await page.goto("/signup");
  await page.getByLabel(/email/i).fill("test@example.com");
  await page.getByRole("button", { name: /sign up/i }).click();

  // TODO: replace with actual OTP code from email
  const code = "123456";

  // Looks reasonable. Fails on every modern OTP component
  // because .fill on each input fires keystrokes, not paste.
  for (let i = 0; i < 6; i++) {
    await page.locator('input[maxlength="1"]').nth(i).fill(code[i]);
  }

  await page.getByRole("button", { name: /verify/i }).click();
  await expect(page.getByText(/welcome/i)).toBeVisible();
});

The right column is real Playwright. No proprietary YAML, no cloud format, no opaque scenario IDs. The two helper functions live in your repo, your CI runs them on every PR, and if you decide to replace temp-mail.io with Mailosaur or Mailslurp tomorrow, you swap the inbox helper and the regex cascade keeps working unchanged.

The honest counterargument: where Mailosaur and Mailslurp earn their keep

A page that pretends Mailosaur and Mailslurp do not exist is dishonest. They are excellent receive-side services, and any team shipping OTP tests in production CI should have one of them or an equivalent. The disposable temp-mail.io endpoint is a great default because it is zero-config and free, but it is unauthenticated, rate-limited per IP, and can be blocked by sender deliverability rules. For real CI you want predictable, paid-for inboxes.

The architectural point of this page is not “use temp-mail.io forever.” It is “the agent needs an inbox-receive primitive at all, and it needs to call it before form fill.” Whether the implementation behind that primitive is temp-mail.io, Mailosaur, Mailslurp, or your own SES-backed catch-all is a downstream choice. Assrt's DisposableEmail class is intentionally a thin wrapper so it can be swapped without changing the agent.

What none of those services solve on their own is the orchestration itself. They give you a stable inbox; they do not give you the system prompt that says “FIRST call create_temp_email” or the canonical evaluate expression for split-digit dispatch. That layer is the one this page is about.

Try it on a real signup flow

The fastest way to see whether your existing AI test generator has the four-tool contract is to point it at a signup form that uses a split-digit OTP component. Most modern stacks (Clerk, Auth0 passwordless, Supabase) ship one by default. If the generated test stops at “TODO: replace with actual code”, you have your answer.

# Run against your app and read the four tool calls in the live stream
npx @m13v/assrt discover https://your-app.com

The repo is at github.com/assrt-ai/assrt-mcp. The four tool definitions are 18 lines total in src/core/agent.ts; the polling and regex cascade are in src/core/email.ts; the system prompt that pins the order lives a few hundred lines below the tool definitions in the same file. Worth reading before deciding whether your current setup is a few prompt edits away or a rewrite.

Frequently asked questions