Assrt vs Manual Playwright: The Email Verification Gap

1. The Gap Manual Playwright Tests Can't Close

Playwright is excellent at browser automation. It clicks buttons, fills forms, asserts text, takes screenshots. But modern web apps don't live entirely inside the browser tab. A signup flow sends a verification email. A password reset flow sends a magic link. A two-factor auth flow sends an OTP code.

Manual Playwright tests have no built-in way to handle these. The @playwright/test API covers browser actions. Everything outside the browser (receiving email, reading SMS, calling webhooks) requires you to integrate external services, write custom helpers, and manage API credentials.

This is where most teams give up on end-to-end testing for auth flows. They mock the email step, or they skip it entirely. Either way, the most critical user journey in the app goes untested.

2. Three Email Tools in agent.ts

The Assrt SDK defines 18 tools that its AI agent can call during test execution. Three of them handle email:

These tools are defined in src/core/agent.ts of the @assrt-ai/assrt npm package. They sit alongside the 15 browser tools (navigate, click, type_text, assert, screenshot, and so on) as equal participants in the agent's tool set. The agent calls them the same way it calls any other tool: by deciding, based on the test step and current state, that it needs to interact with email.

In a plain English test plan, using these tools looks like this:

#Case 1: Full signup with email verification
1. Create a temporary email address
2. Navigate to the signup page
3. Type the temp email into the email field
4. Type a password into the password field
5. Click Sign Up
6. Wait for the verification code in the email
7. Type the verification code into the OTP input
8. Click Verify
9. Verify the dashboard loads

3. What Email Testing Looks Like in Manual Playwright

To test the same signup flow in manual Playwright, you need to:

1. Choose an email service. Mailosaur, Mailhog, Ethereal, or a custom SMTP trap. Each has its own API, pricing, and limitations.
2. Install and configure the SDK.Add the service's npm package, create an API key, store it in environment variables, and initialize the client in your test setup.
3. Write inbox helpers. Create a function that generates a unique inbox address, another that polls for new messages with exponential backoff, and another that parses the HTML email body to extract a 6-digit code or a link.
4. Wire it into the test. Call the helpers between Playwright actions, pass the extracted code back to page.fill(), and add timeout logic for slow email delivery.
5. Handle CI environments. Make sure the email service is accessible from CI, that API keys are available as secrets, and that inbox cleanup runs between test suites.

A typical implementation adds 50 to 100 lines of setup code before you write a single assertion. Here is a simplified version:

import { test, expect } from '@playwright/test';
import Mailosaur from 'mailosaur';

const mailosaur = new Mailosaur('your-api-key');
const serverId = 'your-server-id';

test('signup with email verification', async ({ page }) => {
  const email = `test.${Date.now()}@${serverId}.mailosaur.net`;

  await page.goto('https://myapp.com/signup');
  await page.getByLabel('Email').fill(email);
  await page.getByLabel('Password').fill('SecurePass123!');
  await page.getByRole('button', { name: 'Sign Up' }).click();

  // Poll Mailosaur for the verification email
  const message = await mailosaur.messages.get(serverId, {
    sentTo: email,
  }, { timeout: 30000 });

  // Parse the OTP from the email body
  const match = message.html.body.match(/\d{6}/);
  const code = match?.[0] ?? '';

  await page.getByLabel('Verification code').fill(code);
  await page.getByRole('button', { name: 'Verify' }).click();
  await expect(page.getByText('Dashboard')).toBeVisible();
});

This requires a Mailosaur account (starting at $99/month), an API key stored as a CI secret, regex parsing for the verification code, and timeout handling for slow delivery. If your email template changes and the code is no longer a 6-digit number, the regex breaks silently.

4. Side-by-Side: Signup Flow Test

5. Beyond Email: UX Feedback That Tests Don't Give You

Manual Playwright tests are binary: they pass or they fail. They tell you whether the assertion matched. They never tell you that the loading spinner was missing, or that the error message was confusing, or that the form submitted without visual feedback.

Assrt's agent has a tool called suggest_improvement that it can call at any point during a test run. The structured output includes a title, severity level (minor, major, critical), and a description of the issue. These show up in the results JSON alongside pass/fail data.

For example, while testing a signup flow, the agent might notice that clicking "Sign Up" shows no loading indicator. It calls suggest_improvement with:

{
  "title": "No loading state after form submission",
  "severity": "major",
  "description": "After clicking Sign Up, the button remains
    static with no spinner or disabled state. Users may
    click it multiple times, creating duplicate accounts."
}

This transforms testing from "did the code work?" to "is the experience good?" Manual Playwright tests will never surface these insights because they only check what you explicitly assert. The agent checks everything it sees.

6. Frequently Asked Questions