Open source testing tutorial: real signup, real OTP, no mocks
Every top-ten Google result for this phrase is a list of tools. Selenium, Cypress, Playwright, Appium, JMeter. None of them walk a reader through the first wall a real QA engineer hits on day one: testing a signup flow against a live email server with a multi-digit OTP input. This tutorial ships that exact flow, in 6 lines of plain English, running against a throwaway inbox, with the regex patterns and paste expression the listicles never mention.
What every SERP winner leaves out
Pull up the top ten Google results for this phrase today. softwaretestinghelp, geeksforgeeks, katalon, bugbug, opensource.com. Every one of them is a bullet list of open source tools followed by a short paragraph per tool. There is no plan you can type, no terminal command you can run, no directory you can list when it finishes. The hardest single problem in testing a real SaaS, a signup flow with email verification, does not get one line of treatment. This page is the inverse. It starts from the signup-plus-OTP case and walks through every file that makes it work.
Seeded accounts hide bugs
Pre-created test users skip the exact hop the product breaks on: the verification email itself. If the email never sends, the test still passes because the test never read an inbox.
Multi-digit OTP fields eat keystrokes
Most libraries listen for a paste event on the wrapper and split the value across inputs. Typing one digit at a time triggers focus guards that drop every key except the first.
Regex order is load-bearing
A bare /\d{6}/ would match a reference ID before it matches the code. Patterns must try labeled captures first (code, verification, OTP, pin) and fall back only if nothing matches.
3s poll, 60s timeout
Short enough for transactional emails, long enough for queued ones. Set at email.ts lines 67-77. waitForEmail returns the last message in the inbox so multiple in-flight tests do not collide.
Seven patterns, priority order
Labeled captures bind to the word in the email body. Bare-digit fallbacks are tried last, so the test will not confuse `ref: 1234` with the code. Empty match returns the body text, not null.
Inbox lives outside your infrastructure
Every test gets a fresh 10-character name from temp-mail.io, so you never pollute your production user table with signup@test1@test2@...
0 patterns, 0s poll, 0s timeout, 0 paid services. Every number in this page comes from a line in the open source repo, not a marketing slide.
The whole test, in six lines of English
This is the complete plan for a signup-plus-OTP test. It is not pseudo-code, and it is not a simplification. The file saved to disk is exactly what the parser reads, and the parser is the single regex at agent.ts line 569: /(?:#?\s*(?:Scenario|Test|Case))\s*\d*[:.]\s*/gi. Split on any case header, body is the steps.
Three inputs, one LLM, four endpoints it can reach
The plan file, the target URL, and your LLM key are the only three things you feed the runner. Everything else is code in the open source repo. Claude Haiku 4.5 reads the plan, picks from 18 browser tools, and reaches out to four systems on your behalf: a disposable inbox, a Chromium instance via Playwright MCP, the ClipboardEvent API inside the page, and the filesystem under /tmp/assrt/.
Signup + OTP pipeline
Every message that crosses the wire, once
This is what actually happens when the six-line plan runs. The sequence is not hand-drawn; each row lines up with a tool call the agent emits, and every emission is logged to events.json so you can replay the same diagram from a real run. Note the loop at step 8: GET /messages repeats every 3 seconds until the inbox is non-empty or 60 seconds pass.
scenario.md to /welcome, every hop
The seven regex patterns, in priority order
Here is the actual block of source code that extracts the code. It sits at email.ts lines 101 through 109. Priority matters: a body that reads `verification code 482155, reference ID 1234` would otherwise match the reference ID first. Every labeled pattern is tried before any bare-digit fallback. The function returns the first hit it finds and stops; if all seven miss, it returns the email body text so you can see what actually arrived instead of a null that erases context.
“Labeled captures bind to the word in the body. Bare digits are the last resort. The test will not confuse a reference ID with the code.”
email.ts lines 101-109
One ClipboardEvent fills the 6-box OTP input
Every React signup UI written in the last three years uses a 6-input OTP component whose wrapper intercepts the paste event and splits the value across boxes. Typing one digit at a time bounces off focus guards. The agent's system prompt pins the exact expression it should dispatch: a DataTransfer with the code as text, a ClipboardEvent of type paste, fired at the parent of the first input[maxlength="1"]. The expression is copy-pasted verbatim from agent.ts line 235. Only CODE_HERE gets replaced.
Six steps, six minutes of reading, then rerunnable forever
The story of a signup test, walked through the order the agent executes it. Each step maps to one or more tool calls in events.json. Every step has a line number you can open in the repo; nothing here is a black box.
Allocate a disposable inbox before touching the form
The agent calls create_temp_email first. DisposableEmail.create() posts to https://api.internal.temp-mail.io/api/v3/email/new with min_name_length and max_name_length both set to 10. Returns {email, token}. The token is kept in memory for the polling call; the email address is the value you feed into the signup form.
Submit the form with the throwaway address
Normal Playwright MCP path: snapshot to get the accessibility tree with [ref=eN] ids, type_text into the email field, click the Continue button. The only thing different from a seeded-account test is where the email string came from.
Poll the inbox until the email lands or the timeout fires
wait_for_verification_code calls waitForEmail internally, which GETs /email/{address}/messages every 3 seconds. When the inbox has at least one message, it returns the last one. Max wait is 60 seconds by default; passing timeout_seconds on the tool call overrides.
Run the seven regex patterns against the email body
Priority order: `code:` label, `verification:` label, `OTP:` label, `pin:` label, then bare 6-digit, 4-digit, 8-digit. First match wins. If all miss, returns {code: '', body}. That empty-code path is what you see when the email arrives but the text does not match any pattern.
Paste the code using one ClipboardEvent, not individual keystrokes
The agent calls evaluate with the fixed expression pinned at agent.ts line 235. DataTransfer + ClipboardEvent('paste') on the parent element of the first input[maxlength="1"]. This is the only path most OTP UI libraries consume; typed keystrokes bounce off.
Assert, record, move on
After the paste, the agent calls snapshot to confirm every maxlength=1 input is filled, clicks Verify, waits for the DOM to stabilize with wait_for_stable, and asserts the dashboard heading is visible. Result goes into events.json; the WebM recording plays the whole flow back.
Run it, watch the 11 tool calls, keep the video
This is the real terminal output from running the six-line plan against a live app. Timestamps omitted, tool calls in order. When the run finishes there is a WebM recording on disk plus a numbered screenshot per visual action, so a failure takes 30 seconds to diagnose.
Assrt vs. raw open source framework
The listicle winners ship Playwright / Cypress / Selenium and call it done. That is the box. This is what you have to build yourself on top of the box before a signup test passes.
| Feature | Raw Playwright / Cypress | Assrt (open source) |
|---|---|---|
| Signup flow out of the box | Write your own fixture user factory per test suite | Three tools (create_temp_email, wait_for_verification_code, check_email_inbox) built into the agent runtime |
| OTP extraction | Hand-roll a pattern per email template, hope vendor does not change it | Seven regex patterns in priority order, shipped (email.ts lines 101-109) |
| Multi-digit OTP paste | Loop typing digits, break on React Hook Form, debug for 30 minutes | One fixed ClipboardEvent expression at agent.ts line 235, always works |
| Test file rewrite when UI moves | Update every locator by hand; `.nth(2)` becomes `.nth(3)` | Selectors re-discovered from accessibility tree every run; plan unchanged |
| Inbox cost | Mailosaur at $200/mo for moderate volume | $0 via temp-mail.io public v3 API |
| Vendor lock-in on the plan | Proprietary YAML or cloud visual builder, unexportable | scenario.md in your repo is the whole spec; portable |
CI readiness checklist
Once the signup plan passes locally, move it into CI with seven boxes ticked. Every item here is one commit or one workflow line; nothing requires a new service.
Before you merge the plan to main
- Plan file committed to the repo at tests/signup.md
- Preview URL passed to --url on every preview deployment
- Run uploaded as an artifact on failure (/tmp/assrt/<runId>/)
- LLM key stored in CI secrets, not in a config file
- temp-mail.io rate limit headroom checked for parallel runs
- WebM recording attached to failed-test Slack alerts
- Assertion for the dashboard heading, not just HTTP 200
The unfair advantage of a runnable tutorial
A tutorial you can type into a file and run is worth ten tutorials you can only read. Three things survive the round trip; only the last one stays.
Selenium, Cypress, Playwright. Five paragraphs per tool, zero working code, zero guidance on the first real wall.
Plan in their UI, run in their cloud, read results on their domain. Uninstall and you lose every test you wrote.
scenario.md at 6 lines, @assrt-ai/assrt as one npm dep, /tmp/assrt/ on your disk. Uninstalling leaves the plan; the plan still runs in any future tool that can read #Case blocks.
Want to see the OTP flow run on your app?
Book 20 minutes and we will write your first signup-plus-OTP #Case against a real preview URL.
Book a call →Frequently asked questions
What does this open source testing tutorial cover that the top SERP results do not?
The ten best-ranked results for this phrase on Google are listicles. They name Selenium, Cypress, Playwright, Appium, JMeter, then stop. Not one of them walks a reader through a real signup flow end-to-end: create a throwaway inbox, submit the form, pull the OTP out of the email body, paste it into a split-input verification UI, and assert the user lands on the dashboard. That is the first wall a new tester hits, and it is exactly where the listicles hand waive. This tutorial ships the missing step. The Assrt codebase, which is open source, implements all of it in two files: a DisposableEmail class in assrt-mcp/src/core/email.ts and the OTP-aware paste logic wired into the agent at assrt-mcp/src/core/agent.ts line 235.
Why do you need a disposable inbox instead of a seeded test account?
A seeded account skips the part the product actually has bugs in: the verification email itself. The email might never send, the link might be wrong, the OTP might have the wrong TTL, the signed token in the link might not verify, the multi-digit OTP field might only accept paste and not typed input. None of those failures surface if you log in as a pre-created user. A disposable inbox lets you run the real submission path against a real SMTP hop. In this tutorial the inbox is allocated from temp-mail.io via their internal v3 API (POST https://api.internal.temp-mail.io/api/v3/email/new with a 10-character name length), returning {email, token}. That happens inside DisposableEmail.create() at email.ts lines 43 through 56.
How does the agent extract the OTP code from the email body?
By trying seven regex patterns in priority order and returning the first match. The patterns live at email.ts lines 101 through 109, and the order matters because a body that contains `your verification code is 842155, reference ID 1234` would otherwise match the reference ID first. The list, in order: /(?:code|Code|CODE)[:\s]+(\d{4,8})/, /(?:verification|Verification)[:\s]+(\d{4,8})/, /(?:OTP|otp)[:\s]+(\d{4,8})/, /(?:pin|PIN|Pin)[:\s]+(\d{4,8})/, /\b(\d{6})\b/, /\b(\d{4})\b/, /\b(\d{8})\b/. The first four bind to a label, the last three are bare-digit fallbacks. If every pattern misses, the function returns an empty code plus the email body so you can see what actually arrived instead of a null that erases context.
How does the agent paste an OTP into a UI with multiple single-character inputs?
One ClipboardEvent. The exact expression the system prompt tells the agent to dispatch is pinned at agent.ts line 235: const inp = document.querySelector('input[maxlength="1"]'); const c = inp.parentElement; const dt = new DataTransfer(); dt.setData('text/plain', 'CODE_HERE'); c.dispatchEvent(new ClipboardEvent('paste', {clipboardData: dt, bubbles: true, cancelable: true})). This matters because most OTP UIs register a paste handler on the input group container that splits the pasted value across every box. If you keystroke one digit at a time, React Hook Form or similar controllers intercept the key events and discard all but the focused field. The paste event is the only path the UI library expects, and every agent that does not know this will spend 30 minutes on a phantom focus bug.
How long does the agent wait for the email to arrive, and how often does it poll?
60 seconds default timeout with a 3 second poll interval, set at email.ts lines 67 through 77 inside waitForEmail. Each poll calls GET /email/{address}/messages on the temp-mail.io API and returns the last message if the inbox is non-empty. The agent loop calls the higher-level waitForVerificationCode, which first waits for any email and then runs the seven-pattern regex scan. If you need shorter polling for a transactional email that arrives in under 2 seconds, pass intervalMs. If you need longer because your provider has a slow queue, bump timeoutMs. Neither changes behavior, just cadence.
Where does the verification code come back to the agent, and how does it know to type it?
The tool result for wait_for_verification_code is a string of the shape `Verification code received: 482155\nFrom: noreply@example.com\nSubject: Your verification code`. That string is appended to the conversation as a tool_result, so the next model turn sees it as prior context. The model then picks the next tool call, usually evaluate with the clipboard expression above, or type_text into the visible OTP input, and fills the code. There is no hardcoded control flow that links email receipt to form fill. The agent reasons about it each turn with the full transcript.
Does this tutorial require any paid SaaS or vendor lock-in?
No. Assrt is open source (MIT, see assrt-mcp/LICENSE). The disposable inbox is a free public API. The browser runs via @playwright/mcp, also open source. The only paid dependency is the LLM, and it is bring-your-own: either an Anthropic API key or a Claude Code OAuth token in macOS Keychain (see assrt-mcp/src/core/keychain.ts). Every file the run produces is on your local disk at /tmp/assrt/. The scenario.md plan is yours to commit to your repo. Uninstalling assrt deletes one npm package; it does not lock up your tests in a proprietary dashboard you pay $7.5K/month to read.
Why does this tutorial not use Cypress or Playwright directly?
You absolutely can. Every tool Assrt provides to the agent is a thin wrapper over Playwright MCP primitives. The reason this tutorial is not a Playwright-only tutorial is that writing the signup-plus-OTP flow in raw Playwright code is 60 to 120 lines of TypeScript per case, every line has a locator that rots when the UI changes, and the test code has to be updated by a human with knowledge of the current markup. The #Case format is 6 lines of English that describe intent, and the agent rediscovers the selectors at run time from an accessibility tree. When the Email field moves from aria-label to placeholder, Playwright code fails; the #Case keeps passing because the accessibility tree snapshot still finds an email-shaped input.
Can I run this tutorial in CI without a visible browser?
Yes. Omit the --headed flag. Playwright MCP launches Chromium headless and the recording still works. Add a workflow step that runs `npx @assrt-ai/assrt run --url $PREVIEW_URL --plan-file tests/signup.md --json` and uploads /tmp/assrt/<runId>/ as an artifact on failure. GitHub Actions, Vercel preview deploys, and Fly.io release pipelines all work. The only operational note: temp-mail.io is rate limited per source IP, so a CI runner that burns through 200 signup tests per hour may see throttling. For that scale, swap DisposableEmail for your own inbox service by replacing the fetch URL in email.ts.
What does a failed OTP test look like in the artifacts?
The run directory under /tmp/assrt/<runId>/ keeps everything. events.json has every tool call with timestamps; you can see the exact second wait_for_verification_code returned, and whether the code field was null or populated. screenshots/ has a numbered PNG for each visual tool call: 04_step5_type_text.png will show the OTP field, and 05_step6_assert.png will show whatever the page looked like when the assertion ran. The WebM recording at video/recording.webm plays back the whole run at 1x (default) or 5x (press 5 in the player); a wrong paste is usually obvious at 5x. And because the DisposableEmail logs the email body into the tool result, you can see exactly what text the regex failed to match against.
How do I write my first #Case for a signup + OTP flow?
Six lines. `#Case 1: New user can sign up and verify email` header, then numbered steps: `1. Call create_temp_email to get a throwaway address 2. Navigate to /signup 3. Type that email into the Email field 4. Click the Continue button 5. Call wait_for_verification_code and use evaluate with a ClipboardEvent paste to fill the code 6. Assert the text 'Welcome' appears on the page`. The agent reads each step as English, picks one of 18 named tools per step, and logs the result. The full tool list is in agent.ts between lines 18 and 186; the three email tools you need for this flow are create_temp_email, wait_for_verification_code, and check_email_inbox.
What about tests beyond signup and OTP? Is #Case enough?
For most web applications, yes. The same format covers login, onboarding, profile updates, settings changes, subscription upgrades, and destructive flows like account deletion. For third-party integrations (Telegram bot, Slack webhook, GitHub PR check), the agent has http_request as tool 17, so a #Case can chain `submit form`, `poll the third-party API`, `assert the message arrived`. wait_for_stable (tool 18) handles async content that updates after the initial load: LLM chat responses, search suggestions, lazy-loaded dashboards. The one flow this shape does not handle well is pixel-perfect visual regression; for that you still want a dedicated screenshot-diff tool.
Adjacent pages that dig into specific parts of the runtime.
Keep going
AI Testing Tutorial
The companion to this page: the 18 tools the agent can call and where every line of the plan parser lives.
Open Source Testing Guide
How the scenario.md file survives edits with a 1-second FSWatcher debounce and no codegen step.
Automated Open Source Testing
The runner itself is a forkable file. Everything here is copy-pastable into your own repo.
How did this page land for you?
React to reveal totals
Comments (••)
Leave a comment to see what others are saying.Public and anonymous. No signup.