Auto-Discovers Test Scenarios by Crawling: The Tool That Finds Tests While Running Tests

1. The Crawl-Then-Test Model and Why It Wastes Time

The conventional approach to auto-discovering test scenarios works like this: point a crawler at your root URL, let it spider through every link it can find, build a map of pages and interactive elements, then generate test cases from that map. Tools like AegisRunner, Testaify, and open-source crawlers like test-crawler and python-testing-crawler all follow this pattern.

The problem is that crawling and testing are treated as two completely separate activities. The crawler does not know which pages matter to your users. It treats your marketing blog with the same priority as your checkout flow. It follows every link, including ones that lead to logout endpoints, third-party OAuth redirects, and admin panels that will break the crawl session. Then you wait for the crawl to finish before any testing begins.

For a typical SaaS application with 200+ pages, a full crawl takes anywhere from 3 to 30 minutes depending on the tool. That is time spent before a single test runs. And because crawlers work by parsing the DOM for anchor tags and form actions, they miss pages that are only reachable through JavaScript navigation, modal flows, or multi-step wizards. The discovery is broad but shallow.

Assrt skips this entire phase. There is no crawl step. Discovery happens during test execution itself.

2. How Discovery During Execution Actually Works

When you run assrt_test with a URL and a test plan, an AI agent opens a real browser and starts executing your test scenarios step by step. It clicks buttons, fills forms, navigates between pages, and verifies outcomes. That part is straightforward.

What makes Assrt different is what happens in the background. Every time the agent navigates to a new URL during test execution, that URL gets added to a discovery queue via a function called queueDiscoverPage(). This happens at two specific points: when the agent first navigates to your target URL, and every time the agent follows a link or submits a form that lands on a new page during a scenario.

After each test scenario completes, the agent calls flushDiscovery(). This processes the queued URLs: for each one, Assrt takes a snapshot of the page (using the accessibility tree, not raw HTML), captures a screenshot, and sends both to a lightweight AI model. The model generates 1 to 2 quick test cases for that page, each completable in 3 to 4 actions.

The critical detail is that discovery runs concurrently with test execution. While your main test scenarios are running, discovered pages are being analyzed and turned into test cases in the background. You do not wait for discovery to finish before testing, and you do not wait for testing to finish before discovering. They happen together.

3. Inside the Discovery Queue: Constants, Prompts, and Concurrency

The discovery system is governed by two constants defined in the agent source code (agent.ts). MAX_DISCOVERED_PAGES is set to 20, meaning a single test run will discover and generate scenarios for up to 20 new pages. MAX_CONCURRENT_DISCOVERIES is set to 3, which limits parallel scenario generation to 3 pages at a time to avoid overloading the browser session.

The AI prompt used for discovery is deliberately constrained. The DISCOVERY_SYSTEM_PROMPT instructs the model to act as a QA engineer generating quick test cases for a page the agent just landed on. It enforces several rules: generate only 1 to 2 cases, keep each case to 3 to 4 actions maximum, reference actual buttons and links visible on the page, and skip login/signup cases, CSS checks, responsive layout tests, and performance tests.

This is a deliberate design choice. Discovery cases are meant to be fast, targeted smoke tests for pages that were not in your original plan. They verify that the page loads, that key interactive elements work, and that the basic flow makes sense. They are not meant to be exhaustive regression suites. If a discovered page needs deeper coverage, you can take the generated cases and expand them.

URL normalization strips trailing slashes and reduces each URL to its origin plus pathname, which prevents duplicate discoveries for the same page with different query parameters or fragments.

4. What Gets Skipped (and Why That Matters)

Not every URL the agent visits should become a test case. The discovery system automatically filters out URLs matching several patterns: /logout, /api/, javascript:, about:blank, data:, and chrome:// URLs.

This is worth highlighting because it shows a problem that crawl-first tools often run into. A naive crawler will follow a logout link, destroy the session, and then fail on every subsequent page. It will hit API endpoints and try to generate UI tests for JSON responses. It will follow OAuth redirect chains and end up on third-party login pages. Each of these wastes crawl time and generates useless test cases.

Because Assrt's discovery is agent-driven rather than link-driven, it naturally avoids most of these problems. The agent navigates like a user, so it tends to visit pages that are actually part of the user experience. The skip patterns are a safety net for the edge cases, not the primary filtering mechanism.

5. How This Compares to AegisRunner, Testaify, and Traditional Crawlers

AegisRunner is the closest competitor in approach. It crawls your site with a real Chromium browser, identifies interactive elements, and generates Playwright TypeScript code. It handles JavaScript rendering and SPAs. The key difference is architectural: AegisRunner crawls first, then generates tests from the crawl results. The free tier limits you to 25 pages per crawl and 15 crawls per month. Paid plans start at $9/month and go up to $79/month. Assrt has no page limit, no crawl quota, and is free.

Testaifyuses an "intelligent bot-crawling approach" that takes 30 minutes for smoke testing and up to 6 hours for regression testing. It builds a visual application model from the crawl, then generates test scenarios from that model. The test format is proprietary to Testaify's platform. The service is currently in managed rollout with no public pricing. Assrt outputs standard Playwright code that runs anywhere.

test-crawler (open-source) crawls pages and does screenshot comparison for visual regression, but does not generate functional test scenarios at all. python-testing-crawler focuses on link validation and HTTP status codes rather than interactive scenario generation.

6. What the Discovered Scenarios Look Like

Discovered test cases use the same format as manually created ones. Each case starts with a #Case N: header followed by step-by-step instructions in plain English:

#Case 1: Verify product listing loads
Navigate to /products. Verify at least 3 product cards are visible.
Click the first product card. Verify the product detail page loads
with a title and price.

#Case 2: Test search functionality
Click the search icon in the header. Type "wireless" in the search
input. Press Enter. Verify search results appear with at least
one matching product.

These cases are saved to /tmp/assrt/scenario.md alongside your original test plan. You can edit the file directly to refine, remove, or expand any discovered case. On re-run, Assrt picks up your edits. When executed, these plain-English cases are translated into real Playwright browser actions: clicking elements, typing into inputs, asserting visible text, and navigating between pages.

The discovery prompt explicitly tells the AI to reference actual buttons, links, and inputs visible on the page. This means discovered cases are grounded in what the page actually contains, not in generic test templates. If a page has a "Download CSV" button, the discovered case will reference that specific button. If the page has a date picker, the case will interact with the date picker.

7. Getting Started with Assrt Discovery

Discovery is built into the standard test execution flow. You do not need to enable it or configure it separately. Here is how to see it work:

1. 1.Generate a test plan. Run assrt_plan with your application URL. Assrt takes screenshots at multiple scroll positions, extracts the accessibility tree, and generates 5 to 8 test cases for the page.
2. 2.Run the tests. Execute assrt_test with the URL and your plan. As the agent runs each scenario and navigates to new pages, those pages get queued for discovery automatically.
3. 3.Check discovered scenarios. After the run completes, read /tmp/assrt/scenario.md. You will see your original cases plus any new cases the agent discovered during execution.
4. 4.Edit and re-run. Remove cases you do not want, expand ones that need more depth, and re-run. Each subsequent run may discover additional pages if your tests navigate to new areas of the application.

Because Assrt is self-hosted and runs locally, your application code and test results never leave your machine. There is no account to create, no API key to provision, and no usage quota to worry about.

8. Frequently Asked Questions