How to Auto-Discover Test Scenarios by Crawling: Why the Best Approach Crawls While Testing

1. The Problem with Crawl-First, Test-Second

The traditional approach to auto-discovering test scenarios follows a two-phase pipeline. Phase one: a spider or crawler visits your root URL, follows every link, submits every form, and builds a complete map of your application. Phase two: a test generator reads that map and creates scenarios for each page.

This sounds clean in theory. In practice, it has three serious problems.

First, the crawl phase is slow. Research on automated web application crawling has found that fully automated crawling approaches can take multiple days to complete on large applications. That makes it impractical in continuous development environments where you want feedback on every pull request.

Second, the crawler misses dynamic pages. A single-page application might render entirely different content based on authentication state, query parameters, or client-side routing. A traditional crawler that just follows href attributes will never reach the checkout confirmation page, the settings panel that only appears after onboarding, or the error state that triggers after a failed API call.

Third, the crawl results are stale by the time tests run. If the crawler found 200 pages yesterday, and a developer pushed a change that restructured the navigation today, the generated tests are targeting a map that no longer matches reality.

2. Opportunistic Discovery: Crawling as a Side Effect of Testing

There is an alternative pattern that solves all three problems. Instead of crawling first and testing second, you let the test agent discover pages while it is already running tests. Every time the agent navigates to a new URL during a test scenario, that URL gets queued for analysis. The crawl and the test happen simultaneously.

This is how Assrt works. When you point it at a URL and ask it to generate a test plan, it does not spider your site. It opens a real browser, takes screenshots at three scroll positions (0px, 800px, and 1600px), captures the accessibility tree at each position, and feeds up to 8,000 characters of that content to an AI model that generates 5 to 8 test cases.

Then, as those test cases execute, something interesting happens. Every time the test agent clicks a link, submits a form, or navigates to a new route, that new URL gets intercepted and added to a discovery queue. The agent finishes the current test scenario, then processes the queue: visiting each discovered URL, taking a snapshot, and generating 1 to 2 additional test cases for it.

The result is that testing and discovery feed each other. A test that clicks through a checkout flow discovers the order confirmation page. A test that logs in discovers the dashboard. The agent's own behavior becomes the crawl.

3. How the URL Queue Works Under the Hood

The discovery mechanism in Assrt is built around a simple queue with deduplication. The implementation lives in the test agent (specifically, a function called queueDiscoverPage in agent.ts). Here is what happens each time the agent navigates:

1. The URL is normalized (protocol, trailing slashes, query parameter ordering).
2. The agent checks whether this URL has already been discovered. If it has, the URL is skipped.
3. The agent checks whether the maximum number of discovered pages has been reached. If it has, the URL is skipped.
4. The agent checks the URL against a list of skip patterns (more on this in the next section). If it matches, the URL is skipped.
5. If the URL passes all three checks, it is added to a set of discovered URLs and pushed to a pending queue.

After each test scenario completes, a function called flushDiscovery processes the pending queue. For each URL in the queue, the agent takes a snapshot of the page (accessibility tree plus screenshot), emits a page_discovered event, and calls an AI model to generate test cases for that page.

This is deliberately simple. There is no recursive link extraction, no sitemap parsing, no depth-first search. The crawl surface is exactly whatever the test agent touches during execution. That means it naturally follows real user paths through the application rather than exhaustively enumerating every possible route.

4. Skip Patterns: URLs the Crawler Knows to Avoid

Not every URL the agent encounters should become a test target. Navigating to a logout endpoint would destroy the session state that other tests depend on. Hitting an API route directly would produce a JSON response, not a testable UI. Assrt defines six skip patterns that filter URLs before they enter the discovery queue:

These patterns are applied using case-insensitive regex matching. The logout pattern is particularly important because, in an opportunistic discovery model, the agent maintains a single browser session across all test scenarios. Cookies and authentication state carry over between tests. If the agent followed a logout link, every subsequent test would fail because the session would be gone.

This is a detail that crawl-first tools do not need to worry about, because their crawl phase and test phase are separate. When the crawl is the test, protecting session state becomes a first-class concern.

5. Micro Case Generation: 1-2 Tests Per Discovered Page

When a discovered page is processed, the agent does not try to generate a comprehensive test suite for it. It generates 1 to 2 test cases, each limited to 3 to 4 actions. These are intentionally small and focused.

The generation prompt explicitly instructs the AI model to reference actual buttons, links, and inputs visible on the page. It prohibits generating login/signup cases (since the agent is likely already authenticated), CSS or layout tests (since the agent tests via accessibility tree, not visual rendering), and performance tests (which require different instrumentation).

This micro-case approach serves two purposes. First, it keeps execution time bounded. If the agent discovered 15 new pages and generated 8 cases each, the test run would take an impractical amount of time. With 1 to 2 cases per page, the discovery phase adds minutes, not hours. Second, it prioritizes breadth over depth. A single sanity check on 15 pages catches more regressions than an exhaustive suite on 3 pages.

The generated cases follow the same #Case N: name format used throughout Assrt. They are real Playwright test steps, not proprietary markup or YAML. You can copy them into your own test suite, modify them, and run them independently. There is no vendor lock-in.

6. Hard Caps and Why They Matter

Opportunistic discovery without limits would be a crawl by another name. Assrt enforces three hard caps to keep the process fast and predictable:

These numbers are not arbitrary. A cap of 20 pages means even on a large application, the discovery phase finishes in a predictable window. Three concurrent discoveries balance throughput against the cost of parallel AI calls. The 8,000 character limit on accessibility tree content ensures the model receives enough context to generate useful tests without hitting token limits.

For comparison, traditional crawl-based tools often provide no cap at all, or set it so high (tens of thousands of pages) that the crawl itself becomes the bottleneck. The entire point of opportunistic discovery is that you do not need to visit every page. You need to visit the pages that matter, which are the ones real user flows actually reach.

7. Getting Started with Assrt

Assrt is open-source and free. You can start testing with opportunistic discovery in under a minute. There are two ways to use it:

Via the MCP server (for Claude Code or IDE users)

If you use Claude Code, Cursor, or another tool that supports the Model Context Protocol, Assrt runs as an MCP server. Point it at any URL and it will generate a plan, execute it, and discover additional pages along the way:

Then use the assrt_plan tool to generate test scenarios, or assrt_test to generate and execute in one step. The test agent handles discovery automatically.

Via the web app

Visit app.assrt.ai, paste your URL, and click generate. The web app uses the same underlying engine with a visual interface for watching tests execute in real time. You can see discovered pages appear as the agent navigates, and watch test cases get generated and executed live.

Frequently Asked Questions