AI Changeset Analysis for QA: Smarter Test Selection on Every PR

1. Where AI Provides Practical QA Value Today

The AI hype in testing has produced a lot of noise and relatively few proven practices. But several applications have moved beyond experimentation into daily use at production-scale teams. The two areas delivering the most consistent value are test scenario generation (using AI to suggest what to test based on requirements or code changes) and intelligent test selection (using AI to decide which existing tests to run on each build).

Test scenario generation works well because it plays to AI's strengths: analyzing text, identifying patterns, and generating structured output. Given a feature requirement or user story, an LLM can suggest test scenarios that cover happy paths, edge cases, error conditions, and boundary values. The quality varies, but even imperfect suggestions accelerate the QA planning process by giving teams a starting point instead of a blank page.

Intelligent test selection works well because it is a constrained optimization problem. Given a set of code changes and a mapping of tests to code paths, the system selects the subset of tests most likely to detect regressions introduced by those changes. This does not require the AI to understand your business logic deeply. It requires it to trace dependencies and rank tests by relevance, which is a pattern-matching task where LLMs excel.

2. How Changeset Analysis Works

At its core, AI changeset analysis takes a git diff as input and produces a ranked list of test scenarios as output. The process has several stages. First, the system parses the diff to identify which files changed, what functions were modified, and what the nature of the change was (new feature, bug fix, refactor, configuration update). Second, it maps those changes to application features and user flows using a dependency graph or learned associations.

The dependency mapping is where the intelligence matters. Simple approaches use file-level heuristics: if a file in the checkout module changed, run all checkout tests. More sophisticated approaches trace import chains and call graphs to identify exactly which test scenarios exercise the modified code paths. LLM-based approaches can go further by analyzing the semantic meaning of the change (for example, recognizing that a change to a validation function affects all forms that use it, even if the forms are in different modules).

The third stage is prioritization. Not all affected tests are equally important. Tests that directly exercise the modified code are highest priority. Tests that exercise downstream consumers of the modified code are next. Tests that might be affected through indirect dependencies are lowest priority. The system assigns a relevance score to each test and selects a subset that fits within the target execution time budget.

The practical result is that instead of running your entire regression suite on every PR (which might take sixty minutes), you run a targeted subset that takes thirty minutes and catches the same defects. The full suite still runs on a schedule (nightly or on merge to main) to catch anything the targeted selection missed. This hybrid approach is where teams see the 50% reduction in regression execution time.

3. AI for Test Scenario Generation and Bug Triage

Beyond test selection, AI is proving valuable for generating test scenarios from requirements documents and for triaging bug reports. When a product manager writes a feature specification, an LLM can analyze the text and produce a structured list of test scenarios covering positive flows, negative flows, boundary conditions, and integration points. This does not replace the QA engineer's judgment, but it provides a comprehensive starting point that is faster than brainstorming from scratch.

Bug triage benefits from similar analysis. When a bug report comes in, the AI can analyze the description, identify potentially affected components, suggest which existing tests should have caught the issue (and may need strengthening), and recommend additional test scenarios to prevent regression. This accelerates the triage process and ensures that bug fixes are accompanied by targeted test improvements.

The key insight is that these applications use AI as an accelerator for human decision-making, not a replacement. The QA engineer reviews the AI's suggestions, removes irrelevant scenarios, adds domain-specific cases the AI missed, and prioritizes based on risk. The AI handles the breadth (generating many possible scenarios quickly) while the human handles the depth (evaluating which scenarios actually matter for the application's specific context).

Assrt combines these capabilities by analyzing your application and generating real Playwright test code for discovered scenarios. Rather than producing abstract scenario descriptions that someone needs to implement manually, it outputs runnable tests that can be reviewed, modified, and merged. The setup is minimal: npx @m13v/assrt discover https://your-app.com

4. The Coverage Inflation Problem

One of the biggest risks with AI-generated tests is coverage inflation. An AI can generate hundreds of tests that exercise different code paths, driving line coverage and branch coverage metrics up dramatically. On the dashboard, it looks like your test suite improved overnight. In practice, many of these tests may be verifying trivial behavior, duplicating existing coverage, or making assertions so shallow that they would pass even if the feature was broken.

Coverage inflation is dangerous because it creates false confidence. A team sees 90% code coverage and assumes their application is well-tested. But if half of that coverage comes from AI-generated tests that assert only that pages load without errors, the effective coverage for meaningful defect detection is much lower. The metric becomes a vanity number rather than a useful signal.

The root cause is that coverage metrics measure which code was executed, not whether the tests verified that the code behaved correctly. An AI-generated test that navigates to every page and asserts expect(page).toBeVisible() will hit a lot of code but catch very few defects. Meaningful tests need assertions that verify specific business outcomes: that the correct data appears, that calculations produce expected results, that error states are handled properly.

The solution is to evaluate AI-generated tests on defect detection capability, not coverage contribution. Mutation testing (intentionally introducing bugs and checking which tests catch them) is the gold standard for measuring test effectiveness. If an AI-generated test does not catch any mutations in the code it covers, it is not adding real value regardless of its coverage contribution. Teams should establish a mutation score threshold for AI-generated tests alongside their coverage metrics.

5. Building a Changeset Analysis Pipeline

A practical changeset analysis pipeline integrates into your existing CI workflow. The pipeline triggers on each PR and runs before the test execution phase. It fetches the diff, analyzes the changes, queries the test-to-code mapping, ranks the relevant tests, and outputs a test list that the runner uses instead of running the full suite.

The test-to-code mapping can be built in several ways. The simplest approach is static analysis: parsing import statements and call graphs to trace which test files depend on which source files. A more dynamic approach runs your full test suite with code coverage instrumentation and records which source lines each test executes. This produces a precise mapping but needs to be regenerated periodically as tests and source code evolve.

The LLM component adds semantic understanding on top of the structural mapping. When a change modifies error handling in an API endpoint, the structural mapping identifies tests that call that endpoint. The LLM can additionally flag tests for related error display components in the UI, even if they do not directly import the modified file. This semantic layer catches cross- cutting concerns that structural analysis misses.

Start simple and iterate. Begin with file-level heuristics (if checkout files changed, run checkout tests). Add structural analysis when file-level heuristics select too many tests. Add the LLM semantic layer when structural analysis misses cross-cutting changes. Each layer improves precision but adds complexity, so only add what your scale requires.

6. Validation and Measuring Real Impact

Measuring the impact of AI-powered test selection requires tracking two metrics: execution time reduction and defect escape rate. Execution time reduction is straightforward to measure. Compare the average CI time for PRs before and after enabling changeset analysis. The target is a 40% to 50% reduction in regression execution time on PR builds.

Defect escape rate is harder to measure but more important. Track how many defects reach production that would have been caught by tests that the changeset analysis system chose not to run. This requires running the full suite on a parallel schedule (nightly or on merge) and comparing the results. If the nightly full-suite run catches defects that the PR-level targeted run missed, those are escapes that need investigation.

A healthy changeset analysis system should produce near-zero defect escapes. If the escape rate exceeds 2% to 3%, the test-to-code mapping needs refinement or the confidence thresholds for test selection need lowering. Most teams find that the initial mapping catches 95% or more of relevant tests, with the remaining gaps coming from indirect dependencies that the semantic analysis layer addresses over time.

The broader point is that AI in QA delivers the most value when it is measured rigorously and deployed incrementally. Teams that adopt AI testing tools without validation often end up with inflated metrics and hidden risk. Teams that validate each capability against real defect detection data build genuine confidence in their testing pipeline and use AI where it demonstrably helps, not where it merely looks impressive on a dashboard.