AI QA platform self-heal vs regression detection. Gate auto-decisions on signal type and write every one of them to an audit log.

Self-heal becomes a regression-masking failure mode when it patches a real product bug and reports a green check. Imagine the Upgrade button is removed because of a feature-flag regression and the only nearby button is Downgrade. An ungated self-heal will find 'something close enough', click it, and pass. The CI dot is green. The deploy lands. The user-visible upgrade flow is broken. The leak is the green check that did not need to be green. Saw this happen twice last quarter on real products; both times the platform was working as advertised, and that was the problem.

Because selector drift is real, frequent, and not interesting. UI teams rename labels, swap class names, restructure DOM trees on a weekly cadence. If every one of those changes fails the test suite, the team disables the suite within a sprint. The right answer is to gate self-heal on the signal type. Allow it where the cost of a false positive is low (selector drift, layout drift). Forbid it where the cost is high (business logic, removed flows, numeric values).

Selector drift inside a tight label distance band, layout drift inside a small DOM-path-change budget, timeout extension up to a small multiple of the original timeout. These are the changes a test framework cannot reasonably ask a human to review every time, because they happen on most front-end PRs and almost never indicate a regression. Setting a numeric tolerance band on each one keeps the auto-heal honest.

Business logic deviation outside a tolerance band (a price, a count, a status). Missing assertion targets that the scenario named explicitly (a div with id 'order-total'). Removed flows where the agent cannot find any element with the right role and a label even loosely related to the intent. Email subjects, since the subject is the contract with the user, not a free-text approximation. These are the signals the test exists to catch; healing them defeats the purpose.

The platform proceeds with the heal (so CI does not block on every borderline call) but writes a record to the heal queue with a 24 hour SLA on human review. A pending heal blocks release, not merge. After the SLA, the platform either auto-rejects (strict) or lets the heal stand (lenient), depending on team policy. Most teams start lenient and tighten as confidence grows.

One JSON record per decision. Run id (timestamp), scenario name (path plus case), step number, intent (the human-readable instruction the scenario expressed), signal type (selector_drift, business_logic_deviation, fuzzy_text_match, etc.), policy applied, old value, new value, distance metric, reason in plain English, review status (auto_accepted, pending, rejected). The records persist across runs in a directory committed to the repo or pushed to an artifact store the team controls.

Assrt is one option that exposes the agent's tool surface in source so policy gates can be wired explicitly. Scenarios live in your repo as Markdown, the agent reads them and executes them with an LLM-backed tool loop, and every retry, selector swap, fuzzy match, or timeout extension is recorded. The policy file lives next to the scenario; the override file lives next to the policy; the heal log writes per run. Other platforms can offer the same shape if they expose enough of the agent surface to gate. The principle outlives any single tool.

That is the strong negative signal. If you cannot enumerate the classifications the AI returns, you cannot gate. You are at the vendor's mercy on every borderline call. The right reply is to ask for the signal taxonomy in writing and the policy hooks per signal. If the vendor declines, you are buying a black box. That may still be the right trade for some teams, but the trade should be visible.

Most flakes show up as low-confidence signals: a fuzzy match at distance 0.40, a layout drift with three DOM path changes, a timeout extension to 4x the budget. Gating forces these into the heal_with_pending queue rather than silent auto-heal. The flake stops being a mystery in CI and becomes a queued review. Either the underlying issue gets fixed, or the policy band gets tuned, or the scenario gets rewritten. All three are progress; silent retry is not.

On a 200-scenario suite running on every push, expect 5 to 15 pending heals per day in the first month, dropping to 1 to 3 per day by month three as scenarios stabilize. Total review cost is 20 to 40 minutes per week for the team lead. The investment pays for itself the first time a real regression that would have been auto-healed instead lands as a fail_fast and gets caught before deploy.

Block release. Allow merge. The point is that a pending heal is information, not a verdict; the team has a window to look at it. Blocking merge creates a queue of stuck PRs and the team starts auto-accepting to clear the queue, which defeats the purpose. Blocking release lets feature work move forward while the qa review CLI processes the queue in batch.

Traditional record-and-replay self-heal patches the test file in place, reports green, and offers no per-decision record. The trust model is 'we know better than you'. Signal-type gating inverts this: the platform classifies the deviation, applies a per-signal policy, and writes the decision to a log. The trust model becomes 'we noticed something, here is what, here is the policy we applied, you can override'. The cost is one weekly ritual. The benefit is that green stays meaningful and regressions stop hiding behind heals.