Multi selector candidate ranking in E2E tests: the four-tier scoring function inside an AI test agent

The problem this function solves

Playwright's recommended locator priority is well documented and well repeated. getByRole first, then getByLabel, then getByPlaceholder, then getByText, then getByTestId, and CSS only when nothing else works. That priority assumes you, the test author, are writing the locator. It says nothing useful about what an agent should do when the input is a free-text description like "the primary call-to-action button".

That description is not a valid getByRole call. It is not a CSS selector. It is what a human types into a Reddit comment when they describe a test step. If an AI E2E agent wants to act on that string, the agent has to turn it into one DOM element before Playwright is even involved. Multi selector candidate ranking is the function that does that turn.

The interesting thing about the algorithm is that it's small, deterministic, and runs inside the browser via a single browser_evaluate call. No model is consulted. The LLM only gets involved later, when this function fails to find anything and the agent has to look at the accessibility tree.

The candidate set is exactly nine selector types

The first decision the algorithm makes is what counts as a candidate. Too narrow, and the agent misses real click targets. Too wide, and the scoring function gets noisy because most elements on a page are not interactive. The set in Assrt's resolver is:

a, button, input, [role="button"], select, textarea, label, [onclick], [href]

That's the union of (1) the four native interactive elements every browser ships, (2) anything the page explicitly opted into being a button via ARIA, (3) label because clicking a label fires the associated input, and (4) anything with a click handler or an href so divs-as-buttons are caught.

What's deliberately not in the set: [tabindex], [role="link"], [role="menuitem"], and div generically. Adding any of them broadens recall but the scoring function then has to fight the noise. The trade-off was made in favor of precision; if your app uses one of those patterns heavily, the candidate set is one line in browser.ts and you can fork it.

The whole algorithm fits on one screen

Below is the actual code that runs in the page, copied from src/core/browser.ts in the Assrt repo. It's injected into the browser context via the browser_evaluatetool of Playwright MCP. There is no library involved; it's pure DOM.

// browser.ts, lines 294-344, function showClickAt
// Injected into the page; runs inside the browser, not in Node.
const result = await this.callTool("browser_evaluate", {
  "function": `() => {
    const sel = ${sel};                    // the human-readable description
    const selLower = sel.toLowerCase();
    let el = null;

    // Tier 0: try as a literal CSS selector first.
    try { el = document.querySelector(sel); } catch {}

    if (!el) {
      // Tier 1: scan a fixed candidate set.
      const candidates = document.querySelectorAll(
        'a, button, input, [role="button"], select, textarea, label, [onclick], [href]'
      );

      const words = selLower.split(/\\s+/).filter(w => w.length > 2);
      let bestScore = 0;

      for (const e of candidates) {
        const txt = (e.textContent || '').trim().toLowerCase();
        if (!txt) continue;

        // Tier 2: exact match wins immediately.
        if (txt === selLower) { el = e; break; }

        let score = 0;
        if (txt.includes(selLower)) score = 3;                           // tier 3a
        else if (selLower.includes(txt) && txt.length > 2) score = 2;    // tier 3b
        else {
          const matched = words.filter(w => txt.includes(w)).length;
          if (matched > 0) score = matched / words.length;               // tier 4
        }

        if (score > bestScore) { bestScore = score; el = e; }
      }
    }

    if (el) {
      const r = el.getBoundingClientRect();
      return JSON.stringify({ x: r.left + r.width / 2, y: r.top + r.height / 2 });
    }
    return null;
  }`,
});

A few details worth pointing at. The try/catch around querySelector is necessary because most human descriptions raise SyntaxError when passed as CSS selectors (the string "Sign in button" has unescaped spaces and isn't valid). The txt.length > 2 floor on the candidate-in-target tier is what prevents a one-character label from winning by accident when the input string happens to contain that character. And the partial-word tier ignores words shorter than three characters, which keeps articles and prepositions from scoring as positive matches.

Walking through four realistic scenarios

The scoring tiers are easier to reason about when you watch them resolve a real page. Pretend the page has three candidates: a header <a>Sign in</a>, a hero <button>Sign in to continue</button>, and a footer <a>Sign up</a>.

Where this diverges from Playwright's own first-match rule

Playwright has a strict-by-default policy: when a locator matches more than one element, the locator API throws unless you opt in with .first(), .last(), or .nth(). The official guidance is that you should never opt in; you should make the locator unique instead. That guidance is correct for hand-written test code where the author can see the page and disambiguate.

It doesn't apply to an AI agent that's receiving a description from a model and has to produce somedeterministic answer. The agent doesn't have the luxury of throwing and asking the user to rewrite the locator. It either picks one and acts, or it gives up. The Assrt resolver picks one, using the scoring tiers above, and lets DOM order break ties.

The trade-off is conscious: the resolver is biased toward acting deterministically on ambiguous input. The check on whether it acted correctly is the next snapshot the agent takes after the click. If the click landed on the wrong element, the next snapshot won't contain the expected state, and the agent recovers. The cost of being wrong is one extra LLM round-trip; the cost of refusing to act would be a stuck agent.

What this means for someone choosing an AI E2E tool

Most hosted AI E2E products keep the resolver as a closed black box. You give them a description, you get back a pass or a fail, and you have no way to inspect why a particular element was chosen on a particular run. When tests start clicking the wrong button after a UI rebrand, debugging means filing a ticket with the vendor and waiting.

The honest version of multi selector candidate ranking is one where the algorithm is small enough to read in one sitting, lives in a file you can grep, and uses no proprietary scoring model. The Assrt version is fifty lines of plain DOM JavaScript. It runs in your browser, ships its decision in a snapshot diff you can inspect, and you can change the candidate set or the scoring weights by editing one file. That's the trade-off the project makes: less magic, more legibility.

None of this is a knock on hosted products that get the resolution layer right; it's a statement about what kind of tool you want owning the most important runtime decision in your test suite. If your team values the ability to read, modify, and own the resolver, an open-source pick is the natural fit. If you'd rather pay a vendor to take responsibility, that's a different trade-off, and the difference shows up clearly the first time the resolver picks the wrong button and you need to know why.