Deterministic, reproducible agent testing infrastructure: the seven knobs that turn an LLM into a test runner

No, and any tool that claims otherwise is hiding the variance somewhere. The same model, on the same prompt, with temperature zero, can still produce different tool-call sequences across runs because providers occasionally re-tokenize, batch, or quantize behind the scenes. Even the same byte-for-byte output can land at the browser at slightly different wall-clock times and resolve to different DOM states. What you can build is one level up: reproducibility. Same plan, same variables, same pinned model, same explicit pass criteria, and the run produces the same pass/fail verdict and the same set of addressable artifact URLs. Determinism asks 'will every byte match.' Reproducibility asks 'will the contract hold.' The seven knobs in this guide are the contract.

The Anthropic model default is set in /Users/matthewdi/assrt-mcp/src/core/agent.ts on line 9: `const DEFAULT_ANTHROPIC_MODEL = "claude-haiku-4-5-20251001"`. The Gemini default sits next to it. These are exact dated model snapshots, not aliases like `claude-latest`. If you do not pass a `model` argument to assrt_test, those are what runs. The MCP tool also exposes the override at server.ts:351, so you can pin to a different snapshot per scenario, and the value gets logged on every run start at agent.ts:383 inside the `agent.run.start` event. The point is: the model is a load-bearing input to the test, so the source treats it like one. It is logged with every run, defaulted to a dated snapshot, and overridable per call.

Because every artifact URL has to exist before the artifact does. Look at /Users/matthewdi/assrt-mcp/src/mcp/server.ts lines 407-432: assrt_test creates the scenario UUID first (line 408-418), then mints a fresh run UUID via `crypto.randomUUID()` on line 429, then builds the per-run artifact directory under `tmpdir()/assrt/<runId>` on line 430. By the time the agent fires its first browser action, every cloud URL the test will eventually expose, video, screenshots, log, page summary, has already been computed by `buildCloudUrls` in scenario-store.ts:243-263. CI can record those URLs in a comment on the PR while the test is still running. If the test fails halfway, the URLs still resolve to whatever artifacts did get uploaded.

Variables are not a hint to the model, they are a substitution into the plan text before the model sees it. agent.ts lines 376-381 do a simple regex replace: every `{{KEY}}` in the scenario plan becomes the value you passed in `variables`. If your plan says 'sign up with {{EMAIL}} and product {{SKU}},' and you pass `{ EMAIL: 'qa+canary@yourcompany.com', SKU: 'ENT-001' }`, the agent reads 'sign up with qa+canary@yourcompany.com and product ENT-001.' The model still picks DOM targets, but the input it is steering toward is fixed. That is what makes a parameterized run reproducible across CI matrices: every job sees the same plan with its own variables substituted in, no shared mutable state.

passCriteria. The MCP tool schema at server.ts:343 takes a free-text string of explicit success conditions. Inside the agent, agent.ts:670 wraps that string in a labelled section: '## Pass Criteria (MANDATORY)\nThe test MUST verify ALL of the following conditions. Mark the scenario as FAILED if any condition is not met.' The agent is then forced to call its `complete_scenario` tool with one assertion per criterion. If you pass `passCriteria: 'Cart total displays $42.99 exactly. Confirmation email arrives within 30s.'`, you do not get a pass on a green page that did not actually verify those two facts. This converts the test from 'did the agent feel like it worked' to 'did each named criterion get a passing assertion,' which is what reproducibility actually needs.

Three layers. First, every run gets its own UUID and its own artifact directory under `tmpdir()/assrt/<runId>` (server.ts:429-432), so screenshots and videos from run A cannot land on run B's paths. Second, the `isolated: true` option in the tool schema (server.ts:354) keeps the browser profile in memory only, so cookies and localStorage do not leak across runs. Third, if you stay on the persistent profile and two runs race, the second one runs `killOrphanChromeProcesses` and tries to scrub `SingletonLock`; if the lock cannot be removed because the first Chrome is still alive, it silently falls back to `--isolated` for that one call (browser.ts:336-340). In practice that means two parallel persistent runs degrade to one persistent and one isolated, never to a corrupt shared profile.

Because it is the only artifact that survives the LLM. Every other tool in this category emits a proprietary YAML, JSON DSL, or 'visual flow' that only their cloud can run. If their cloud goes down, deprecates the format, or hikes prices, your test infrastructure stops being reproducible because nobody can run it. Assrt emits standard Playwright .spec.ts files. You commit them. They run in your CI today, your CI in five years, on Chromium, Firefox, and WebKit, and the LLM is no longer in the loop for the deterministic execution path. The agent's role collapses to: discover scenarios, draft the spec, propose updates when selectors break. The shipping artifact is plain code.

agent.ts:383 emits a structured `agent.run.start` line with the URL, mode, model, whether passCriteria was set, and the variable count. The screenshot files are written under the per-run UUID dir at server.ts:467-486, with filenames like `00_step1_navigate.png`, so the order in which the agent acted is on disk, not just in memory. The cloud URLs from buildCloudUrls (scenario-store.ts:243-263) are deterministic given (scenarioId, runId), so two runs with the same IDs would point at the same artifacts; in practice a fresh runId is minted per call, so each contract is uniquely addressable. If you want to prove a CI job ran with the inputs you think it ran with, the inputs are in `agent.run.start` and the outputs are at predictable paths.