The part most guides on this topic quietly skip
Anthropic Claude Code testing automation, powered by the exact OAuth token Claude Code already wrote to your Keychain
Every other piece on this topic explains the same thing. Claude Code runs your unit tests. Claude Code reads the failures. Claude Code edits the code. Rinse, repeat. The harder, uncovered path is the other direction: using the same OAuth session Claude Code writes to ~/Library/Keychains to authenticate a local browser-driving agent. That is what this runner does. Service name: Claude Code-credentials. JSON path: claudeAiOauth.accessToken. Header: anthropic-beta: oauth-2025-04-20. Default model: claude-haiku-4-5-20251001. No separate API key, no vendor dashboard, no second billing relationship.
Anthropic Claude Code testing automation, viewed from the credential side
Almost every piece on this topic answers the wrong question. They describe how Claude Code, the IDE, picks a test command, runs it, and iterates. That is fine; it is also one of the most-covered stories on the internet. The question nobody answers is: how do you get the same Claude model driving a real browser through a real Playwright session, reusing the same auth you already have, without creating a second API key or pasting a token into a .env file?
The answer lives in 66 lines of code. When you sign in to the Claude Code desktop app on a Mac, it writes an OAuth credential blob into your login Keychain under the service name Claude Code-credentials. The shape of that blob is stable and documented in the runner source: a top-level claudeAiOauth object with accessToken, refreshToken, expiresAt, and scopes. The runner shells out to the macOS security command, reads that blob, pulls the access token, and hands it to the Anthropic SDK with one extra header. That is the whole auth story. The model is now on the hook for driving the browser; your Claude Code subscription pays for the tokens.
Keychain → Anthropic → Chromium
The full credential lookup, in the one file that matters
If you only read one file to understand how this is wired, it is assrt-mcp/src/core/keychain.ts. Sixty-six lines. Three branches. No sugar, no framework, no abstractions. It is a function called getCredential that returns either { token, type: "oauth" } or { token, type: "apiKey" }. Nothing else. Everything downstream branches on that type and does the right thing.
Read it top to bottom. The order matters. ANTHROPIC_API_KEY wins if it is set, which is how CI survives without a Keychain. The platform gate means Linux and Windows without the env var fail fast with a clear message instead of silently hanging on a security command that does not exist. And the actual Keychain read is a single execSync call with a five-second timeout. The result is a discriminated union the rest of the runner can branch on.
How the credential reaches Anthropic
Once the token is in hand, the question is how the Anthropic Node SDK knows to send it as a Bearer token with the OAuth beta header rather than as an X-Api-Key. The answer is the constructor argument name: authToken triggers the Bearer path, and apiKey triggers the X-Api-Key path. You need to add the OAuth beta header yourself via defaultHeaders or the API will reject the token. Here is the exact branch that does both.
The two Anthropic branches differ by three lines. Everything else about the agent, including the tool schema, the loop, the snapshot truncation, and the assertion format, is identical between OAuth-authed runs and API-key-authed runs. The model does not know which credential shape authenticated it; the API treats the model call the same way.
Six things that hold the credential path together
None of these are decorative. Each one is a specific choice that makes the zero-setup path possible.
One Keychain entry, exact service name
KEYCHAIN_SERVICE = "Claude Code-credentials" at keychain.ts line 10. No Keychain search, no wildcard, no token file on disk. The runner hits the one entry the Claude Code app owns and nothing else.
Reuses Claude Code's subscription
If you are paid for Claude Code (Pro, Max, or Team), that same OAuth session drives the browser. No second API key, no second billing relationship, no need to create an organization-scoped key in the Console.
oauth-2025-04-20 beta header
The token is a Bearer OAuth token, not an X-Api-Key. The Anthropic SDK requires anthropic-beta: oauth-2025-04-20 on every request. Set at agent.ts line 361 and mirrored at server.ts lines 782 and 881.
Falls back to ANTHROPIC_API_KEY
The env var is checked first (keychain.ts line 33). If present, the runner uses the plain API-key path with X-Api-Key. That is the portable path for Linux CI where macOS Keychain is not available.
Haiku 4.5 by default
DEFAULT_ANTHROPIC_MODEL = "claude-haiku-4-5-20251001" at agent.ts line 9. Fast, tool-happy, cheap per scenario. Override with --model or ANTHROPIC_MODEL when a flow needs Sonnet-sized reasoning.
Nothing touches the filesystem
The token is read, held in the process, and attached to outbound requests to api.anthropic.com. It is never written to ~/.assrt, never logged, never uploaded, never passed to the Playwright MCP child process.
Verify it yourself in under two minutes
None of this is abstract. Every claim above is something you can inspect with tools already on your Mac. The first command below proves the Keychain entry exists and shows you the top-level JSON keys. The second proves the runner resolves the credential type to "oauth". The third runs a trivial end-to-end scenario against any URL with no ANTHROPIC_API_KEY in the environment.
The full flow, tracked across five actors
Follow one interaction end to end. The Claude Code desktop app wrote the token into Keychain last time you signed in. The Assrt runner reads it fresh on every invocation. Anthropic gets a Bearer token with the OAuth beta header. Playwright MCP never sees the token at all; it only talks to the local Chromium process.
From Keychain entry to a clicked button
What it actually does once authenticated
The credential path is the interesting half. The browser path is the familiar one. Six well-defined steps run for every scenario, from the first Keychain read to the last artifact landing on disk.
You open Claude Code once and sign in
The Claude Code app writes your OAuth access token into the macOS Keychain under the service name "Claude Code-credentials" as a JSON blob: { claudeAiOauth: { accessToken, refreshToken, expiresAt, scopes } }. That is the entire per-user setup. No Assrt config file, no API key paste.
You run npx assrt run (or call the MCP tool)
cli.ts line 470 calls getCredential(). keychain.ts runs through its ordered fallback: ANTHROPIC_API_KEY env var first, then macOS security find-generic-password, then a clear error. On a normal dev Mac, the second hop wins and returns { token, type: "oauth" } back to the CLI.
The Anthropic client is built with the right shape
agent.ts line 358 branches on the credential type. For "oauth" it calls new Anthropic({ authToken: token, defaultHeaders: { "anthropic-beta": "oauth-2025-04-20" } }). For "apiKey" it calls new Anthropic({ apiKey: token }). The rest of the agent does not care which branch ran; it just uses the client.
@playwright/mcp@0.0.70 is spawned on stdio
McpBrowserManager in browser.ts resolves the Playwright MCP package from node_modules and launches it as a stdio child process with --viewport-size 1600x900, --output-mode file, --caps devtools. No network, no cloud, no extra auth. The browser runs on your machine; the model calls go to Anthropic.
The agent loop reads your scenario and makes tool calls
The Claude model reads /tmp/assrt/scenario.md, picks a tool (navigate, snapshot, click, type_text, evaluate, assert), and emits tool_use. McpBrowserManager catches each tool and forwards it to Playwright MCP as browser_click / browser_type / browser_snapshot. The accessibility tree from each snapshot is resolved from a .yml file on disk and truncated to 120k chars before being passed back to the model.
Results land locally and the video player opens
complete_scenario writes /tmp/assrt/results/latest.json with the full tool trace, screenshots, and a webm recording. The HTML player at generateVideoPlayerHtml in server.ts opens in your browser at 5x playback by default. You did not create an account on any vendor dashboard; the artifacts are on your disk.
What you need in place before the first run
Five required preconditions. Two optional levers. Checked items are the normal Mac dev-machine state; unchecked items are for the API-key path or for a stronger model.
Preflight for the OAuth-driven path
- Claude Code is installed on this machine and signed in (this is the only setup; no API key needed)
- macOS Keychain has an entry under the service name "Claude Code-credentials" (you can verify with the security CLI)
- Node 20 or newer is on PATH (npx assrt pulls the runner with no global install)
- The test target (your app, your staging env, or any URL) is reachable from this host
- A /tmp/assrt/scenario.md file exists, or you are passing --plan inline; either works
- Optional: ANTHROPIC_API_KEY is set if you want to force the API-key path instead of the OAuth path
- Optional: ANTHROPIC_MODEL is set to a stronger Claude variant if Haiku 4.5 is underpowered for your flow
Counted against a traditional vendor setup
Cloud QA vendors built their entire auth story on the assumption that you do not already have a credential. Claude Code broke that assumption. The table below lines up the differences at the credential layer where it now matters.
| Feature | Vendor cloud QA (API-key path) | Assrt (Claude Code OAuth path) |
|---|---|---|
| Credential source | Dashboard-provisioned API key, copied to .env.local | Claude Code OAuth token, read live from macOS Keychain service "Claude Code-credentials" |
| Separate billing relationship | Yes. Vendor account, seat count, monthly invoice | No. Model calls bill against the Claude Code subscription already on your workstation |
| Default model | Vendor-chosen, usually not disclosed | claude-haiku-4-5-20251001 (agent.ts line 9), overridable per run |
| Auth header on Anthropic calls | X-Api-Key: sk-ant-... (or proxy) | Authorization: Bearer <oauth-token> + anthropic-beta: oauth-2025-04-20 |
| Where the token lives on disk | Plaintext in .env or vault-synced secret | Keychain-managed, owned by Claude Code, never touched by Assrt's own code |
| Who drives the browser | Cloud runner in vendor infra, IP-unknown | Local @playwright/mcp@0.0.70 process, your Chromium, your network |
| Test artifacts | Vendor cloud dashboard, proprietary format | webm + latest.json under /tmp/assrt/results on your filesystem |
| Vendor lock-in | Proprietary YAML DSL, pricing starts around 7,500 USD per month at the top end | MIT-licensed, scenarios are plain text, zero per-seat cost on top of Claude Code |
This is not a criticism of API keys. API keys are the right choice on Linux CI, in shared-runner environments, and when the agent must run as a service account rather than as a specific human. The point is that on a developer's Mac, where Claude Code is already signed in, asking for a second credential is redundant; reusing the Keychain entry is the shortest honest path.
A few measurements worth knowing
Each of these comes from the source, not from marketing. Lines of code, pinned versions, timeouts, and defaults are all verifiable in the repository at the paths noted in the FAQ.
The uncopyable anchor
Open /Users/matthewdi/assrt-mcp/src/core/keychain.ts. Line 10 is const KEYCHAIN_SERVICE = "Claude Code-credentials". That string is the entire handshake with the Claude Code desktop app; if Anthropic ever renamed the service, this runner would need one constant edit and nothing else. Line 47 is the security find-generic-password call that actually reads it. Line 55 is where the discriminated union is built and returned. Then in /Users/matthewdi/assrt-mcp/src/core/agent.ts line 361, defaultHeaders: { "anthropic-beta": "oauth-2025-04-20" } is the one-line difference between an OAuth-authed Anthropic client and a vanilla one. No other page on this topic names those four strings, because no other page ships the runner.
What you no longer have to maintain
Each chip below is a line item from a traditional vendor QA setup. Once the credential is the Claude Code OAuth token, most of this infrastructure quietly disappears.
When an API key is still the right answer
Two cases. First, CI. GitHub Actions, CircleCI, Buildkite, anything that does not run on a human's macOS machine will not have security find-generic-password available. Set ANTHROPIC_API_KEY in the CI secret store and the same runner works, same scenarios, same artifacts. Second, multi-tenant agents or service accounts where you want a token scoped to the automation and detachable from any one human's Claude Code login. Create a regular Anthropic API key in the Console, keep it in a vault, pass it through the env. The runner does not care which path it took; the Claude model behaves the same on the other side.
The OAuth path is a specific optimisation for the specific case where a human developer, signed in to Claude Code, wants to drive a browser from that same session without doubling their credential bookkeeping. For everything else, the portable API-key path is one flag away.
Want this run against your own checkout, signup, or onboarding flow?
Twenty minutes on a screenshare. A #Case written against your real app. A webm recording and a latest.json you can carry away. The only credential you need is the one already in your Keychain.
Questions this topic usually raises
What Anthropic credential does the runner use to drive the browser?
It reuses the exact OAuth access token Claude Code already wrote to your macOS Keychain. The service name is literally "Claude Code-credentials" (defined at assrt-mcp/src/core/keychain.ts line 10). Assrt shells out to the system keychain with security find-generic-password -s "Claude Code-credentials" -w, parses the returned JSON (shape: { claudeAiOauth: { accessToken, refreshToken, expiresAt, scopes } }), and pulls claudeAiOauth.accessToken. That token then goes to new Anthropic({ authToken }) in the official SDK, with defaultHeaders: { "anthropic-beta": "oauth-2025-04-20" }, which is the beta header the Anthropic API expects for Claude Code OAuth tokens. You do not set ANTHROPIC_API_KEY anywhere.
What model runs the browser by default, and can you override it?
Default is claude-haiku-4-5-20251001, pinned at agent.ts line 9 as DEFAULT_ANTHROPIC_MODEL. You can override with --model on the CLI, the model argument on the assrt_test MCP tool, or ANTHROPIC_MODEL in the environment. The resolution order is: explicit argument wins, then env var, then the constant. Haiku is chosen because browser automation is tool-heavy and latency-sensitive, and Haiku 4.5 is fast enough that a typical end-to-end scenario of ten to fifteen tool hops finishes in well under a minute. If your scenario is complex (long multi-step onboarding, OTP verification, email magic-link, Stripe redirect) you can bump to a larger Claude model by passing model: "claude-sonnet-4-5" in the MCP call.
Does the runner fall back to an API key, or is the Keychain path the only path?
Keychain is preferred but not required. keychain.ts lines 33 to 38 check process.env.ANTHROPIC_API_KEY first. If it is set, the runner uses that and marks the credential type as apiKey (passed to the SDK as a normal X-Api-Key). Only if the env var is absent does it try security find-generic-password on macOS. The fallback matters for Linux CI where a keychain does not exist: set ANTHROPIC_API_KEY in the CI secret store and the same runner binary works, same scenarios, same reports. The OAuth path is the zero-setup path; the API-key path is the portable path.
If the OAuth token expires mid-run, what happens?
The run fails on the next tool call with a 401 from Anthropic. The refresh token is present in the Keychain JSON (the claudeAiOauth object carries refreshToken and expiresAt), but Assrt does not refresh the token itself today. It relies on Claude Code to have a live token, which is almost always the case because the Claude Code app refreshes on its own schedule. If a run starts and the token happens to be stale, rerun after opening Claude Code once; the app will refresh the Keychain entry on start. Long term, the right fix is to detect expiresAt in-process and call the Anthropic refresh endpoint before the next model call. Tracked as a follow-up; not in the current code path.
What does the agent actually do once it is authenticated?
It drives a real Chromium via @playwright/mcp@0.0.70, spawned over stdio from the same Node process. The agent knows eleven high-level tools (navigate, snapshot, click, type_text, select_option, scroll, press_key, wait, screenshot, evaluate, assert, plus helpers like create_temp_email and wait_for_verification_code for signup flows). Each maps to one Playwright MCP call (browser_click, browser_type, browser_snapshot, browser_evaluate). The Claude model decides which tool to call next based on the plain-English #Case blocks you wrote in /tmp/assrt/scenario.md. Every snapshot is written to a .yml file in ~/.assrt/playwright-output so the accessibility tree never pollutes the model context. Full mapping lives at assrt-mcp/src/core/browser.ts lines 560 to 670.
Why pass a beta header instead of a plain Bearer token to the Anthropic API?
Because Claude Code OAuth tokens are a different credential type than regular Anthropic API keys. Regular keys go in X-Api-Key; OAuth tokens go in Authorization: Bearer and require the client to acknowledge it is using the OAuth flow via the anthropic-beta: oauth-2025-04-20 header. The Anthropic Node SDK handles both: constructing the client with new Anthropic({ apiKey }) produces an X-Api-Key client, constructing it with new Anthropic({ authToken }) plus the beta header produces a Bearer client. Assrt picks the shape based on the credential type returned from keychain.ts; see agent.ts lines 358 to 366 for the branch.
Is my Keychain token ever sent anywhere other than Anthropic?
No. The token is read from security find-generic-password, held in the runner process memory, and included on outbound HTTPS requests to api.anthropic.com via the official SDK. It is never written to disk by Assrt, never logged (the telemetry module at telemetry.ts masks credentials), never sent to Assrt's own cloud, and never shared with the Playwright MCP process (which does not need it; it only drives the browser). The only external destination is Anthropic, and only for model calls the agent initiates to complete your test plan.
How does this compare to Anthropic's own testing story inside Claude Code?
Claude Code's built-in story is write code, run tests. It reads the codebase, proposes edits, runs the project's test command (pytest, npm test, go test), reads failures, iterates. That loop is fast and great for unit and integration tests that already exist. What it does not do is stand up a real browser, click through a checkout, and verify the confirmation page looks right. Assrt fills that gap using the same OAuth session. End-to-end browser testing becomes an extension of the Claude Code experience rather than a separate tool with its own billing and auth.
What happens on Linux or Windows where there is no macOS Keychain?
keychain.ts line 40 checks process.platform !== "darwin" and returns a clear error if ANTHROPIC_API_KEY is not set: "No credentials found. Set ANTHROPIC_API_KEY, or run on macOS with Claude Code installed." On Linux CI you set the env var and everything else works exactly the same; the agent.ts constructor takes an authType of "apiKey" or "oauth" and builds the right Anthropic client. Windows users without Claude Code run it the same way: ANTHROPIC_API_KEY in the shell, no Keychain lookup, no beta header. The OAuth optimisation is macOS-specific because that is where Claude Code stores its credentials.
Can two Claude Code accounts drive two concurrent runs on the same machine?
Not from the same Keychain. macOS stores one entry per service name, so security find-generic-password -s "Claude Code-credentials" returns whichever account is currently logged in to Claude Code. To run a second agent with a different identity, set ANTHROPIC_API_KEY to a regular API key in that second shell; keychain.ts prefers the env var over the Keychain read (line 33 runs first). You can have one OAuth-driven run and one API-key-driven run side by side on the same host without collision.
Which exact files should I open to audit the credential path myself?
Three files, about twenty minutes of reading. First, assrt-mcp/src/core/keychain.ts — all 66 lines; this is the whole credential lookup including the service name, the JSON shape, the platform gate, and the returned discriminated union. Second, assrt-mcp/src/core/agent.ts lines 342 to 366 — the TestAgent constructor that takes authType and builds either an apiKey Anthropic client or an oauth Anthropic client with the beta header. Third, assrt-mcp/src/mcp/server.ts lines 777 to 882 — the MCP server entry points that call getCredential() for assrt_test, assrt_plan, and assrt_diagnose. Together, those three files are the entire surface area between macOS Keychain and the browser-driving loop.