AI Config vs CI: Fixing Test Command Drift as a Compilation Problem

1. What AI Config Drift Actually Looks Like

A real example from the Django audit. The AGENTS.md file, last edited fourteen months ago, tells the agent to run tests with a simple command. The CI workflow runs something different: the same runner, but with three flags that materially change the outcome.

# AGENTS.md says:
pytest tests/

# .github/workflows/ci.yml actually runs:
pytest tests/ --cov=src --cov-fail-under=85 \
  --strict-markers --maxfail=1 -p no:cacheprovider

An agent reading only AGENTS.md writes a patch that passes locally with no coverage drop, no marker usage, and a cached plugin. CI rejects it on the strict-markers flag because the agent introduced a new test with an unregistered marker. The error message is buried in pytest output, the agent does not know to read it, and the PR stalls.

2. Why Test Commands Are the First to Drift

Three reasons, in order of how often we saw them in the audit.

• Test runners accrete flags faster than any other command. A coverage threshold gets added, then strict markers, then a maxfail, then a custom reporter. Each change lands in CI YAML without touching the docs.
• Test commands often wrap preflight steps (prisma generate, go generate, protoc) that live in a reusable workflow or composite action. The config author linked to the top level job, not the job that actually runs.
• Matrix builds multiply the command. The config picks one row of the matrix, freezes its flags, and the other rows silently diverge. Node 18 uses npm test, Node 20 uses npm run test:ci, nobody notices until a Node 20 only failure hits production.

None of this is malicious or even careless. It is the natural entropy of two documents that nobody has promoted to a compiled output.

3. Treating Configs as Compiled Artifacts

The shift in framing is small but load bearing. Stop treating AGENTS.md as a document. Treat it as the output of a compiler whose input is the CI workflow. The compiler has three jobs:

• Read the workflow YAML, resolve includes and reusable actions, and extract the canonical invocation for each command class (test, lint, build, typecheck).
• Emit a flat markdown file with those commands verbatim, in the exact form CI runs them.
• Fail the build if the committed markdown does not match the generated output, same way a stale generated client fails.

The agent now gets one thing to read, that one thing is guaranteed fresh, and any human who tries to paper over a CI change by editing the markdown directly is caught by the next CI run.

4. A Thirty-Line Divergence Detector

You do not need a build system overhaul to start. A small Node or Python script walks the workflow files, collects every line that looks like a test invocation, normalizes whitespace, and diffs it against the invocations grep finds in AGENTS.md and README.md. Anything that does not match prints a diff and exits non zero.

# Pseudocode, pipeline step
ci_cmds = extract_run_steps(".github/workflows/*.yml")
doc_cmds = extract_code_blocks("AGENTS.md", "README.md")

drift = []
for c in ci_cmds:
  if normalize(c) not in map(normalize, doc_cmds):
    drift.append(c)

if drift:
  print("Config drift detected:")
  for d in drift:
    print("  ", d)
  sys.exit(1)

Run it in a pre-commit hook and as a CI job. It does not catch every failure mode (environment variables, secret scopes, matrix only flags) but it catches the common case in under a second, and it catches it at the moment the drift is introduced rather than the moment an agent run fails.

5. The E2E Test Harness Wrapper

The detector is a static check. The ground truth is whether the patch the agent wrote actually ships to staging and renders the page the user sees. That is what the E2E harness covers.

A minimal harness runs three things in order, in a container that mirrors the CI image:

1. the canonical test command, verbatim, from the compiled config
2. a merge preview deploy (Vercel, Fly, Render)
3. an E2E suite that hits the preview URL with a real browser

Step three is where tools diverge. You can hand write Playwright specs, which works well if the surface is stable. You can use Cypress if your team already runs it. Or you can let an AI browser agent like Assrt generate specs from natural language scenarios, which removes the selector maintenance cost every time the UI shifts. The point is not which tool you pick; it is that step three exists, and that it runs on every agent authored PR.

6. How to Roll This Out Without a Big Bang

Three stages, two weeks each, no flag days.

• Week one and two. Write the divergence detector. Run it on a schedule, not a blocker. Collect drift reports. Do nothing else.
• Week three and four. Promote the detector to a PR blocker. Fix the drift it reports by editing AGENTS.md, not CI. This is the stage where you learn which divergences were intentional.
• Week five and six. Invert the flow. AGENTS.md becomes compiled from CI. The detector becomes a compiler invariant. Add one E2E smoke test against the merge preview and wire it to the same blocker.

At the end, the agent reads one file, that file is guaranteed current, and every PR the agent opens is verified against a real browser on a real preview deploy. Forty-six percent drift becomes zero, and stays there.

Frequently Asked Questions