A green E2E run in CI/CD only ships when each assertion writes down why it passed

What "confidence" actually means in a CI/CD context

When a team says they want confidence from their E2E suite in CI, they mean something specific. They mean: when the build is green, we ship without a human re-reading the diff. When the build is red, we trust the signal enough to roll back or block the deploy. The whole point of the runner sitting in the pipeline is to remove a human from the loop.

That contract has two halves. The red half is well-served by every modern runner: a failed assertion throws, the stack trace points at the line, the trace viewer replays the DOM. You know what broke. The green half is where most suites silently fall short. A green Playwright run carries the information "none of the assertions threw". That is a weaker statement than "the app worked". The gap between those two statements is where shipping anxiety lives.

The closing move is not more tests. It is a different shape for the existing tests, so the green path carries the same kind of evidence the red path already does.

The structural change: three fields per assertion, not two

Vanilla Playwright assertions are two-field: the thing you are checking, and the boolean outcome. The boolean lives implicitly in "did the line throw". There is no recorded reason for the passes. There cannot be: the API does not have a place to put one.

Assrt's assertion shape adds a third field. The tool declaration at src/core/agent.ts:134-142 requires three properties on every assert call:

{
  name: "assert",
  description: "Make a test assertion about the current page state.",
  input_schema: {
    type: "object",
    properties: {
      description: { type: "string", description: "What you are asserting" },
      passed:      { type: "boolean", description: "Whether the assertion passed" },
      evidence:    { type: "string", description: "Evidence for the result" },
    },
    required: ["description", "passed", "evidence"],
  },
}

The required array is the load-bearing part. The agent cannot call assert with the evidence field empty; the tool call is rejected at the model layer before any test state is mutated. That is why every entry in the final report carries a written reason.

The TestAssertion type that lands in the JSON report is at src/core/types.ts:13-17. The same three fields, persisted:

export interface TestAssertion {
  description: string;
  passed: boolean;
  evidence: string;
}

How the runner talks to CI: one exit code, one JSON artifact

The other half of a usable CI step is the wiring. A runner that carries rich assertion data internally but only prints to stdout is still opaque to the pipeline. Two things have to happen at the boundary: the process must exit non-zero on any failure (so the CI step actually fails the build), and the report must land somewhere the pipeline can grab it (so the failure summary is one curl away).

The exit-code wiring in Assrt is one line, at src/cli.ts:601:

process.exit(report.failedCount > 0 ? 1 : 0);

The JSON artifact is the --json flag, parsed at src/cli.ts:590-598. Pass the flag, redirect stdout to a file, and you have a build artifact you can upload from any pipeline (GitHub Actions upload-artifact, GitLab artifacts:paths, Buildkite artifact upload). The report contains every scenario, every step, every assertion, every evidence string, the total duration, and the timestamp. Reading the artifact tells you enough to decide whether to ship; you only open the video when you cannot decide from text.

What a green run looks like, end to end

Concretely: this is what a CI step running the Assrt CLI prints on a simple checkout scenario. Two assertions, both passing. Notice that each PASSED line is followed by an evidence line describing what the agent actually observed. The shell exit code at the end is 0, which is what the pipeline reads to decide green-or-red.

The thing to look at is not the PASSED lines. It is the evidence lines below them. "Subtotal text was \"$29.00\" matching the unit price of the single SKU in the cart" tells you the assertion was actually exercising the price calculation, not silently passing on an unrelated DOM. Six months later, reading the same artifact, you will know what that scenario covered without rerunning it.

Where this matters in practice: a regression slips in and the test still passes for the wrong reason. With evidence-per-pass, you can grep the historical reports for the assertion description and watch the evidence string drift over time. A pass whose evidence used to say "total was $29.00" and now says "heading text included '29'" is a pass that used to mean something and no longer does. That is a signal vanilla green-or-red does not carry.

The honest pushback

The fair objection: you can already write the equivalent in vanilla Playwright. Wrap every expect in a try/catch, attach a custom annotation on success, write the annotation to a side file. Mechanically true. In practice no one does it, because the framework does not require it and the discipline collapses the third time someone writes a test under deadline pressure.

The structural argument is that the tool schema is the right place to enforce the discipline. If the runner's tool schema makes the evidence field required (Assrt's does, at agent.ts:142), every test author has to write the string or the call does not go through. The discipline is enforced once, in the tool definition, not 200 times across a growing test suite. That is what separates a structure that survives from a convention that erodes.

The other fair objection: evidence strings can lie. The agent could write "page loaded correctly" on a page that loaded incorrectly. True, and the mitigation is the same as for any test: the evidence string is reviewed alongside the description in code review, and historical evidence drift is watched on the trend graph. Lying evidence is at least visibly present in the report; it is a thing you can look at and challenge. Silent passes are not.

What to do on Monday

Three concrete moves, in order of cost.

1. 1.Pick one critical user flow that currently has a green E2E test in CI. Read the test source. Ask: can I tell, from the code, what each pass actually proves? If the answer is "no, I would have to rerun it in headed mode", that test is a candidate for the structural rewrite.
2. 2.Run Assrt against the same flow: npx @m13v/assrt run --url <your-app> --plan "Walk the checkout flow with a valid card" --json > report.json. Read the resulting JSON. The evidence strings on the passes are the data your current suite does not produce.
3. 3.Add the run to your existing pipeline as a second step, behind a feature flag if the team is cautious. The exit code at cli.ts:601 is already standard. The JSON artifact goes into the same uploader your existing tests use. You do not need to rewrite the suite to start seeing the difference; you need one flow producing evidence-per-pass and a habit of reading it.

Common questions about CI confidence and assertion structure

A short closing

E2E test confidence in CI/CD is a question about the shape of the test artifact, not about the volume of tests. A small suite where every assertion carries written evidence will out-ship a large suite of silent passes. The runner's tool schema is the right place to enforce that, because it is the only place the discipline cannot erode under deadline pressure. The rest is paperwork.