Visual regression testing screenshots, the three pathways teams confuse for one.

It is a captured image of UI state, taken at a known moment, used to detect a visual change you did not intend. The capture itself is the simple part. The hard parts are deciding the format (lossless PNG so a single pixel of drift is detectable, or lossy JPEG because a vision model is doing the judging), the timing (after fonts load, after animations settle, after the network goes quiet), the viewport size (so two runs on different machines produce comparable frames), and what you do with the bytes after (commit them as a baseline, send them to a model, drop them into a video, throw them away). The format and lifecycle decisions matter more than the capture step. Most flake comes from getting one of those wrong, not from the screenshot itself.

A PNG baseline lives on disk forever. It is committed to the repo, it gets diffed against a fresh PNG on every CI run, and the diff engine flags any pixel-level deviation that exceeds a numeric threshold. The format is lossless on purpose, because a 1px border or a 1-shade color shift is exactly what the test exists to catch. A JPEG quality 50 vision-model screenshot lives in memory for one model turn. It is base64-encoded into a single image part of an Anthropic or Google API request, the model reads it against a plain-English checklist, and the bytes are discarded. The format is lossy on purpose, because the model is judging layout meaning and the compression artifacts at q50 are below the model's perceptual threshold while halving the per-request payload. Same word, different artifact, different lifecycle, different cost line.

It is the human-replay artifact. Pixel diffs and vision-model checks both fail the test or pass it, but neither lets a human re-watch what happened. A WebM recording captures every compositor frame at a known viewport (Assrt records at 1600x900, declared at /Users/matthewdi/assrt-mcp/src/core/browser.ts:628), and the trick is that the recording itself can be enriched. Assrt injects a DOM overlay before recording starts: a 20px red cursor at rgba(239,68,68,0.85), a click ripple that scales 0.5 to 2.0 over 50ms, a green monospace keystroke toast that fades after 2500ms, and a 6px green heartbeat dot at the bottom-right pulsing every 800ms. The overlay only appears in the WebM, not in the JPEGs the model judges, because the screenshot tool reads the page snapshot and the recorder reads the screencast. Different surfaces, different content, same source page.

It is not decorative, it is a compositor primitive. WebM recordings driven by Chrome's screencast API only emit a frame when the compositor decides something changed. If your scenario sits in a 6-second wait_for_stable while a streaming response settles, those 6 seconds can collapse into a single repeated frame, which makes the recording look like the browser hung. A 6px dot at the bottom-right that pulses every 800ms forces a guaranteed change in pixel state on a regular cadence, so the screencast keeps emitting frames during idle periods. The recording stays continuous, so the human reviewer can scrub through it and see real wall-clock time. The line that does this is at /Users/matthewdi/assrt-mcp/src/core/browser.ts:44-48 (the heartbeat.animate call with iterations: Infinity).

Because the model is paying for it twice and would not see the difference. Every screenshot Assrt sends to the LLM is billed by image pixel count and by base64 payload size. JPEG quality 50 cuts the payload to roughly 30 to 50% of a lossless PNG of the same viewport, which compounds across a 30-step scenario. The compression artifacts at q50 are visible to a pixel-diff engine but not to a vision model judging layout, color, and text legibility. The exact line that makes the call is at /Users/matthewdi/assrt-mcp/src/core/browser.ts:601: callTool('browser_take_screenshot', { type: 'jpeg', quality: 50 }). Drop the quality lower and the model starts mis-reading small text. Bump it higher and the bill goes up without affecting any judgment. q50 is empirical, not arbitrary.

After every visual action, but only after the DOM has stopped mutating. Assrt takes a fresh screenshot only when the prior tool was a navigate, click, type_text, scroll, press_key, or select_option. Anything else (snapshot, wait, assert, evaluate, http_request) does not trigger a capture, because the screen has not changed. The exclusion list is hard-coded at /Users/matthewdi/assrt-mcp/src/core/agent.ts:1024. Before the screenshot is taken, the agent calls wait_for_stable when needed: a MutationObserver attached to document.body polls every 500ms and the agent only proceeds once 2 seconds pass with zero childList, subtree, or characterData mutations (default, configurable, ceiling 30s). That is the difference between 'we caught the regression' and 'we caught the spinner mid-rotation'.

Yes, and it is what most mature suites end up doing. The two are complementary, not competing. PNG baselines belong on isolated component pages (Storybook routes, design-system primitives, marketing screenshots, icon sets) where a 1px shift is a real regression and the surface area is small enough that the baseline-approval workflow does not consume the team. Vision-model screenshots belong on full user journeys (checkout, signup, dashboard, search) where the question is functional rather than pixel-level and the surface area is too large for baseline review to scale. Run both, scope each to what it is good at, and stop pretending one tool covers the whole problem.

Each Assrt run writes its screenshots to /tmp/assrt/<runId>/screenshots/, with filenames following the pattern {index:02d}_step{N}_{action}.png (the directory is created at /Users/matthewdi/assrt-mcp/src/mcp/server.ts:431, the filename pattern is at server.ts:468). One small wrinkle worth knowing: the bytes inside those files are JPEG quality 50, even though the extension is .png. The pattern preserves a stable filename ordering for the replay player, but if you open one in an image viewer, your viewer is reading JPEG headers under a .png extension. Rename to .jpg if your tooling cares, or just let the files exist as-is. The replay player serves them as bytes regardless of extension.

Pick one viewport, hard-code it everywhere, and never change it without explicitly retaking baselines. Assrt uses 1600x900 across the entire pipeline: the @playwright/mcp launcher is started with --viewport-size 1600x900 at /Users/matthewdi/assrt-mcp/src/core/browser.ts:296, and the WebM recording is started with size: { width: 1600, height: 900 } at browser.ts:628. The reason for picking once and pinning is that responsive layouts cross breakpoint boundaries at viewport changes, and a screenshot taken at 1280px wide is genuinely a different screen than the same scenario at 1600px wide, even when no code has changed. Pin it once, document the choice, and cross-reference any responsive testing as a separate suite.

Three rules cover most of it. First, never screenshot before the DOM is stable. A network-driven SPA is not done rendering 50ms after navigate; wait for mutations to settle (Assrt uses a 2s quiet window with a 30s ceiling). Second, pin everything that the rendered pixel state depends on: the viewport size, the operating system if you are doing pixel-diff, the timezone if your UI shows times, the locale if your UI shows numbers, the fonts (preload them, do not let them lazy-load mid-test). Third, mask elements that are honest non-determinism (timestamps, generated order IDs, ad iframes) so the diff engine does not flag them as regressions. If you are using a vision-model pathway instead, the second and third rules relax considerably, because the model tolerates anti-aliasing and font-load timing the way a human reviewer does.