Skip to content

test: golden files for the stdout and exit-code contract - #5

Closed
DibbayajyotiRoy wants to merge 1 commit into
mainfrom
test/golden-stdout
Closed

DibbayajyotiRoy wants to merge 1 commit into
mainfrom
test/golden-stdout

Conversation

@DibbayajyotiRoy

Copy link
Copy Markdown
Collaborator

This PR locks the stdout contract ahead of the TUI work (spinners, colour). Scripts and AI agents parse this CLI's stdout and exit codes, so the terminal work must not change either.

What it adds

Only new files: internal/cmd/*_golden_test.go and internal/cmd/testdata/golden/.... No existing file is touched.

TestGolden runs every command and its main modes in-process against a fake API, the way a non-interactive caller sees them (stdout is not a terminal, so JSON by default). For each case it compares:

  • stdout, byte for byte (<case>.stdout), and the exit code (<case>.exit)
  • stderr (<case>.stderr) only where it is itself machine output: the JSON-mode problem document
  • files the command writes with -o / --screenshot-dir (<case>.file.*)

Every case runs three times: plain, NO_COLOR=1, CI=true. All three must produce the same stdout and exit code, stdout and stderr must contain no ANSI escape, and stderr no carriage return (an in-place spinner line). The tests run with a colour-capable TERM, so only the non-terminal output keeps colour off.

Coverage

  • scrape: html, markdown, text, json envelope, pdf (base64), --jsonl, --json, --extract, --ai, --autoparse, --links, --network-capture, --screenshot, -o, target 4xx/5xx and --allowed-status, scrape - (stdin, many URLs), API and usage errors
  • batch: submit (file, jsonl, stdin, --wait), wait (completed, failed, cancelled, pending exit 11), get, list, results, content, cancel, close, retry, append
  • sessions, logs, status, usage, browser url, config, auth, login, logout, schema, docs, version, exit-codes, mcp install, skill install, init, and usage errors from the root command
  • The same commands in human mode (*_human cases), reached through the existing humanOutput hook because that is the only way to get there without a pty

Timestamps, request ids, build identity (version, Go, OS, arch), the fake server address and temp paths are replaced by placeholders in goldenNormalize (documented there). The embedded JSON Schemas are locked by sha256.

Updating

After an intended output change:

go test ./internal/cmd -run TestGolden -update     # or GOLDEN_UPDATE=1 go test ./...

then review the diff. -update also removes golden files no case produces.

Not covered

  • Behaviour that needs a real terminal: the interactive prompts (login, init, batch cancel, sessions delete), and what NO_COLOR / CI do on a TTY. That needs a pty.
  • --help text and shell completion output.
  • stderr prose and OS-specific error text (a refused connection), which are not a stable contract.

Lock what scripts and AI agents parse, byte for byte, before any terminal
work (spinners, colour, TUI) lands on top of it.

internal/cmd/*_golden_test.go run every command and its main modes
in-process against a fake API, as a non-interactive caller would: stdout
is not a terminal, so JSON by default. Per case the stdout bytes and the exit
code are compared with testdata/golden/<group>/<case>.{stdout,exit}. In JSON
mode the stderr problem document is locked too (.stderr), and files a command
writes with -o / --screenshot-dir (.file.*).

Every case runs three times (plain, NO_COLOR=1, CI=true); all must give the
same stdout and exit code, and stdout/stderr must contain no ANSI escape and
stderr no carriage return (an in-place spinner line).

Cases named *_human run the human (terminal) code path through the existing
humanOutput hook, since that is the only way to reach it without a pty.

Timestamps, request ids, build identity, the fake server address and temp
paths are replaced by placeholders (goldenNormalize). Regenerate with
`go test ./internal/cmd -run TestGolden -update` (or GOLDEN_UPDATE=1) and
review the diff. Big outputs (the embedded JSON Schemas) are locked by digest.
@DibbayajyotiRoy

Copy link
Copy Markdown
Collaborator Author

Merged into #4, so the golden tests run against the terminal UI changes they protect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant