Repository navigation
test: golden files for the stdout and exit-code contract - #5
Closed
DibbayajyotiRoy wants to merge 1 commit into
Closed
DibbayajyotiRoy wants to merge 1 commit into
DibbayajyotiRoy wants to merge 1 commit into
Conversation
Lock what scripts and AI agents parse, byte for byte, before any terminal
work (spinners, colour, TUI) lands on top of it.
internal/cmd/*_golden_test.go run every command and its main modes
in-process against a fake API, as a non-interactive caller would: stdout
is not a terminal, so JSON by default. Per case the stdout bytes and the exit
code are compared with testdata/golden/<group>/<case>.{stdout,exit}. In JSON
mode the stderr problem document is locked too (.stderr), and files a command
writes with -o / --screenshot-dir (.file.*).
Every case runs three times (plain, NO_COLOR=1, CI=true); all must give the
same stdout and exit code, and stdout/stderr must contain no ANSI escape and
stderr no carriage return (an in-place spinner line).
Cases named *_human run the human (terminal) code path through the existing
humanOutput hook, since that is the only way to reach it without a pty.
Timestamps, request ids, build identity, the fake server address and temp
paths are replaced by placeholders (goldenNormalize). Regenerate with
`go test ./internal/cmd -run TestGolden -update` (or GOLDEN_UPDATE=1) and
review the diff. Big outputs (the embedded JSON Schemas) are locked by digest.
Collaborator
Author
|
Merged into #4, so the golden tests run against the terminal UI changes they protect. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR locks the stdout contract ahead of the TUI work (spinners, colour). Scripts and AI agents parse this CLI's stdout and exit codes, so the terminal work must not change either.
What it adds
Only new files:
internal/cmd/*_golden_test.goandinternal/cmd/testdata/golden/.... No existing file is touched.TestGoldenruns every command and its main modes in-process against a fake API, the way a non-interactive caller sees them (stdout is not a terminal, so JSON by default). For each case it compares:<case>.stdout), and the exit code (<case>.exit)<case>.stderr) only where it is itself machine output: the JSON-mode problem document-o/--screenshot-dir(<case>.file.*)Every case runs three times: plain,
NO_COLOR=1,CI=true. All three must produce the same stdout and exit code, stdout and stderr must contain no ANSI escape, and stderr no carriage return (an in-place spinner line). The tests run with a colour-capableTERM, so only the non-terminal output keeps colour off.Coverage
scrape: html, markdown, text, json envelope, pdf (base64),--jsonl,--json,--extract,--ai,--autoparse,--links,--network-capture,--screenshot,-o, target 4xx/5xx and--allowed-status,scrape -(stdin, many URLs), API and usage errorsbatch: submit (file, jsonl, stdin,--wait), wait (completed, failed, cancelled, pending exit 11), get, list, results, content, cancel, close, retry, appendsessions,logs,status,usage,browser url,config,auth,login,logout,schema,docs,version,exit-codes,mcp install,skill install,init, and usage errors from the root command*_humancases), reached through the existinghumanOutputhook because that is the only way to get there without a ptyTimestamps, request ids, build identity (version, Go, OS, arch), the fake server address and temp paths are replaced by placeholders in
goldenNormalize(documented there). The embedded JSON Schemas are locked by sha256.Updating
After an intended output change:
then review the diff.
-updatealso removes golden files no case produces.Not covered
login,init,batch cancel,sessions delete), and what NO_COLOR / CI do on a TTY. That needs a pty.--helptext and shell completion output.