Skip to content

test: cover help rendering and the output layer's suppression rules - #33

Merged
patramsey merged 1 commit into
mainfrom
test/output-and-help
Aug 3, 2026
Merged

test: cover help rendering and the output layer's suppression rules#33
patramsey merged 1 commit into
mainfrom
test/output-and-help

Conversation

@patramsey

Copy link
Copy Markdown
Owner
Before After
cmd/help.go 0.0% 68.8%
internal/output/output.go 64.2% 70.3%
Repo total 69.2% 71.4%

On my earlier objection

I argued against testing help output, on the grounds that it becomes a change-detector — failing whenever someone rewords a string, never when anything breaks. That objection is about how the tests are written, not whether the code should be tested, so these assert structure rather than wording:

  • Available commands appear; hidden ones do not — a hidden command in help advertises something unsupported
  • Hidden flags aren't rendered; names, shorthands and usage strings survive
  • printFilteredFlags shows only what the allow list permits
  • With colour off, no escape sequence reaches the writer — help is routinely piped into a file, a pager, or an agent
  • Long falls back to Short; a leaf command with no subcommands and no flags doesn't panic

Reword any description in this repo and these still pass. Delete a command from the list and they fail.

essentialGlobalFlagNames is the one place a literal name is pinned, deliberately: --dry-run and --yes gate destructive actions, so dropping either from subcommand help hides the safety controls on exactly the pages where someone is about to mutate something.

The output layer is all about suppression

Nearly every one of these guards something that must not appear:

  • Hint is silent outside table mode — emitted into JSON it corrupts the document a caller is about to parse
  • WarnBox degrades to plain prefixed lines rather than drawing a box into a pipe, but must not go silent; its warnings are the ones that earned extra weight
  • Spinners are inert on a non-TTY, and Stop/Update stay safe to call. Stopping twice doesn't panic
  • YAMLList omits nextPage when there is no next page, so callers test presence rather than comparing to zero
  • DefaultConfig picks JSON when stdout isn't a terminal — the contract every piped invocation depends on

One test was worthless, and mutation testing caught it

The NO_COLOR presence cases passed vacuously. Under go test stdout isn't a TTY, so ColorEnabled() returns false whatever NO_COLOR does — a boolean misreading of the spec passed just as happily as the correct presence check.

They now set CLICOLOR_FORCE=1 alongside, which makes the two readings diverge: correct code returns false because NO_COLOR is checked first, a boolean reading returns true. The assertion can now fail, and it pins that NO_COLOR outranks CLICOLOR_FORCE into the bargain.

Verification

Eleven mutations, all caught:

Mutation Mutation
NO_COLOR read as boolean Hint leaks into structured output
env var precedence swapped WarnBox goes silent
CLICOLOR_FORCE ignored hidden commands leak into help
DefaultConfig always table hidden flags leak into help
YAMLList emits zero nextPage --dry-run dropped from help
(+ the NO_COLOR fix above)

Also fixed a fixture bug the tests surfaced immediately: the first helpFixture built subcommands without a Run, and cobra's IsAvailableCommand reports false for those — so they were excluded from help for the wrong reason and the assertions would have passed vacuously.

  • golangci-lint run — no issues
  • go test -race -count=1 ./... — full suite clean
  • 30 subtests confirmed running via -v

cmd/help.go 0% -> 68.8%, internal/output/output.go 64.2% -> 70.3%,
repo 69.2% -> 71.4%.

I argued against this on the grounds that help tests become
change-detectors — failing whenever someone rewords a string, never when
anything breaks. That objection is about HOW the tests are written, not
whether the code can be tested, so these assert structure rather than
wording:

- Available commands appear; hidden ones do not. A hidden command in help
  advertises something unsupported.
- Hidden flags are not rendered, and flag names, shorthands and usage
  strings survive.
- printFilteredFlags shows only what the allow list permits.
- With colour off, no escape sequence reaches the writer — help is
  routinely piped into a file, a pager, or an agent.
- Long falls back to Short; a leaf command with no subcommands and no
  flags does not panic.

essentialGlobalFlagNames is the one place a literal name is pinned, and
deliberately: --dry-run and --yes are the flags that gate destructive
actions, so dropping either from subcommand help hides the safety
controls on exactly the pages where someone is about to mutate something.

On the output side the theme is suppression — nearly every one of these
guards something that must NOT appear:

- Hint is silent outside table mode; emitted into JSON it corrupts the
  document a caller is about to parse.
- WarnBox degrades to plain prefixed lines rather than drawing a box into
  a pipe, but must not go silent — its warnings are the ones that earned
  extra weight.
- Spinners are inert on a non-TTY, and Stop/Update stay safe to call.
  Stopping twice does not panic.
- YAMLList omits nextPage when there is no next page, so callers can test
  for presence rather than compare against zero.
- DefaultConfig picks JSON when stdout is not a terminal, which is the
  contract every piped invocation depends on.

One test was initially worthless and mutation testing caught it. The
NO_COLOR presence cases passed vacuously: under `go test` stdout is not a
TTY, so ColorEnabled returns false whatever NO_COLOR does, and a boolean
reading of it passed too. They now set CLICOLOR_FORCE=1 alongside, which
makes the two readings diverge — correct code returns false because
NO_COLOR is checked first, a boolean reading returns true — so the
assertion can fail, and it pins that NO_COLOR outranks CLICOLOR_FORCE
into the bargain.

Eleven mutations, all caught: NO_COLOR read as a boolean, the two env
vars' precedence swapped, CLICOLOR_FORCE ignored, DefaultConfig always
returning table, YAMLList emitting a zero nextPage, Hint leaking into
structured output, WarnBox going silent, hidden commands and hidden flags
leaking into help, and --dry-run dropped from subcommand help.

Also fixed a fixture bug the tests surfaced immediately: the first
helpFixture built subcommands without a Run, and cobra's
IsAvailableCommand reports false for those, so they were excluded from
help for the wrong reason and the assertions would have passed vacuously.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@patramsey
patramsey merged commit 18c7749 into main Aug 3, 2026
3 checks passed
@patramsey
patramsey deleted the test/output-and-help branch August 3, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants