Skip to content

cli: live progress and a cleaner terminal view, with a golden stdout contract - #4

Merged
DibbayajyotiRoy merged 8 commits into
mainfrom
feat/tui-progress
Oct 22, 2026
Merged

DibbayajyotiRoy merged 8 commits into
mainfrom
feat/tui-progress

Conversation

@DibbayajyotiRoy

@DibbayajyotiRoy DibbayajyotiRoy commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

Live progress on stderr for a person at a terminal. Until now scrape of a rendered page, scrape -, batch submit, paged logs/sessions/batch results and every other API call sat silent until the answer arrived (4 to 60 s for a JavaScript-rendered page), except batch wait, which rewrote one counter line.

⠹ Scraping example.com/pricing (rendering JavaScript) · 4.2s
⠸ Batch job 3/10 done (1 failed) · 12.1s
⠴ Fetching the request log · 1.5s

Each is cleared before the result prints. scrape, scrape - and batch wait then end with one summary line on stderr:

✓ example.com/pricing  200 · 5 credits · 4.2s
✓ batch 01J9Z6V0Q8M4K2T7R3N5B1C9XA  completed · 10/10 items · 1 failed · 10 credits · 8.1s
✗ 12 URLs  10 ok · 2 failed · 31.0s

API errors keep the ERR:: code and hint:

✗ ERR::TARGET::BLOCKED: the site refused the request
  hint     retry with --render, or add --premium-proxy
  target   returned 403
  request  01JABCDEF (spicrawl logs get 01JABCDEF)
  url      https://example.com/blocked
  after    1.2s

Rules it follows

  • stdout is untouched. Everything goes to stderr. Verified byte-for-byte against a build of main (see below).
  • Off unless it is for a person. Live output needs stderr to be a terminal and is disabled by --json, -q/--quiet, NO_COLOR, TERM=dumb and CI (anything but empty/0/false), and on a legacy Windows console. When off, the output is exactly what it was, including batch wait's old progress line and the plain error:/hint: block.
  • A wait under 300 ms shows nothing, so fast calls do not flash a spinner.
  • Never an internal name. Labels say only what the request asked for (rendering JavaScript for --render, using a full browser for --stealth/a pinned browser engine/--actions, extracting with AI), not which tier the API picked. The scrape call is a single blocking POST, so there is no stage to report as it climbs.
  • Ctrl-C clears the line, restores the cursor, prints ✗ interrupted and exits 130 as before. Execute also stops any spinner a command left running.
  • Palette is the terminal's own green/red/cyan plus bold and faint, so it follows the theme on light and dark backgrounds. No white, black or yellow.

How

  • internal/ui (new): gating (ui.Live), Spinner (label, elapsed, one at a time, single Write per frame, label truncated to the terminal width), Done (summary line), Fail (error block), colour helpers. Every method is a no-op on a nil or off *UI, so call sites do not branch.
  • output.Printer.UI() hands it out; Info/Warn write around a running spinner (retry notes land above it).
  • api.Client.Wait hook: cmd.Client() uses it so every API call gets a spinner with a plain label. A command that runs its own (scrape, batch wait) keeps its label, because only one spinner runs at a time.
  • scrape - steps the spinner aside around each JSON line it writes to stdout, so a terminal on both streams stays readable.

New dependency: golang.org/x/term v0.6.0

Used for one thing: the terminal width, so a long URL cannot wrap the spinner line (a wrapped line cannot be cleared with \r). v0.6.0 is the release that matches the golang.org/x/sys v0.6.0 already in go.mod, so nothing else moves. TTY detection keeps using go-isatty, which was already a dependency. No spinner or TUI library: the spinner itself is about 120 lines.

Evidence

Built main and this branch and ran both against a local fake API with delays (no live key was available), on a pseudo-TTY with script.

  • Before: scrape --render printed nothing for 4.2 s, then the markdown. After: spinner frames 0.3s to 4.1s, cleared, markdown, then ✓ example.com 200 · 5 credits · 4.2s.
  • 7 cases (scrape markdown, --render --json, a blocked page, scrape -, batch submit --wait, batch wait, logs), each run with stderr piped and with stderr on a pty and stdout in a file: stdout, stderr (when piped) and exit code are byte-identical between the main and branch binaries.
  • In a pty, with NO_COLOR=1, TERM=dumb, CI=true, --json or --quiet: 0 escape bytes, same as main. With none of them: spinner.
  • 40-column pty with a long URL: the spinner line stays within 39 cells.
  • SIGINT during a spinner: line cleared, cursor shown, ✗ interrupted, exit 130.

Tests

internal/ui: gating table (TTY, NO_COLOR, TERM, CI, Windows), spinner lifecycle (draws after the delay, never newlines, hides then restores the cursor, idempotent Stop, one at a time), Fprintf around a spinner, off-is-inert, summary and error block, elapsed and truncation. internal/cmd/live_test.go: for scrape, scrape -, batch wait and a bare call, the same run with and without a live stderr yields identical stdout; no escape codes with --json/--quiet; a fast call draws no spinner; the error block keeps code and hint while the plain one is unchanged; no engine name reaches the live output even when the API returns one in X-Engine; Ctrl-C restores the cursor. go test -race ./..., go vet ./... and builds for linux, darwin and windows pass.

Not changed: --meta still prints engine: from the response header, and the JSON wrapper still carries engine; both are existing, explicitly requested output.

Also in this PR

  • The other commands get the same look (init, auth, config, status, skill, mcp, docs, logs, sessions). There are spinners on their network waits, a ✓/✗ checklist for status, verdict lines, coloured status columns, and next hints after init/auth/skill install. Styling on stdout applies only when stdout is a terminal. Piped output is unchanged, and config get <key> / config path stay bare for substitution.
  • Golden stdout contract (was test: golden files for the stdout and exit-code contract #5): 242 cases, covering every command and its main modes, each run plain, with NO_COLOR=1 and with CI=true. They lock stdout, exit codes and written files byte for byte, and fail on any \x1b or \r when not a TTY. They pass on this branch, so the new terminal UI leaves piped output untouched. Regenerate with go test ./internal/cmd -run TestGolden -update.

go vet ./... and go test -race ./... pass.

internal/ui is the shared live-terminal layer: a spinner with a label and
elapsed time that appears only after 300 ms, a one-line check/cross summary,
a readable error block and a small colour palette. It is off unless stderr is
a terminal and nothing asked for plain output (--json, --quiet, NO_COLOR,
TERM=dumb, CI), and every method is a no-op when off. output.Printer gains
UI(), and Info/Warn write around a running spinner.
A spinner with a label and elapsed time appears after 300 ms on any wait:
scrape ("Scraping example.com/pricing (rendering JavaScript)"), scrape -
("Scraping 3/10 done"), batch submit and wait ("Batch job 3/10 done"), and
every other API call through Client(). A one-line check/cross summary with
status, credits and duration follows scrape, scrape - and batch wait, and API
errors keep their ERR:: code and hint in a readable block.

All of it is on stderr and only when stderr is a terminal and none of --json,
--quiet, NO_COLOR, TERM=dumb or CI is set; otherwise the output is unchanged.
Execute stops a leftover spinner so Ctrl-C always restores the cursor.
README, DESIGN and CHANGELOG say when the spinner and summary line appear,
that they are stderr-only, and what turns them off.
…logs and sessions

On a terminal: a spinner with a label for each wait (the skill download, the
key check, the status checks, docs, log and session calls), a ✓/✗ checklist
for status and auth status, marks and a count under the init/mcp/skill report,
coloured status cells in the logs and sessions tables, dimmed keys in the
key/value views, and a short next-steps list after login, init and install.

Plain output is unchanged: styling is applied only when the stream is a
terminal and --json, --quiet, NO_COLOR, TERM=dumb and CI are not in play.
Hints and spinners go to stderr; stdout carries the same bytes when piped.
Lock what scripts and AI agents parse, byte for byte, before any terminal
work (spinners, colour, TUI) lands on top of it.

internal/cmd/*_golden_test.go run every command and its main modes
in-process against a fake API, as a non-interactive caller would: stdout
is not a terminal, so JSON by default. Per case the stdout bytes and the exit
code are compared with testdata/golden/<group>/<case>.{stdout,exit}. In JSON
mode the stderr problem document is locked too (.stderr), and files a command
writes with -o / --screenshot-dir (.file.*).

Every case runs three times (plain, NO_COLOR=1, CI=true); all must give the
same stdout and exit code, and stdout/stderr must contain no ANSI escape and
stderr no carriage return (an in-place spinner line).

Cases named *_human run the human (terminal) code path through the existing
humanOutput hook, since that is the only way to reach it without a pty.

Timestamps, request ids, build identity, the fake server address and temp
paths are replaced by placeholders (goldenNormalize). Regenerate with
`go test ./internal/cmd -run TestGolden -update` (or GOLDEN_UPDATE=1) and
review the diff. Big outputs (the embedded JSON Schemas) are locked by digest.
@DibbayajyotiRoy DibbayajyotiRoy changed the title cli: live progress, spinner and summary line on stderr cli: live progress and a cleaner terminal view, with a golden stdout contract Oct 22, 2026
@DibbayajyotiRoy DibbayajyotiRoy self-assigned this Oct 22, 2026
@DibbayajyotiRoy DibbayajyotiRoy added the enhancement New feature or request label Oct 22, 2026
@DibbayajyotiRoy
DibbayajyotiRoy merged commit b17f71b into main Oct 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant