cli: live progress and a cleaner terminal view, with a golden stdout contract

CHANGELOG.md

@@ -2,8 +2,9 @@
 
 ## Unreleased
 
-Bug fixes from a review of 0.1.3. No new commands; two behaviour changes are marked.
+Bug fixes from a review of 0.1.3, and live progress on a terminal. No new commands; two behaviour changes are marked.
 
+- **Live progress on a terminal.** A spinner with a label and elapsed time covers every wait over about 300 ms (`scrape`, `scrape -`, `batch submit`/`wait`, and any other API call), and `scrape`, `scrape -` and `batch wait` end with a `✓`/`✗` summary line (page, status, credits, duration) on stderr. API errors keep their `ERR::` code and hint in a short block. Everything is on stderr, only when stderr is a terminal, and off with `--json`, `-q`, `NO_COLOR`, `TERM=dumb` or `CI`: stdout and the plain output are unchanged. New dependency: `golang.org/x/term`, for the terminal width.
 - **Retries are safe.** A connection that drops after the request was sent now exits 12 ("may have run and been billed"), not 10 ("nothing was sent, safe to retry"). A gateway 5xx without an API problem document is retried only for GET/HEAD, and is no longer mistaken for the target's status under `--original-status`. A 301/302/303 on a POST is an error naming the Location instead of a silent GET. `Retry-After` waits are capped at 60 s and announced on stderr.
 - **Ctrl-C exits 130** with "interrupted", including while `scrape -` waits on stdin. A mistyped subcommand (`spicrawl batch wiat`) is a usage error with a suggestion, not help and exit 0.
 - **Keys:** a trailing CR/LF or space on the key (flag, env or file) is trimmed. The config file is written via a private temp file, short keys are fully masked, and commands that get the key and base URL from flags work without `$HOME`.

DESIGN.md

@@ -10,6 +10,12 @@ never invent a field the API does not accept (the API rejects unknown fields).
 ## Contract agents rely on
 
 1. **stdout is data only.** Progress, warnings and errors go to stderr.
+   Live progress (spinner, colour, the `✓`/`✗` summary line, the error block) is
+   the one thing a person gets that an agent does not, so it is gated hard: only
+   when stderr is a terminal, and never with `--json`, `-q`, `NO_COLOR`,
+   `TERM=dumb` or `CI`. Switched off, the output is the plain text it always was.
+   It never prints an engine or browser name; it says what the request asked for
+   ("rendering JavaScript"), not which tier the API picked.
 2. **JSON when not a terminal.** `--json` forces it. Human mode prints tables
    or the document itself.
 3. **Never prompts without a TTY.** Anything interactive has a flag form.
@@ -85,6 +91,7 @@ internal/api        HTTP client, Problem, DoWithRetry
 internal/config     key/base resolution, config file
 internal/exitcode   exit codes
 internal/output     Printer (JSON/human/JSONL, stderr info)
+internal/ui         stderr spinner, summary line, error block, colour (off unless a terminal)
 internal/cmd        one file per command group; each registers in init()
 ```
 

README.md

@@ -166,6 +166,12 @@ For other clients (Gemini CLI, OpenCode, Factory Droid, Devin and more) use the
 
 `spicrawl --help` and `spicrawl <command> --help` list every flag. Global flags: `--api-key`, `--base-url`, `--json`, `-q/--quiet` and `--timeout` (default `3m`).
 
+### Progress on a terminal
+
+When stderr is a terminal, a wait longer than about 300 ms shows a spinner with what the CLI is doing and how long it has taken (`⠹ Scraping example.com/pricing (rendering JavaScript) · 4.2s`, `⠸ Batch job 3/10 done · 12.1s`). It is cleared before the result prints, and `scrape`, `scrape -` and `batch wait` end with a `✓` or `✗` line on stderr: page, target status, credits charged, duration. API errors keep their `ERR::` code and hint in a short block.
+
+All of it is on stderr and stdout is byte-for-byte unchanged. It switches itself off, and the CLI prints exactly what it printed before, when stderr is not a terminal (a pipe, a file, an agent or CI log), with `--json` or `-q/--quiet`, with `NO_COLOR` set, with `TERM=dumb`, or when `CI` is set.
+
 ### How to convert a web page to Markdown from the command line
 
 ```sh

go.mod

@@ -6,6 +6,7 @@ require (
 	github.com/mattn/go-isatty v0.0.20
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/pflag v1.0.9
+	golang.org/x/term v0.6.0
 )
 
 require (

go.sum

@@ -11,4 +11,6 @@ github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
 golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw=
+golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

internal/api/client.go

@@ -33,6 +33,10 @@ type Client struct {
 	// Notify, when set, receives progress lines (DoWithRetry waiting between
 	// attempts). The caller decides where they go and whether --quiet mutes them.
 	Notify func(format string, a ...any)
+	// Wait, when set, is called as each round trip starts and the func it
+	// returns as the round trip ends (answer read, or failed). The commands use
+	// it to show a spinner while a call is outstanding.
+	Wait func(method, path string) (done func())
 }
 
 func New(baseURL, apiKey string, timeout time.Duration) *Client {
@@ -188,6 +192,12 @@ func (c *Client) Do(ctx context.Context, req Request) (*Response, error) {
 	hr.Header.Set("Accept", accept)
 	hr.Header.Set("User-Agent", "spicrawl-cli/"+Version)
 
+	if c.Wait != nil {
+		if done := c.Wait(method, req.Path); done != nil {
+			defer done()
+		}
+	}
+
 	var sent atomic.Bool
 	hr = hr.WithContext(httptrace.WithClientTrace(hr.Context(), &httptrace.ClientTrace{
 		WroteHeaders: func() { sent.Store(true) },

internal/cmd/agents.go

@@ -36,6 +36,7 @@ import (
 
 	"github.com/OfficialSpicrawl/cli/internal/config"
 	"github.com/OfficialSpicrawl/cli/internal/output"
+	"github.com/OfficialSpicrawl/cli/internal/ui"
 )
 
 const (
@@ -422,11 +423,13 @@ func agentApply(rs []*agentResult) error {
 }
 
 // agentReport prints results (JSON list, or a table) plus secret guidance.
+// On a terminal the table gets marks and colour and a count line under it.
 func agentReport(p *output.Printer, rs []*agentResult, s agentScope) error {
 	if rs == nil {
 		rs = []*agentResult{}
 	}
 	err := p.Result(rs, func(w io.Writer) {
+		u := outUI(p)
 		rows := make([][]string, 0, len(rs))
 		for _, r := range rs {
 			note := r.Reason
@@ -436,14 +439,75 @@ func agentReport(p *output.Printer, rs []*agentResult, s agentScope) error {
 				}
 				note += "backup: " + agentDisplayPath(r.Backup, s)
 			}
-			rows = append(rows, []string{r.Client, r.Kind, r.Action, agentDisplayPath(r.File, s), note})
+			action := r.Action
+			if u.On() {
+				action = agentActionMark[r.Action] + " " + action
+			}
+			rows = append(rows, []string{r.Client, r.Kind, action, agentDisplayPath(r.File, s), note})
+		}
+		viewTable(p, u, []string{"CLIENT", "KIND", "ACTION", "FILE", "NOTE"}, rows, func(col int, cell string) string {
+			switch {
+			case col == 2 && strings.HasPrefix(cell, "✓"):
+				return u.Green(cell)
+			case col == 2 || col == 4:
+				return u.Dim(cell)
+			}
+			return cell
+		})
+		if u.On() {
+			agentSummary(w, u, rs)
 		}
-		p.Table([]string{"CLIENT", "KIND", "ACTION", "FILE", "NOTE"}, rows)
 	})
 	agentSecretGuidance(p, rs, s)
 	return err
 }
 
+// agentActionMark is the mark in front of an action on a terminal.
+var agentActionMark = map[string]string{
+	agentActionCreated:  "✓",
+	agentActionUpdated:  "✓",
+	agentActionUnchange: "=",
+	agentActionSkipped:  "–",
+}
+
+// agentSummary is the line under the table: what was written, kept, skipped.
+func agentSummary(w io.Writer, u *ui.UI, rs []*agentResult) {
+	n := map[string]int{}
+	for _, r := range rs {
+		n[r.Action]++
+	}
+	var parts []string
+	if c := n[agentActionCreated] + n[agentActionUpdated]; c > 0 {
+		parts = append(parts, fmt.Sprintf("%d written", c))
+	}
+	for _, a := range []string{agentActionUnchange, agentActionSkipped} {
+		if n[a] > 0 {
+			parts = append(parts, fmt.Sprintf("%d %s", n[a], a))
+		}
+	}
+	if len(parts) == 0 {
+		return
+	}
+	m := markOK
+	if n[agentActionCreated]+n[agentActionUpdated]+n[agentActionUnchange] == 0 {
+		m = markNeutral
+	}
+	fmt.Fprintf(w, "\n%s %s\n", viewMark(u, m), strings.Join(parts, " · "))
+}
+
+// agentChanged is the names of the clients whose file was just written.
+func agentChanged(rs []*agentResult) []string {
+	var names []string
+	seen := map[string]bool{}
+	for _, r := range rs {
+		if (r.Action == agentActionCreated || r.Action == agentActionUpdated) && !seen[r.Client] {
+			seen[r.Client] = true
+			names = append(names, agentClientByID(r.Client).Name)
+		}
+	}
+	return names
+}
+
 // agentSecretGuidance warns about literal keys, with .gitignore advice for
 // files inside the project.
 func agentSecretGuidance(p *output.Printer, rs []*agentResult, s agentScope) {

internal/cmd/agents_golden_test.go

@@ -0,0 +1,57 @@
+package cmd
+
+import (
+	"net/http"
+	"testing"
+)
+
+// The agent commands (init, mcp install, skill install) write config files and
+// print what they did. Their base URL is an unreachable address so the MCP and
+// docs URLs they write are fixed; the skill comes from a fake docs server.
+const (
+	gAgentBase  = "http://127.0.0.1:1"
+	gAgentSkill = "---\nname: spicrawl-web-data-api\ndescription: golden test skill\n---\n\n# Spicrawl\n\nUse https://api.spicrawl.com for scraping.\n"
+)
+
+func gSkillAPI() func() http.HandlerFunc {
+	return goldenServe(map[string]http.HandlerFunc{"GET /skill.md": func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "text/markdown")
+		_, _ = w.Write([]byte(gAgentSkill))
+	}})
+}
+
+func init() {
+	env := map[string]string{agentEnvSkillURL: "{server}/skill.md"}
+	// agent builds a case that runs in a fresh project directory.
+	agent := func(name string, human bool, files []string, setup func(t *testing.T, e *goldenEnv), args ...string) goldenCase {
+		return goldenCase{name: name, human: human, args: args, baseURL: gAgentBase, env: env, server: gSkillAPI(), files: files, setup: setup}
+	}
+	allMCP := []string{".mcp.json", ".cursor/mcp.json", ".vscode/mcp.json", ".codex/config.toml"}
+	existing := func(t *testing.T, e *goldenEnv) {
+		goldenWrite(t, e.Dir, ".mcp.json", `{"zeta": 1, "mcpServers": {"other": {"command": "x"}}}`)
+	}
+
+	c := agent("agents/mcp_print_all", false, nil, nil, "mcp", "install", "--client", "all", "--dir", "{dir}", "--print")
+	c2 := agent("agents/mcp_install_claude", false, []string{".mcp.json"}, nil, "mcp", "install", "--client", "claude", "--dir", "{dir}")
+	c3 := agent("agents/mcp_install_all", false, allMCP, nil, "mcp", "install", "--client", "all", "--dir", "{dir}")
+	c4 := agent("agents/mcp_install_merges_existing", false, []string{".mcp.json"}, existing, "mcp", "install", "--client", "claude", "--dir", "{dir}")
+	c5 := agent("agents/mcp_err_no_client_detected", false, nil, nil, "mcp", "install", "--dir", "{dir}")
+	c5.stderr = true
+	c6 := agent("agents/skill_print", false, nil, nil, "skill", "install", "--print", "--dir", "{dir}")
+	c7 := agent("agents/skill_install_claude", false, []string{".claude/skills/spicrawl/SKILL.md"}, nil, "skill", "install", "--client", "claude", "--dir", "{dir}")
+	c8 := agent("agents/skill_install_codex_agents_md", false, []string{".agents/skills/spicrawl/SKILL.md", "AGENTS.md"}, nil, "skill", "install", "--client", "codex", "--agents-md", "--dir", "{dir}")
+	c9 := agent("agents/init_yes", false, []string{".mcp.json", ".claude/skills/spicrawl/SKILL.md", ".cursor/mcp.json"}, nil, "init", "--client", "claude,cursor", "--yes", "--dir", "{dir}")
+	c10 := agent("agents/init_needs_yes_without_terminal", false, nil, nil, "init", "--client", "claude", "--dir", "{dir}")
+	c10.stderr = true
+	c11 := agent("agents/init_err_no_client_detected", false, nil, nil, "init", "--yes", "--dir", "{dir}")
+	c11.stderr = true
+
+	goldenAdd(c, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11,
+		agent("agents_human/mcp_print_all", true, nil, nil, "mcp", "install", "--client", "all", "--dir", "{dir}", "--print"),
+		agent("agents_human/mcp_install_all", true, nil, nil, "mcp", "install", "--client", "all", "--dir", "{dir}"),
+		agent("agents_human/mcp_install_merges_existing", true, nil, existing, "mcp", "install", "--client", "claude", "--dir", "{dir}"),
+		agent("agents_human/skill_print", true, nil, nil, "skill", "install", "--print", "--dir", "{dir}"),
+		agent("agents_human/skill_install_claude", true, nil, nil, "skill", "install", "--client", "claude", "--dir", "{dir}"),
+		agent("agents_human/init_yes", true, nil, nil, "init", "--client", "claude,cursor", "--yes", "--dir", "{dir}"),
+	)
+}

internal/cmd/agents_test.go

@@ -7,6 +7,7 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
 )
 
 const agentTestSkill = "---\nname: spicrawl-web-data-api\ndescription: test skill\n---\n\n# Spicrawl\n"
@@ -879,3 +880,145 @@ func TestMCPInstallNonSecretKeepsMode(t *testing.T) {
 		t.Errorf("mode = %v, want 0664 kept", fi.Mode().Perm())
 	}
 }
+
+// ---- terminal output ----
+
+func TestInitHumanPlainIsUnchanged(t *testing.T) {
+	dir, _ := agentTestEnv(t)
+	agentWrite(t, filepath.Join(dir, "CLAUDE.md"), "x")
+	out, errOut, code := runCLIHuman(t, agentTestBase, "init", "--yes", "--dir", dir)
+	if code != 0 {
+		t.Fatalf("exit %d: %s", code, errOut)
+	}
+	if !strings.HasPrefix(out, "CLIENT  KIND   ACTION   FILE") || strings.Contains(out, "✓") {
+		t.Errorf("table changed:\n%s", out)
+	}
+	if !strings.HasPrefix(errOut, "clients: Claude Code\nMCP server: "+agentTestMCPURL+"\n") {
+		t.Errorf("stderr changed:\n%s", errOut)
+	}
+	if strings.Contains(errOut, "next") {
+		t.Errorf("hints without a terminal:\n%s", errOut)
+	}
+	noEscapes(t, "stdout", out)
+	noEscapes(t, "stderr", errOut)
+}
+
+func TestInitLive(t *testing.T) {
+	liveUI(t)
+	dir, _ := agentTestEnv(t)
+	agentWrite(t, filepath.Join(dir, "CLAUDE.md"), "x")
+	out, errOut, code := runCLIHuman(t, agentTestBase, "init", "--yes", "--dir", dir)
+	if code != 0 {
+		t.Fatalf("exit %d: %s", code, errOut)
+	}
+	plain := stripANSI(out)
+	for _, want := range []string{"CLIENT", "✓ created", ".mcp.json", ".claude/skills/spicrawl/SKILL.md", "✓ 2 written"} {
+		if !strings.Contains(plain, want) {
+			t.Errorf("stdout lacks %q:\n%s", want, plain)
+		}
+	}
+	e := stripANSI(errOut)
+	for _, want := range []string{"Setting up Spicrawl", "clients     Claude Code", "MCP server  " + agentTestMCPURL, "next", "restart Claude Code"} {
+		if !strings.Contains(e, want) {
+			t.Errorf("stderr lacks %q:\n%s", want, e)
+		}
+	}
+	if strings.Contains(e, "clients: ") {
+		t.Errorf("plain intro lines next to the styled one:\n%s", e)
+	}
+
+	// Nothing changed the second time: no "next" list, and the count says so.
+	out, errOut, _ = runCLIHuman(t, agentTestBase, "init", "--yes", "--dir", dir)
+	if !strings.Contains(stripANSI(out), "✓ 2 unchanged") || strings.Contains(stripANSI(errOut), "next") {
+		t.Errorf("second run:\n%s\n%s", stripANSI(out), stripANSI(errOut))
+	}
+
+	// --quiet and --json stay plain.
+	for _, flag := range []string{"-q", "--json"} {
+		out, errOut, _ = runCLIHuman(t, agentTestBase, flag, "init", "--yes", "--dir", dir)
+		noEscapes(t, flag+" stdout", out)
+		noEscapes(t, flag+" stderr", errOut)
+	}
+}
+
+func TestInitLivePlanAndSkillSpinner(t *testing.T) {
+	spinNow(t)
+	dir, _ := agentTestEnv(t)
+	slow, _ := newRecordingServer(t, func(w http.ResponseWriter, _ *http.Request) {
+		time.Sleep(40 * time.Millisecond)
+		_, _ = w.Write([]byte(agentTestSkill))
+	})
+	t.Setenv(agentEnvSkillURL, slow.URL+"/skill.md")
+	orig, origIn := initStdinIsTerminal, initStdin
+	initStdinIsTerminal = func() bool { return true }
+	t.Cleanup(func() { initStdinIsTerminal, initStdin = orig, origIn })
+	initStdin = strings.NewReader("n\n")
+
+	out, errOut, code := runCLIHuman(t, agentTestBase, "init", "--client", "codex", "--dir", dir)
+	if code != 0 {
+		t.Fatalf("exit %d: %s", code, errOut)
+	}
+	if !strings.Contains(errOut, "Downloading the Spicrawl skill") {
+		t.Errorf("no spinner while the skill downloads: %q", errOut)
+	}
+	e := stripANSI(errOut)
+	if !strings.Contains(e, "plan:\n  create  codex   mcp") || !strings.Contains(e, "apply these changes? [y/N]") {
+		t.Errorf("plan:\n%s", e)
+	}
+	if !strings.Contains(stripANSI(out), "– skipped") || !strings.Contains(stripANSI(out), "not confirmed") {
+		t.Errorf("report:\n%s", stripANSI(out))
+	}
+}
+
+func TestMCPAndSkillInstallLive(t *testing.T) {
+	liveUI(t)
+	dir, _ := agentTestEnv(t)
+	out, errOut, code := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "claude", "--dir", dir)
+	if code != 0 {
+		t.Fatalf("exit %d: %s", code, errOut)
+	}
+	if !strings.Contains(stripANSI(out), "✓ created") || !strings.Contains(stripANSI(errOut), "restart Claude Code to load the MCP server") {
+		t.Errorf("mcp install:\n%s\n%s", stripANSI(out), stripANSI(errOut))
+	}
+	out, errOut, code = runCLIHuman(t, agentTestBase, "skill", "install", "--client", "claude", "--dir", dir)
+	if code != 0 {
+		t.Fatalf("exit %d: %s", code, errOut)
+	}
+	if !strings.Contains(stripANSI(out), "✓ created") || !strings.Contains(stripANSI(errOut), "start a new session in Claude Code to use the skill") {
+		t.Errorf("skill install:\n%s\n%s", stripANSI(out), stripANSI(errOut))
+	}
+	// Idempotent: nothing written, nothing to restart.
+	_, errOut, _ = runCLIHuman(t, agentTestBase, "skill", "install", "--client", "claude", "--dir", dir)
+	if strings.Contains(stripANSI(errOut), "next") {
+		t.Errorf("hint after a no-op:\n%s", errOut)
+	}
+}
+
+func TestMCPInstallPrintHeadingStyleOnlyOnATerminal(t *testing.T) {
+	dir, _ := agentTestEnv(t)
+	plain, _, _ := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "claude", "--print", "--dir", dir)
+	if !strings.HasPrefix(plain, "# Claude Code: merge into .mcp.json\n{") {
+		t.Fatalf("plain:\n%s", plain)
+	}
+	noEscapes(t, "plain", plain)
+
+	liveUI(t)
+	live, _, _ := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "claude", "--print", "--dir", dir)
+	if !strings.Contains(live, "\x1b[") || stripANSI(live) != plain {
+		t.Errorf("styled heading must be the only difference:\n%q\n%q", live, plain)
+	}
+}
+
+func TestSkillInstallPrintUnchangedWhenLive(t *testing.T) {
+	dir, _ := agentTestEnv(t)
+	plain, _, code := runCLIHuman(t, agentTestBase, "skill", "install", "--print", "--dir", dir)
+	if code != 0 || !strings.Contains(plain, "# Spicrawl") {
+		t.Fatalf("exit %d %q", code, plain)
+	}
+	spinNow(t)
+	live, _, code := runCLIHuman(t, agentTestBase, "skill", "install", "--print", "--dir", dir)
+	if code != 0 || live != plain {
+		t.Errorf("--print must write the skill and nothing else: exit %d\n%q\n%q", code, live, plain)
+	}
+	noEscapes(t, "stdout", live)
+}