From 5e4ab173b2c2a73f2c16c9ff8bbc8cbcea80eb6d Mon Sep 17 00:00:00 2001 From: Dibbayajyoti Date: Thu, 22 Oct 2026 14:06:43 +0530 Subject: [PATCH] cli: show rendering tiers, never engine names Customers see three tiers (fetch, js, browser), not the engines behind them. The API already reports tiers on every public endpoint, so the CLI follows. - sessions create/list and browser url take a tier with --engine. The names older CLIs took still work (sent as typed) with a stderr notice that does not repeat them. - scrape/batch submit keep --engine and the display flag hidden: the API ignores both (ENGINE_IGNORED). The spinner no longer reads an engine pin. - Human output maps any engine name the API sends to its tier (logs, sessions, usage, --meta) and rewrites engine names in warnings, hints and errors. --json and --jsonl stay exactly as sent. - openapi.yaml, the schemas and the embedded skill are re-synced from the docs site (make spec); README, DESIGN.md and CHANGELOG use tier names. - Tests: help, human output, live summary, errors, the embedded skill and the specs never name an engine; goldens updated to the API's tier values. --- CHANGELOG.md | 1 + DESIGN.md | 19 +- README.md | 20 +- internal/cmd/assets/skill.md | 84 ++-- internal/cmd/batch.go | 8 +- internal/cmd/batch_test.go | 11 +- internal/cmd/browser.go | 38 +- internal/cmd/browser_test.go | 29 +- internal/cmd/engine_names_test.go | 225 +++++++++ internal/cmd/logs.go | 13 +- internal/cmd/ops_golden_test.go | 4 +- internal/cmd/root.go | 8 +- internal/cmd/scrape.go | 17 +- internal/cmd/scrape_test.go | 2 +- internal/cmd/scrapeflags.go | 10 +- internal/cmd/sessions.go | 35 +- internal/cmd/sessions_golden_test.go | 10 +- internal/cmd/sessions_test.go | 38 +- .../cmd/testdata/golden/browser/url.stdout | 2 +- .../golden/browser/url_quiet_json.stdout | 2 +- .../browser/url_quiet_prints_bare_url.stdout | 2 +- .../testdata/golden/browser_human/url.stdout | 2 +- .../golden/schema/batch.stdout.sha256 | 4 +- .../golden/schema/batch_item.stdout.sha256 | 4 +- .../golden/schema/scrape.stdout.sha256 | 4 +- .../golden/schema/session.stdout.sha256 | 4 +- .../golden/schema_human/batch.stdout.sha256 | 4 +- .../schema_human/batch_item.stdout.sha256 | 4 +- .../golden/schema_human/scrape.stdout.sha256 | 4 +- .../golden/schema_human/session.stdout.sha256 | 4 +- .../testdata/golden/sessions/create.stdout | 4 +- .../create_session_context_file.stdout | 2 +- .../cmd/testdata/golden/sessions/get.stdout | 2 +- .../cmd/testdata/golden/sessions/list.stdout | 4 +- .../golden/sessions/list_all_pages.stdout | 4 +- .../golden/sessions/list_next_cursor.stdout | 2 +- .../testdata/golden/sessions/release.stdout | 2 +- .../golden/sessions_human/create.stdout | 2 +- .../testdata/golden/sessions_human/get.stdout | 2 +- .../golden/sessions_human/get_released.stdout | 2 +- .../golden/sessions_human/list.stdout | 6 +- .../sessions_human/list_next_cursor.stdout | 4 +- .../golden/sessions_human/release.stdout | 2 +- .../usage_human/by_engine_unattributed.stdout | 10 +- internal/cmd/usage.go | 4 +- internal/output/output.go | 10 +- internal/output/tier.go | 48 ++ internal/output/tier_test.go | 32 ++ internal/schema/batch-item.json | 24 +- internal/schema/batch.json | 48 +- internal/schema/scrape.json | 40 +- internal/schema/session.json | 11 +- openapi.yaml | 434 +++++++++--------- 53 files changed, 842 insertions(+), 469 deletions(-) create mode 100644 internal/cmd/engine_names_test.go create mode 100644 internal/output/tier.go create mode 100644 internal/output/tier_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ee3479..21d9cf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman - **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API. - **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`. - **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`). +- **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names. - **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary. ## 0.1.3 — 2026-10-05 diff --git a/DESIGN.md b/DESIGN.md index d8cd746..a766d89 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -60,6 +60,17 @@ never invent a field the API does not accept (the API rejects unknown fields). the key) once on stdout; its connection hints on stderr use `$SPICRAWL_BROWSER_URL` and are suppressed by `-q`. A `ws://` URL warns on stderr, always, that the key travels unencrypted. +10. **Engine routing is internal.** A customer sees three rendering tiers, + `fetch`, `js` and `browser`, and never an engine or browser name. The API + already reports tiers. `sessions create` and `browser url` take a tier with + `--engine`; the engine names older CLIs took (which the API still honours) + are sent as typed, with a deprecation notice on stderr that does not repeat + them. `scrape` and `batch submit` keep `--engine` and the display flag next + to it hidden: the API ignores both and says so (`ENGINE_IGNORED`). In human output any + engine name the API sends is shown as its tier (`output.Tier`, + `output.PublicText`). JSON and `--jsonl` are the API's data and stay exactly + as sent. `internal/cmd/engine_names_test.go` locks the help, the human + output and the embedded skill and specs. ## Configuration @@ -111,11 +122,11 @@ spicrawl scrape [flags] POST /v1/scrape --format html|markdown|text|json|pdf response_format --render js_render --stealth --impersonate - --engine fetch|obscura|chromium|camoufox --mode auto + --mode auto --premium-proxy --country CC --proxy URL --proxy-verify --sticky-key K --session ID --wait MS --wait-for SEL --wait-for-timeout MS - --block images,fonts,... --headless=false + --block images,fonts,... --header 'Name: value' (repeatable) custom_headers --extract JSON|@file --ai PROMPT --ai-schema JSON|@file --autoparse --links --main-content/--no-main-content --include SEL --exclude SEL --no-parse-pdf @@ -141,10 +152,10 @@ spicrawl batch cancel [--yes] --yes required with spicrawl batch retry|close spicrawl batch append -spicrawl sessions create [--engine E] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K] +spicrawl sessions create [--engine fetch|js|browser] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K] spicrawl sessions list | get | context | release | delete [--yes] -spicrawl browser url [--engine chromium|obscura] [--country CC] [--ttl S] [--headless=false] +spicrawl browser url [--engine browser|js] [--country CC] [--ttl S] prints the wss:// CDP URL once (stdout) spicrawl logs [--errors] [--status S] [--limit N] [--all] [--all-projects] diff --git a/README.md b/README.md index fb2aac5..99883f8 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ The result is one JSON object with the page under `content`: `spicrawl` is a single native binary that calls the [Spicrawl](https://spicrawl.com) web scraping API: - **Five output formats.** `--format` takes `html` (the API default), `markdown`, `text`, `json` or `pdf`; Markdown keeps only the main content unless you pass `--no-main-content`. -- **JavaScript rendering.** `--render` runs the page in a browser, `--wait-for` waits for a CSS selector, and `--engine` pins `fetch`, `obscura` or `chromium` (`camoufox` is coming soon). +- **JavaScript rendering.** `--render` runs the page's JavaScript and `--wait-for` waits for a CSS selector. The API picks the rendering tier (`fetch`, `js` or `browser`) from what the request needs; there is no engine to choose. - **Batch jobs of up to 10,000 URLs.** `spicrawl batch submit` sends up to 10,000 URLs and 1 MiB per request as one server-side job, runs up to 50 items at once (server default 10) and tries each item up to 10 times (default 3). - **Parallel scrapes from stdin.** `spicrawl scrape -` reads one URL per line and runs 4 requests at once by default (`--concurrency`), with every scrape flag available. - **Structured data without a model.** `--extract` (CSS selectors), `--autoparse` (JSON-LD, OpenGraph) and `--links`. @@ -146,8 +146,8 @@ For other clients (Gemini CLI, OpenCode, Factory Droid, Devin and more) use the |---|---| | `spicrawl scrape ` | Fetch one page, or many from stdin, as Markdown, HTML, text, JSON, PDF or extracted data | | `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` | -| `spicrawl sessions` | Keep one engine and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` | -| `spicrawl browser` | Cloud browser over the Chrome DevTools Protocol (`browser url`). Coming soon | +| `spicrawl sessions` | Keep one rendering tier and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` | +| `spicrawl browser` | Cloud browser over a DevTools Protocol (CDP) WebSocket (`browser url`). Coming soon | | `spicrawl init` | Set up AI agent clients: MCP server config and the agent skill | | `spicrawl mcp` | `mcp install`: add the hosted MCP server to a client's config | | `spicrawl skill` | `skill install`: write the agent skill (`SKILL.md`) where each client looks for it | @@ -215,11 +215,11 @@ AI extraction with a plain-language prompt or a JSON Schema (`--ai`, `--ai-schem ### How to take a screenshot or save a page as PDF -Screenshots and PDF output need the Chromium engine: +Screenshots and PDF output are rendered in a full browser, which the API picks for you: ```sh -spicrawl scrape https://example.com --engine chromium --screenshot --screenshot-full-page -o home.html -spicrawl scrape https://example.com/report --engine chromium --format pdf -o report.pdf +spicrawl scrape https://example.com --screenshot --screenshot-full-page -o home.html +spicrawl scrape https://example.com/report --format pdf -o report.pdf ``` Screenshots are saved next to the `-o` file as `.