diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ee3479..21d9cf3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman - **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API. - **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`. - **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`). +- **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names. - **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary. ## 0.1.3 — 2026-10-05 diff --git a/DESIGN.md b/DESIGN.md index d8cd746..a766d89 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -60,6 +60,17 @@ never invent a field the API does not accept (the API rejects unknown fields). the key) once on stdout; its connection hints on stderr use `$SPICRAWL_BROWSER_URL` and are suppressed by `-q`. A `ws://` URL warns on stderr, always, that the key travels unencrypted. +10. **Engine routing is internal.** A customer sees three rendering tiers, + `fetch`, `js` and `browser`, and never an engine or browser name. The API + already reports tiers. `sessions create` and `browser url` take a tier with + `--engine`; the engine names older CLIs took (which the API still honours) + are sent as typed, with a deprecation notice on stderr that does not repeat + them. `scrape` and `batch submit` keep `--engine` and the display flag next + to it hidden: the API ignores both and says so (`ENGINE_IGNORED`). In human output any + engine name the API sends is shown as its tier (`output.Tier`, + `output.PublicText`). JSON and `--jsonl` are the API's data and stay exactly + as sent. `internal/cmd/engine_names_test.go` locks the help, the human + output and the embedded skill and specs. ## Configuration @@ -111,11 +122,11 @@ spicrawl scrape [flags] POST /v1/scrape --format html|markdown|text|json|pdf response_format --render js_render --stealth --impersonate - --engine fetch|obscura|chromium|camoufox --mode auto + --mode auto --premium-proxy --country CC --proxy URL --proxy-verify --sticky-key K --session ID --wait MS --wait-for SEL --wait-for-timeout MS - --block images,fonts,... --headless=false + --block images,fonts,... --header 'Name: value' (repeatable) custom_headers --extract JSON|@file --ai PROMPT --ai-schema JSON|@file --autoparse --links --main-content/--no-main-content --include SEL --exclude SEL --no-parse-pdf @@ -141,10 +152,10 @@ spicrawl batch cancel [--yes] --yes required with spicrawl batch retry|close spicrawl batch append -spicrawl sessions create [--engine E] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K] +spicrawl sessions create [--engine fetch|js|browser] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K] spicrawl sessions list | get | context | release | delete [--yes] -spicrawl browser url [--engine chromium|obscura] [--country CC] [--ttl S] [--headless=false] +spicrawl browser url [--engine browser|js] [--country CC] [--ttl S] prints the wss:// CDP URL once (stdout) spicrawl logs [--errors] [--status S] [--limit N] [--all] [--all-projects] diff --git a/README.md b/README.md index fb2aac5..99883f8 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ The result is one JSON object with the page under `content`: `spicrawl` is a single native binary that calls the [Spicrawl](https://spicrawl.com) web scraping API: - **Five output formats.** `--format` takes `html` (the API default), `markdown`, `text`, `json` or `pdf`; Markdown keeps only the main content unless you pass `--no-main-content`. -- **JavaScript rendering.** `--render` runs the page in a browser, `--wait-for` waits for a CSS selector, and `--engine` pins `fetch`, `obscura` or `chromium` (`camoufox` is coming soon). +- **JavaScript rendering.** `--render` runs the page's JavaScript and `--wait-for` waits for a CSS selector. The API picks the rendering tier (`fetch`, `js` or `browser`) from what the request needs; there is no engine to choose. - **Batch jobs of up to 10,000 URLs.** `spicrawl batch submit` sends up to 10,000 URLs and 1 MiB per request as one server-side job, runs up to 50 items at once (server default 10) and tries each item up to 10 times (default 3). - **Parallel scrapes from stdin.** `spicrawl scrape -` reads one URL per line and runs 4 requests at once by default (`--concurrency`), with every scrape flag available. - **Structured data without a model.** `--extract` (CSS selectors), `--autoparse` (JSON-LD, OpenGraph) and `--links`. @@ -146,8 +146,8 @@ For other clients (Gemini CLI, OpenCode, Factory Droid, Devin and more) use the |---|---| | `spicrawl scrape ` | Fetch one page, or many from stdin, as Markdown, HTML, text, JSON, PDF or extracted data | | `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` | -| `spicrawl sessions` | Keep one engine and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` | -| `spicrawl browser` | Cloud browser over the Chrome DevTools Protocol (`browser url`). Coming soon | +| `spicrawl sessions` | Keep one rendering tier and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` | +| `spicrawl browser` | Cloud browser over a DevTools Protocol (CDP) WebSocket (`browser url`). Coming soon | | `spicrawl init` | Set up AI agent clients: MCP server config and the agent skill | | `spicrawl mcp` | `mcp install`: add the hosted MCP server to a client's config | | `spicrawl skill` | `skill install`: write the agent skill (`SKILL.md`) where each client looks for it | @@ -215,11 +215,11 @@ AI extraction with a plain-language prompt or a JSON Schema (`--ai`, `--ai-schem ### How to take a screenshot or save a page as PDF -Screenshots and PDF output need the Chromium engine: +Screenshots and PDF output are rendered in a full browser, which the API picks for you: ```sh -spicrawl scrape https://example.com --engine chromium --screenshot --screenshot-full-page -o home.html -spicrawl scrape https://example.com/report --engine chromium --format pdf -o report.pdf +spicrawl scrape https://example.com --screenshot --screenshot-full-page -o home.html +spicrawl scrape https://example.com/report --format pdf -o report.pdf ``` Screenshots are saved next to the `-o` file as `.