Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman
- **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API.
- **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`.
- **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`).
- **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names.
- **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary.

## 0.1.3 — 2026-10-05
Expand Down
19 changes: 15 additions & 4 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,17 @@ never invent a field the API does not accept (the API rejects unknown fields).
the key) once on stdout; its connection hints on stderr use
`$SPICRAWL_BROWSER_URL` and are suppressed by `-q`. A `ws://` URL warns on
stderr, always, that the key travels unencrypted.
10. **Engine routing is internal.** A customer sees three rendering tiers,
`fetch`, `js` and `browser`, and never an engine or browser name. The API
already reports tiers. `sessions create` and `browser url` take a tier with
`--engine`; the engine names older CLIs took (which the API still honours)
are sent as typed, with a deprecation notice on stderr that does not repeat
them. `scrape` and `batch submit` keep `--engine` and the display flag next
to it hidden: the API ignores both and says so (`ENGINE_IGNORED`). In human output any
engine name the API sends is shown as its tier (`output.Tier`,
`output.PublicText`). JSON and `--jsonl` are the API's data and stay exactly
as sent. `internal/cmd/engine_names_test.go` locks the help, the human
output and the embedded skill and specs.

## Configuration

Expand Down Expand Up @@ -111,11 +122,11 @@ spicrawl scrape <url|-> [flags] POST /v1/scrape
--format html|markdown|text|json|pdf response_format
--render js_render
--stealth --impersonate
--engine fetch|obscura|chromium|camoufox --mode auto
--mode auto
--premium-proxy --country CC --proxy URL --proxy-verify --sticky-key K
--session ID
--wait MS --wait-for SEL --wait-for-timeout MS
--block images,fonts,... --headless=false
--block images,fonts,...
--header 'Name: value' (repeatable) custom_headers
--extract JSON|@file --ai PROMPT --ai-schema JSON|@file --autoparse --links
--main-content/--no-main-content --include SEL --exclude SEL --no-parse-pdf
Expand All @@ -141,10 +152,10 @@ spicrawl batch cancel <id> [--yes] --yes required with
spicrawl batch retry|close <id>
spicrawl batch append <id> <file|->

spicrawl sessions create [--engine E] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K]
spicrawl sessions create [--engine fetch|js|browser] [--ttl S] [--country CC] [--rotate-ip] [--sticky-key K]
spicrawl sessions list | get <id> | context <id> | release <id> | delete <id> [--yes]

spicrawl browser url [--engine chromium|obscura] [--country CC] [--ttl S] [--headless=false]
spicrawl browser url [--engine browser|js] [--country CC] [--ttl S]
prints the wss:// CDP URL once (stdout)

spicrawl logs [--errors] [--status S] [--limit N] [--all] [--all-projects]
Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ The result is one JSON object with the page under `content`:
`spicrawl` is a single native binary that calls the [Spicrawl](https://spicrawl.com) web scraping API:

- **Five output formats.** `--format` takes `html` (the API default), `markdown`, `text`, `json` or `pdf`; Markdown keeps only the main content unless you pass `--no-main-content`.
- **JavaScript rendering.** `--render` runs the page in a browser, `--wait-for` waits for a CSS selector, and `--engine` pins `fetch`, `obscura` or `chromium` (`camoufox` is coming soon).
- **JavaScript rendering.** `--render` runs the page's JavaScript and `--wait-for` waits for a CSS selector. The API picks the rendering tier (`fetch`, `js` or `browser`) from what the request needs; there is no engine to choose.
- **Batch jobs of up to 10,000 URLs.** `spicrawl batch submit` sends up to 10,000 URLs and 1 MiB per request as one server-side job, runs up to 50 items at once (server default 10) and tries each item up to 10 times (default 3).
- **Parallel scrapes from stdin.** `spicrawl scrape -` reads one URL per line and runs 4 requests at once by default (`--concurrency`), with every scrape flag available.
- **Structured data without a model.** `--extract` (CSS selectors), `--autoparse` (JSON-LD, OpenGraph) and `--links`.
Expand Down Expand Up @@ -146,8 +146,8 @@ For other clients (Gemini CLI, OpenCode, Factory Droid, Devin and more) use the
|---|---|
| `spicrawl scrape <url\|->` | Fetch one page, or many from stdin, as Markdown, HTML, text, JSON, PDF or extracted data |
| `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` |
| `spicrawl sessions` | Keep one engine and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` |
| `spicrawl browser` | Cloud browser over the Chrome DevTools Protocol (`browser url`). Coming soon |
| `spicrawl sessions` | Keep one rendering tier and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` |
| `spicrawl browser` | Cloud browser over a DevTools Protocol (CDP) WebSocket (`browser url`). Coming soon |
| `spicrawl init` | Set up AI agent clients: MCP server config and the agent skill |
| `spicrawl mcp` | `mcp install`: add the hosted MCP server to a client's config |
| `spicrawl skill` | `skill install`: write the agent skill (`SKILL.md`) where each client looks for it |
Expand Down Expand Up @@ -215,11 +215,11 @@ AI extraction with a plain-language prompt or a JSON Schema (`--ai`, `--ai-schem

### How to take a screenshot or save a page as PDF

Screenshots and PDF output need the Chromium engine:
Screenshots and PDF output are rendered in a full browser, which the API picks for you:

```sh
spicrawl scrape https://example.com --engine chromium --screenshot --screenshot-full-page -o home.html
spicrawl scrape https://example.com/report --engine chromium --format pdf -o report.pdf
spicrawl scrape https://example.com --screenshot --screenshot-full-page -o home.html
spicrawl scrape https://example.com/report --format pdf -o report.pdf
```

Screenshots are saved next to the `-o` file as `<name>.<label>.<format>`, or into `--screenshot-dir`; `--screenshot-selector` and `--screenshot-format png|jpeg|webp` narrow them. PDF is never written to a terminal: pass `-o FILE`, or `--json` to get it base64-encoded.
Expand All @@ -239,10 +239,10 @@ Each batch item is one fetch or render, returned as `html` (the default), `markd

### How to keep cookies and logins across scrapes

A session keeps one engine and one cookie jar across many scrapes, so a login or a cart survives between requests:
A session keeps one rendering tier and one cookie jar across many scrapes, so a login or a cart survives between requests:

```sh
SID=$(spicrawl sessions create --engine chromium --ttl 7200 --json | jq -r .id)
SID=$(spicrawl sessions create --engine browser --ttl 7200 --json | jq -r .id)
spicrawl scrape https://example.com/account --session "$SID" --format markdown
spicrawl sessions release "$SID"
```
Expand Down Expand Up @@ -340,11 +340,11 @@ Up to 10,000 URLs, and 1 MiB of request body, per `spicrawl batch submit`. A job

### Does it render JavaScript?

Yes: add `--render` to run the page in a browser, and `--wait-for '<css selector>'` to wait for the content. `--engine chromium` pins real Chromium, which screenshots and PDF need.
Yes: add `--render` to run the page in a browser, and `--wait-for '<css selector>'` to wait for the content. Screenshots and PDF are rendered in a full browser automatically; you do not pick one.

### Is Spicrawl free?

Spicrawl is priced in credits: each successful request draws on your organization's monthly credit allowance. The [credits page](https://docs.spicrawl.com/credits) lists 1 credit for a plain fetch, 3 with `--render` and 8 with `--engine chromium`; failed requests cost 0, and a cache hit is billed at the price of the fetch that stored it. `spicrawl usage summary` shows the current period against your allowance.
Spicrawl is priced in credits: each successful request draws on your organization's monthly credit allowance. The [credits page](https://docs.spicrawl.com/credits) lists 1 credit for a plain fetch, 3 with `--render` and 8 for a screenshot or PDF (full-browser rendering); failed requests cost 0, and a cache hit is billed at the price of the fetch that stored it. `spicrawl usage summary` shows the current period against your allowance.

### Does it work with Claude Code, Cursor, Codex and VS Code?

Expand Down
Loading
Loading