Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 181 additions & 40 deletions README.md

Large diffs are not rendered by default.

12 changes: 9 additions & 3 deletions skills/firecrawl-scrape/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
name: firecrawl-scrape
description: |
Extract a URL's content as clean markdown, including JS-rendered pages. Use whenever the user provides a URL and wants its content; prefer over WebFetch.
Extract a URL's content as clean markdown, including JS-rendered pages, or execute a known Alexandria data-provider capability. Use when the user supplies a page URL or has selected a provider tool and wants its data.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl scrape

Scrape one or more URLs. Returns clean, LLM-optimized markdown. Multiple URLs are scraped concurrently.
Scrape one or more URLs, or execute a selected Alexandria capability with `--alexandria`. Page scraping returns content; Alexandria execution returns structured provider results.

## Quick start

Expand All @@ -35,7 +35,13 @@ firecrawl scrape "https://example.com/pricing" --query "What is the enterprise p

Run `firecrawl scrape --help` for the full option list.

**Done when:** you have the scraped content — on stdout, in your `-o` file, or under `.firecrawl/` for multi-URL scrapes — and have inspected it with bounded reads (`head`, `grep`) to answer the request.
## Alexandria provider execution

Use `firecrawl scrape --alexandria <provider>/<capability> --options '<JSON>'` after reading the tool's contract. This is a URL-less request; do not combine it with page URLs or page-scraping options. Inspect each result in `data.alexandria`, including per-item errors and `data.creditsCost` for the total charge.

Keep the returned request ID. An identical retry uses `--request-id <same-id>`; a pending or uncertain execution must not be retried under a fresh ID. See the [Alexandria workflow](../firecrawl/rules/alexandria.md) for discovery, required inputs, and a complete example.

**Done when:** you have inspected the page content or provider results with bounded reads and used them to answer the request. Report per-item provider failures instead of treating the outer response as proof of success.

## Tips

Expand Down
28 changes: 22 additions & 6 deletions skills/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
name: firecrawl-search
description: |
Web search with full page content. Use when no URL is known: finding sources, articles, or news. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.
Web search with optional page content and Alexandria tool discovery. Use when no URL is known: finding sources, articles, news, or data-provider tools by intent. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl search

Web search with optional content scraping. Returns search results as JSON, optionally with full page content.
Web search with optional content scraping and Alexandria tool discovery. JSON groups web results under `data.web` and tool contracts under `data.tools`.

## Quick start

Expand All @@ -28,7 +28,23 @@ Run `firecrawl search --help` for the full option list.

`--categories developer` weighs the developer index beside ordinary web results in this same call (no passage control, no index filters). `--categories research` is a website filter, not the paper index. Dedicated skills: [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) and [firecrawl-research-index](../firecrawl-research-index/SKILL.md).

**Done when:** results are saved under `.firecrawl/`, verified non-empty, processed for the request, and one feedback event is sent within the time window (unless opted out).
## Alexandria tools

```bash
# Find tool contracts by meaning
firecrawl search "podcast conversations about AI agents" \
--sources alexandria --limit 2 --json -o .firecrawl/tools.json

# Web pages and semantic tools, with optional domain matches
firecrawl search "podcast conversations about AI agents" \
--sources web,alexandria --domain-tools --limit 2 --json -o .firecrawl/web-and-tools.json
```

Search always needs a non-empty query. `--sources alexandria` searches tools semantically; `--domain-tools` opts into additional contextual matches. Both return contracts in `data.tools`, with `matchedBy` and `matchedUrls` explaining relevance. Omit `--domain-tools` when semantic matches are enough. Discovery does not execute the returned tools.

For a known URL or provider, use `firecrawl find-tools`; catalogue filters belong there. Follow the [Alexandria workflow](../firecrawl/rules/alexandria.md) to read contracts, follow `next`, and execute a selected tool through `scrape --alexandria`.

**Done when:** fresh results have been inspected and used for the request, or no matches have been reported. Apply the feedback guidance below when web results were used; free tool discovery does not require a web-search feedback call.

## Tips

Expand All @@ -38,9 +54,9 @@ Run `firecrawl search --help` for the full option list.
- Use `jq` to extract URLs or titles: `jq -r '.data.web[].url' .firecrawl/search.json`
- Naming convention: `.firecrawl/search-{query}.json` or `.firecrawl/search-{query}-scraped.json`

## Send feedback after every search (refunds 1 credit)
## Send feedback after using web search results

Search costs 2 credits. After you've actually used the results (or decided they were useless), send structured feedback with `firecrawl search-feedback <id>`. The first feedback per search refunds 1 credit and helps us improve search quality. **Do this once per search**, in the background, after you finish processing the results — it does not block your main task.
For ordinary web search, after you've actually used the results (or decided they were useless), send structured feedback with `firecrawl search-feedback <id>`. Eligible feedback can refund 1 credit and helps us improve search quality. **Do this once per search**, in the background, after you finish processing the results. Alexandria-only discovery and Find Tools cost 0 credits; do not apply this refund workflow to them. For mixed results, feedback should describe the web results you used.

**Opt out:** if `FIRECRAWL_NO_SEARCH_FEEDBACK=1` (or `FIRECRAWL_DISABLE_SEARCH_FEEDBACK=1`) is set, the CLI silently skips the call and never sends anything. Respect that — do not try to work around it. (Team admins can also disable this server-side; the API will return `feedbackErrorCode: "TEAM_OPTED_OUT"` and the CLI will exit 0 silently.)

Expand All @@ -62,7 +78,7 @@ Verify the search returned results before reading its `id`. Zero-result searches
# Send once per search. Rate honestly and replace the placeholder with the
# rating that matches what actually happened. The two fields shown
# satisfy the substantive-content rule for every rating.
if SEARCH_ID=$(jq -er 'select(any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json); then
if SEARCH_ID=$(jq -er 'select(any(.data | .web, .images, .news, .developer; length > 0)) | .id // empty' .firecrawl/search-react-hooks.json); then
firecrawl search-feedback "$SEARCH_ID" \
--rating "<good|partial|bad>" \
--valuable-sources '[{"url":"https://react.dev/reference/react/hooks","reason":"Most authoritative"}]' \
Expand Down
36 changes: 20 additions & 16 deletions skills/firecrawl/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
name: firecrawl
description: |
Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login — web only; local files route to firecrawl-parse. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.
Live web data and Alexandria tools via the Firecrawl CLI: web research, reading or extracting pages, discovering data-provider tools, retrieving provider data, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login. Local files route to firecrawl-parse. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl-cli *)
---

# Firecrawl CLI

Search, scrape, and interact with the web. Returns clean markdown optimized for LLM context windows.
Search, scrape, and interact with the web, or discover and execute Alexandria data tools. Returns page content, tool contracts, or structured provider data.

Run `firecrawl --help` or `firecrawl <command> --help` for full option details. For app integration or outcome workflows (research briefs, SEO audits, etc.), route to the `firecrawl-build` / `firecrawl-workflows` skills — see [When to Load References](#when-to-load-references).

Expand All @@ -30,19 +30,22 @@ Follow this escalation pattern:
5. **Monitor** - Need recurring checks or ongoing alerts. Prefer setting a monitor with `--page` plus `--goal` instead of doing repeated one-off scrapes.
6. **Interact** - Scrape first, then interact with the page (pagination, modals, form submissions, multi-step navigation).

| Need | Command | When |
| --------------------------- | --------------------- | --------------------------------------------------------------- |
| Find pages on a topic | `search` | No specific URL yet |
| Find research papers | `research` | Biomedical/clinical/scientific literature — use the paper index |
| Answer a coding question | `developer` | Issues, merged PRs, READMEs, and docs — not a general web page |
| Get a page's content | `scrape` | Have a URL, page is static or JS-rendered |
| Find URLs within a site | `map` | Need to locate a specific subpage |
| Bulk extract a site section | `crawl` | Need many pages (e.g., all /docs/) |
| AI-powered data extraction | `agent` | Need structured data from complex sites |
| Interact with a page | `scrape` + `interact` | Content requires clicks, form fills, pagination, or login |
| Download a site to files | `x download` | Save an entire site as local files |
| Parse a local file | `parse` | File on disk (PDF, DOCX, XLSX, etc.) — not a URL |
| Watch pages for changes | `monitor` | Schedule recurring scrapes/crawls, diff against snapshots |
| Need | Command | When |
| -------------------------------- | ----------------------------- | --------------------------------------------------------------- |
| Find pages on a topic | `search` | No specific URL yet |
| Find data tools by intent | `search --sources alexandria` | Describe the data needed; returns tool contracts |
| Explore tools by URL or provider | `find-tools` | Contextual lookup and progressive disclosure |
| Execute a known data tool | `scrape --alexandria` | Have a provider, capability, and its required inputs |
| Find research papers | `research` | Biomedical/clinical/scientific literature — use the paper index |
| Answer a coding question | `developer` | Issues, merged PRs, READMEs, and docs — not a general web page |
| Get a page's content | `scrape` | Have a URL, page is static or JS-rendered |
| Find URLs within a site | `map` | Need to locate a specific subpage |
| Bulk extract a site section | `crawl` | Need many pages (e.g., all /docs/) |
| AI-powered data extraction | `agent` | Need structured data from complex sites |
| Interact with a page | `scrape` + `interact` | Content requires clicks, form fills, pagination, or login |
| Download a site to files | `x download` | Save an entire site as local files |
| Parse a local file | `parse` | File on disk (PDF, DOCX, XLSX, etc.) — not a URL |
| Watch pages for changes | `monitor` | Schedule recurring scrapes/crawls, diff against snapshots |

For detailed command reference, run `firecrawl <command> --help`.

Expand All @@ -64,6 +67,7 @@ For detailed command reference, run `firecrawl <command> --help`.
## When to Load References

- **Searching the web or finding sources first** -> [firecrawl-search](../firecrawl-search/SKILL.md)
- **Discovering Alexandria tools, walking providers and capabilities, or executing provider data requests** -> [rules/alexandria.md](rules/alexandria.md). Semantic search and optional domain matches share `data.tools`; Find Tools reveals contracts without executing the selected provider.
- **Finding research papers (biomedical, clinical, or scientific literature; PubMed, bioRxiv, medRxiv, arXiv)** -> [firecrawl-research-index](../firecrawl-research-index/SKILL.md). Use the paper index instead of scraping PubMed or Google Scholar by hand; `search --categories research` is a website filter, not the paper index.
- **Answering a library, API, error, or known-bug question from issues, merged PRs, READMEs, or docs** -> [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md)
- **Scraping a known URL** -> [firecrawl-scrape](../firecrawl-scrape/SKILL.md)
Expand Down Expand Up @@ -107,7 +111,7 @@ Single format outputs raw content. Multiple formats (e.g., `--format markdown,li

## Feedback

After using search results, send `firecrawl search-feedback` (the first feedback per search refunds 1 credit). The full pattern, guard, and rules live in [firecrawl-search](../firecrawl-search/SKILL.md).
After using web search results, follow the feedback guidance in [firecrawl-search](../firecrawl-search/SKILL.md). Free Alexandria-only discovery and Find Tools do not earn a web-search credit refund.

For non-search endpoint jobs, use `firecrawl feedback <endpoint> <jobId>` to send concise job-level feedback through `/v2/feedback`. Supported endpoints are `search`, `scrape`, `parse`, and `map`.

Expand Down
62 changes: 62 additions & 0 deletions skills/firecrawl/rules/alexandria.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Alexandria: discover a tool, read its contract, get its data

Use Alexandria when the task needs a provider's structured data. These commands require a CLI and API deployment with Alexandria support and an authenticated team with access. Check `firecrawl find-tools --help` for the installed command surface.

## Choose the discovery request

- **Describe the data needed:** `firecrawl search "your intent" --sources alexandria --limit 2 --json`. Add `web` to the sources to also retrieve web results. Search requires a query.
- **Match tools to result domains:** add `--domain-tools` to Search. This is optional and can add lookup latency. Semantic and contextual matches share `data.tools`.
- **Already have a URL or provider:** use `find-tools`, without a search query. It looks up the catalogue; it does not fetch the supplied page or execute the tools it finds.

Inspect a Search tool's `provider`, `capability`, `options`, `requiresOneOf`, `response`, `examples`, `creditsCost`, and `perRecord`. `matchedBy` and `matchedUrls` explain why it appeared. A discovery `warning` means lookup was unavailable, rather than proving no tools match. A zero-result CLI search may leave no new output file; do not read a stale file from a previous run.

## Reveal only the detail needed

```bash
# Start with a provider; lookup depth is inferred
firecrawl find-tools --providers particle --limit 2 --json \
-o .firecrawl/find-tools.json

# Or start from a page returned by search or scrape
firecrawl find-tools "https://podcasts.apple.com" --limit 2 --json \
-o .firecrawl/page-tools.json

# Read the inputs, response, and examples for a known capability
firecrawl find-tools --providers particle \
--capabilities podcasts/episodes/search \
--expand options,response,examples --limit 2 --json \
-o .firecrawl/episode-contract.json
```

Find Tools returns its catalogue page inside `data.alexandria[0].data`:

```bash
jq '.data.alexandria[0].data | {level, items, next}' .firecrawl/find-tools.json
```

Follow the selected item's `next` to reveal more detail. Follow the catalogue page's top-level `next` for another page. Pass the complete returned request unchanged:

```bash
firecrawl find-tools --request '<next request JSON>' --json \
-o .firecrawl/find-tools-next.json
```

`--request` cannot be combined with URLs or lookup filters. Provider, category, group, and capability selectors are optional; add only those needed to narrow the lookup. If necessary, `--level providers|groups|tools` overrides the inferred depth. Reuse contracts already present in Search results instead of fetching them again.

## Execute the selected capability

Use the inputs from the current contract. For Particle episode search, `keyword_search` supplies the exact words to find:

```bash
firecrawl scrape --alexandria particle/podcasts/episodes/search \
--options '{"keyword_search":"AI agents","limit":2}' \
--json -o .firecrawl/episodes.json
```

Inspect `data.alexandria` for provider results and per-item errors; an overall successful response can contain a failed item. `data.creditsCost` is the total charge. Tool discovery and Find Tools are free. Web search retains its own cost, and executing a selected provider uses its published price, including per-record pricing where specified. Browsing a contract does not accept provider terms or execute the provider.

The CLI generates a request ID and prints it on stderr, including on failure; successful JSON output also includes `requestId`. Retry an identical payload with `--request-id <same-id>`. If execution is pending or uncertain, report that state rather than creating a new ID to trigger another execution. Credit and provider-terms rejections must be resolved before execution can proceed.

## Provider terms (THIRD_PARTY_DATA_TERMS_REQUIRED)

Paid providers need an organization admin to accept their terms once. Until then, `--json` execution writes `{"success":false,"code":"THIRD_PARTY_DATA_TERMS_REQUIRED","requiresAction":{"type":"accept_terms","terms":"<provider>","version":"...","url":"..."}}` to stdout and exits 1; no credits are charged. When this happens, show the user `requiresAction.url` and stop. Do not retry, and do not try to accept: acceptance is a legal act by a human. A human admin accepts in the dashboard at that URL, or reads and accepts from their own terminal with `firecrawl alexandria terms <provider>` and `firecrawl alexandria terms accept <provider>` (interactive only, needs the admin's own API key). Once they confirm, rerun the identical command with the same `--request-id`.
Loading
Loading