diff --git a/docs/guides/extending_crawlee.mdx b/docs/guides/extending_crawlee.mdx new file mode 100644 index 0000000000..6e4a3d3977 --- /dev/null +++ b/docs/guides/extending_crawlee.mdx @@ -0,0 +1,191 @@ +--- +id: extending-crawlee +title: Extending Crawlee +description: The extension points Crawlee exposes, the contract each one defines, and how to choose between them. +--- + +import ApiLink from '@site/src/components/ApiLink'; + +Crawlee covers the common cases out of the box, but sooner or later you'll hit something it doesn't do: a parser for a format no built-in crawler understands, an HTTP backend your company mandates, a database you want your data in, or a browser that isn't launched through the standard Playwright path. Rather than forking, you can plug your own implementation into the layer that differs and keep everything else. + +This guide covers the extension points, the contract each one defines, and links to the guides that go deeper on each. If you maintain a third-party integration, you can build it against these contracts and host your own guide for it. + +## Extension points + +The four component families covered here contain the main extension points, and they're where most integrations plug in. They don't cover every extensible class in Crawlee. Other examples include `RequestLoader`, `FingerprintGenerator`, and `RenderingTypePredictor`. In the diagram, the `extension point` marker means a class you can extend or implement. + +```mermaid +--- +config: + class: + hideEmptyMembersBox: true +--- + +classDiagram + +%% ======================== +%% Crawlers +%% ======================== + +class BasicCrawler { + <> +} + +class AbstractHttpCrawler { + <> +} + +class AbstractHttpParser { + <> +} + +class PlaywrightCrawler { + <> +} + +class StagehandCrawler + +%% ======================== +%% HTTP clients +%% ======================== + +class HttpClient { + <> +} + +%% ======================== +%% Storage clients +%% ======================== + +class StorageClient { + <> +} + +class DatasetClient { + <> +} + +class KeyValueStoreClient { + <> +} + +class RequestQueueClient { + <> +} + +%% ======================== +%% Browsers +%% ======================== + +class BrowserPool + +class BrowserPlugin { + <> +} + +class BrowserController { + <> +} + +class PlaywrightBrowserPlugin { + <> +} + +%% ======================== +%% Relations +%% ======================== + +BasicCrawler --|> AbstractHttpCrawler +BasicCrawler --|> PlaywrightCrawler +AbstractHttpCrawler --> AbstractHttpParser : parses with +PlaywrightCrawler --|> StagehandCrawler +BasicCrawler --> HttpClient : uses +BasicCrawler --> StorageClient : uses +StorageClient --> DatasetClient : creates +StorageClient --> KeyValueStoreClient : creates +StorageClient --> RequestQueueClient : creates +PlaywrightCrawler --> BrowserPool : uses +BrowserPool --> BrowserPlugin : manages +BrowserPlugin --|> PlaywrightBrowserPlugin +BrowserPlugin --> BrowserController : new_browser() returns +``` + +### Crawlers + +A crawler drives the whole run. It takes requests from the queue, fetches each one, builds the context object your handler receives, and manages retries, concurrency, sessions, and storage along the way. `BasicCrawler` implements that orchestration and stays agnostic about how a page is fetched or parsed, which is what makes it the base every other crawler builds on. Extend it directly when a crawler has to fetch pages in a way neither the HTTP nor the browser layer covers. + +For HTTP-based crawling, `AbstractHttpCrawler` adds the fetch-and-parse layer. Its contract pairs a parser, a crawling context type, and a crawler class. The parser implements `AbstractHttpParser`: + +- `parse` turns an `HttpResponse` into your parsed type, and `parse_text` does the same for a string. +- `select` and `is_matching_selector` apply selectors. +- `find_links` extracts URLs for link enqueuing. + +The context exposes the parsed data to handlers, and the crawler ties the parser and context together. + +Browser crawlers use the same orchestration with a browser-backed context. Extend `PlaywrightCrawler` when an integration needs crawler-level browser behavior or a different handler context. `StagehandCrawler` is an example. It extends `PlaywrightCrawler` with a Stagehand-specific context and browser behavior. If only browser launch or lifecycle differs, a browser plugin is the narrower extension point. + +See the [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and the [Stagehand crawler guide](./stagehand-crawler) for `StagehandCrawler` itself. The [Architecture overview](./architecture-overview) explains how HTTP and browser crawlers relate to the other components. + +### HTTP clients + +An HTTP client performs network calls for crawlers. Swapping it changes the transport, including the TLS stack, connection pooling, proxy handling, and browser impersonation. It doesn't change how pages are parsed or how the crawl is orchestrated. + +The contract is `HttpClient`: + +- `crawl` performs a request inside the crawler's pipeline and returns the result the crawler consumes. +- `send_request` covers standalone calls made from a handler. +- `stream` yields a response you read incrementally. +- `cleanup` releases whatever the client holds open. + +Crawlee ships `ImpitHttpClient`, `HttpxHttpClient`, and `CurlImpersonateHttpClient`. + +See the [HTTP clients guide](./http-clients) for the libraries these clients wrap and their installation requirements. + +### Storage clients + +A storage client is the backend behind Crawlee's three storages. `Dataset`, `KeyValueStore`, and `RequestQueue` are the API you write against, and the storage client decides where that data actually lives. That separation is what lets you move a crawler from the local file system to a database or a cloud service without changing crawl code. + +The `StorageClient` contract defines three factory methods: `create_dataset_client`, `create_kvs_client`, and `create_rq_client`. The returned clients define the rest of the contract: + +- `DatasetClient` handles appending and reading items. +- `KeyValueStoreClient` handles record access and iteration. +- `RequestQueueClient` handles adding, fetching, and marking requests as handled. + +A custom backend implements all four classes. + +See the [Storage clients guide](./storage-clients) for the built-in implementations and a custom client example. + +### Browser plugins + +A browser plugin launches browsers for `PlaywrightCrawler`. The crawler delegates that work to `BrowserPool`. The pool initializes its plugins, forwards browser context options when creating pages, and manages each browser's lifecycle. + +The abstract contract is `BrowserPlugin`: + +- `new_browser` launches a browser and returns a `BrowserController`, which the pool uses to open pages and tear down the browser. +- `browser_launch_options` and `browser_new_context_options` hold the options the pool forwards. +- `browser_type` tells the pool what the plugin launches, and `max_open_pages_per_browser` is the per-browser page limit the plugin passes to the controllers it creates. The pool asks the controller for free capacity, not the plugin. +- `__aenter__` and `__aexit__` start and stop whatever the plugin holds open, and `active` reports whether the plugin is started. + +Implement this base contract directly when the launch and lifecycle are too specific for Crawlee's Playwright integration. A plugin that launches something other than a standard Playwright browser also needs its own `BrowserController` implementation. + +Most integrations should start with `PlaywrightBrowserPlugin`. Configure it when its launch and context options cover the required browser. Extend it when you need a custom Playwright-compatible launch path while preserving its standard lifecycle and context handling. + +See the [Playwright crawler guide](./playwright-crawler) for the responsibilities a subclass has to preserve, and the [Camoufox example](../examples/playwright-crawler-with-camoufox) for a complete integration. + +## Choosing an extension point + +Start with configuration before writing a subclass. You can parse a response with a third-party library inside an `HttpCrawler` handler, pass an existing `http_client` to any crawler, or configure `PlaywrightBrowserPlugin`. Use an extension contract only when configuration doesn't cover the required behavior. + +- If reusable HTTP parsing and the handler context both need to change, extend `AbstractHttpCrawler` and implement `AbstractHttpParser`. +- If browser-level orchestration or the handler context needs to change, extend `PlaywrightCrawler`. +- If the network transport needs to change while crawler behavior stays the same, implement `HttpClient` and pass it to the crawler. +- If the storage backend needs to change while the storage API stays the same, implement `StorageClient` and its three per-storage clients. +- If browser launch needs to change while the Playwright lifecycle stays the same, extend `PlaywrightBrowserPlugin`. Implement `BrowserPlugin` directly only when the Playwright plugin's launch and lifecycle contract needs a different implementation. + +When more than one fits, pick the narrowest. A custom HTTP client works with every HTTP crawler, so it's easier to maintain than a crawler subclass that hard-codes the same transport. + +## Conclusion + +Each extension point has a documented class contract, and everything built on top of it keeps working once you implement that contract. Crawlee changes those contracts only in a major release, which makes them the stable surface for a third-party integration and its documentation. + +If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!