feat(playwright-crawler): add `BrowserFetchHttpClient` by noahg9 · Pull Request #4177 · apify/crawlee

noahg9 · 2026-09-30T13:46:00Z

Some sites block HTTP clients but not a logged-in browser, so people currently hand-write a crawler loop inside a PlaywrightCrawler handler to reuse the browser's session. This adds an HTTP client that sends each request with fetch() from pages of a Playwright browser context you have already set up. Used as the httpClient of CheerioCrawler, requests get the browser's cookies and TLS fingerprint, and the crawler keeps its queue, retries and statistics.

it lives in @crawlee/playwright and takes a BrowserContext, so @crawlee/http-client doesn't depend on Playwright; the playwright peer dependency is now >=1.52.0
each origin gets its own page, opened on an empty document of that origin without loading the site, and its requests run concurrently there; up to maxOpenPages (20 by default) stay open, and the least recently used idle ones are closed
a 401 fails the request like other blocked statuses, so errorHandler can log in again once for all of them before they are retried (the docs example shows how)
aborting a request (e.g. on a handler timeout) also cancels it inside the page, so timed-out requests don't pile up there
proxyUrl, fingerprint and ignoreTlsErrors are ignored, since the browser decides them, and redirects to another origin fail, as the browser treats them as cross-origin requests
the HTTP clients guide has a new section with an example and the limits, and the public API report is updated

Closes #4145

An HTTP client that sends requests with `fetch()` inside a logged-in Playwright page, so HTTP crawlers can use the browser's session. Closes apify#4145

The client needs a Playwright page to work, and only Playwright's `Page` satisfies the structural `BrowserFetchPage` interface, so it belongs next to the other Playwright code rather than in the dependency-free `@crawlee/http-client`. It is now also exported from `crawlee`.

The client now takes a Playwright `BrowserContext` instead of a page, and opens one page per origin on an empty document served by a route. Any origin can be fetched with the cookies of the context without loading the site, and a login in another page of the context no longer destroys the requests in flight. The `login` callback is gone. A 401 already throws a `SessionError` and `errorHandler` runs before the request is retried, so that is the place to log in again.

…ient The example passes a browser context and logs in again from `errorHandler` on a 401, and the same-origin limit is gone from the guide. It also mentions that the browser sends only a few HTTP/1.1 requests to one host at a time.

Close a page that crashes instead of leaving it open next to the one that replaces it. The docs now say that redirects to another origin fail, as the browser treats them as cross-origin requests, and that the 401 handling relies on the default `blockedStatusCodes`.

Since Playwright 1.52, `evaluate` serializes typed arrays in both directions, so the client no longer base64-encodes request and response bodies itself. The `playwright` peer dependency of `@crawlee/playwright` is raised to `^1.52.0` accordingly, as older versions would mangle the bodies.

The new `maxOpenPages` option (20 by default, like `maxOpenPagesPerBrowser`) caps the pages the client keeps open. Once there are more, the least recently used ones are closed. Pages with requests in flight are never closed, as that would fail the requests, so the limit can be exceeded while more origins are fetched at once.

`#fetchInPage` now takes the `Request` and returns the `Response`, which drops the intermediate request and response shapes. The abort and the in-flight page tests wait for the held request to reach the server instead of relying on fixed delays.

…andler

barjin · 2026-10-05T08:58:29Z

cc @janbuchar I hijacked this PR and shaped it in a slightly different direction.

Would you mind reviewing this, as I am biased now? 🙏

janbuchar · 2026-10-06T10:23:02Z

Thanks @noahg9 and @barjin! Two initial observations:

the PR description is stale
crawlee-python has a PlaywrightHttpClient, which uses playwright's APIRequestContext to implement a default HTTP client for PlaywrightCrawler — a different use case, but we should take extra care with naming so that we don't confuse users too much when we add a JS counterpart of it further down the road.

janbuchar · 2026-10-07T08:57:10Z

@barjin Another high-level question — how does this integrate with BrowserPool and SessionPool? If I want to rotate sessions and open a new browser if my login routine stops working, how painful will that be to set up?

janbuchar

I like the general shape of this, but we should think about the important details of how this fits in with the rest of the framework.

janbuchar · 2026-10-07T08:59:21Z

+
+<CodeBlock language="ts">{CheerioBrowserFetchExample}</CodeBlock>
+
+The client opens one page per origin on an empty document of that origin, without loading the site, so URLs of any origin can be fetched with the cookies of the context. It keeps up to `maxOpenPages` of them open (20 by default) and closes the least recently used idle ones. With the default `blockedStatusCodes`, a `401` response fails the request, and `errorHandler` runs before it is retried, so log in again there, in a new page of the context. Keep in mind that:


one page per origin

What exactly does "origin" mean in this context?

janbuchar · 2026-10-07T09:01:28Z

+- the browser uses its own cookies and hides the `Cookie` and `Set-Cookie` headers, so the session's cookie jar is not used,
+- the `proxyUrl`, `fingerprint` and `ignoreTlsErrors` options are ignored, as the browser context decides them,


I understand why that happens, but can I beat the browser into the shape dictated by my session? The cookie jar discrepancy sounds like a real footgun...

janbuchar · 2026-10-07T09:02:56Z

+- the browser uses its own cookies and hides the `Cookie` and `Set-Cookie` headers, so the session's cookie jar is not used,
+- the `proxyUrl`, `fingerprint` and `ignoreTlsErrors` options are ignored, as the browser context decides them,


Worth mentioning you probably want to adjust your concurrency settings to match that? Or add a load signal to your concurrency system so that you don't try to scale past the limit?

janbuchar · 2026-10-07T09:13:54Z

+    let server: Server;
+    let otherPortServer: Server;
+    let baseUrl: string;
+    let otherUrl: string;
+    let otherPortUrl: string;
+    let browser: Browser;
+    let context: BrowserContext;
+    let logLevel: number;
+
+    const sessions = new Set<string>();
+    let sessionCounter = 0;
+    let logoutAfterHits = Infinity;
+    let protectedHits = 0;
+    let inFlight = 0;
+    let maxInFlight = 0;
+    let flakyHits = 0;
+    let hangReached: PromiseWithResolvers<void>;
+    let hangReleased: PromiseWithResolvers<void>;
+    let hangClosed: boolean;
+    const requestedPaths: string[] = [];
+    const allBytes = Buffer.from(Array.from({ length: 256 }, (_, i) => i));


This would benefit from 1. structure and 2. documentation

janbuchar · 2026-10-07T09:48:27Z

+    test('CheerioCrawler crawls pages that need the session of the context', async () => {
+        const titles: string[] = [];
+        const crawler = new CheerioCrawler({
+            httpClient: new BrowserFetchHttpClient({ context }),
+            maxRequestRetries: 1,
+            requestHandler: async ({ $ }) => {
+                titles.push($('title').text());
+            },
+        });
+
+        const stats = await crawler.run([
+            ...Array.from({ length: 8 }, (_, i) => `${baseUrl}/private/${i}`),
+            `${baseUrl}/flaky`,
+        ]);
+
+        expect(titles.sort()).toEqual(['Flaky', ...Array.from({ length: 8 }, (_, i) => `Private ${i}`)].sort());
+        expect(stats.requestsSucceeded).toBe(9);
+        expect(stats.requestsFailed).toBe(0);
+        expect(stats.retryHistogram).toEqual([8, 1]);
+    });
+
+    test('runs requests concurrently on one page', async () => {
+        const crawler = new CheerioCrawler({
+            httpClient: new BrowserFetchHttpClient({ context }),
+            minConcurrency: 4,
+            maxConcurrency: 4,
+            requestHandler: async () => {},
+        });
+
+        const stats = await crawler.run(Array.from({ length: 8 }, (_, i) => `${baseUrl}/private/${i}?delay=300`));
+
+        expect(maxInFlight).toBe(4);
+        expect(stats.requestsSucceeded).toBe(8);
+        expect(stats.requestsFailed).toBe(0);
+    });
+
+    test('lets errorHandler log in again when the session is lost while requests are in flight', async () => {
+        let logins = 0;
+        let relogin: Promise<void> | undefined;
+        const crawler = new CheerioCrawler({
+            httpClient: new BrowserFetchHttpClient({ context }),
+            minConcurrency: 4,
+            maxConcurrency: 4,
+            maxRequestRetries: 1,
+            errorHandler: async ({ response }) => {
+                if (response?.status !== 401) return;
+                if (!relogin) {
+                    logins++;
+                    relogin = login().finally(() => {
+                        relogin = undefined;
+                    });
+                }
+                await relogin;
+            },
+            requestHandler: async () => {},
+        });
+
+        logoutAfterHits = 3;
+        const stats = await crawler.run(Array.from({ length: 12 }, (_, i) => `${baseUrl}/private/${i}?delay=200`));
+
+        expect(logins).toBe(1);
+        expect(stats.requestsSucceeded).toBe(12);
+        expect(stats.requestsFailed).toBe(0);
+    });


I have mixed feelings about using CheerioCrawler as a test harness for the http client, basically. Some of the tests are not even that relevant to the client itself.

feat(http-client): add BrowserFetchHttpClient

a78537e

An HTTP client that sends requests with `fetch()` inside a logged-in Playwright page, so HTTP crawlers can use the browser's session. Closes apify#4145

barjin self-requested a review October 1, 2026 08:05

barjin added 10 commits October 1, 2026 11:47

chore: accept any Playwright version from 1.52 on in @crawlee/playwright

c0f8e5a

chore: drop the empty icon from the BrowserFetchHttpClient page document

e3abfc6

chore: drop the void operator from the BrowserFetchHttpClient crash h…

0de5105

…andler

barjin requested a review from janbuchar October 1, 2026 13:25

barjin changed the title ~~feat(http-client): add BrowserFetchHttpClient~~ feat(playwright-crawler): add BrowserFetchHttpClient Oct 2, 2026

Merge branch 'master' into feat/browser-fetch-http-client

3019b18

janbuchar reviewed Oct 7, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(playwright-crawler): add `BrowserFetchHttpClient` - #4177

feat(playwright-crawler): add `BrowserFetchHttpClient`#4177
noahg9 wants to merge 12 commits into
apify:masterfrom
noahg9:feat/browser-fetch-http-client

noahg9 commented Sep 30, 2026 •

edited by barjin

Loading

Uh oh!

barjin commented Oct 5, 2026

Uh oh!

janbuchar commented Oct 6, 2026

Uh oh!

janbuchar commented Oct 7, 2026

Uh oh!

janbuchar left a comment

Uh oh!

janbuchar Oct 7, 2026

Uh oh!

janbuchar Oct 7, 2026

Uh oh!

janbuchar Oct 7, 2026

Uh oh!

janbuchar Oct 7, 2026

Uh oh!

janbuchar Oct 7, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		<CodeBlock language="ts">{CheerioBrowserFetchExample}</CodeBlock>

		The client opens one page per origin on an empty document of that origin, without loading the site, so URLs of any origin can be fetched with the cookies of the context. It keeps up to `maxOpenPages` of them open (20 by default) and closes the least recently used idle ones. With the default `blockedStatusCodes`, a `401` response fails the request, and `errorHandler` runs before it is retried, so log in again there, in a new page of the context. Keep in mind that:

		- the browser uses its own cookies and hides the `Cookie` and `Set-Cookie` headers, so the session's cookie jar is not used,
		- the `proxyUrl`, `fingerprint` and `ignoreTlsErrors` options are ignored, as the browser context decides them,

Conversation

noahg9 commented Sep 30, 2026 • edited by barjin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

barjin commented Oct 5, 2026

Uh oh!

janbuchar commented Oct 6, 2026

Uh oh!

janbuchar commented Oct 7, 2026

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

noahg9 commented Sep 30, 2026 •

edited by barjin

Loading