http_scraper.py: the three modes with no browser

[jehrr]

Option B from the 2026-09-30 audit ("check whether Chromium is needed on the normal path"). The audit was right: the three data endpoints are ungated, so the browser is insurance for the day AWS WAF moves in front of them, not something the normal path needs. This PR adds a fourth engine that takes the browser away and keeps everything else.

What it is

http_scraper.py runs the same page_flow loop over requests. It produces the same output files, sidecar and exit codes.

• Flags: the browser engines' set minus a named list of flags that describe a browser (--headless/--headful, --cdp-endpoint, --fingerprint*, --locale, --twocaptcha-key, --captcha-api, --solve-captcha, --min-score). The suite pins the set both ways, so that list is the documentation.
• Proxies and rotation: they work, and a rotation is a fresh session (§8).
• Identity: it sends requests' own User-Agent, not a Chrome one over a TLS handshake that is not Chrome's. §24 records a site refusing exactly that contradiction. Here P2P answered both identities 6 of 6.
• AWS WAF: it cannot pass it. If the WAF answers instead of an endpoint, the run ends with exit 3 at once and the log names the engine that can get through.

Measured live, 2026-09-30, same VPS, one 3-page run per mode

mode	http	playwright	rows
p2p USDT/EUR	3.0 s	5.2 s	60 = 60, same order, identical
copytrading	3.4 s	8.8 s	90 = 90, same order. One row differed: one portfolio's AUM and copier PnL changed in the seconds between the two runs
announcements	2.7 s	9.1 s	150 = 150, identical

All six runs were complete, exit 0, with core_field_shortfall: {}.

A shared-loop change a control found

When landing on a WAF challenge, land() waited up to 15 s for the challenge script to run. With no browser, nothing runs it. Worse, the poll re-read the empty 202 body without its status, classified it unknown, and a plain block (exit 3) became "never arrived" (exit 5).

The engine now declares runs_scripts = False and the wait is skipped. Browser engines are unchanged: they default to True.

The first version of the test for this asserted "waited under 5 s" and stayed green against the fault, because the loop left after one 1-s poll. It now asserts no wait at all.

Evidence

• Suite: 389 checks pass in a core-only venv and 394 with Playwright installed. The HTTP engine needs only requests, so its group never skips.
• It is now included in the operations-from-AST, signature-binding, credential-masking, module-level-import and banned-flag checks. A new check drives it through the loop with requests stubbed: served → 0; WAF challenge → 3 with no wait; dead proxy → 5 with the password masked and the exit kept; announcements GET → 0.
• Six controls, run on a copy with no bytecode cache. Each went red on the check named for it, and none crashed:
  • runs_scripts True;
  • the loop ignoring runs_scripts;
  • no timeout;
  • unmasked error text;
  • an added --locale;
  • proxy failure not recognised.
• The canary gets a fourth matrix row: P2P through http_scraper.py, with no browser installed.

Not changed

• Playwright stays the primary engine and the Docker image's entrypoint.
• The HTTP engine is not in the image. The image is built around Chromium, and a browserless run needs no image beyond pip install -r requirements.txt.
• The shared loop's refusal advice still reads browser-first ("re-fetching from a fresh browser", --cdp-endpoint). The HTTP engine adds its own line naming the engine that can pass the WAF.

🤖 Generated with Claude Code