http_scraper.py: the three modes with no browser

.github/ci_checks.py

@@ -34,7 +34,7 @@
 ENGINE_LIBS = ("playwright", "pyppeteer", "selenium", "webdriver_manager")
 
 CLIS = ["playwright_scraper.py", "puppeteer_scraper.py", "selenium_scraper.py",
-        "scraper_api_client.py", "fingerprint_client.py", "env_config.py",
+        "http_scraper.py", "scraper_api_client.py", "fingerprint_client.py", "env_config.py",
         "diff_runs.py"]
 
 # One real run per mode: three row classes, three samples.

.github/workflows/canary.yml

@@ -33,7 +33,7 @@ permissions:
 
 jobs:
   run:
-    name: ${{ matrix.mode }}
+    name: ${{ matrix.mode }} (${{ matrix.engine }})
     runs-on: ubuntu-latest
     timeout-minutes: 20
     strategy:
@@ -43,14 +43,24 @@ jobs:
           # USDT/EUR had ~190-200 adverts on 2026-09-24, so three pages of 20
           # are full; the floor leaves room for a thin morning.
           - mode: p2p
+            engine: playwright
             args: "--asset USDT --fiat EUR --side buy"
             floor: 40
           - mode: copytrading
+            engine: playwright
             args: "--mode copytrading --time-range 30D --sort-by roi"
             floor: 60
           - mode: announcements
+            engine: playwright
             args: "--mode announcements --category new-listings"
             floor: 100
+          # The browserless engine, so the README's "no browser needed" is
+          # under test every morning too. One mode: the loop is shared, and
+          # P2P is the POST that needs the most of it.
+          - mode: p2p
+            engine: http
+            args: "--asset USDT --fiat EUR --side buy"
+            floor: 40
     env:
       BINANCE_PROXY: ${{ secrets.BINANCE_PROXY }}
     steps:
@@ -62,16 +72,19 @@ jobs:
       - name: Install
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt -r requirements-playwright.txt
-          playwright install --with-deps chromium
+          pip install -r requirements.txt
+          if [ "${{ matrix.engine }}" = "playwright" ]; then
+            pip install -r requirements-playwright.txt
+            playwright install --with-deps chromium
+          fi
 
       # >= 3 pages, because with one page pagination is never exercised (§11).
       - name: Run
         id: run
         run: |
           set +e   # `run:` is `bash -e`; without this a non-zero exit aborts
                    # the step before the echo below records it.
-          python playwright_scraper.py ${{ matrix.args }} --pages 3 \
+          python ${{ matrix.engine }}_scraper.py ${{ matrix.args }} --pages 3 \
             --out canary --format both --dump-html canary_dump.json
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 
@@ -137,7 +150,7 @@ jobs:
       - uses: actions/upload-artifact@v4
         if: always()
         with:
-          name: canary-${{ matrix.mode }}
+          name: canary-${{ matrix.mode }}-${{ matrix.engine }}
           path: |
             canary.json
             canary.csv

CHANGELOG.md

@@ -7,6 +7,20 @@ toolkit can: a patch release means **fixes**, not that every flag and
 default is frozen. A default that changes behaviour for an existing user is
 said so at the top of its release notes.
 
+## [Unreleased]
+
+### Added
+
+- `http_scraper.py`: the three modes with no browser, through the same
+  fetch loop, output and exit codes. Same flags as the browser engines
+  minus the ones that describe a browser. It sends requests' own
+  User-Agent rather than a Chrome one over a non-Chrome TLS handshake.
+  Measured 2026-09-30: identical rows to Playwright in all three modes, in
+  about a third to a half of the time. If AWS WAF answers instead of an
+  endpoint it reports exit 3 straight away, without waiting for a
+  challenge script that nothing here can run.
+- The canary runs P2P through it daily, with no browser installed.
+
 ## [0.1.1] — 2026-09-30
 
 > **A moved payload shape is no longer reported as an empty listing.** A

README.md

@@ -5,7 +5,7 @@
 [![canary](https://github.com/2scraper/binance-scraper/actions/workflows/canary.yml/badge.svg)](https://github.com/2scraper/binance-scraper/actions/workflows/canary.yml)
 ![python](https://img.shields.io/badge/python-3.9%20%E2%80%93%203.14-blue)
 [![licence](https://img.shields.io/badge/licence-MIT-green)](LICENSE)
-![engines](https://img.shields.io/badge/engines-playwright%20%7C%20selenium%20%7C%20pyppeteer-lightgrey)
+![engines](https://img.shields.io/badge/engines-playwright%20%7C%20selenium%20%7C%20pyppeteer%20%7C%20http-lightgrey)
 ![runs without an account](https://img.shields.io/badge/all%20three%20modes-no%20account%20needed-brightgreen)
 
 Scrapes three things from [binance.com](https://www.binance.com) that its
@@ -43,6 +43,15 @@ one of those endpoints and issues every page as a same-origin `fetch()`, so
 no page is ever rendered. A full USDT/EUR buy-side order book, 10 pages,
 came back as **196 of 196 adverts** in 13 seconds with nothing configured.
 
+Which also means the data needs **no browser at all**. `http_scraper.py`
+sends the same requests with `requests` and nothing else, through the same
+loop, into the same files. Measured 2026-09-30, one 3-page run per mode
+from the same VPS: P2P 3.0 s against Playwright's 5.2 s, copy-trading 3.4 s
+against 8.8 s, announcements 2.7 s against 9.1 s, with the same rows in the
+same order. What the browser buys is the day the WAF moves in front of the
+endpoints: then only a browser engine can run its challenge or have its
+CAPTCHA solved, and the HTTP engine says so and exits 3.
+
 What the paid products buy here is insurance and scale, and the section
 below says exactly which one does what.
 
@@ -59,6 +68,14 @@ python3 -m venv venv
 Install **one** engine per virtualenv: the three libraries pin versions of
 their dependencies that cannot all be satisfied at once.
 
+Or no browser at all, for the HTTP engine:
+
+```bash
+python3 -m venv venv
+./venv/bin/pip install -r requirements.txt
+./venv/bin/python http_scraper.py --asset USDT --fiat EUR --pages 3
+```
+
 ## Run
 
 ```bash
@@ -172,11 +189,12 @@ retried, and the exit code is 5 ("the data never arrived"), not 3.
 | `playwright_scraper.py` | **Primary.** Authenticates a proxy and a remote CDP endpoint. |
 | `puppeteer_scraper.py` | pyppeteer is effectively unmaintained; here for parity. Authenticates a proxy and a CDP endpoint. `--chromium-path` points it at another browser if its own will not start. |
 | `selenium_scraper.py` | Drives the Chrome you already have. **Cannot authenticate a proxy or a remote CDP endpoint** (`debuggerAddress` is a bare `host:port`), so it refuses a credentialled `--cdp-endpoint` with exit 2. None of the modes needs either. |
+| `http_scraper.py` | **No browser**: `requests` only, the same flags minus the ones that describe a browser (`--headless`, `--cdp-endpoint`, `--fingerprint*`, `--locale`, the solver's). Proxies and rotation work. Cannot pass AWS WAF: if it answers, the run is exit 3 at once, naming the engine that can. |
 | `scraper_api_client.py` | No local browser: the 2Captcha Scraper API fetches the page. Reads `--mode announcements` only, the one endpoint with a URL. The P2P and copy-trading endpoints answer POST, and **this repo does not implement a POST through the Scraper API**. Measured: 2 pages, 100 rows, $0.0005 a page. |
 
 The fetch loop itself (landing, the WAF, retries, throttling, rotation,
-parsing) is one implementation in `page_flow.py` that all three browser
-engines drive, so they cannot disagree about a page. Each was run live on
+parsing) is one implementation in `page_flow.py` that all four engines
+drive, so they cannot disagree about a page. Each was run live on
 2026-09-24 through the same seven scenarios (P2P buy and sell, copy-trading,
 announcements, `--concurrency`, a refused `--pay-type`, `--url`) with the
 same results.