Repository navigation
Low-level primitives: RawConnection + blasthttp.h2 (+ fix #15) - #16
Merged
Merged
Conversation
Splits dispatch_raw into two pieces. connect_stream owns the TCP + TLS setup and returns a Box<dyn IoReadWrite + Send + Unpin>. dispatch_direct keeps the rest: hyper handshake, request building, response parsing. Semantically, dispatch_direct is a better name for what the function actually does (one-shot HTTP request over a fresh, un-pooled connection via hyper's low-level SendRequest). The new name clears the word "raw" for a forthcoming RawConnection primitive that exposes a long-lived unframed handle to Python callers. No behavior change. Rust unit tests, Python binding tests, and BBOT's excavate + lightfuzz test suites all pass against the refactored code.
Adds a long-lived handle to a TCP or TLS stream that bypasses HTTP framing entirely. The caller writes and reads arbitrary bytes: no framing applied, no validation, no normalization, no decoding. Use this when the desired bytes on the wire are something hyper refuses to construct — deliberately malformed framing for smuggling tests, pipelined requests on one socket, raw-protocol probes, etc. Rust API (src/client/raw.rs): - RawConnection::connect(url, config) -> Result<Self> - send_bytes(data) - read_raw(max_bytes, timeout_ms) - close() - cert_info() for HTTPS connections TLS comes for free via the connect_stream helper we already extracted (scheme in the URL picks TCP vs TLS). TLS settings reuse the existing RequestConfig surface: verify_certs, cipher_string, min/max_tls_version, resolve_ip. Python bindings (src/python.rs): - BlastHTTP.raw_connect(url, ...) -> RawConnection (async) - RawConnection.send_bytes(bytes) (async) - RawConnection.read_raw(max_bytes, timeout_ms=None) (async) - RawConnection.close() (async) - RawConnection.cert_info (getter) raw_connect is a method on BlastHTTP so that opening a raw connection consumes one token from the client's rate limiter when set (same semantics as normal request(), request_batch()). Visibility: connect_stream and IoReadWrite in hyper.rs are now pub(crate) so raw.rs can call them. No public API change. Tests: - 10 new Rust unit tests in src/client/raw.rs exercising a local asyncio TCP echo server (roundtrip, timeout, close semantics, peer-close, invalid URL, unreachable address, idempotent close, zero max_bytes, multiple sends). - Matching Python-side tests in the loose test_python_bindings.py suite (ignored in-tree, run manually). All green: 69 Rust unit tests, 87 Python binding tests, and BBOT's excavate + lightfuzz module tests (88) all pass against this change. A walking-skeleton integration test reproduces a known CL.TE request-smuggling scenario end-to-end through the new primitive.
Creates a proper tracked Python test suite under tests/python/ using pytest + pytest-asyncio. The existing test_python_bindings.py (gitignored by a prior commit) remains in place for local-only smoke testing; its tests can migrate in a later pass. What lands now: - tests/python/test_raw_connection.py — 11 tests exercising the RawConnection primitive end-to-end through the PyO3 bindings against a local asyncio TCP echo server. - pyproject.toml — pytest + pytest-asyncio in [dependency-groups] dev, [tool.pytest.ini_options] with asyncio_mode = "auto" and testpaths = ["tests/python"]. - .github/workflows/python-tests.yml — now installs pytest in the venv and runs `pytest tests/python/ -v` on every PR across Python 3.10-3.14. The python-tests workflow previously did a build-and-import smoke test despite its name. With this change the name matches behavior: the workflow actually runs Python tests.
Caught these slipping into the prior commit. Remove from tracking and add to .gitignore so they stay out.
Disables Nagle's algorithm on sockets opened through connect_stream, which is used by RawConnection and dispatch_direct (the resolve_ip / request_target specialized path). Both are byte-level / timing-sensitive code paths where Nagle's coalescing delay can split consecutive writes across packets with a 40-200ms gap, during which cross-tenant traffic can interleave on the server side — ruining the ordering guarantee smuggling and protocol-attack probes depend on. hyper-util's HttpConnector remains at its default (nodelay=false), so the main HyperClient path used by request() / request_batch() is unchanged. BBOT's normal HTTP request workload is unaffected. Failure to set the socket option is non-fatal — logged via debug_record and the connection proceeds. This matches hyper-util's own behavior. All Rust unit tests pass. Python bindings tests pass. The scenario-1 CL.TE smuggle reproduces end-to-end through the now-nodelay-enabled RawConnection.
Adds proxy handling to the raw-path connect_stream. When config.proxy is
set, the TCP connection routes through the proxy using one of two
protocols based on the proxy URL scheme:
- http://host:port/ -> HTTP CONNECT tunneling. Sends CONNECT
target:port HTTP/1.1 to the proxy and
expects 200; the resulting socket is a
byte-transparent tunnel to target:port.
TLS handshake (if any) then runs inside
the tunnel using the target's hostname
for SNI.
- socks5://[user:pass@]host:port -> SOCKS5 method negotiation
(no-auth or username/password),
CONNECT command carrying the target
hostname, and drain of the bound-addr
reply. Socket becomes a byte-transparent
TCP tunnel.
Both modes preserve the byte-level fidelity smuggling probes need: the
proxy never parses the tunneled HTTP (CONNECT) or any protocol at all
(SOCKS5). Whatever the caller writes reaches the target unmodified.
New file src/client/proxy.rs holds:
- ProxyConfig struct + ProxyScheme enum + parse_proxy_url helper
- perform_http_connect
- perform_socks5
- 13 unit tests covering URL parsing, CONNECT success/rejection,
SOCKS5 no-auth, SOCKS5 userpass auth, SOCKS5 refused replies,
SOCKS5 all-methods-rejected errors
Python API: BlastHTTP.raw_connect() now accepts an optional
`proxy=<url>` kwarg that flows into RequestConfig.proxy and is honored
by connect_stream.
No behavior change for callers that don't set proxy.
Tests: full Rust suite (82 tests, was 69) green. Python binding tests
unchanged (still green).
Adds RequestConfig.alpn_protocols: Option<Vec<String>> and wires it
through connect_stream's TLS handshake. Callers can now opt into
HTTP/2 on a raw connection by passing alpn_protocols=["h2"] (or
["h2", "http/1.1"] to let the server choose). Direct-connect paths
(request_target/resolve_ip) still default to http/1.1-only since
they don't meaningfully negotiate h2.
RawConnection.negotiated_alpn() exposes the server's selected
protocol ("h2" / "http/1.1" / None for plain HTTP), so callers
that requested multiple protocols can branch on what was actually
negotiated.
Python side: raw_connect(..., alpn_protocols=[...]) and
RawConnection.negotiated_alpn property. Unblocks badhttp's HTTP/2
smuggling probes.
New helper encode_alpn_protocols() serializes protocol names into
the wire format OpenSSL expects (length-prefixed). Validates that
no protocol name is empty or exceeds 255 bytes.
New `blasthttp::h2` module + PyO3-exposed `blasthttp.h2` submodule. Ships a raw HPACK encoder and the full set of HTTP/2 frame builders with explicit knobs for every real-world parser-variation axis we want to probe. Unblocks smuggling research that standard H2 clients can't reach — where the vulnerable wire byte sequence is something Python's `hpack` or Go/Rust `h2` crates refuse to emit. Permissive knobs (all opt-in, defaults are spec-compliant): - allow_invalid_value: emit CRLF / NUL / other RFC 9113 §8.2.1- forbidden bytes in header values - allow_invalid_name: emit uppercase, spaces, CRLF, or other non- lowercase-token bytes in header names — the key for header-name CRLF injection attacks (Kettle's H2 tunneling primitive) - huffman_name / huffman_value: force Huffman on, off, or auto (shortest wins). Separates code paths in decoders that validate Huffman-decoded and raw-octet strings differently - indexing: with / without / never — HPACK's three literal- representation modes, decoders sometimes apply validation inconsistently across them - length_bloat_name / length_bloat_value: emit deliberately overlong HPACK variable-length integer encodings for length prefixes, to fuzz decoder bugs like CVE-2023-44487 / integer handling quirks - force_static_index: emit header as a static-table reference even when the value doesn't match the canonical entry Frame builders (Tier 2 researcher-level primitives): - build_raw_frame: arbitrary type/flags/stream-id/payload escape hatch for unusual frames - build_{headers,continuation,data,settings,ping,rst_stream, goaway,window_update,priority}_frame - High-level build_probe assembles preface + SETTINGS + HEADERS (with optional CONTINUATION splitting + padding + priority) + optional DATA for the common single-request case Python bindings exposed at blasthttp.h2: - Header class mirroring the Rust struct, all knobs as kwargs - build_probe(headers, body=None, **frame_opts) -> bytes - Full set of frame builders + PREFACE constant + frame-type / flag / settings-id module constants Stateless by design: no dynamic-table state across calls. Each build produces a fresh header block assuming empty dynamic table. Real H2 decoders track dynamic tables per-connection, but since we control one probe per connection and don't reuse encoders, stateless avoids a whole class of cross-probe bugs. Tests: - RFC 7541 Appendix C.2.1-C.2.3 header examples (strict mode byte-for-byte match) - RFC 7541 Appendix C.4.1-C.4.3 Huffman encoding test vectors - Integer encoding C.1 examples - Each permissive knob individually (rejects strict, emits with flag) - Frame-layer: length encoding, settings pairs, HEADERS flags, padding, RST_STREAM, GOAWAY - Probe-layer: preface+settings+headers ordering, DATA frame on body, CONTINUATION splitting - 32 new h2::* tests, total Rust test count 114 (was 82) Version bumped 0.2.0 → 0.3.0. Adds thiserror = "2" dependency.
Adds the read side of HPACK to round out the h2 module: Decoder class with persistent dynamic-table state, Huffman decode via canonical prefix-tree walk, full RFC 7541 §6 representation handling (indexed, literal with/without/never indexing, dynamic table size updates). Validated against all RFC 7541 Appendix C canonical examples: - §C.1 integer encoding (10, 1337, 42 round-trips) - §C.2.1-C.2.4 literal and indexed header fields - §C.3.1 request-sequence dynamic-table growth - §C.4.1-C.4.3 Huffman-encoded string test vectors - Round-trip: encoder -> decoder on all 256 byte values Also: - Dynamic-table eviction (oldest-first on overflow) - Single-entry-larger-than-table clears table (§4.4) - Integer overflow detection on malformed continuation bytes - Huffman EOS-in-stream rejection + trailing <=7-bit 1s padding Python binding: blasthttp.h2.Decoder class with decode(block) returning list of (name, value) byte tuples. Construct once per connection (dynamic-table state is shared across all HEADERS / CONTINUATION frames of that connection). Unblocks dropping python hpack dependency from badhttp — response parsing can now use the Rust decoder end-to-end. 27 new hpack-module tests total (up from 12). Huffman tests +10. Total Rust test count now 124 (was 114).
Forces uv to rebuild the editable install via maturin so the HPACK decoder added in 130e491 actually lands when consumers (badhttp) run under `uv run`. Same-version editable reinstalls pull from uv's wheel cache, which stomps maturin-installed .so's with symbols not in the cached wheel. Version bump is the standard-issue fix for this pattern.
When BlastHTTP.set_rate_limit() is set, the rate limiter now propagates into the RawConnection returned by raw_connect(). Every send_bytes and read_raw call on that handle consumes one token, not just the initial connect. This prevents a single-connection caller who pipelines many ops from bursting past the configured rate. PyRawConnection holds an Option<Arc<RateLimiter>> inherited from the owning BlastHTTP at connect time; send_bytes / read_raw acquire one token before dispatching to the underlying I/O. Test pins the behavior: at 10 RPS (100ms interval), connect + send + read is 3 token acquisitions, must take ≥180ms (~first immediate, next two paced).
README additions: - New bullet in Key Advantages for RawConnection + blasthttp.h2 - Dedicated "RawConnection — byte-level TCP/TLS I/O" section with usage example + explanation of raw_connect's TLS knobs and ALPN - Dedicated "HTTP/2 primitives — blasthttp.h2" section explaining the permissive HPACK toolkit, listing every exported symbol, and covering encode/decode, frame builders, and build_probe with examples - One-line mention under Global Rate Limiting that RawConnection handles inherit the client's limiter (per preceding commit) New tests (pytest-asyncio): - test_alpn.py (6 tests) — ALPN negotiation via raw_connect: h2-only, http1-only, server-preference ordering, http1 default, server-with-no-alpn edge case, plain-http negative - test_h2.py (12 tests) — module surface, preface-matches-RFC, HPACK roundtrip, dynamic-table state across decodes, permissive CRLF-in-value roundtrip, SETTINGS/HEADERS/DATA frame builders, build_probe ordering (preface + SETTINGS + HEADERS + optional DATA) Total Python test count: 11 → 30. Coverage gap on the PyO3 surface (ALPN + blasthttp.h2) closed before publishing v0.3.1. Stray debug script `test_request.py` in repo root removed.
The original RateLimiter held a tokio::sync::Mutex across sleep_until, which had two compounding failures at sub-millisecond intervals: 1. Tokio's timer-wheel resolution (~1ms) floored sleep_until, so a configured 100k RPS interval (10µs) was enforced at ~1k QPS. 2. The mutex-across-await serialized every caller single-file through the limiter, preventing any parallel progress. Users setting a rate_limit as a "safety ceiling" well above realistic throughput got 50-215x regressions vs the unlimited path instead of the expected near-zero overhead. Replace with a single AtomicU64 nanosecond cursor + CAS bump, with the sleep happening outside any critical section. The cursor clamps to max(current, now) on every acquire so idle periods can't stockpile back-dated slots into a burst. Sub-millisecond sleep deficits are skipped (cursor still advances) and aggregated into real 1ms sleeps when the debt accumulates — aggregate rate is preserved without per-acquire timer-granularity cost. Two regression tests pin the fix: - test_rate_limiter_high_rps_does_not_collapse_to_timer_tick: direct unit-level assertion that 1000 acquires at 100k RPS complete under 250ms (was 1.08s on the broken impl, now ~19ms). - test_send_batch_high_rate_limit_matches_unlimited: end-to-end mirror of the issue's scenario. 1000-req batch with a 100k RPS cap must not take >200ms longer than unlimited (was +1.08s, now +14ms). All 15 pre-existing rate-limit tests (1/10/20/100 rps, shared limiter, concurrent batches) still pass unchanged — slow-path semantics are identical. 134 Rust lib tests pass total. Fixes #15.
All 27 clippy warnings on the raw-connection branch resolved, bringing
it in line with the clean baseline on stable. The CI workflow
(.github/workflows/rust-tests.yml) runs cargo fmt --all --check and
cargo clippy --all-targets --all-features --locked -- -D warnings; both
now pass.
Categories of fixes:
- src/python.rs: pyo3 deprecation — PyObject → Py<PyAny> (13 sites),
.downcast::<T>() → .cast::<T>() (2 sites). Add
#[allow(clippy::too_many_arguments)] on raw_connect to match the
existing convention on other Python-binding surfaces in this file
(request, request_batch, download, and three H2 frame builders) —
the arg count is dictated by the Python keyword-arg API the function
exposes.
- src/h2/probe.rs: three test helpers converted from
`let mut opts = Default::default(); opts.x = y;` to struct-update
syntax `ProbeOpts { x: y, ..Default::default() }`.
- src/h2/frame.rs: two `for _ in 0..padding { payload.push(0) }` loops
collapsed to `payload.resize(...)`.
- src/h2/hpack.rs: extracted `Vec<(Vec<u8>, Vec<u8>)>` into a
`DecodedHeader` type alias for `Decoder::decode_headers`'s return
type.
- src/batch.rs: added blank line in doc comment so the paragraph after
the numbered list parses as a paragraph, not a continuation.
- Assorted formatting normalizations across the new h2/ and client/
files from cargo fmt.
All tests still pass: 134 Rust lib tests, 30 Python tests, and 368
badhttp downstream tests (via the rebuilt wheel).
maturin 1.13+ uses `pip install --group` to install the `[dependency-groups]` block in pyproject.toml. That flag requires pip >= 25.1; actions/setup-python ships older pip (23.x on 3.10) so the build step fails with 'no such option: --group'. Upgrade pip first, independent of maturin version.
TheTechromancer
approved these changes
Apr 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two low-level primitives for callers that need byte-level control over the wire (protocol fuzzers, smuggling detectors, anything that can't fit through the HTTP framing layer), and fixes a rate-limiter collapse at high RPS (#15). Version bumped 0.3.0 → 0.3.2.
New features
RawConnection— unframed TCP/TLS I/OBlastHTTP.raw_connect(url, ...)returns a handle exposingsend_bytes/read_raw/close. No HTTP parsing, no framing, no re-emission. TLS knobs pass through (verify_certs,cipher_string,min_tls_version,max_tls_version,resolve_ip,proxy). Post-handshake attributes:cert_info,negotiated_alpn.ALPN negotiation on
raw_connectNew
alpn_protocols=["h2", "http/1.1"]kwarg.conn.negotiated_alpnreports what the server picked (orNonefor plain HTTP / no-ALPN handshakes).Proxy support on
raw_connectHTTP
CONNECT(for HTTPS via proxy) and SOCKS5 (with username/password auth) for arbitrary protocol. Auto-selected by proxy-URL scheme (http://vssocks5://).blasthttp.h2— permissive HTTP/2 toolkitHand-rolled (no
h2crate dependency) so callers can emit bytes a strict encoder refuses — the building block for H2 smuggling / CRLF-injection tooling.Headerwith permissiveness flags:allow_invalid_name,allow_invalid_value,huffman_name,huffman_value,indexing,force_static_index,length_bloat_name,length_bloat_valueencode_headers()— HPACK header-block-fragmentDecoder— stateful HPACK decoder (dynamic-table-aware across calls)HEADERS/DATA/SETTINGS/CONTINUATION/RST_STREAM/GOAWAY/PING/PRIORITY/WINDOW_UPDATE+build_raw_frameescape hatch for malformed framesbuild_probe(headers, body=None, ...)— one-shot preface + SETTINGS + HEADERS + optional DATAPREFACEbytes + everyFRAME_*/FLAG_*constant exposedRate limiter inherited by
RawConnectionWhen
BlastHTTP.set_rate_limit()is set, the returnedRawConnectionnow inherits the limiter — everysend_bytes/read_rawcall on the handle also consumes one token. Single-connection callers can't burst past the configured rate.TCP_NODELAY on connect
Raw sockets enable
TCP_NODELAYso small writes flush immediately (matters for the low-level-I/O use cases where the caller is crafting bytes byte-by-byte).Bug fix
RateLimiter collapse at high RPS — closes #15
Old impl held
tokio::sync::Mutexacrosssleep_until, hitting two compounding issues:Replaced with a single
AtomicU64nanosecond cursor + CAS bump, with sleep happening outside any critical section. Sub-ms sleep deficits are coalesced (cursor still advances, debt aggregates into real 1ms sleeps). Idle limiter resets tonowon next acquire — no back-dated slot bursts.Measured impact (1000-request batch, 100k RPS cap):
Low-RPS semantics unchanged — all 7 pre-existing rate-limit tests still pass.
Test coverage added
build_probelayout, proxy CONNECT/SOCKS5, 2 issue-RateLimiter collapses above ~1k RPS #15 regressions (unit + end-to-end)RawConnectionlifecycle, rate-limit inheritance, ALPN negotiation (6 cases covering server-preference order / no-ALPN / plain-HTTP),blasthttp.h2surface (12 cases)Docs
README: new bullet in Key Advantages; dedicated "RawConnection — byte-level TCP/TLS I/O" and "HTTP/2 primitives — blasthttp.h2" sections with usage examples; rate-limit section notes limiter inheritance into RawConnection.