Skip to content

fix(http): classic shutdown waits for every handler before the app goes - #362

Merged
gg582 merged 3 commits into
devfrom
fix/classic-shutdown-uaf
Oct 12, 2026
Merged

gg582 merged 3 commits into
devfrom
fix/classic-shutdown-uaf

Conversation

@gg582

@gg582 gg582 commented Oct 12, 2026

Copy link
Copy Markdown
Member

Follow-up to #351 (the open TSan report in test_http_client).

Bug

Classic pool handlers run on detached workers. A keep-alive connection keeps its worker in a blocking read for as long as the client holds it open (the BDR fast path's recv has no timeout at all), and a client that keeps sending requests keeps it dispatching. Shutdown waited for the in-flight count only up to the drain timeout, then cwist_app_destroy() freed the route table under a worker still in cwist_route_table_lookup.

Fix

Each pool worker registers the connection it serves; cwist_http_pool_destroy() drains them before it returns (and before the caller destroys the app):

  1. idle keep-alive connections are shut down at once;
  2. in-flight requests get the drain timeout (CWIST_DRAIN_TIMEOUT) to finish;
  3. what is left is shut down (SHUT_RDWR) and the pool waits until every handler has returned. A handler stuck in route code is waited for (warning every 5 s) rather than freed under, the same contract C1M already has (it joins its handler threads).
  • Handlers mark the wait for the next request with cwist_http_live_idle_enter() / _leave() (private src/net/http/http_live.h): the header wait in cwist_http_receive_request() and the BDR recv in cwist_app_http_handler(). Once the pool is stopping, idle_enter returns false and the handler closes instead of waiting. Outside a classic pool worker both are no-ops.
  • The handler serves a dup of the socket; the pool keeps the original open until the registry entry is unlinked, so a shutdown() issued under the registry lock never reaches a reused descriptor. Cost: one dup + close per classic connection (not per request), and one extra descriptor while it is served.

Scope: the plain-HTTP classic pool. The HTTPS pool and the forking path are unchanged.

Tests

  • New tests/test_classic_shutdown.c (classic, one worker, 4 s drain):
    • idle keep-alive connection: shutdown in 0.02 s (before: 4.01 s, the full drain, then the app was destroyed under the worker — verified by running the test against dev);
    • in-flight /slow (1 s handler) still gets its complete response;
    • a client hammering its keep-alive connection cannot keep the server alive; every response it got was complete.
  • test_http_client removed from TSAN_EXCLUDE.
  • Under TSan, 3 runs each: test_http_client, test_classic_shutdown, test_listen_ex, test_classic_pool_scaling, test_shutdown — all pass. Normal build: the same plus test_static_and_range, test_async_defer, test_https_metrics pass.
  • malloc_lint.sh, check_api_baseline.py, check_test_wiring.py OK (no public API change).

gg582 added 2 commits October 12, 2026 18:57
Classic pool handlers run on detached workers. A keep-alive connection
keeps its worker in a blocking read for as long as the client holds it
open, and a client that keeps sending requests keeps it dispatching.
Shutdown only waited for the in-flight count up to the drain timeout and
then destroyed the app, so cwist_app_destroy() freed the route table under
a worker still in cwist_route_table_lookup (the open TSan report in
test_http_client, #351 follow-up).

Each pool worker now registers the connection it serves, and
cwist_http_pool_destroy() drains them before returning:
  1. idle keep-alive connections are shut down at once;
  2. in-flight requests get the drain timeout to finish;
  3. what is left is shut down (SHUT_RDWR), and the pool waits until every
     handler has returned. A handler stuck in route code is waited for,
     with a warning every 5 s, rather than freed under.

Handlers mark the wait for the next request with
cwist_http_live_idle_enter()/_leave() (src/net/http/http_live.h): the
header wait in cwist_http_receive_request() and the BDR fast path's recv in
cwist_app_http_handler(). Once the pool is stopping, idle_enter returns
false and the handler closes the connection instead of waiting. The
handler serves a dup of the socket; the pool keeps the original open until
the entry is unlinked, so shutdown() never reaches a reused descriptor.

tests/test_classic_shutdown.c: an idle keep-alive connection no longer
holds the drain (0.02 s vs 4.01 s before), an in-flight /slow request still
gets its full response, and a client hammering its keep-alive connection
cannot keep the server alive. test_http_client runs under TSan again.
WASI has no dup() or shutdown(), so wasm32-wasip2 failed to build
(component pipeline). There the pool serves connections unregistered, as
before, and the shutdown kick is a no-op.
@gg582
gg582 merged commit 982ba38 into dev Oct 12, 2026
23 checks passed
@gg582
gg582 deleted the fix/classic-shutdown-uaf branch October 13, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant