Repository navigation
Faster cross-thread wakes, asyncio turn shape and native getaddrinfo behind threaded servers - #122
Merged
egorsmkv merged 7 commits intoOct 19, 2026
Conversation
A cross-thread wake stores the root future's ready flag (or pushes the task onto the remote queue) and then loads the runtime's `waiting` flag; the park stores `waiting` and then loads ready. With release/acquire only, both stores can still sit in their store buffers when the loads run, so the waker skips the interrupt and the park sleeps through the wake. The loop thread recovers at its next park deadline, but the coordination thread parks with no timer and never wakes again: callbacks queued before a run_forever (a run_until_complete task's first step) are then never forwarded while loop-local work keeps running. The greenlet repro of RustedBytes#115 hung this way in about 1 of 300-450 runs under Docker (epoll), on v0.1.64 as well. A SeqCst fence now separates store and load on the park side and on both notifiers (root waker and remote task queue). Same-thread wakes skip it. Two stress tests ping-pong 100k wakes against the park on the mio driver and fail when progress stops for 5 s; without the fences the root-waker test stalls in 2 of 3 runs.
…e run Every off-thread handle used to travel through the command channel to the coordination thread, which then pushed it into the run's queue and woke the loop thread: two thread wakeups per handle where one is enough. Behind a server that resumes requests with call_soon_threadsafe from its own I/O threads (Granian), that is two wakeups per request start. While a run is active and no earlier handle is still on the channel, the handle now joins the run's pending queue directly (one wake). A counter of handles still on the channel keeps each producer's order: once one is queued there (before EnterRun, between runs), later handles follow it. finish_run now puts the run's queue (older work) ahead of commands received since the last dispatch. Microbenchmark on the v0.1.64 base (Docker epoll, --cpus 2): cross-thread ping-pong CPU 11.4 -> 7.8 us/op, task ping-pong 10.4 -> 6.9 us/op (uvloop 5.9 / 6.9).
The loop thread spun for up to 50 us, with the GIL released, waiting for a cross-thread wake before parking. Behind Granian those wakes come from Python threads (call_soon_threadsafe in its future watcher) that still hold the GIL when the flag flips, so the spinning loop thread queued for the GIL behind them, paying up to the 5 ms switch interval per reattach. Request starts and completions then reached the loop in bursts. With a small SQLAlchemy pool (4 + 2 per worker at 8 concurrent requests) the pool kept discarding returned connections into a full idle queue and opening new ones: 33-55 new Postgres sessions per 1k requests vs 3-4 on uvloop, the whole CPU-per-request gap at c16. RSLOOP_WAKE_SPIN_US now defaults to 0 (park at once) and remains an opt-in for workloads whose wakes come from GIL-free threads. Microbenchmarks on the v0.1.64 base (Docker epoll, --cpus 2, medians of 3): Granian-shaped cross-thread request CPU 54.6 -> 41.1 us/req (uvloop 50.0), cross-thread ping-pong 14.1 -> 6.5 us/op (uvloop 7.6), and a 4 + 2 pool churn repro 598 -> 2,593 req/s with 61.3 -> 0.2 connects per 1k (uvloop 2,166, 0.1). v0.1.65 keeps the spin: the same repro opens 60-74 connections per 1k there.
run_forever drained each turn to quiescence: callbacks scheduled during the turn joined it (every 64 items, up to 1024), and the reactor was polled only once nothing was left. Under load, database responses piled up between polls and completed in one burst, so more connections came back to a SQLAlchemy pool in one turn than its idle queue holds before any woken waiter ran; the overflow was closed and reopened (a Litestar API at c64: ~20 new Postgres sessions per 1k requests vs 0.1-3.7 on uvloop, ~690 full-queue check-ins in 20 s). A turn now runs the callbacks queued when it started, as asyncio's _run_once does; newer ones wait for the next turn, after another non-blocking poll. That poll runs with the GIL released, as uvloop's does: back-to-back turns would otherwise hold the GIL for the switch interval while call_soon_threadsafe producers wait. A stop() whose Stop item is still queued keeps the turn going until it runs, so stop keeps taking effect in the turn that requested it. The loop thread schedules into its own lock-free queue and other threads into the run's pending queue, so a generation has to be merged from both. Off-thread items now carry the loop thread's enqueue count, read when they are queued, and the drain merges them behind exactly the local items scheduled before them: call_soon and call_soon_threadsafe callbacks run in the order they were scheduled, as on asyncio (before, 50 alternating call_soon / thread-posted call_soon_threadsafe calls from one callback did not interleave in call order). Marks are stored once per change, the loop thread keeps a single-writer counter, and work queued before a run goes ahead of the run's own. On the v0.1.64 base, the same API at c64: 1,618 -> 1,936 rps (1.3-3.0 new sessions per 1k; uvloop 1,746). Microbenchmark cost against v0.1.65: a self-rescheduling call_soon chain 0.20 -> 0.51 us/op and a cancelled timer 1.52 -> 2.21 us/op (uvloop 0.79 and 2.61), now one turn per step as in asyncio and uvloop; benches/compare_event_loops.py is level. New tests: a call_soon chain no longer delays a readable protocol by hundreds of callbacks (513 before, 1 after), and three ordering tests compare call_soon / call_soon_threadsafe order with asyncio.
loop.getaddrinfo() of a host name goes through run_in_executor: a blocking worker takes the GIL to call socket.getaddrinfo and takes it again after the lookup. On a busy loop each acquisition waits for the loop thread to drop the GIL. uvloop resolves in libuv's pool without it. Services that connect by name (a NATS or Postgres client per new connection) pay this on every connect. - resolver.rs: on Unix, an exact loop without a custom default executor, with the standard library's socket.getaddrinfo installed and plain host and port arguments (an ASCII name the IDNA codec passes unchanged or bytes; an int port in range, str or bytes), calls getaddrinfo(3) on the blocking pool without the GIL. The loop thread raises the socket.getaddrinfo audit event through PySys_Audit, as _socket does (a replaced sys.audit skips no hook), and builds the same rows (enum members, glibc's inet_ntop text) or the same gaierror / OSError (EAI_SYSTEM as PyErr_SetFromErrno raises it, errno 0 included). Every other shape keeps the executor path. - A replaced or wrapped socket.getaddrinfo still receives every lookup, as on asyncio: the standard function is recognised by what a functools.wraps wrapper cannot copy (its globals are the socket module's namespace, its code is named getaddrinfo, it has no __wrapped__). A replaced _socket.getaddrinfo, the delegate it calls, keeps the executor path too. Both are checked again on every lookup. - blocking.rs: at most min(32, CPUs + 4) lookups run at once, the size of asyncio's default executor (CPUs from the affinity mask, as os.process_cpu_count counts them); further lookups wait in order, so a burst of 1,000 lookups no longer starts hundreds of threads. run_in_executor jobs never wait for a lookup slot. - engine: a LoopWork ready item runs the loop-thread half of such work, so the worker never takes the GIL. Tests: tests/test_native_addrinfo.py compares 25 argument shapes with socket.getaddrinfo (rows, enum types and errors, including the shapes left to CPython), 64 concurrent lookups, cancelled lookups, a lookup created before run_until_complete, the audit event (also with sys.audit replaced), a functools.wraps wrapper installed before the first lookup, a replaced _socket.getaddrinfo before and after native lookups, and the thread cap under a burst. Rust tests cover the bounded slots, glibc's IPv6 text, the IDNA fast path, the cap formula and the EAI_SYSTEM mapping.
perf/ANALYSIS.md records how the gap behind Granian was found (per-thread profiles, perf and strace, a standalone SQLAlchemy pool repro, a pool trace inside the service), what each commit of this series changes and its tests, and microbenchmark and pool-repro measurements against v0.1.65 and uvloop, the full-service comparison against v0.1.65 (not a quiet run, so only the "ahead" cells are differences) and the quiet-gated run on the v0.1.64 base. perf/bench/ holds the tools those measurements used: - harness.py: the full-service harness (quiet-gated, interleaved and rotated arms, CPU and new Postgres sessions per 1k requests); it needs an application stack, described by RSPERF_* variables, which also name the endpoints and test-account login of the API under test. - quiet.py: CPU used outside the benchmark's containers over a window. - compare.py: per-row medians, round ranges and deltas for any arms; a measurement with non-2xx responses or transport errors is counted as failed and left out; oha's end-of-run "aborted due to deadline" entries are not failures (tests/tooling/test_perf_compare.py). - micro.py, pool_churn.py: standalone microbenchmarks and the pool repro. - sitecustomize.py: an opt-in SQLAlchemy pool trace for the service workers.
Preserve callback FIFO across loop restarts, serialize subprocess lifecycle notifications, and use a portable native driver in wake-race tests. Skip canceled native DNS work before it reaches libc and recheck immutable stdlib code plus wrapper metadata after in-place instrumentation or reload. Add deterministic regressions for queue activation, subprocess EOF ordering, DNS cancellation and Future cleanup, and instrumentation changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Behind Granian, a Litestar web service with an async SQLAlchemy/asyncpg pool
ran up to 44% slower on rsloop 0.1.64 than on uvloop at c16. The cause was not raw loop speed: the
way the loop woke up for
call_soon_threadsafeand how much work it ran perturn made request starts and database completions arrive in bursts, and a
small SQLAlchemy pool then kept discarding and reopening connections (33–55 new
Postgres sessions per 1k requests vs 3–4 on uvloop, each one a SCRAM login).
v0.1.65 still does this: in the run below it is 17–35% behind uvloop at c16,
with 27–53 new sessions per 1k. This PR fixes that and one lost-wake race.
Each commit stands alone and can be reviewed or dropped separately:
stores the ready flag and then loads
waiting, while the park storeswaitingand then loads the ready flag. With only release/acquire both loadscan see the old value, so the coordination thread (which parks with no
deadline) can sleep through a wake for good. Callbacks queued before a
run_foreverthen never reach the run. It shows as a hang in about 1 in300–450 runs of the Task wakeup lost (or SIGSEGV) when a future is resolved inside a greenlet after the loop is re-entered from a deeper C stack — SQLAlchemy asyncio hangs under Granian #115 repro under Docker. Adds a SeqCst fence on both
sides, plus two stress tests (100k wakes each). On master without the fences
the
block_ontest stalls in 2 of 3 runs.run. One wake per handle instead of a hop through the coordination thread.
A counter of handles still on the command channel keeps each producer's FIFO
order across run start and end.
RSLOOP_WAKE_SPIN_USnow defaults to 0 instead of 50. Most cross-threadwakes come from Python threads that still hold the GIL when they flip the
flag, so a spinning loop thread queues for the GIL behind them, paying up to
the 5 ms switch interval. This is the root cause of the burst/churn pattern.
The spin stays available as an opt-in.
GIL. This is asyncio's
_run_onceshape. A turn runs what was ready whenit started, then does a non-blocking reactor poll with the GIL released, as
uvloop does. It no longer runs to quiescence (refill every 64, up to 1024).
call_soonandcall_soon_threadsafecallbacks are merged in schedulingorder: each off-thread item carries the loop thread's enqueue count. A
queued
stop()still takes effect in the turn that requested it.exact loop with the standard
socket.getaddrinfoand plain arguments,getaddrinfo(3)runs on the blocking pool without the GIL. The loop threadraises the audit event and builds the same rows and errors CPython would.
At most
min(32, CPUs + 4)lookups run at once, like asyncio's defaultexecutor. Anything unusual (a subclass, a custom default executor, a
replaced or
functools.wraps-wrappedsocket.getaddrinfo, IDNA, odd porttypes) keeps the executor path.
perf/ANALYSIS.mdandperf/bench/. This commit is optional for you to take (see below).Measurements
All figures below were measured before this branch was rebased onto master
f045b7b (on the v0.1.65 base, 37a3d3f); they were not re-run after the rebase.
All arms are release wheels (
maturin build --release --locked), eachunpacked into its own directory and put first on
PYTHONPATH, so every armruns on the same interpreter. "v0.1.65" is
37a3d3f.Pool repro (
perf/bench/pool_churn.py). A SQLAlchemy async engine(
pool_size=4, max_overflow=2, asyncpg, Postgres) on one loop, with 8 driverthreads posting requests through
call_soon_threadsafeas Granian does,2 queries per request. Docker epoll,
--cpus 1, 3 runs per arm. Other buildswere running on the machine (load1 13–25); the connection count does not
depend on that load.
So v0.1.65 does not match uvloop on this shape yet. Every new connection is a
Postgres SCRAM login, and that login is where the service's extra CPU went.
Microbenchmarks (
perf/bench/micro.py). Docker epoll,--cpus 2, 3rounds with the arm order rotated, median CPU µs per op. Measured under load
(load1 11–21), so read a few percent as noise.
call_soon_threadsafeping-pongcall_soonchainasyncio.timeoutaround a resolved awaitgetaddrinfo("localhost")getaddrinforun_in_executor(None, int)open_connection("localhost", ...)+ round trip + closeYour
benches/compare_event_loops.py, pinned to one CPU, 1,000,000callbacks / 200,000 tasks / 20,000 TCP round trips. Median seconds of 5 runs,
two interleaved passes per build:
These are level.
getaddrinfoburst. 1,000 concurrent lookups oflocalhoston a20-CPU host. v0.1.65 grows smol's pool to its 500-thread limit and takes
114–123 ms. The branch uses 24 threads (cap
min(32, 20 + 4)) and takes29–30 ms.
Full service against v0.1.65 (Granian, 2 workers,
--cpus 2, ohac16/c64, 15 s warm-up and 60 s per row, 3 rounds with the arm order
rotated). The endpoints are
authenticated JSON reads, named by workload: two session reads (light
authenticated GETs on different routes), an aggregate read (a GET with a DB
aggregate), a count read (a GET with a small DB count) and a list read (a GET
with a small DB list). Each cell is the median of the 3 rounds, with the
min–max over the rounds in brackets. "ahead" means the branch's range lies
entirely above the other arm's, and "overlap" means the two ranges overlap.
CPU-ms is the API container's cgroup CPU per 1k requests, and new PG sessions
are counted from
pg_stat_database.This was measured while other builds were running on the 20-CPU host
(load1 3.1–9.7, 1.5–5.9 cores of CPU outside the benchmark, P-cores at
2.1–4.5 GHz), so the API container was pinned to two physical P-cores
(CPUs 0–3), oha to CPUs 4–7 and Postgres/Redis/NATS to CPUs 8–11, and
rows above 6 foreign cores were measured again; even so, 77 of the 90 rows,
spread over all three rounds and all arms, were above the 2.5-core
quiet-machine threshold, so every round counts as contaminated and only the
"ahead" cells should be read as differences.
At c16, v0.1.65 still opens 27–53 new Postgres sessions per 1k requests,
against 3–11 on uvloop and 3–8 on the branch. There the branch is ahead of
v0.1.65 on four of the five rows (+17.5% to +64.9%), and session read 1
(+18.7%) overlaps. The branch also uses 14–39% less CPU per request than
v0.1.65, and it is within noise of uvloop on all five rows. At c64, the branch
is ahead of v0.1.65 on session read 1 (+47.8%, where v0.1.65 still
churns 28 sessions per 1k). The other four c64 rows are within noise of
v0.1.65 (−3.5% to +5.9%). No row has the branch outside noise of uvloop
(−8.7% to +15.4%).
On session read 1, contention also widens v0.1.65's deficit. One
quiet-gated, unpinned round of session read 1 and the aggregate read on the
same three wheels, taken the
same morning before the builds started (every row at or below 2.5 foreign
cores), gave these rps for uvloop / v0.1.65 / branch:
New PG sessions per 1k in that round were 12.7 for v0.1.65 against 1.7 for
the branch on session read 1 at c16, and 33.8 against 10.1 on the aggregate
read at c16. So on a quieter machine, v0.1.65 is level on session read 1 at
c16 and the gap there is smaller than the table shows. The c64 rows and the
aggregate read at c16 point the same way in both runs.
Full service on the v0.1.64 base (same harness, earlier quiet run of the
same changes):
The measured stack: Litestar 2.23.0 on Granian 2.7.5 (2 workers),
SQLAlchemy 2.0.50 (async engine,
QueuePool) over asyncpg 0.31.0 to Postgres,redis-py 5.3.1 and nats-py 2.15.0, with uvloop 0.22.1 as the reference loop and
oha as the load generator. What each dependency showed:
call_soon_threadsafefromits own threads, which still hold the GIL when the wake flag flips. That is
the wake pattern behind the spin problem (commit 3).
QueuePool+ asyncpg turned the resulting bursts intoconnection churn. Every new asyncpg connection is a SCRAM login
(
_generate_salted_password,libcrypto), and that is where the extra CPUwent (commits 3 and 4).
loop.getaddrinfo(a health probe that opens a NATS connection per requestspent ~250 µs of CPU on it on v0.1.64; commit 5).
the same application code.
perf/ANALYSIS.mdhas the method, the profiles and the rest of the rows.Things you may want to change or drop
call_soon/ timer cost (commit 4). A self-reschedulingcall_soonchain goes from 0.20 to 0.51 µs per step, and a cancelled timerfrom 1.52 to 2.21 µs. Both are still below uvloop (0.79 and 2.61), and your
compare_event_loops.pycallbacks workload is level. The turn shape is thepoint of the change: a self-rescheduling chain now yields to I/O every step,
as on asyncio and uvloop. Without it, a
call_soonchain held back a readablesocket for 513 callbacks in the new test. If you prefer the old drain, commits
1–3 and 5 still apply without it, but commit 4 is what fixed the session read 1
and aggregate read c64 rows.
(rsloop's own reader workers) can still opt in with
RSLOOP_WAKE_SPIN_US=50.On the v0.1.64 base the spin let rsloop beat uvloop by 4–7% on the
saturated c64 rows of session read 2, count read and list read, while costing 22–47% at c16. Against v0.1.65, which
still spins by default, those c64 rows in the run above are within noise
of the branch (−3.5% to +5.9%).
perf/(commit 6, benchmark tooling) is optional and can be dropped.Commits 1–5 do not depend on it. The harness needs a private application
stack, so it documents the method rather than being runnable as-is.
micro.pyandpool_churn.pyrun standalone. Drop the commit if you wouldrather keep only
benches/.smol's blocking pool keeps its idle count ina static, so
blocking::runin a child forked within 500 ms of the parent'slast blocking job waits for a thread that does not exist (a small Rust test
reproduces it). In Python, however,
run_in_executorandrsloop.runalreadyhang in a child forked after the parent ran any rsloop loop, on v0.1.64,
v0.1.65 and this branch alike. So this PR does not try to fix fork support.
thread-per-call
blocking::runwith its own cached pool. v0.1.65 alreadymoved it to
smol::unblock, so that part was dropped. Only the boundedlookup slots remain, on top of
smol::unblock.Tests added
vibeio::executor::tests::{block_on,spawned_task}_never_misses_a_wake_racing_with_the_park,engine::loop_core::generation_order_tests(3),blocking::tests(boundedslots: cap, blocked bounded jobs don't delay others, a panicking job passes
its slot on),
resolver::tests(glibc IPv6 text, IDNA fast path, cap formulavs asyncio's,
EAI_SYSTEMvsPyErr_SetFromErrno).tests/test_fast_callbacks.py(producer order across run restarts,call_soon/call_soon_threadsafeordering, 3 tests against asyncio andrsloop),
tests/test_local_sockets.py::test_tcp_data_is_delivered_between_call_soon_generations,tests/test_native_addrinfo.py(25 argument shapes vssocket.getaddrinfo,concurrency, cancellation, audit event, a
functools.wrapswrapper, thethread cap).
Verification
cargo fmt --check.cargo clippy --all-targets --all-features --locked -- -D warningsis clean on every commit.scripts/run_rust_tests.py: 385 passed. With--all-features: 476 passed.scripts/test-supported-pythons.sh --debug(CPython 3.14.5): 608 passed,124 skipped.
test_native_addrinfo.pyandtest_free_threading.py, 40 passed.StreamTransportdoes not support cyclic GC: every Protocol that keeps its transport leaks (asyncpg ~32 KB per connection) #116 andCancelled
TimerHandlekeeps its callback, args and context alive until every earlier timer has fired (asyncio.timeout, asyncpg statement cache) #118: clean.