Mpsc scheduler#227
Conversation
Virtual threads with sticky affinity preserve carrier locality: - start/unpark from a sticky VT uses lazy submission (local queue, no signal) - sub-pollers (mode 2 and 3) are made sticky, simplifying Poller.polled()
…llers Mode 2 sub-pollers no longer use stickyAffinity — they revert to the original useLazyUnpark dispatch in polled(). Only mode 3 (per-carrier) sub-pollers are sticky.
Alternative scheduler using dual MPSC queues per carrier (sticky + external), ticket-lock work stealing, FJP-style probing, and alternating drain. Adds roundRobinAffinity() and stickyAffinity() support via preferredCarrier and affinityHint routing. Poller Mode 4: each carrier is its own master poller — sub-pollers register with carrier-local epoll fd, no global master poller thread. Activated with -Djdk.virtualThreadScheduler.useMpsc=true and -Djdk.pollerMode=4.
Carrier count now matches parallelism exactly. Steal scan uses precomputed int[] siblings and skips parked carriers. @contended on carrierState and ticket lock fields to avoid false sharing.
Mode 4 (carrier-master-poller) added epoll_wait syscall overhead that reduced peak throughput. Reverted to plain carrier loop with LockSupport.park/unpark and Mode 3 (global master poller + per-carrier sub-pollers). Also fixed setVolatile for PARKED state (StoreLoad barrier) and added isEmpty guard on external queue poll.
Single queue, no ticket lock, no alternation, no steal scan. The carrier loop is just poll → run → poll. Carrier affinity via attachment handles all routing after first run.
Single object per carrier: CarrierThread holds queue, state, id directly. No indirection, no shutdown check. Plain volatile for carrierState. Skip attach when attachment is already current carrier.
Task attachment set once at onStart, never in the carrier loop. Brief spin (16 iterations with onSpinWait) before parking to catch imminent work without OS scheduler round-trip.
Each carrier owns its own epoll fd with direct VT fd registration. No sub-pollers, no master poller, no ConcurrentHashMap — plain HashMap since only the carrier accesses it. Event loop interleaves task drain (50us budget) with non-blocking I/O poll. Blocks in epoll_wait only when both queue and I/O are idle. Uses VThreadTask (lightweight) with affinityHint for carrier routing instead of CustomVThreadTask.
Continuation.pin() was held through LockSupport.park(), blocking the carrier. Fixed by unpinning after the poller lookup but before parking. Replaced ThreadLocal with ConcurrentHashMap keyed by carrier thread id since VTs can't see carrier thread-locals.
Add @contended groups to MpscUnboundedQueue to separate producer fields (producerIndex, producerLimit) from consumer fields (consumerIndex) on different cache lines. Add Thread.onSpinWait() in the poll spin loop for lagging producer stores. Defer System.nanoTime() in drain budget to after first task polled.
|
👋 Welcome back franz1981! A progress list of the required criteria for merging this PR into |
|
❗ This change is not yet ready to be integrated. |
|
As a design point, given the low core assumption, this scheduler could use a mpmc shared queue where to spill work which exceed a threshold for each carrier (I would make the mpsc mailboxes soft bounded) - in case the VT to spill has not sticky affinity. |
|
this will be faster for the eventloop think, in pekko stream, the mailbox is a mpsc queue now and faster than it was |
|
Some more benchmarking data from the CPU bound test: 
30K tps: 40K tps: 
50K tps: 
60K tps: 
and p50/p99 vs load: 
And at 30K tps load: |
|
@AlanBateman By moving the poller to be per carrier (as this pr), un-shared, would allow me to more easily use edge-triggered poll, further reducing the difference between Netty event loops/async I/O frameworks - and what Loom I/O does, saving one syscall per blocking read. At this point a better name for this scheduler would be "event loop scheduler" |
|
@AlanBateman IIRC the way eventFd is setup to not accumulate writes and a single poll would drain them all: so I've pushed be4dd8f as an additional optimization, although rare, in case on a cycle of VT draining, none register for some I/O events; if that happen, the per carrier local poller just skip the non blocking poll. |
|
As a side note for Netty uring integration: in modern Linux kernel versions the uring ring buffer can provide its own (pollable) fd which can be registered for POLLIN ONE SHOT against the per carrier poller allowing Netty VT to safely park/unpack. |
2 participants
This is a first PoC for an alternative scheduler targeting low number of cores.
It has no work-stealing implemented (so it would be better to have VT non-voluntary preemption, but can still suffer from unbalanced work-loads (obviously!) - but to keep latencies and cpu usage tight, it collapse the subpollers as carrier-confined in the carrier loop.
I've introduced a pollerMode 4 for this.
In term of peak tps it is in between the #226 performance and the custom scheduler with native transport i.e. the number of hops from I/O polling is the same.
Architecture
affinityHint, set once atonStart,onContinuealways routes homeroundRobinAffinity()for initial VT distribution across carriersHashMap<Integer, Thread>for fd→VT mapping (carrier-local, single-thread access)epoll_wait(0), blocks only when genuinely idle@Contendedon queue producer/consumer fields to avoid false sharing-Djdk.virtualThreadScheduler.useMpsc=true -Djdk.pollerMode=4If pwait2 would be available it could collapse into it a timer wheel structure (or just a priority q, since there's no sharing) for timers.
I have not performed any specific optimization on wakeups; eventFd is knew to be terrible and in Netty indeed we shield concurrent eventFd writes with a slightly more complex state machine. Here I made it simple as the most of the interaction(s) are local.
Sustained Load Results
Benchmark: same as #226
MPSC (Mode 4): this new schedulerFJP (sticky): the FJP schedulerBoth are running Netty event loops as "sticky" VirtualThreads, assigned (if the scheduler allow it) round robin to each carrier.
CPU-bound (50K req/s, 2 server CPUs, 1ms mock delay, 100 connections)
MPSC matches or beats FJP at p50-p75 and p99 with 6% less CPU. Context switches are 4x lower. p90 is wider.
I/O-bound (120K req/s, 12 server CPUs, 30ms mock delay, 7000 connections)
Same throughput. MPSC uses 14% less CPU with half the context switches. Tail latency (p99) is worse under heavy I/O.
Progress
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.org/loom.git pull/227/head:pull/227$ git checkout pull/227Update a local copy of the PR:
$ git checkout pull/227$ git pull https://git.openjdk.org/loom.git pull/227/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 227View PR using the GUI difftool:
$ git pr show -t 227Using diff file
Download this PR as a diff file:
https://git.openjdk.org/loom/pull/227.diff