release 0.6.6 — swarm health under CGNAT - #16
Merged
Conversation
The dedup table for background metadata lookups was created by whichever Fetcher happened to run first, so it died with that torrent's fetcher and every later background task crashed with ArgumentError on insert. An ETS table has to be owned by a process that outlives its users, so own it from a supervised GenServer instead. Co-authored-by: Cursor <cursoragent@cursor.com>
…check BEP 3 verifies a piece against the SHA-1 in the metainfo, but a failure only told us the piece was bad, never who sent it: the worker dropped the data, re-requested every block, and a peer serving garbage was picked again. One torrent sat at 99.84% for hours re-downloading the same index. A piece now remembers which peer supplied each block, so a failure whose blocks all came from one peer is reported to that peer's controller. A peer stops being asked for an index it has already corrupted, and is dropped once it has ruined @max_hash_failures distinct pieces — distinct, because retrying one poisoned piece must not count as several offences. Co-authored-by: Cursor <cursoragent@cursor.com>
Whether a piece can still be sourced is a question about that index, not about the torrent. The abort check also required the swarm to be empty, so on any torrent with peers a worker whose holders had all disconnected was never released: it held one of the @max_parallel_pieces slots with no peer monitored and its full block list unclaimed, forever. Live this was 7 of 12 slots, capping a torrent with 26 unchoked peers at 5 pieces in flight. Ask whether any connected peer still has this index. Co-authored-by: Cursor <cursoragent@cursor.com>
"Does this piece still have blocks" and "does it have blocks for you" are different questions, and piece_has_waiting?/2 answered the first: it counts blocks in flight to other peers. Outside endgame such a block cannot be handed to a second peer, so a peer pinned to a fully claimed piece just sat there until something else moved it. Live: 27 of 37 peers pinned to 4 claimed pieces while 8 active pieces with 49-64 free blocks had no peer at all. Add piece_has_unclaimed?/2 and use it both to refuse a repin onto such a piece and to free a pin that has drained. Endgame keeps the old test, since duplicating in-flight blocks is its point. Co-authored-by: Cursor <cursoragent@cursor.com>
A {:next_piece} that cannot place a piece reschedules itself, and every
independent trigger started its own such chain: the 2s reconcile tick, each
peer handoff kick, every piece's requests_are_dealt closure. Nothing
deduplicated them, so the chains accumulated. A live debug build went from
99 to about 2600 discovery dial cycles per minute over twelve minutes with
no change in swarm size, and the tracker began answering 403.
Every wake runs the same full pump attempt, so one pending wake per
controller is enough. Route them all through one timer, cancelling the
previous, and fold the queued strategy into the new one.
Co-authored-by: Cursor <cursoragent@cursor.com>
A TCP socket defaults to send_timeout: :infinity, so once a peer stopped draining its receive window our sender blocked inside :prim_inet.send/4 and never came back. Live one Peer.Sender held a 20863-message mailbox that only grows: every block, have and keepalive queues behind a write that will not complete. Give both accepted and dialled sockets a 30s send timeout and close on it, so the write fails and Peer.Sender.do_send/2 can stop the connection. 30s is long enough for a genuinely slow peer on a congested path, short enough that a dead one cannot park memory indefinitely. Co-authored-by: Cursor <cursoragent@cursor.com>
…e task The delivery task only tolerated :noproc, so a peer that shut down mid-call or was too slow to accept the block took the task down with it: 333 [error] crash reports in fifteen minutes, all restating one fact once per in-flight block, drowning the log we read to find real faults. Treat "this block is never reaching that peer" as a cancellation and log one debug line. A timeout counts, because it means the peer's socket has stopped draining; BEP 3 lets us simply not answer a request, and the peer will ask again if it still wants it. Any other exit still crashes, so a genuine fault in the delivery path is not swallowed. Co-authored-by: Cursor <cursoragent@cursor.com>
…fill Three compounding effects starved the swarm. Live, eight of nine torrents sat at 1-3 connected peers while discovery knew 37-53 candidates each: only 3-8 were dialable, and the scarce v6 pool — the only family with real yield on this host — was blocked 2-6 of 5-8. With 1-3 peers and no pieces to trade nothing unchokes us, so nothing downloaded at all. Only mark_productive/3 cleared a row, which asks "was this endpoint useful" rather than "is it reachable". A leecher with nothing to trade is kept choked, so a reachable peer never delivers bytes and carries its failure history for the whole session. Registration now clears it, for inbound peers too: if they reached us, our timeouts were about their NAT, not a dead host. Retention was refreshed on every failure, so a row re-dialled at least once per 30 min never aged out and its counter only climbed — rows reached fail_count 107-109, a permanent sticky block on an endpoint that merely fails intermittently. Escalation now counts failures within a 10 min streak window. Sticky blocks were absolute, which is right while anything else is dialable and starves the torrent once nothing is: 1151 of 1162 active blocks refused resurrection, so a 50-endpoint batch request returned 3-8. Under min_count pressure they are now last-resort, ordered productive, then soft before sticky, then v6 before v4, then fewest failures. Co-authored-by: Cursor <cursoragent@cursor.com>
Peer.Endpoints monitors the per-peer supervisor, and that supervisor is auto_shutdown: :any_significant — it exits with a bare :shutdown as soon as either child stops, whatever the child's reason was. So every disconnect logged reason=:shutdown: true of the supervisor, and useless for the one question worth asking when a swarm keeps shrinking, which is whether the peer left or we dropped it. The controller now hands its reason over from terminate/2, which runs before the supervisor exits, so the note is in place when the :DOWN arrives. A note is discarded when the endpoint registers again, since it describes the connection that ended rather than the one starting. Co-authored-by: Cursor <cursoragent@cursor.com>
The blacklist was a MapSet that only grew, so one bad frame excluded a peer for the rest of the session, across every torrent. A ten-minute live run banned 352 peer IDs while eight of nine torrents could not get past three connections — on a CGNAT host, where a torrent runs on 1-8 peers, that list *is* the swarm. A ban is a hedge against a peer that will misbehave again, and re-testing it costs one connection, so it does not need to outlive the failure by much: 30 min with a cap, swept in the background. Each ban now also logs the rule that fired, because "protocol_error" alone gave no way to tell a broken client from a rule of ours that is too strict. Co-authored-by: Cursor <cursoragent@cursor.com>
BEP 6 sends have_all, have_none, suggest_piece, allowed_fast and reject only once both sides advertised the Fast extension, and we do advertise it, so a peer sending one without it is at odds with the spec. Dropping the connection over it was still wrong, because terminate/2 also blacklists the peer ID for every torrent: 71 of 98 disconnects in one three-minute window, all of them qBittorrent and Transmission builds that do support Fast. have_all and have_none now go through the normal handler — they carry what a full or empty bitfield carries, and a seeder we record as having nothing is a connection we can never request from. The rest are advisory, or self-healing in reject's case since the request timeout already covers it, so they are logged and ignored. A frame that stops half-way is a congested path, not a slow-loris: a uTP window that collapsed, a peer that swapped. Dropping the connection is the defence there; the ban only removed a mainstream client from every torrent at once. Co-authored-by: Cursor <cursoragent@cursor.com>
A protocol error drops the peer and blacklists its ID for every torrent, but the log only carried the verdict, and a dozen rules share it. A run that banned 100 peers in five minutes gave no way to tell which one fired — so no way to tell a broken client from a rule of ours that is too strict. The generic cast handler now logs the wire message that was rejected, and the piece-bounds check logs the block it refused alongside the torrent's own geometry, which is what makes an off-by-one in our bookkeeping visible. Also updates the DialBackoff state-machine model to the current resurrection order: sticky blocks are last-resort now, not excluded. Co-authored-by: Cursor <cursoragent@cursor.com>
A cancel is not atomic. When we cancel a block, get choked, or repin a peer to another piece, we dropped the request locally the same instant — but the block may already be on the wire, and BEP 6 requires the peer to answer every request exactly once, so a cancelled request comes back as a piece or a reject one RTT later. Both landed in the "not outstanding" branch, which returned :protocol_error: disconnect plus a blacklist of the peer ID across every torrent. So the scheduler was feeding the blacklist. Every repin opened a window in which the peer's correct answer got it banned, and the Fast clients — the ones obliged to send the reject — were hit hardest. A five-minute window showed 46 bans from handle_piece and 29 from handle_reject, with the swarm never growing past a handful of peers. Withdrawn blocks are now remembered for a bounded window: an answer for one is expected, and a piece is still counted since the payload reaches storage either way. A block matching neither set is unrequested — wasteful rather than malicious, so it ends the connection at 512 blocks without banning the ID. Co-authored-by: Cursor <cursoragent@cursor.com>
The window was cleared wholesale once it passed its cap, because a MapSet has no order to evict by. A peer that gets repinned often withdraws far more blocks than the window holds, so the clear kept landing on blocks that were still genuinely in flight: a four-minute run logged 4392 answers as unsolicited, and six peers crossed the disconnect threshold on nothing but their own correct rejects. Retiring a full generation instead keeps at least one cap's worth of recent withdrawals recognisable at all times, and eviction now drops the oldest rather than everything. Co-authored-by: Cursor <cursoragent@cursor.com>
The withdrawn-request window was a set, so it could only record *that* a block had been withdrawn, not how many answers were still owed for it. Live logs show the same block requested, cancelled and requested again up to seven times against one peer, and the first answer erased the block from the window, so every later one read as unsolicited: 71% of the 3891 "unsolicited" answers in a 25-minute run were for blocks we had demonstrably requested from that same peer. BEP 6 gives an exact rule to count with — one answer per request, always — so the window is now a count per block, decremented as answers arrive. That also makes the bookkeeping independent of arrival order, which matters because a re-requested block sits in both the live set and the window at once. Co-authored-by: Cursor <cursoragent@cursor.com>
…t peer A dial batch resolved only when its slowest endpoint did, and one endpoint can hold a slot for the whole connect + handshake budget: measured successes came back at 16s, 42s and 42s, with a worst case near 55s. Until then the manager refused to start anything new for that torrent. That is worst exactly where it hurts. A starved torrent whose queue is all IPv4 gets capped to a four-endpoint probe batch, so it was making four dial attempts per minute against a CGNAT success rate near 1% — one torrent sat at zero peers for the whole run with 18 candidates queued. The torrents that did fine were the ones with deep enough queues to fill a 40-endpoint batch. Batches may now overlap, bounded by endpoints in flight (40) and concurrent batches (3), with in-flight endpoints excluded from selection so overlapping batches cannot dial the same peer twice. `dialing?` keeps its meaning as a hard stop and is now set only at saturation. Co-authored-by: Cursor <cursoragent@cursor.com>
A pin counted as "drained" whenever the piece had no unclaimed blocks left, without asking who had claimed them. A peer that had claimed every remaining block on its piece therefore made that piece look drained, was moved off it, and the move cancelled the very requests that drained it: the blocks returned to `waiting`, the piece looked workable again, and the next reconcile tick pinned the peer straight back. That oscillated at the 2 s tick. One peer re-sent the same block 79 times in ten minutes and delivered none of it, and across the swarm we sent 17 wire requests per block received while the cancels churned the withdrawn-answer window behind them. A piece now counts as drained for a peer only when it has neither unclaimed blocks nor in-flight requests from that peer, so the rule still frees peers parked on someone else's in-flight blocks — its original purpose — without evicting the peer doing the work. Measured over seven minutes on the live node: requests per block 17:1 -> 1.1:1, re-request factor 5.66x -> 1.08x, worst peer 55.8x -> 1.0x, swarm 28 -> 73 peers, aggregate 1.79 -> 2.47 MB/s. Co-authored-by: Cursor <cursoragent@cursor.com>
Two defects held peers on a piece that had nothing left for them, both
in the endgame path, and together they parked torrents just short of
completion.
First, `pin_drained?/4` branched on the *torrent's* mode. A torrent
enters `:endgame` while the piece workers it already started keep
running in normal mode, and for those the endgame branch probed
`piece_has_waiting?/2`, which counts blocks in flight to *other* peers.
A normal-mode worker never hands out a block that is already in flight,
so that is exactly the wrong answer: the pin was held for work this peer
could never be given. The probe is now `piece_serves_peer?/3` in both
modes, which is well-defined because `waiting` already encodes the mode
— a normal worker drops a block when it hands it out, an endgame worker
keeps it until delivery, so redundant re-requests stay possible and the
endgame pin stays useful until the piece is genuinely finished.
Second, `endgame_preferred_index/2` destructured the peer key as
`{_hash, peer_id}`, but `Peer.make_key/2` builds `{id, hash}`. It was
hashing the torrent hash, so every peer on a torrent got the same
answer: the mechanism meant to spread peers across all remaining indices
funnelled the whole swarm onto one. That gate now also guards the
drained-pin path, which endgame needs because `target_accepts_repin?/3`
accepts any active index there — without it the peers freed by the first
fix would all follow `reconcile_refresh_interest/4` onto whichever index
it walks first.
Measured live on a torrent stopped at 99.18% for over fifteen minutes:
8 of its 10 peers pinned to one piece with no unclaimed blocks left,
three of them unchoked and idle, while six pieces sat with all 64 blocks
unclaimed and no peer at all.
Co-authored-by: Cursor <cursoragent@cursor.com>
`stale_useless_pin?/1` required `choke_me`, so the only pin it ever released belonged to a peer the remote had choked — and a choked peer holds no requests at all, because `handle_choke/1` clears them. The harmful case was the one it skipped: an unchoked peer keeps a full 64-request pipeline, so every block it sits on is unavailable to anyone else and is merely re-timed-out every `@timeout_request` and handed straight back to it. Live on a torrent stopped at 99%: two peers held one whole piece each, 61 and 64 blocks in flight, having delivered 48 KiB and 0 B, while two pieces with all 64 blocks unclaimed had no peer at all. The same two peers re-requested their 64 blocks 821 and 622 times in five minutes, the interval alternating ~30 s / ~24 s — the block timeout reclaiming them and the pump returning them to the peer that had just failed them. Zero bytes on the pin now releases it whichever way the choke went, on a longer threshold when unchoked (60 s) than when choked (15-20 s): longer than a block timeout, so a merely slow peer gets a full request cycle to produce something before it loses the pin. This is what other clients call snubbing. Co-authored-by: Cursor <cursoragent@cursor.com>
…sition A piece worker reads the torrent's mode once, in State.download/3, so a piece that was already downloading when the torrent crossed the endgame threshold ran in normal mode for the rest of its life -- and normal mode gives a block to exactly one peer. Redundancy was therefore absent from precisely the pieces endgame exists for. Live: a torrent sat at 99.939% for over an hour on one 1 MiB piece. The worker was mode=nil under an :endgame torrent with waiting=[] and all 27 remaining blocks in flight to a single peer that had logged 297 request timeouts on that piece and zero hash failures, while three unchoked peers holding the piece had nothing they were allowed to ask for. Each block was only retried on its own 30s timeout and handed straight back to the peer that had just failed it. :reconcile_pump now upgrades active workers level-triggered. enter_endgame/1 is idempotent and one-way, and re-queues the in-flight subpieces into waiting so other peers may request them while the requests already out stay in place -- endgame adds sources for a block rather than taking it from the peer fetching it. The pending stall/orphan timer is cancelled and flushed, since delivered late it would abort a worker that has just started making progress. Redundancy stays capped at @endgame_redundancy, and pieces started after the transition already behaved this way. Co-authored-by: Cursor <cursoragent@cursor.com>
Swarm health under CGNAT. Three separate mechanisms were shedding peers a handful-of-peers swarm cannot afford — cancelled requests answered correctly and punished as protocol errors, Fast messages treated the same way, and a blacklist that never expired — while the piece scheduler parked the survivors on work they were not allowed to do, leaving torrents at 99% with unchoked idle peers beside unclaimed blocks. Also: endgame now reaches pieces already in flight, dial batches overlap instead of serialising on their slowest endpoint, and writes to a peer that stopped reading are bounded. The peer ID prefix is derived from the package version, so this also moves the advertised BEP 20 prefix from ET0-6-5 to ET0-6-6 — recorded in README and PROTOCOL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Overlapping dial batches pushed two functions past Credo's ABC ceiling of 30,
which fails the strict Credo gate: dial_batch/2 reached 42, because selecting
the peers, the empty-queue replenish branch and the task launch all shared one
body, and the {:dial_done, ...} clause reached 35 for rebuilding six state
fields inline.
No behaviour change. select_dial_peers/3 and launch_dial_batch/3 split
dial_batch/2 along the branch it already had, and release_dial_batch/3 takes the
state rebuild out of handle_info/2 — which also gives the rule a name: a failed
endpoint stays queued so backoff ordering decides when to retry it, while the
rest leave the queue with the batch, and the slots are freed whatever the
results were so the next batch can start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mix sobelow --out writes sobelow.sarif into the repo root. CI regenerates it per run on a throwaway runner, but reproducing the CI gates locally before a PR leaves it in the working tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts 0.6.6 on top of 20 swarm-health fixes. The BEP 20 peer-id prefix is derived from the package version, so this also moves the advertised prefix from
ET0-6-5toET0-6-6(recorded inREADME.mdandPROTOCOL.md).Why
On a CGNAT host a torrent runs on a handful of peers, and three separate mechanisms were shedding peers we could not afford to lose:
handle_pieceand 29 fromhandle_rejectin five minutes, hitting Fast clients hardest precisely because they are the ones obliged to reject.MapSetthat only grew: 352 IDs in ten minutes, while eight of nine torrents could not exceed three connections.Meanwhile the piece scheduler parked the survivors on work they were not allowed to do — torrents sat at 99% for hours with unchoked idle peers beside unclaimed blocks.
What changed
Peer bans / protocol errors
have_all/have_nonego through the normal bitfield handler — a seeder recorded as having nothing is a peer we can never request from. The advisory Fast messages are logged and ignored. A truncated frame still drops the connection but no longer bans.Peer.Endpointsmonitors a supervisor withauto_shutdown: :any_significant, which exits with a bare:shutdownwhatever the child's reason was, so every disconnect readreason=:shutdown.Piece scheduling
State.download/3, so exactly the pieces endgame exists for ran without redundancy — one 1 MiB piece held a torrent at 99.939% for over an hour.choke_me, but a choked peer holds no requests at all — the harmful case was the one it skipped.endgame_preferred_index/2destructured the peer key backwards, funnelling the whole swarm onto one index.Dial / transport
min_countpressure instead of absolute (1151 of 1162 active blocks were refusing resurrection).perf(dial): batches overlap instead of serialising on their slowest endpoint, bounded by endpoints in flight (40) and concurrent batches (3).send_timeout: :infinity, and one sender held a 20863-message mailbox. Both accepted and dialled sockets now take a 30 s send timeout.{:next_piece}chains accumulated to ~2600 discovery dial cycles/min until the tracker answered 403.Validation
mix compile --warnings-as-errorsclean.test/peer_id_test.exs,test/dht_config_test.exs— the DHTvfield takes the first 4 bytes of the prefix): 11 passed.origin/master(picks up the dependabot github-actions bump); linear, no conflicts.Full detail per fix is in the
CHANGELOG.mdentry.🤖 Generated with Claude Code