Fix native CDN downloader: full host pool, per-host tokens, chunk requeue + diagnostics - #145
Open
teneburu wants to merge 1 commit into
Open
Fix native CDN downloader: full host pool, per-host tokens, chunk requeue + diagnostics#145teneburu wants to merge 1 commit into
teneburu wants to merge 1 commit into
Conversation
…ueue + diagnostics Closes the chunk-abort failure mode behind issue Midrags#144: - _get_cdn_servers now tries the Steam client's server list (normalizing the deque of ContentServer objects - the old isinstance(list) check silently rejected it and always fell back), then the public GetServersForSteamPipe webapi via httpx (bypassing the bundled requests/gevent stack that stalls in the AppImage), then a multi-host hardcoded fallback instead of a single steampipe host. - CDN auth tokens are minted per host (they are host-scoped) instead of one token applied to every host; a deny blob (tiny 200 that fails decrypt) or HTTP 401/403 triggers a token re-mint. - Chunk fetch rotates across the FULL unique host pool (2 rounds) and a chunk only fails after every host denies it, instead of aborting the whole depot on the first failure. - Per-attempt diagnostics (host, HTTP status, size, outcome, elapsed) are logged at DEBUG and printed per chunk on final failure, so a dead chunk is debuggable instead of a bare 'Failed chunk <sha>'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #144
Fixes the chunk-abort failure mode documented in #144 — the Linux native downloader dying on a small subset of chunks (e.g.
e85d1aa4663e84da…,9afceb218059d981…,33940e070ebbf0b4…) while everything else downloads.What was wrong
_get_cdn_serverscheckedisinstance(cdn_client.servers, list)— the steam lib returns adequeofContentServerobjects, so the check was always False and the fallback[{"host": "steampipe.akamaized.net"}]was used every run. All chunk retries ((index + retry) % len(server_hosts)) then hit the same edge.What this PR does
_get_cdn_serverstries, in order: the Steam client's server list (normalized, deque fixed) → the publicGetServersForSteamPipewebapi via httpx (the bundled requests/gevent stack stalls inside the AppImage on this call, issue Linux native downloader aborts on specific chunks; CDN server-list fetch times out inside AppImage (single-host fallback) #144, while the endpoint answers in ~0.3s standalone) → a multi-host hardcoded fallback (steampipe.akamaized.net+cs.steampowered.com).Failed chunk <sha>.[native] CDN servers: N host(s) (…)on start, DEBUG per failed attempt (host/status/size/outcome/elapsed), and a per-chunk diagnostic line per host on final failure.Verification
Hermetic tests (fake CDN + fake Steam client, no network) cover: full-pool rotation finding a chunk only one non-preferred host serves (the #144 regression), per-host token minting, deny-blob → re-mint recovery, missing-everywhere diagnostics, and server-list source ordering. Run locally with
python -m pytest tests/test_native_downloader.py(tests/ is gitignored per project convention, kept out of the diff).