Skip to content

Fix native CDN downloader: full host pool, per-host tokens, chunk requeue + diagnostics - #145

Open
teneburu wants to merge 1 commit into
Midrags:mainfrom
teneburu:fix-native-cdn-downloader
Open

Fix native CDN downloader: full host pool, per-host tokens, chunk requeue + diagnostics#145
teneburu wants to merge 1 commit into
Midrags:mainfrom
teneburu:fix-native-cdn-downloader

Conversation

@teneburu

@teneburu teneburu commented Aug 21, 2026

Copy link
Copy Markdown

Fixes #144

Fixes the chunk-abort failure mode documented in #144 — the Linux native downloader dying on a small subset of chunks (e.g. e85d1aa4663e84da…, 9afceb218059d981…, 33940e070ebbf0b4…) while everything else downloads.

What was wrong

  1. Server list always collapsed to one host. _get_cdn_servers checked isinstance(cdn_client.servers, list) — the steam lib returns a deque of ContentServer objects, so the check was always False and the fallback [{"host": "steampipe.akamaized.net"}] was used every run. All chunk retries ((index + retry) % len(server_hosts)) then hit the same edge.
  2. One auth token applied to all hosts. Steam CDN auth tokens are host-scoped; the code minted a single token and appended it to every host's chunk URL, so chunks routed to other hosts got deny blobs.
  3. Abort-on-first-failure. Any chunk failing its 3+1 attempts killed the whole 54k-chunk depot.

What this PR does

  • _get_cdn_servers tries, in order: the Steam client's server list (normalized, deque fixed) → the public GetServersForSteamPipe webapi via httpx (the bundled requests/gevent stack stalls inside the AppImage on this call, issue Linux native downloader aborts on specific chunks; CDN server-list fetch times out inside AppImage (single-host fallback) #144, while the endpoint answers in ~0.3s standalone) → a multi-host hardcoded fallback (steampipe.akamaized.net + cs.steampowered.com).
  • Per-host tokens, minted lazily; a deny blob (tiny HTTP 200 that fails decrypt) or 401/403 triggers a one-time re-mint.
  • Chunk fetch rotates the full unique host pool (2 rounds); a chunk fails only after every host denies it, then the depot reports the per-attempt diagnostics instead of a bare Failed chunk <sha>.
  • Observability: [native] CDN servers: N host(s) (…) on start, DEBUG per failed attempt (host/status/size/outcome/elapsed), and a per-chunk diagnostic line per host on final failure.

Verification

Hermetic tests (fake CDN + fake Steam client, no network) cover: full-pool rotation finding a chunk only one non-preferred host serves (the #144 regression), per-host token minting, deny-blob → re-mint recovery, missing-everywhere diagnostics, and server-list source ordering. Run locally with python -m pytest tests/test_native_downloader.py (tests/ is gitignored per project convention, kept out of the diff).

…ueue + diagnostics

Closes the chunk-abort failure mode behind issue Midrags#144:

- _get_cdn_servers now tries the Steam client's server list (normalizing
  the deque of ContentServer objects - the old isinstance(list) check
  silently rejected it and always fell back), then the public
  GetServersForSteamPipe webapi via httpx (bypassing the bundled
  requests/gevent stack that stalls in the AppImage), then a multi-host
  hardcoded fallback instead of a single steampipe host.
- CDN auth tokens are minted per host (they are host-scoped) instead of
  one token applied to every host; a deny blob (tiny 200 that fails
  decrypt) or HTTP 401/403 triggers a token re-mint.
- Chunk fetch rotates across the FULL unique host pool (2 rounds) and a
  chunk only fails after every host denies it, instead of aborting the
  whole depot on the first failure.
- Per-attempt diagnostics (host, HTTP status, size, outcome, elapsed)
  are logged at DEBUG and printed per chunk on final failure, so a dead
  chunk is debuggable instead of a bare 'Failed chunk <sha>'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linux native downloader aborts on specific chunks; CDN server-list fetch times out inside AppImage (single-host fallback)

1 participant