Skip to content

Release: the ~1 GB Windows asset upload fails or hangs on every release #166

Description

@luthermonson

The Windows release asset upload fails or hangs on every release. Twice in a row now, and both needed manual intervention.

v0.2.2Create draft release sat in progress for 516 minutes (8.6 hours) holding a self-hosted runner slot, with 4 of 5 assets uploaded. Cancelling and re-running the publish job succeeded.

v0.2.3 — same step, same asset, failed outright instead of hanging:

⬆️ Uploading ephemerd_v0.2.3_windows_amd64.zip...
##[error]other side closed

Both times the other four assets uploaded fine and checksums.txt already contained the Windows entry — the checksum step runs before the upload, so the file demonstrably exists and only the upload fails.

The distinguishing factor is size

ephemerd_v0.2.3_windows_amd64.zip    984 MB
ephemerd_v0.2.2_darwin_arm64.tar.gz  361 MB
ephemerd_v0.2.2_linux_amd64.tar.gz   294 MB
ephemerd_v0.2.2_linux_arm64.tar.gz   205 MB

The Windows asset is ~3x the next largest. On disk the installed binary is ~1.1 GB, because it embeds platform-specific VM assets (kernel, initrd, rootfs) for the Linux sidecar.

Why it matters beyond the annoyance

  • Every release needs a manual retry, which is exactly the kind of step that gets skipped or forgotten. A draft release that never publishes 404s every node — the [ephemerd] config comment already warns about this, and v0.1.5 apparently sat undeployable for that reason.
  • A hung publish holds a self-hosted runner for as long as it hangs. 8.6 hours on v0.2.2.
  • Every node downloads it on upgrade. ~1 GB per Windows node per upgrade, which cuts against the bandwidth reduction work.

Possible directions, unevaluated

  • Retry with backoff around the upload step specifically, rather than failing the whole job.
  • A step timeout so a hang fails fast instead of holding a runner for hours.
  • Look at whether the Windows binary needs everything it currently embeds — if the Linux VM assets are only used when [vm.linux] is enabled, shipping them unconditionally is ~700 MB per download for a feature many nodes never turn on.

The last one is speculative; I have not checked what the embed actually contains or whether it can be made conditional.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions