Skip to content

npm: read the stored publish time in the cooldown download check - #296

Merged
andrew merged 1 commit into
git-pkgs:mainfrom
VictorCodesseira:feat/npm-cooldown-published-at
Sep 2, 2026
Merged

npm: read the stored publish time in the cooldown download check#296
andrew merged 1 commit into
git-pkgs:mainfrom
VictorCodesseira:feat/npm-cooldown-published-at

Conversation

@VictorCodesseira

@VictorCodesseira VictorCodesseira commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

With cooldown enabled, versionInCooldown runs on every npm tarball request —
cache hits included — and loads the package's full packument through
FetchOrCacheMetadata, unmarshalling the entire body to read one entry of the
time map. For packages like typescript or @swc/core that is a multi-megabyte
buffer and parse per tarball, and with an S3 storage backend it is also one
storage GET per tarball.

Under a heavy load (bursts of several hundred concurrent requests) this
dominates the proxy's cost: I measured ~12-16ms of CPU per request with cooldown on versus
well under 1ms with it off, install fetch times of ~100s that dropped to ~9s
when cooldown was disabled, and higher memory usage with it enabled, causing OOM kills.

A version's publish time is immutable, so the check now reads the stored
versions.published_at first and only falls back to the packument for a
version the proxy has never seen, persisting the parsed time so the packument
is fetched and parsed at most once per version. Same shape as the PyPI fix
from #242.

Changes:

  • versionInCooldown consults versions.published_at before fetching
    metadata, and persists the parsed time after the packument fallback
  • Add DB.SetVersionPublishedAt, an upsert that writes only the publish time
    so it never disturbs enrichment data on an existing row
  • UpsertVersion now preserves a stored published_at when the incoming
    value is NULL (COALESCE, both dialects). Without this, caching the
    artifact right after the check upserts the versions row without a publish
    time and erases the value the same request just stored; it also means a
    null-bearing upsert can no longer erase enrichment data
  • Tests: downloads with a stored publish time never touch the metadata
    upstream (served and withheld cases); two downloads of an uncached version
    fetch the packument exactly once, proving the stored time survives the
    artifact-cache upsert; database test for preserve-on-NULL and
    update-on-value in both dialects

go test ./..., go fmt, go vet clean.

Companion PR: #297 (serving publish times without enabling cooldown).

- Consult versions.published_at before fetching the packument, and persist
  the parsed time after the packument fallback, so each version's metadata
  is fetched and parsed at most once
- Add DB.SetVersionPublishedAt, an upsert that writes only the publish time
- Preserve a stored published_at in UpsertVersion when the incoming value
  is NULL, so the artifact-cache upsert cannot erase it
- Add handler tests for stored-time downloads and single-fetch behavior,
  and a database test for preserve-on-NULL in both dialects

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice — same shape as the PyPI fix in #242, and the COALESCE on UpsertVersion is a strict improvement (only caller is updateCacheDB, which never sets PublishedAt). Thanks for the measurements.

@andrew
andrew merged commit 76fcd07 into git-pkgs:main Sep 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants