diff --git a/README.md b/README.md index 2dfbb13..16eb45a 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,26 @@ A 3-day cooldown means that when `lodash` publishes version `4.18.0`, your build Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates. See [docs/configuration.md](docs/configuration.md) for the full config reference. +## Artifact Scanning + +Cooldown only looks at a version's publish timestamp — it never inspects the actual bytes. Artifact scanning closes that gap: when enabled, every artifact is staged into storage and scanned by one or more external services (trivy, ClamAV, Wiz, or anything else that speaks a small HTTP/JSON contract) before it's committed to the cache and served to clients. + +```yaml +scanning: + enabled: true + signing_key: ${PROXY_SCANNING_SIGNING_KEY} + scanners: + - name: clamav + url: http://clamav-adapter:8080/scan + mode: block # a block verdict deletes the artifact and returns 403 + - name: trivy + url: http://trivy-adapter:8081/scan + mode: monitor # findings are logged, never gate caching + ecosystems: [npm, pypi] +``` + +The proxy never uploads artifact bytes to a scanner. Each scanner is notified with package metadata plus a short-lived signed URL; the scanner pulls the bytes itself from the proxy's own storage. Scanners run concurrently, and the first `block`-mode scanner to report a verdict of not-allowed wins immediately, canceling the rest. See [docs/configuration.md](docs/configuration.md) for the full config reference and the scanner HTTP contract. + ## Supported Registries | Registry | Language/Platform | Cooldown | Completed | diff --git a/config.example.yaml b/config.example.yaml index 277b58c..f348ba4 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -280,3 +280,32 @@ cooldown: # packages: # "pkg:npm/lodash": "0" # "pkg:npm/@babel/core": "14d" + +# Pre-cache artifact scanning. When enabled, every artifact is staged into +# storage and scanned by the configured scanners before it is committed to +# the cache and served to clients. Scanners never receive artifact bytes +# directly — each notify call includes a short-lived signed URL that the +# scanner fetches itself, so the proxy stays agnostic to trivy/ClamAV/Wiz/ +# any custom service. Scanners run concurrently; the first "block" verdict +# wins and cancels the rest. +# scanning: +# enabled: true +# fail_open: false +# timeout: 30s +# +# # Authenticates pull requests to the internal scan-fetch route. +# # Required whenever enabled is true. Supports ${VAR_NAME} expansion. +# signing_key: ${PROXY_SCANNING_SIGNING_KEY} +# +# # Address scanners use to reach this proxy to pull staged artifacts. +# # Defaults to base_url. +# # fetch_base_url: http://proxy.internal:8080 +# +# scanners: +# - name: clamav +# url: http://clamav-adapter:8080/scan +# mode: block +# - name: trivy +# url: http://trivy-adapter:8081/scan +# mode: monitor +# ecosystems: [npm, pypi] diff --git a/docs/configuration.md b/docs/configuration.md index 1b70410..6bf1954 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -370,6 +370,103 @@ Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyG Note: Hex cooldown requires disabling registry signature verification since the proxy re-encodes the protobuf payload without the original signature. Set `HEX_NO_VERIFY_REPO_ORIGIN=1` or configure your repo with `no_verify: true`. +## Artifact Scanning + +Cooldown only ever looks at a version's *publish timestamp* — it never inspects the actual bytes of an artifact. Artifact scanning runs after a fetched artifact is staged into storage but before it becomes visible from cache, so an external scanner (trivy, ClamAV, Wiz, or any custom service) can block a bad verdict from ever reaching a client. + +```yaml +scanning: + enabled: true + fail_open: false + timeout: 30s + signing_key: ${PROXY_SCANNING_SIGNING_KEY} + fetch_base_url: http://proxy.internal:8080 + scanners: + - name: clamav + url: http://clamav-adapter:8080/scan + mode: block + - name: trivy + url: http://trivy-adapter:8081/scan + mode: monitor + ecosystems: [npm, pypi] +``` + +| Config | Environment | Description | +|--------|-------------|-------------| +| `scanning.enabled` | `PROXY_SCANNING_ENABLED` | Turn on the scan gate. When false (default), artifacts are cached exactly as if scanning didn't exist | +| `scanning.fail_open` | `PROXY_SCANNING_FAIL_OPEN` | Treat scanner errors/timeouts as allow instead of block. Default is fail-closed | +| `scanning.timeout` | `PROXY_SCANNING_TIMEOUT` | Per-scan-call timeout, Go duration syntax (default `30s`) | +| `scanning.signing_key` | `PROXY_SCANNING_SIGNING_KEY` | Signs pull requests to the internal scan-fetch route. Required whenever `enabled` is true | +| `scanning.fetch_base_url` | `PROXY_SCANNING_FETCH_BASE_URL` | Address scanners use to reach this proxy to pull staged artifacts. Defaults to `base_url` | +| `scanning.scanners` | - | List of external scanning services (YAML only) | +| `scanning.scanners[].name` | - | Identifies this scanner in logs and metrics | +| `scanning.scanners[].url` | - | Endpoint the proxy POSTs scan notifications to | +| `scanning.scanners[].mode` | - | `block` (default) or `monitor` | +| `scanning.scanners[].ecosystems` | - | Restricts this scanner to specific ecosystems (e.g. `npm`, `pypi`). Empty means all ecosystems | +| `scanning.scanners[].headers` | - | Extra HTTP headers sent with every scan request (e.g. for authenticating to the scanner service). Values support `${VAR_NAME}` expansion | + +### How caching defers to a scan verdict + +The proxy never uploads artifact bytes to a scanner. When an artifact is fetched from upstream, it's stored to the configured storage backend first, exactly as without scanning. If scanning is enabled for the artifact's ecosystem, the proxy then notifies each applicable scanner with package metadata and a short-lived, HMAC-signed URL pointing at the internal `/_internal/scan-fetch` route; each scanner GETs that URL itself to pull the exact bytes staged in storage and runs its own scan against them. + +Scanners configured for the same ecosystem all run concurrently, never sequentially. The moment any `block`-mode scanner reports a not-allowed verdict (or errors, unless `fail_open` is set), the proxy cancels the in-flight calls to the other scanners and deletes the staged artifact — it's never committed to the cache database, so it was never visible to a client. If nothing blocks, the proxy waits for every `block`-mode scanner to finish before caching the artifact and serving it. A `monitor`-mode scanner's findings are logged and never gate the wait or the caching decision, even when it reports not-allowed. + +A blocked download surfaces to the client as `403 Forbidden` with the scanner's reason, across every ecosystem handler. + +### Scanner HTTP contract + +Any external service that implements this contract can act as a scanner — a trivy wrapper, a clamav-rest bridge, a Wiz connector, or an in-house service. The proxy POSTs a notify request to `scanning.scanners[].url` and waits for a JSON verdict. + +**Request** + +| Field | Type | Description | +|-------|------|-------------| +| `ecosystem` | string | e.g. `npm`, `pypi`, `cargo` | +| `name` | string | Package name | +| `version` | string | Package version | +| `filename` | string | Artifact filename | +| `purl` | string | Package URL (PURL) identifying this exact version | +| `content_type` | string | Artifact content type | +| `size` | integer | Artifact size in bytes | +| `fetch_url` | string | Short-lived signed URL; GET this to retrieve the exact staged bytes | + +```json +{ + "ecosystem": "npm", "name": "left-pad", "version": "1.0.0", + "filename": "left-pad-1.0.0.tgz", "purl": "pkg:npm/left-pad@1.0.0", + "content_type": "application/octet-stream", "size": 1234, + "fetch_url": "https://proxy.internal/_internal/scan-fetch?path=...&exp=...&sig=..." +} +``` + +**Response** + +| Field | Type | Description | +|-------|------|-------------| +| `allowed` | boolean | Whether the artifact may be cached and served | +| `reason` | string | Human-readable reason, surfaced to the client when `allowed` is false | +| `findings` | array | Optional list of `{"severity", "title", "description"}` objects | + +```json +{ + "allowed": false, + "reason": "malware detected", + "findings": [ + {"severity": "critical", "title": "Trojan.GenericKD", "description": "..."} + ] +} +``` + +The scanner must respond within `scanning.timeout` (default `30s`); a timeout is treated the same as a `block` verdict unless `fail_open` is set. + +### The `/_internal/scan-fetch` route + +`fetch_url` points at an internal route, `/_internal/scan-fetch`, that streams a staged object straight from the proxy's storage backend via a short-lived HMAC-signed token (`path`, `exp`, `sig` query parameters). This works identically across every storage backend — local filesystem, S3, GCS, Azure — since it never depends on a backend-specific presigned URL, only on the one storage operation every backend already implements. + +This route is not part of the public API. It's meant only for scanners to pull artifacts they've been notified about, and should be restricted to internal-network access at the ingress/network-policy layer — the HMAC scoping (one object, a short TTL) limits what a leaked token can do, but isn't a substitute for network restriction. Its query parameters are also documented in the generated [OpenAPI spec](../README.md#openapi-swagger). + +The route only exists when scanning is actually configured: it's not mounted at all unless at least one scanner is enabled and `scanning.signing_key` is set, and it also refuses every request with `404` if either condition somehow isn't met at request time. There is no way to reach it, even with a forged token, when scanning is disabled. + ## Metadata Caching By default the proxy fetches metadata fresh from upstream on every request. Enable `cache_metadata` to store metadata responses in the database and storage backend for offline fallback. When upstream is unreachable, the proxy serves the last cached copy. ETag-based revalidation avoids re-downloading unchanged metadata. diff --git a/docs/swagger/docs.go b/docs/swagger/docs.go index 76b4828..cc88b4c 100644 --- a/docs/swagger/docs.go +++ b/docs/swagger/docs.go @@ -15,6 +15,61 @@ const docTemplate = `{ "host": "{{.Host}}", "basePath": "{{.BasePath}}", "paths": { + "/_internal/scan-fetch": { + "get": { + "description": "Streams the exact bytes staged in storage for a pre-cache security scan.\nRequires a short-lived HMAC-signed token minted by the proxy itself and\ndelivered via the fetch_url field of the scan notify request (see the\nArtifact Scanning section of docs/configuration.md). Not part of the\npublic API; restrict access to the scanner network at the ingress layer.", + "produces": [ + "application/octet-stream" + ], + "tags": [ + "scanning" + ], + "summary": "Fetch a staged artifact for scanning", + "parameters": [ + { + "type": "string", + "description": "Storage path of the staged artifact", + "name": "path", + "in": "query", + "required": true + }, + { + "type": "integer", + "description": "Token expiry, Unix seconds", + "name": "exp", + "in": "query", + "required": true + }, + { + "type": "string", + "description": "HMAC-SHA256 signature over the string path|exp", + "name": "sig", + "in": "query", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "file" + } + }, + "403": { + "description": "invalid, expired, or tampered token", + "schema": { + "type": "string" + } + }, + "404": { + "description": "object not found in storage, or scanning is not configured", + "schema": { + "type": "string" + } + } + } + } + }, "/api/bulk": { "post": { "consumes": [ diff --git a/docs/swagger/swagger.json b/docs/swagger/swagger.json index 42a2ec2..5db9166 100644 --- a/docs/swagger/swagger.json +++ b/docs/swagger/swagger.json @@ -8,6 +8,61 @@ }, "basePath": "/", "paths": { + "/_internal/scan-fetch": { + "get": { + "description": "Streams the exact bytes staged in storage for a pre-cache security scan.\nRequires a short-lived HMAC-signed token minted by the proxy itself and\ndelivered via the fetch_url field of the scan notify request (see the\nArtifact Scanning section of docs/configuration.md). Not part of the\npublic API; restrict access to the scanner network at the ingress layer.", + "produces": [ + "application/octet-stream" + ], + "tags": [ + "scanning" + ], + "summary": "Fetch a staged artifact for scanning", + "parameters": [ + { + "type": "string", + "description": "Storage path of the staged artifact", + "name": "path", + "in": "query", + "required": true + }, + { + "type": "integer", + "description": "Token expiry, Unix seconds", + "name": "exp", + "in": "query", + "required": true + }, + { + "type": "string", + "description": "HMAC-SHA256 signature over the string path|exp", + "name": "sig", + "in": "query", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "file" + } + }, + "403": { + "description": "invalid, expired, or tampered token", + "schema": { + "type": "string" + } + }, + "404": { + "description": "object not found in storage, or scanning is not configured", + "schema": { + "type": "string" + } + } + } + } + }, "/api/bulk": { "post": { "consumes": [ diff --git a/internal/config/config.go b/internal/config/config.go index 61aa490..eef19b3 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -115,6 +115,10 @@ type Config struct { // Cooldown configures version age filtering to mitigate supply chain attacks. Cooldown CooldownConfig `json:"cooldown" yaml:"cooldown"` + // Scanning configures pre-cache artifact scanning (trivy, ClamAV, Wiz, + // or a custom service) to mitigate supply chain attacks. + Scanning ScanningConfig `json:"scanning" yaml:"scanning"` + // CacheMetadata enables caching of upstream metadata responses for offline fallback. // When enabled, metadata is stored in the database and storage backend. // The mirror command always enables this regardless of this setting. @@ -189,6 +193,140 @@ func (c *CooldownConfig) NormalizedPackages() map[string]string { return normalized } +// ScanningConfig configures pre-cache artifact scanning (e.g. trivy, +// ClamAV, Wiz, or a custom service) to mitigate supply chain attacks. +// Unlike Cooldown, which only looks at a version's publish timestamp, +// scanning inspects the actual artifact bytes before they become +// servable from cache. +type ScanningConfig struct { + // Enabled turns on the scan gate. When false (default), artifacts are + // cached exactly as if scanning didn't exist. + Enabled bool `json:"enabled" yaml:"enabled"` + + // FailOpen treats scanner errors and timeouts as an allow verdict + // instead of a block. Default is fail-closed, since the default + // posture for a security gate should block on infrastructure failure. + FailOpen bool `json:"fail_open" yaml:"fail_open"` + + // Timeout bounds each scan call. Uses Go duration syntax (e.g. "30s"). + // Default: "30s". + Timeout string `json:"timeout" yaml:"timeout"` + + // SigningKey authenticates pull requests to the internal scan-fetch + // route used by every storage backend. Required whenever Enabled is + // true. Supports ${VAR_NAME} expansion like AuthConfig fields. + SigningKey string `json:"signing_key" yaml:"signing_key"` + + // FetchBaseURL is the address scanners use to reach this proxy to pull + // staged artifacts. Defaults to BaseURL. Set this separately when + // scanners reach the proxy over an internal address different from the + // public-facing BaseURL (mirrors DirectServeBaseURL/UIBaseURL). + FetchBaseURL string `json:"fetch_base_url" yaml:"fetch_base_url"` + + // Scanners is the list of external scanning services to call. + Scanners []ScannerConfig `json:"scanners" yaml:"scanners"` +} + +// ScannerConfig configures a single external scanning service. +type ScannerConfig struct { + // Name identifies this scanner in logs and metrics. + Name string `json:"name" yaml:"name"` + + // URL is the endpoint the proxy POSTs scan notifications to. + URL string `json:"url" yaml:"url"` + + // Mode is "block" (default) or "monitor". A "block" scanner's verdict + // can prevent caching; a "monitor" scanner's findings are logged but + // never gate caching. + Mode string `json:"mode" yaml:"mode"` + + // Ecosystems restricts this scanner to specific ecosystems (e.g. + // "npm", "pypi"). Empty means all ecosystems. + Ecosystems []string `json:"ecosystems" yaml:"ecosystems"` + + // Headers are additional HTTP headers sent with every scan request + // (e.g. for authenticating to the scanner service). Values support + // ${VAR_NAME} expansion like AuthConfig fields. + Headers map[string]string `json:"headers" yaml:"headers"` +} + +// SigningKeyExpanded returns SigningKey with ${VAR_NAME} references expanded. +func (s *ScanningConfig) SigningKeyExpanded() string { + return expandEnv(s.SigningKey) +} + +// HeadersExpanded returns Headers with ${VAR_NAME} references expanded in +// each value. +func (s *ScannerConfig) HeadersExpanded() map[string]string { + if len(s.Headers) == 0 { + return nil + } + expanded := make(map[string]string, len(s.Headers)) + for k, v := range s.Headers { + expanded[k] = expandEnv(v) + } + return expanded +} + +// Validate checks the scanning configuration for errors, applying the +// default timeout if unset. +func (s *ScanningConfig) Validate() error { + if !s.Enabled { + return nil + } + + if s.SigningKeyExpanded() == "" { + return fmt.Errorf("scanning.signing_key is required when scanning.enabled is true") + } + + if len(s.Scanners) == 0 { + return fmt.Errorf("scanning.scanners must not be empty when scanning.enabled is true") + } + + if s.FetchBaseURL != "" { + if err := validateAbsoluteURL("scanning.fetch_base_url", s.FetchBaseURL); err != nil { + return err + } + } + + if s.Timeout == "" { + s.Timeout = defaultScanningTimeoutStr + } + if d, err := time.ParseDuration(s.Timeout); err != nil { + return fmt.Errorf("invalid scanning.timeout %q: %w", s.Timeout, err) + } else if d <= 0 { + return fmt.Errorf("invalid scanning.timeout %q: must be > 0", s.Timeout) + } + + for i := range s.Scanners { + if err := s.Scanners[i].Validate(); err != nil { + return fmt.Errorf("scanning.scanners[%d]: %w", i, err) + } + } + + return nil +} + +// Validate checks a single scanner's configuration, applying the default +// mode ("block") if unset. +func (s *ScannerConfig) Validate() error { + if s.Name == "" { + return fmt.Errorf("name is required") + } + if err := validateAbsoluteURL("url", s.URL); err != nil { + return err + } + if s.Mode == "" { + s.Mode = "block" + } + switch s.Mode { + case "block", "monitor": + default: + return fmt.Errorf("invalid mode %q (must be block or monitor)", s.Mode) + } + return nil +} + // StorageConfig configures artifact storage. type StorageConfig struct { // URL is the storage backend URL. @@ -742,6 +880,11 @@ func (c *Config) LoadFromEnv() { setEnvString(&c.Upstream.Debian, "PROXY_UPSTREAM_DEBIAN") setEnvString(&c.Upstream.RPM, "PROXY_UPSTREAM_RPM") setEnvString(&c.Cooldown.Default, "PROXY_COOLDOWN_DEFAULT") + setEnvBool(&c.Scanning.Enabled, "PROXY_SCANNING_ENABLED") + setEnvBool(&c.Scanning.FailOpen, "PROXY_SCANNING_FAIL_OPEN") + setEnvString(&c.Scanning.Timeout, "PROXY_SCANNING_TIMEOUT") + setEnvString(&c.Scanning.SigningKey, "PROXY_SCANNING_SIGNING_KEY") + setEnvString(&c.Scanning.FetchBaseURL, "PROXY_SCANNING_FETCH_BASE_URL") setEnvBool(&c.CacheMetadata, "PROXY_CACHE_METADATA") setEnvBool(&c.MirrorAPI, "PROXY_MIRROR_API") setEnvString(&c.MetadataTTL, "PROXY_METADATA_TTL") @@ -858,6 +1001,10 @@ func (c *Config) validateComponents() error { return err } + if err := c.Scanning.Validate(); err != nil { + return err + } + return c.Gradle.BuildCache.Validate() } @@ -925,6 +1072,7 @@ const ( defaultGradleBuildCacheSweepInterval = 10 * time.Minute defaultGradleMaxUploadSizeStr = "100MB" defaultGradleSweepIntervalStr = "10m" + defaultScanningTimeoutStr = "30s" ) // ParseMaxSize returns the maximum cache size in bytes. diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 6e64391..1735c21 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -748,6 +748,117 @@ func TestValidateHealthStorageProbeInterval(t *testing.T) { } } +func TestScanningConfigValidate(t *testing.T) { + tests := []struct { + name string + cfg ScanningConfig + wantErr bool + }{ + { + name: "disabled skips validation entirely", + cfg: ScanningConfig{Enabled: false, Timeout: "not-a-duration"}, + }, + { + name: "enabled without signing key fails", + cfg: ScanningConfig{Enabled: true}, + wantErr: true, + }, + { + name: "enabled with signing key and no scanners fails", + cfg: ScanningConfig{Enabled: true, SigningKey: "s3cret"}, + wantErr: true, + }, + { + name: "invalid fetch_base_url fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + FetchBaseURL: "not-a-url", + Scanners: []ScannerConfig{{Name: "clamav", URL: "http://scanner.invalid/scan"}}, + }, + wantErr: true, + }, + { + name: "invalid timeout fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Timeout: "not-a-duration", + Scanners: []ScannerConfig{{Name: "clamav", URL: "http://scanner.invalid/scan"}}, + }, + wantErr: true, + }, + { + name: "zero timeout fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Timeout: "0", + Scanners: []ScannerConfig{{Name: "clamav", URL: "http://scanner.invalid/scan"}}, + }, + wantErr: true, + }, + { + name: "empty timeout defaults and is valid", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Timeout: "", + Scanners: []ScannerConfig{{Name: "clamav", URL: "http://scanner.invalid/scan"}}, + }, + }, + { + name: "scanner missing name fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Scanners: []ScannerConfig{{URL: "http://scanner.invalid/scan"}}, + }, + wantErr: true, + }, + { + name: "scanner with invalid url fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Scanners: []ScannerConfig{{Name: "clamav", URL: "not-a-url"}}, + }, + wantErr: true, + }, + { + name: "scanner with invalid mode fails", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Scanners: []ScannerConfig{ + {Name: "clamav", URL: "http://scanner.invalid/scan", Mode: "quarantine"}, + }, + }, + wantErr: true, + }, + { + name: "scanner with default mode is valid", + cfg: ScanningConfig{ + Enabled: true, + SigningKey: "s3cret", + Scanners: []ScannerConfig{{Name: "clamav", URL: "http://scanner.invalid/scan"}}, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := tt.cfg.Validate() + if tt.wantErr && err == nil { + t.Error("Validate() error = nil, want error") + } + if !tt.wantErr && err != nil { + t.Errorf("Validate() unexpected error: %v", err) + } + }) + } +} + func TestParseHTTPTimeout(t *testing.T) { tests := []struct { name string diff --git a/internal/handler/container.go b/internal/handler/container.go index 722e8f8..7467fa9 100644 --- a/internal/handler/container.go +++ b/internal/handler/container.go @@ -155,6 +155,10 @@ func (h *ContainerHandler) handleBlobDownload(w http.ResponseWriter, r *http.Req h.containerError(w, http.StatusNotFound, "BLOB_UNKNOWN", "blob unknown to registry") return } + if errors.Is(err, ErrArtifactBlocked) { + h.containerError(w, http.StatusForbidden, "DENIED", err.Error()) + return + } h.proxy.Logger.Error("failed to fetch blob", "error", err) h.containerError(w, http.StatusBadGateway, "INTERNAL_ERROR", "failed to fetch blob") return diff --git a/internal/handler/go.go b/internal/handler/go.go index a03b10c..b562aca 100644 --- a/internal/handler/go.go +++ b/internal/handler/go.go @@ -117,6 +117,10 @@ func (h *GoHandler) handleDownload(w http.ResponseWriter, r *http.Request, modul http.Error(w, "not found", http.StatusNotFound) return } + if errors.Is(err, ErrArtifactBlocked) { + http.Error(w, err.Error(), http.StatusForbidden) + return + } h.proxy.Logger.Error("failed to get artifact", "error", err) http.Error(w, "failed to fetch module", http.StatusBadGateway) return diff --git a/internal/handler/handler.go b/internal/handler/handler.go index 6f4ceaf..7384f66 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -20,6 +20,7 @@ import ( "github.com/git-pkgs/proxy/internal/database" "github.com/git-pkgs/proxy/internal/metrics" "github.com/git-pkgs/proxy/internal/packageurl" + "github.com/git-pkgs/proxy/internal/scanner" "github.com/git-pkgs/proxy/internal/storage" "github.com/git-pkgs/purl" "github.com/git-pkgs/registries/fetch" @@ -145,6 +146,19 @@ type Proxy struct { DirectServeBaseURL string HTTPClient *http.Client AuthForURL func(string) (headerName, headerValue string) + + // Scanners runs pre-cache artifact scanning (e.g. trivy, ClamAV, Wiz). + // Nil or disabled means artifacts are cached without scanning. + Scanners *scanner.Group + + // ScanSigningKey authenticates pull requests to the internal + // /_internal/scan-fetch route used by scanners to retrieve staged + // artifacts, for every storage backend. + ScanSigningKey []byte + + // ScanFetchBaseURL is the address scanners use to reach this proxy to + // pull staged artifacts. + ScanFetchBaseURL string } // NewProxy creates a new Proxy with the given dependencies. @@ -357,20 +371,53 @@ func (p *Proxy) fetchAndCache(ctx context.Context, ecosystem, name, version, fil } metrics.RecordUpstreamFetch(ecosystem, fetchDuration) - // Store in cache + return p.storeArtifact(ctx, ecosystem, name, version, filename, pkgPURL, versionPURL, info.URL, "", artifact) +} + +// storeArtifact writes a fetched artifact to storage, verifies it against +// upstreamHash if non-empty, runs it through the scan gate if scanning is +// enabled, and commits it to the cache database. +// +// The scan gate sits between Storage.Store and updateCacheDB: updateCacheDB +// is the only thing that makes an artifact visible to clients (checkCache +// looks up its row before touching Storage), so deferring it until after a +// verdict means a blocked artifact was never reachable by any client. On +// block, the just-stored bytes are deleted and ErrArtifactBlocked is +// returned; updateCacheDB is never called. +func (p *Proxy) storeArtifact(ctx context.Context, ecosystem, name, version, filename, pkgPURL, versionPURL, upstreamURL, upstreamHash string, artifact *fetch.Artifact) (*CacheResult, error) { storagePath := storage.ArtifactPath(ecosystem, "", name, version, filename) + storeStart := time.Now() size, hash, err := p.Storage.Store(ctx, storagePath, artifact.Body) _ = artifact.Body.Close() metrics.RecordStorageOperation("write", time.Since(storeStart)) - if err != nil { metrics.RecordStorageError("write") return nil, fmt.Errorf("storing artifact: %w", err) } + if !artifactHashMatches(hash, upstreamHash) { + if delErr := p.Storage.Delete(ctx, storagePath); delErr != nil { + p.Logger.Warn("failed to discard artifact with mismatched checksum", "path", storagePath, "error", delErr) + } + return nil, fmt.Errorf("artifact checksum mismatch: upstream declared %s, got %s", upstreamHash, hash) + } + + if p.Scanners != nil && p.Scanners.Enabled() { + if err := p.runScan(ctx, ecosystem, name, version, filename, versionPURL, storagePath, size, artifact.ContentType); err != nil { + // Detached from ctx: a client disconnecting must not abort + // cleanup of a genuinely blocked artifact and leave its bytes + // orphaned in storage with no DB row pointing at them. + if delErr := p.Storage.Delete(context.WithoutCancel(ctx), storagePath); delErr != nil { + p.Logger.Warn("failed to delete blocked artifact from storage", + "path", storagePath, "error", delErr) + } + return nil, err + } + } + // Update database - if err := p.updateCacheDB(ecosystem, name, filename, pkgPURL, versionPURL, info.URL, storagePath, hash, size, artifact.ContentType); err != nil { + if err := p.updateCacheDB(ecosystem, name, filename, pkgPURL, versionPURL, upstreamURL, storagePath, hash, size, artifact.ContentType); err != nil { p.Logger.Warn("failed to update cache database", "error", err) // Continue anyway - we have the file } @@ -394,6 +441,43 @@ func (p *Proxy) fetchAndCache(ctx context.Context, ecosystem, name, version, fil }, nil } +// runScan generates a signed fetch URL for the just-staged artifact and +// asks the configured scanners for a verdict. Returns a wrapped +// ErrArtifactBlocked if any scanner blocks, or a scan-infrastructure error. +// +// The scan call runs on a context detached from ctx's cancellation +// (context.WithoutCancel): ctx is the original client request's context, and +// a client disconnecting mid-scan must not be indistinguishable from a real +// scanner verdict. Group.Scan still bounds the call with its own configured +// timeout, so a detached context cannot hang forever. +func (p *Proxy) runScan(ctx context.Context, ecosystem, name, version, filename, purlStr, storagePath string, size int64, contentType string) error { + fetchURL := p.scanFetchURL(storagePath, p.Scanners.Timeout()) + result := p.Scanners.Scan(context.WithoutCancel(ctx), scanner.Request{ + Ecosystem: ecosystem, + Name: name, + Version: version, + Filename: filename, + PURL: purlStr, + FetchURL: fetchURL, + Size: size, + ContentType: contentType, + }) + if !result.Allowed { + p.Logger.Warn("artifact blocked by security scan", + "ecosystem", ecosystem, "name", name, "version", version, "filename", filename, + "scanner", result.ScannerName, "reason", result.Reason, "infra_error", result.InfraError) + reason := result.Reason + if result.InfraError { + // result.Reason may contain raw scanner-infrastructure details + // (internal hostnames, ports, connection errors) that must not + // reach an untrusted client via the 403 response body. + reason = "scan could not be completed" + } + return fmt.Errorf("%w: %s", ErrArtifactBlocked, reason) + } + return nil +} + func (p *Proxy) updateCacheDB(ecosystem, name, filename, pkgPURL, versionPURL, upstreamURL, storagePath, hash string, size int64, contentType string) error { now := time.Now() @@ -549,13 +633,21 @@ func JSONError(w http.ResponseWriter, status int, message string) { // ErrUpstreamNotFound indicates the upstream returned 404. var ErrUpstreamNotFound = fmt.Errorf("upstream: %w", fetch.ErrNotFound) +// ErrArtifactBlocked indicates a pre-cache security scan blocked the artifact. +var ErrArtifactBlocked = errors.New("artifact blocked by security scan") + // serveArtifactError writes response for a failed fetch: -// 404 when upstream reports artifact missing, 502 otherwise. +// 404 when upstream reports artifact missing, 403 when a security scan +// blocked the artifact, 502 otherwise. func (p *Proxy) serveArtifactError(w http.ResponseWriter, err error, clientMsg string) { if errors.Is(err, ErrUpstreamNotFound) { http.Error(w, "not found", http.StatusNotFound) return } + if errors.Is(err, ErrArtifactBlocked) { + JSONError(w, http.StatusForbidden, err.Error()) + return + } p.Logger.Error("failed to get artifact", "error", err) http.Error(w, clientMsg, http.StatusBadGateway) } @@ -967,41 +1059,7 @@ func (p *Proxy) fetchAndCacheFromURL(ctx context.Context, ecosystem, name, versi return nil, fmt.Errorf("fetching from upstream: %w", err) } - storagePath := storage.ArtifactPath(ecosystem, "", name, version, filename) - storeStart := time.Now() - size, hash, err := p.Storage.Store(ctx, storagePath, artifact.Body) - _ = artifact.Body.Close() - metrics.RecordStorageOperation("write", time.Since(storeStart)) - if err != nil { - metrics.RecordStorageError("write") - return nil, fmt.Errorf("storing artifact: %w", err) - } - if !artifactHashMatches(hash, upstreamHash) { - if err := p.Storage.Delete(ctx, storagePath); err != nil { - p.Logger.Warn("failed to discard artifact with mismatched checksum", "path", storagePath, "error", err) - } - return nil, fmt.Errorf("artifact checksum mismatch: upstream declared %s, got %s", upstreamHash, hash) - } - - if err := p.updateCacheDB(ecosystem, name, filename, pkgPURL, versionPURL, downloadURL, storagePath, hash, size, artifact.ContentType); err != nil { - p.Logger.Warn("failed to update cache database", "error", err) - } - - readStart := time.Now() - reader, err := p.Storage.Open(ctx, storagePath) - metrics.RecordStorageOperation("read", time.Since(readStart)) - if err != nil { - metrics.RecordStorageError("read") - return nil, fmt.Errorf("opening cached artifact: %w", err) - } - - return &CacheResult{ - Reader: reader, - Size: size, - ContentType: artifact.ContentType, - Hash: hash, - Cached: false, - }, nil + return p.storeArtifact(ctx, ecosystem, name, version, filename, pkgPURL, versionPURL, downloadURL, upstreamHash, artifact) } func artifactHashMatches(got, expected string) bool { diff --git a/internal/handler/handler_test.go b/internal/handler/handler_test.go index 80909d1..87e74d8 100644 --- a/internal/handler/handler_test.go +++ b/internal/handler/handler_test.go @@ -65,7 +65,13 @@ func (s *mockStorage) Exists(_ context.Context, path string) (bool, error) { return ok, nil } -func (s *mockStorage) Delete(_ context.Context, path string) error { +func (s *mockStorage) Delete(ctx context.Context, path string) error { + // Real backends (S3/GCS SDKs) fail fast on an already-cancelled + // context; mirror that here so tests can catch cleanup calls that + // forgot to detach from a cancelled client context. + if err := ctx.Err(); err != nil { + return err + } delete(s.files, path) return nil } diff --git a/internal/handler/npm.go b/internal/handler/npm.go index 166b944..2cd8885 100644 --- a/internal/handler/npm.go +++ b/internal/handler/npm.go @@ -287,12 +287,15 @@ func (h *NPMHandler) handleDownload(w http.ResponseWriter, r *http.Request) { r.Context(), "npm", packageName, version, filename, downloadURL, ) if err != nil { - if errors.Is(err, ErrUpstreamNotFound) { + switch { + case errors.Is(err, ErrUpstreamNotFound): JSONError(w, http.StatusNotFound, "package not found") - return + case errors.Is(err, ErrArtifactBlocked): + JSONError(w, http.StatusForbidden, err.Error()) + default: + h.proxy.Logger.Error("failed to get artifact", "error", err) + JSONError(w, http.StatusBadGateway, "failed to fetch package") } - h.proxy.Logger.Error("failed to get artifact", "error", err) - JSONError(w, http.StatusBadGateway, "failed to fetch package") return } diff --git a/internal/handler/npm_test.go b/internal/handler/npm_test.go index 9005925..c4a5f7e 100644 --- a/internal/handler/npm_test.go +++ b/internal/handler/npm_test.go @@ -2,6 +2,7 @@ package handler import ( "encoding/json" + "errors" "io" "log/slog" "net/http" @@ -651,3 +652,60 @@ func TestNPMDownloadCooldownFetchesMetadataOnce(t *testing.T) { t.Errorf("metadata requests = %d, want 1", got) } } + +// TestNPMDownloadErrorResponsesAreJSON guards against a regression where +// routing handleDownload's error path through the shared serveArtifactError +// helper silently switched npm's 404/502 tarball error bodies from JSON to +// plain text; npm clients expect a JSON {"error": "..."} body on every +// download failure, including the newer scan-blocked (403) case. +func TestNPMDownloadErrorResponsesAreJSON(t *testing.T) { + tests := []struct { + name string + fetchErr error + blocked bool + wantStatus int + }{ + {"upstream not found", fetch.ErrNotFound, false, http.StatusNotFound}, + {"upstream failure", errors.New("connection refused"), false, http.StatusBadGateway}, + {"blocked by scan", nil, true, http.StatusForbidden}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + proxy, _, _, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + if tt.blocked { + proxy.Scanners = newTestScanGroup(t, newTestScanServer(t, false, "malware detected").URL, false) + } + fetcher.fetchErr = tt.fetchErr + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("tarball data")), + ContentType: "application/octet-stream", + } + + h := NewNPMHandler(proxy, "http://proxy.test", "http://upstream.invalid") + srv := httptest.NewServer(h.Routes()) + defer srv.Close() + + resp, err := http.Get(srv.URL + "/leftpad/-/leftpad-1.0.0.tgz") + if err != nil { + t.Fatalf("request failed: %v", err) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != tt.wantStatus { + t.Errorf("status = %d, want %d", resp.StatusCode, tt.wantStatus) + } + if ct := resp.Header.Get("Content-Type"); ct != contentTypeJSON { + t.Errorf("Content-Type = %q, want %q", ct, contentTypeJSON) + } + var body map[string]any + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + t.Fatalf("response body is not valid JSON: %v", err) + } + if _, ok := body["error"]; !ok { + t.Errorf("response body %v missing \"error\" key", body) + } + }) + } +} diff --git a/internal/handler/scan_test.go b/internal/handler/scan_test.go new file mode 100644 index 0000000..7480bad --- /dev/null +++ b/internal/handler/scan_test.go @@ -0,0 +1,310 @@ +package handler + +import ( + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/config" + "github.com/git-pkgs/proxy/internal/scanner" + "github.com/git-pkgs/purl" + "github.com/git-pkgs/registries/fetch" +) + +// newTestScanServer returns an httptest.Server implementing the HTTPScanner +// notify contract, always replying with the given verdict. +func newTestScanServer(t testing.TB, allowed bool, reason string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var body map[string]any + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + t.Errorf("decode scan notify body: %v", err) + } + if body["fetch_url"] == "" || body["fetch_url"] == nil { + t.Error("scan notify body missing fetch_url") + } + _ = json.NewEncoder(w).Encode(map[string]any{"allowed": allowed, "reason": reason}) + })) + t.Cleanup(srv.Close) + return srv +} + +func newTestScanGroup(t testing.TB, scanURL string, failOpen bool) *scanner.Group { + t.Helper() + g, err := scanner.NewGroup(config.ScanningConfig{ + Enabled: true, + FailOpen: failOpen, + Timeout: "15s", + SigningKey: "test-signing-key", + Scanners: []config.ScannerConfig{ + {Name: "test-scanner", URL: scanURL, Mode: "block"}, + }, + }, slog.New(slog.NewTextHandler(io.Discard, nil))) + if err != nil { + t.Fatalf("scanner.NewGroup() error: %v", err) + } + return g +} + +func TestGetOrFetchArtifact_ScanAllowed(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newTestScanGroup(t, newTestScanServer(t, true, "").URL, false) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("clean content")), + ContentType: "application/gzip", + } + + result, err := proxy.GetOrFetchArtifact(context.Background(), "npm", "leftpad", "1.0.0", "leftpad-1.0.0.tgz") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + defer func() { _ = result.Reader.Close() }() + + body, _ := io.ReadAll(result.Reader) + if string(body) != "clean content" { + t.Errorf("body = %q, want %q", body, "clean content") + } + + cached, err := db.GetCachedArtifact( + purl.MakePURLString("npm", "leftpad", ""), purl.MakePURLString("npm", "leftpad", "1.0.0"), "leftpad-1.0.0.tgz") + if err != nil { + t.Fatalf("GetCachedArtifact() error: %v", err) + } + if cached == nil { + t.Error("expected allowed artifact to be committed to the cache database") + } + if len(store.files) == 0 { + t.Error("expected allowed artifact bytes to remain in storage") + } +} + +func TestGetOrFetchArtifact_ScanBlocked(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newTestScanGroup(t, newTestScanServer(t, false, "malware detected").URL, false) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("evil content")), + ContentType: "application/gzip", + } + + _, err := proxy.GetOrFetchArtifact(context.Background(), "npm", "evilpkg", "1.0.0", "evilpkg-1.0.0.tgz") + if err == nil { + t.Fatal("expected error for blocked artifact") + } + if !errors.Is(err, ErrArtifactBlocked) { + t.Errorf("error = %v, want wrapped ErrArtifactBlocked", err) + } + if !strings.Contains(err.Error(), "malware detected") { + t.Errorf("error %q does not include scanner reason", err.Error()) + } + + cached, err := db.GetCachedArtifact( + purl.MakePURLString("npm", "evilpkg", ""), purl.MakePURLString("npm", "evilpkg", "1.0.0"), "evilpkg-1.0.0.tgz") + if err != nil { + t.Fatalf("GetCachedArtifact() error: %v", err) + } + if cached != nil { + t.Error("blocked artifact must never be committed to the cache database") + } + if len(store.files) != 0 { + t.Errorf("blocked artifact bytes must be deleted from storage, got %d files", len(store.files)) + } +} + +func TestGetOrFetchArtifact_BlockedDeleteSurvivesClientDisconnect(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + + const scanDelay = 150 * time.Millisecond + blockingSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + time.Sleep(scanDelay) + _ = json.NewEncoder(w).Encode(map[string]any{"allowed": false, "reason": "malware detected"}) + })) + t.Cleanup(blockingSrv.Close) + proxy.Scanners = newTestScanGroup(t, blockingSrv.URL, false) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("evil content")), + ContentType: "application/gzip", + } + + // The client disconnects long before the (genuinely malicious) verdict + // comes back; cleanup of the blocked bytes must not be skipped just + // because the client is gone. + ctx, cancel := context.WithCancel(context.Background()) + time.AfterFunc(20*time.Millisecond, cancel) + + _, err := proxy.GetOrFetchArtifact(ctx, "npm", "evilpkg", "1.0.0", "evilpkg-1.0.0.tgz") + if err == nil { + t.Fatal("expected error for blocked artifact") + } + if !errors.Is(err, ErrArtifactBlocked) { + t.Errorf("error = %v, want wrapped ErrArtifactBlocked", err) + } + + cached, _ := db.GetCachedArtifact( + purl.MakePURLString("npm", "evilpkg", ""), purl.MakePURLString("npm", "evilpkg", "1.0.0"), "evilpkg-1.0.0.tgz") + if cached != nil { + t.Error("blocked artifact must never be committed to the cache database") + } + if len(store.files) != 0 { + t.Errorf("blocked artifact bytes must still be deleted even though the client disconnected mid-scan, got %d orphaned files", len(store.files)) + } +} + +func TestGetOrFetchArtifact_ScanErrorFailClosed(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + + brokenSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + })) + t.Cleanup(brokenSrv.Close) + proxy.Scanners = newTestScanGroup(t, brokenSrv.URL, false) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("content")), + ContentType: "application/gzip", + } + + _, err := proxy.GetOrFetchArtifact(context.Background(), "npm", "flaky", "1.0.0", "flaky-1.0.0.tgz") + if err == nil { + t.Fatal("expected error when scanner infrastructure fails") + } + if !errors.Is(err, ErrArtifactBlocked) { + t.Errorf("error = %v, want wrapped ErrArtifactBlocked (fail-closed default)", err) + } + if strings.Contains(err.Error(), brokenSrv.URL) { + t.Errorf("error %q leaks the internal scanner URL to the client-facing message", err.Error()) + } + if !strings.Contains(err.Error(), "scan could not be completed") { + t.Errorf("error %q does not use the generic infra-failure message", err.Error()) + } + + cached, _ := db.GetCachedArtifact( + purl.MakePURLString("npm", "flaky", ""), purl.MakePURLString("npm", "flaky", "1.0.0"), "flaky-1.0.0.tgz") + if cached != nil { + t.Error("artifact must not be committed when scanning fails fail-closed") + } + if len(store.files) != 0 { + t.Errorf("artifact bytes must be deleted on scan infra failure, got %d files", len(store.files)) + } +} + +func TestGetOrFetchArtifact_ScanErrorFailOpen(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + + brokenSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + })) + t.Cleanup(brokenSrv.Close) + proxy.Scanners = newTestScanGroup(t, brokenSrv.URL, true) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("content")), + ContentType: "application/gzip", + } + + result, err := proxy.GetOrFetchArtifact(context.Background(), "npm", "flaky", "1.0.0", "flaky-1.0.0.tgz") + if err != nil { + t.Fatalf("unexpected error: %v (FailOpen must treat scanner infra failure as allowed)", err) + } + defer func() { _ = result.Reader.Close() }() + + cached, err := db.GetCachedArtifact( + purl.MakePURLString("npm", "flaky", ""), purl.MakePURLString("npm", "flaky", "1.0.0"), "flaky-1.0.0.tgz") + if err != nil { + t.Fatalf("GetCachedArtifact() error: %v", err) + } + if cached == nil { + t.Error("expected artifact to be committed to the cache when scanning fails fail-open") + } + if len(store.files) == 0 { + t.Error("expected artifact bytes to remain in storage when scanning fails fail-open") + } +} + +func TestGetOrFetchArtifact_ScanSurvivesClientDisconnect(t *testing.T) { + proxy, db, store, fetcher := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + + const scanDelay = 150 * time.Millisecond + slowSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + time.Sleep(scanDelay) + _ = json.NewEncoder(w).Encode(map[string]any{"allowed": true}) + })) + t.Cleanup(slowSrv.Close) + proxy.Scanners = newTestScanGroup(t, slowSrv.URL, false) + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("clean content")), + ContentType: "application/gzip", + } + + // Simulate a client that disconnects shortly after issuing the request: + // its context is cancelled well before the scanner replies, but the + // scan itself must run to completion rather than being torn down with + // it. + ctx, cancel := context.WithCancel(context.Background()) + time.AfterFunc(20*time.Millisecond, cancel) + + start := time.Now() + result, err := proxy.GetOrFetchArtifact(ctx, "npm", "leftpad", "1.0.0", "leftpad-1.0.0.tgz") + elapsed := time.Since(start) + if err != nil { + t.Fatalf("unexpected error: %v (a cancelled client context must not be mistaken for a scanner failure)", err) + } + defer func() { _ = result.Reader.Close() }() + + if elapsed < scanDelay { + t.Errorf("GetOrFetchArtifact returned after %v, want it to wait out the full scan (%v) despite client cancellation", elapsed, scanDelay) + } + + cached, err := db.GetCachedArtifact( + purl.MakePURLString("npm", "leftpad", ""), purl.MakePURLString("npm", "leftpad", "1.0.0"), "leftpad-1.0.0.tgz") + if err != nil { + t.Fatalf("GetCachedArtifact() error: %v", err) + } + if cached == nil { + t.Error("expected artifact to be committed to the cache; a client disconnect must not cause a false block") + } + if len(store.files) == 0 { + t.Error("expected artifact bytes to remain in storage; a client disconnect must not delete a legitimately allowed artifact") + } +} + +func TestGetOrFetchArtifact_ScanDisabledIsNoOp(t *testing.T) { + proxy, db, _, fetcher := setupTestProxy(t) + // proxy.Scanners left nil: scanning disabled. + + fetcher.artifact = &fetch.Artifact{ + Body: io.NopCloser(strings.NewReader("content")), + ContentType: "application/gzip", + } + + result, err := proxy.GetOrFetchArtifact(context.Background(), "npm", "plainpkg", "1.0.0", "plainpkg-1.0.0.tgz") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + defer func() { _ = result.Reader.Close() }() + + cached, err := db.GetCachedArtifact( + purl.MakePURLString("npm", "plainpkg", ""), purl.MakePURLString("npm", "plainpkg", "1.0.0"), "plainpkg-1.0.0.tgz") + if err != nil { + t.Fatalf("GetCachedArtifact() error: %v", err) + } + if cached == nil { + t.Error("expected artifact to be cached when scanning is disabled") + } +} diff --git a/internal/handler/scanfetch.go b/internal/handler/scanfetch.go new file mode 100644 index 0000000..832da9c --- /dev/null +++ b/internal/handler/scanfetch.go @@ -0,0 +1,89 @@ +package handler + +import ( + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "net/http" + "net/url" + "strconv" + "time" +) + +// scanFetchURL builds a short-lived, HMAC-signed URL for the internal +// /_internal/scan-fetch route, so an external scanner can pull the exact +// bytes staged at path without going through cooldown or the scan hook +// itself. This is generated the same way for every storage backend: it +// never depends on Storage.SignedURL, which not every backend implements. +func (p *Proxy) scanFetchURL(path string, ttl time.Duration) string { + exp := time.Now().Add(ttl).Unix() + return fmt.Sprintf("%s/_internal/scan-fetch?path=%s&exp=%d&sig=%s", + p.ScanFetchBaseURL, url.QueryEscape(path), exp, hmacHex(p.ScanSigningKey, path, exp)) +} + +func hmacHex(key []byte, path string, exp int64) string { + mac := hmac.New(sha256.New, key) + _, _ = fmt.Fprintf(mac, "%s|%d", path, exp) + return hex.EncodeToString(mac.Sum(nil)) +} + +// ServeScanFetch streams a storage object to a caller presenting a valid +// short-lived HMAC token, so external scanners can pull a staged artifact +// without going through cooldown or the scan hook themselves. This handler +// never calls GetOrFetchArtifact/fetchAndCache/storeArtifact — the +// separation from the normal request path is structural, not a +// conditional bypass flag. +// +// This route exists only for scanners configured under ScanningConfig; the +// URL is minted by scanFetchURL and passed as fetch_url in the scan notify +// request. It is not part of the public API and should be restricted to +// internal-network access at the ingress/network-policy layer — the HMAC +// scoping (one object, short TTL) limits what a leaked token can do, but +// isn't a substitute for network restriction. +// +// @Summary Fetch a staged artifact for scanning +// @Description Streams the exact bytes staged in storage for a pre-cache security scan. +// @Description Requires a short-lived HMAC-signed token minted by the proxy itself and +// @Description delivered via the fetch_url field of the scan notify request (see the +// @Description Artifact Scanning section of docs/configuration.md). Not part of the +// @Description public API; restrict access to the scanner network at the ingress layer. +// @Tags scanning +// @Produce application/octet-stream +// @Param path query string true "Storage path of the staged artifact" +// @Param exp query int true "Token expiry, Unix seconds" +// @Param sig query string true "HMAC-SHA256 signature over the string path|exp" +// @Success 200 {file} file +// @Failure 403 {string} string "invalid, expired, or tampered token" +// @Failure 404 {string} string "object not found in storage, or scanning is not configured" +// @Router /_internal/scan-fetch [get] +func (p *Proxy) ServeScanFetch(w http.ResponseWriter, r *http.Request) { + if p.Scanners == nil || !p.Scanners.Enabled() || len(p.ScanSigningKey) == 0 { + http.Error(w, "not found", http.StatusNotFound) + return + } + + path := r.URL.Query().Get("path") + exp, err := strconv.ParseInt(r.URL.Query().Get("exp"), 10, 64) + if err != nil || containsPathTraversal(path) || time.Now().Unix() > exp { + http.Error(w, "invalid or expired token", http.StatusForbidden) + return + } + + want := hmacHex(p.ScanSigningKey, path, exp) + if !hmac.Equal([]byte(r.URL.Query().Get("sig")), []byte(want)) { + http.Error(w, "invalid signature", http.StatusForbidden) + return + } + + reader, err := p.Storage.Open(r.Context(), path) + if err != nil { + http.Error(w, "not found", http.StatusNotFound) + return + } + defer func() { _ = reader.Close() }() + + w.Header().Set("Content-Type", "application/octet-stream") + _, _ = io.Copy(w, reader) +} diff --git a/internal/handler/scanfetch_test.go b/internal/handler/scanfetch_test.go new file mode 100644 index 0000000..280f1a2 --- /dev/null +++ b/internal/handler/scanfetch_test.go @@ -0,0 +1,153 @@ +package handler + +import ( + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/config" + "github.com/git-pkgs/proxy/internal/scanner" +) + +// newEnabledScanGroup returns a scanner.Group that reports Enabled() true, +// so tests can exercise ServeScanFetch's normal signature-checking path +// rather than tripping its "scanning not configured" guard. +func newEnabledScanGroup(t testing.TB) *scanner.Group { + t.Helper() + g, err := scanner.NewGroup(config.ScanningConfig{ + Enabled: true, + Timeout: "15s", + SigningKey: "test-signing-key", + Scanners: []config.ScannerConfig{ + {Name: "test-scanner", URL: "http://localhost/scan", Mode: "block"}, + }, + }, slog.Default()) + if err != nil { + t.Fatalf("scanner.NewGroup() error: %v", err) + } + return g +} + +func TestServeScanFetch_ValidToken(t *testing.T) { + proxy, _, store, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newEnabledScanGroup(t) + store.files["npm/lodash/4.17.21/lodash-4.17.21.tgz"] = []byte("artifact bytes") + + target := proxy.scanFetchURL("npm/lodash/4.17.21/lodash-4.17.21.tgz", time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body: %s", w.Code, w.Body.String()) + } + if w.Body.String() != "artifact bytes" { + t.Errorf("body = %q, want %q", w.Body.String(), "artifact bytes") + } +} + +func TestServeScanFetch_Expired(t *testing.T) { + proxy, _, store, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newEnabledScanGroup(t) + store.files["npm/lodash/4.17.21/lodash-4.17.21.tgz"] = []byte("artifact bytes") + + target := proxy.scanFetchURL("npm/lodash/4.17.21/lodash-4.17.21.tgz", -time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusForbidden { + t.Errorf("status = %d, want 403", w.Code) + } +} + +func TestServeScanFetch_TamperedSignature(t *testing.T) { + proxy, _, store, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newEnabledScanGroup(t) + store.files["npm/lodash/4.17.21/lodash-4.17.21.tgz"] = []byte("artifact bytes") + + target := proxy.scanFetchURL("npm/lodash/4.17.21/lodash-4.17.21.tgz", time.Minute) + tampered := strings.Replace(target, "sig=", "sig=deadbeef", 1) + + req := httptest.NewRequest(http.MethodGet, tampered, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusForbidden { + t.Errorf("status = %d, want 403", w.Code) + } +} + +func TestServeScanFetch_PathTraversal(t *testing.T) { + proxy, _, _, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newEnabledScanGroup(t) + + target := proxy.scanFetchURL("../../etc/passwd", time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusForbidden { + t.Errorf("status = %d, want 403", w.Code) + } +} + +func TestServeScanFetch_ScanningDisabled(t *testing.T) { + proxy, _, store, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + // proxy.Scanners left nil: scanning disabled. + store.files["npm/lodash/4.17.21/lodash-4.17.21.tgz"] = []byte("artifact bytes") + + target := proxy.scanFetchURL("npm/lodash/4.17.21/lodash-4.17.21.tgz", time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusNotFound { + t.Errorf("status = %d, want 404 when scanning is disabled", w.Code) + } +} + +func TestServeScanFetch_NoSigningKey(t *testing.T) { + proxy, _, store, _ := setupTestProxy(t) + // proxy.ScanSigningKey left empty. + proxy.Scanners = newEnabledScanGroup(t) + store.files["npm/lodash/4.17.21/lodash-4.17.21.tgz"] = []byte("artifact bytes") + + target := proxy.scanFetchURL("npm/lodash/4.17.21/lodash-4.17.21.tgz", time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusNotFound { + t.Errorf("status = %d, want 404 when no signing key is configured", w.Code) + } +} + +func TestServeScanFetch_MissingObject(t *testing.T) { + proxy, _, _, _ := setupTestProxy(t) + proxy.ScanSigningKey = []byte("test-signing-key") + proxy.Scanners = newEnabledScanGroup(t) + + target := proxy.scanFetchURL("npm/missing/1.0.0/missing-1.0.0.tgz", time.Minute) + + req := httptest.NewRequest(http.MethodGet, target, nil) + w := httptest.NewRecorder() + proxy.ServeScanFetch(w, req) + + if w.Code != http.StatusNotFound { + t.Errorf("status = %d, want 404", w.Code) + } +} diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index df47222..aff5768 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -137,6 +137,32 @@ var ( }, []string{"step"}, ) + + // Scanning metrics + ScanDuration = prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "proxy_scan_duration_seconds", + Help: "Pre-cache artifact scan duration in seconds, by ecosystem and scanner", + Buckets: prometheus.DefBuckets, + }, + []string{"ecosystem", "scanner"}, + ) + + ScanBlocked = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "proxy_scan_blocked_total", + Help: "Total number of artifacts blocked by a pre-cache scan, by ecosystem and scanner", + }, + []string{"ecosystem", "scanner"}, + ) + + ScanErrors = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "proxy_scan_errors_total", + Help: "Total number of pre-cache scan errors, by ecosystem, scanner, and error type", + }, + []string{"ecosystem", "scanner", "error_type"}, + ) ) func init() { @@ -157,6 +183,9 @@ func init() { ActiveRequests, IntegrityFailures, HealthProbeFailures, + ScanDuration, + ScanBlocked, + ScanErrors, ) } @@ -213,6 +242,21 @@ func RecordStorageError(operation string) { StorageErrors.WithLabelValues(operation).Inc() } +// RecordScanResult tracks a completed pre-cache scan call. +func RecordScanResult(ecosystem, scannerName string, allowed bool, duration time.Duration) { + ecosystem = purl.NormalizeEcosystem(ecosystem) + ScanDuration.WithLabelValues(ecosystem, scannerName).Observe(duration.Seconds()) + if !allowed { + ScanBlocked.WithLabelValues(ecosystem, scannerName).Inc() + } +} + +// RecordScanError increments the scan error counter. +// errorType is one of: "error" (scanner call failed), "timeout", "cancelled". +func RecordScanError(ecosystem, scannerName, errorType string) { + ScanErrors.WithLabelValues(purl.NormalizeEcosystem(ecosystem), scannerName, errorType).Inc() +} + // UpdateCacheStats updates cache size and artifact count gauges. func UpdateCacheStats(sizeBytes, artifactCount int64) { CacheSize.Set(float64(sizeBytes)) diff --git a/internal/scanner/group.go b/internal/scanner/group.go new file mode 100644 index 0000000..ed0be57 --- /dev/null +++ b/internal/scanner/group.go @@ -0,0 +1,226 @@ +package scanner + +import ( + "context" + "errors" + "fmt" + "log/slog" + "net/http" + "sync" + "time" + + "github.com/git-pkgs/proxy/internal/config" + "github.com/git-pkgs/proxy/internal/metrics" +) + +const ( + modeBlock = "block" + modeMonitor = "monitor" + + defaultTimeout = 30 * time.Second +) + +type entry struct { + scanner Scanner + mode string + ecosystems map[string]struct{} // nil/empty means all ecosystems +} + +// Group runs a set of configured Scanners concurrently and turns their +// individual verdicts into a single decision. +type Group struct { + entries []entry + timeout time.Duration + failOpen bool + logger *slog.Logger +} + +// NewGroup builds a Group from cfg. If cfg.Enabled is false, the returned +// Group has no entries and Enabled() reports false, so callers can skip +// the scan path entirely. +func NewGroup(cfg config.ScanningConfig, logger *slog.Logger) (*Group, error) { + g := &Group{ + timeout: defaultTimeout, + failOpen: cfg.FailOpen, + logger: logger, + } + if !cfg.Enabled { + return g, nil + } + if cfg.SigningKeyExpanded() == "" { + return nil, fmt.Errorf("scanning.signing_key is required when scanning.enabled is true") + } + if d, err := time.ParseDuration(cfg.Timeout); err == nil && d > 0 { + g.timeout = d + } + + for _, sc := range cfg.Scanners { + mode := sc.Mode + if mode == "" { + mode = modeBlock + } + if mode != modeBlock && mode != modeMonitor { + return nil, fmt.Errorf("scanner %q: invalid mode %q (must be %q or %q)", sc.Name, mode, modeBlock, modeMonitor) + } + + var ecosystems map[string]struct{} + if len(sc.Ecosystems) > 0 { + ecosystems = make(map[string]struct{}, len(sc.Ecosystems)) + for _, eco := range sc.Ecosystems { + ecosystems[eco] = struct{}{} + } + } + + g.entries = append(g.entries, entry{ + scanner: NewHTTPScanner(sc.Name, sc.URL, sc.HeadersExpanded(), http.DefaultClient), + mode: mode, + ecosystems: ecosystems, + }) + } + + return g, nil +} + +// Enabled reports whether any scanner is configured. +func (g *Group) Enabled() bool { + return g != nil && len(g.entries) > 0 +} + +// Timeout returns the per-scan-call timeout used to bound the signed fetch +// URL's validity. +func (g *Group) Timeout() time.Duration { + return g.timeout +} + +func (g *Group) applicable(ecosystem string) []entry { + var out []entry + for _, e := range g.entries { + if len(e.ecosystems) == 0 { + out = append(out, e) + continue + } + if _, ok := e.ecosystems[ecosystem]; ok { + out = append(out, e) + } + } + return out +} + +// Scan runs every scanner applicable to req.Ecosystem concurrently, never +// sequentially, and returns a single decision. +// +// The moment any "block" mode scanner reports Allowed: false (or errors, +// unless FailOpen is set), Scan cancels a context shared by every +// goroutine: in-flight calls to the other scanners are aborted rather than +// waited out, since a single block already decides the outcome. Scan still +// waits for all goroutines to observe that cancellation and return before +// it itself returns, so no scan call outlives this method call. +// +// If nothing blocks, Scan waits for every "block" mode scanner to finish +// before reporting Allowed: true — an allow decision can't be finalized +// until all of them have answered. "monitor" mode scanners never gate the +// wait or trigger cancellation: a monitor verdict of Allowed: false is +// logged and folded into Result.Findings, but never blocks. +func (g *Group) Scan(ctx context.Context, req Request) Result { + entries := g.applicable(req.Ecosystem) + if len(entries) == 0 { + return Result{Allowed: true} + } + + scanCtx, cancel := context.WithTimeout(ctx, g.timeout) + defer cancel() + + var ( + mu sync.Mutex + findings []Finding + blocked *Result + ) + + var wg sync.WaitGroup + for _, e := range entries { + wg.Add(1) + go func(e entry) { + defer wg.Done() + entryFindings, entryBlock := g.evaluate(scanCtx, req, e) + + mu.Lock() + findings = append(findings, entryFindings...) + if entryBlock != nil && blocked == nil { + blocked = entryBlock + cancel() + } + mu.Unlock() + }(e) + } + + wg.Wait() + + if blocked != nil { + blocked.Findings = findings + return *blocked + } + + return Result{Allowed: true, Findings: findings} +} + +// evaluate runs a single scanner and reports its findings plus, if this +// scanner's verdict should block the artifact, the Result to block with +// (nil otherwise). It never sets Result.Findings on a returned block +// Result — the caller assembles Findings from every entry once all of them +// have finished. +func (g *Group) evaluate(scanCtx context.Context, req Request, e entry) (findings []Finding, block *Result) { + start := time.Now() + res, err := e.scanner.Scan(scanCtx, req) + duration := time.Since(start) + + if err != nil { + errType := "error" + switch { + case errors.Is(scanCtx.Err(), context.DeadlineExceeded): + errType = "timeout" + case errors.Is(scanCtx.Err(), context.Canceled): + // scanCtx was cancelled because another scanner in the group + // already decided the verdict, not because this call itself + // timed out or failed on its own. + errType = "cancelled" + } + metrics.RecordScanError(req.Ecosystem, e.scanner.Name(), errType) + + if errType == "cancelled" { + // Another scanner already decided the verdict and cancelled + // scanCtx; this call didn't fail on its own, so don't log it + // as if it did. + return nil, nil + } + + if e.mode == modeMonitor || g.failOpen { + g.logger.Warn("scanner call failed, treating as allowed", + "scanner", e.scanner.Name(), "mode", e.mode, "error", err) + return nil, nil + } + + g.logger.Warn("scanner call failed, blocking artifact", + "scanner", e.scanner.Name(), "mode", e.mode, "error", err) + return nil, &Result{ + Allowed: false, + Reason: fmt.Sprintf("scanner %q failed: %v", e.scanner.Name(), err), + ScannerName: e.scanner.Name(), + InfraError: true, + } + } + + metrics.RecordScanResult(req.Ecosystem, e.scanner.Name(), res.Allowed, duration) + + if e.mode == modeMonitor { + if !res.Allowed { + g.logger.Warn("monitor scanner flagged artifact", + "scanner", e.scanner.Name(), "reason", res.Reason) + } + return res.Findings, nil + } + + if !res.Allowed { + return res.Findings, &Result{Allowed: false, Reason: res.Reason, ScannerName: e.scanner.Name()} + } + return res.Findings, nil +} diff --git a/internal/scanner/group_test.go b/internal/scanner/group_test.go new file mode 100644 index 0000000..e5b40ee --- /dev/null +++ b/internal/scanner/group_test.go @@ -0,0 +1,245 @@ +package scanner + +import ( + "context" + "errors" + "io" + "log/slog" + "strings" + "testing" + "time" + + "github.com/git-pkgs/proxy/internal/config" + "github.com/git-pkgs/proxy/internal/metrics" + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func discardLogger() *slog.Logger { + return slog.New(slog.NewTextHandler(io.Discard, nil)) +} + +// fakeScanner is a Scanner test double that can simulate a delay, a fixed +// result or error, and report whether its context was cancelled before it +// returned. +type fakeScanner struct { + name string + delay time.Duration + result Result + err error + cancelled *bool +} + +func (f *fakeScanner) Name() string { return f.name } + +func (f *fakeScanner) Scan(ctx context.Context, _ Request) (Result, error) { + select { + case <-time.After(f.delay): + case <-ctx.Done(): + if f.cancelled != nil { + *f.cancelled = true + } + return Result{}, ctx.Err() + } + return f.result, f.err +} + +func newTestGroup(entries []entry, failOpen bool) *Group { + return &Group{ + entries: entries, + timeout: time.Second, + failOpen: failOpen, + logger: discardLogger(), + } +} + +func TestGroup_Enabled(t *testing.T) { + disabled, err := NewGroup(config.ScanningConfig{Enabled: false}, discardLogger()) + if err != nil { + t.Fatalf("NewGroup() error: %v", err) + } + if disabled.Enabled() { + t.Error("Enabled() = true for disabled config, want false") + } + + enabled, err := NewGroup(config.ScanningConfig{ + Enabled: true, + Timeout: "10s", + SigningKey: "test-signing-key", + Scanners: []config.ScannerConfig{ + {Name: "clamav", URL: "http://clamav.invalid", Mode: "block"}, + }, + }, discardLogger()) + if err != nil { + t.Fatalf("NewGroup() error: %v", err) + } + if !enabled.Enabled() { + t.Error("Enabled() = false for configured scanner, want true") + } +} + +func TestGroup_NewGroup_InvalidMode(t *testing.T) { + _, err := NewGroup(config.ScanningConfig{ + Enabled: true, + SigningKey: "test-signing-key", + Scanners: []config.ScannerConfig{ + {Name: "bad", URL: "http://bad.invalid", Mode: "quarantine"}, + }, + }, discardLogger()) + if err == nil { + t.Fatal("NewGroup() error = nil, want error for invalid mode") + } +} + +func TestGroup_NewGroup_MissingSigningKey(t *testing.T) { + _, err := NewGroup(config.ScanningConfig{ + Enabled: true, + Scanners: []config.ScannerConfig{ + {Name: "clamav", URL: "http://clamav.invalid", Mode: "block"}, + }, + }, discardLogger()) + if err == nil { + t.Fatal("NewGroup() error = nil, want error for missing signing key") + } +} + +func TestGroup_Scan_NoApplicableScanners(t *testing.T) { + g := newTestGroup([]entry{ + { + scanner: &fakeScanner{name: "npm-only", result: Result{Allowed: false}}, + mode: modeBlock, + ecosystems: map[string]struct{}{"npm": {}}, + }, + }, false) + + result := g.Scan(context.Background(), Request{Ecosystem: "pypi"}) + if !result.Allowed { + t.Error("Allowed = false, want true when no scanner applies to the ecosystem") + } +} + +func TestGroup_Scan_Allowed(t *testing.T) { + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "clamav", result: Result{Allowed: true}}, mode: modeBlock}, + }, false) + + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + if !result.Allowed { + t.Error("Allowed = false, want true") + } +} + +func TestGroup_Scan_Blocked(t *testing.T) { + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "clamav", result: Result{Allowed: false, Reason: "malware"}}, mode: modeBlock}, + }, false) + + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + if result.Allowed { + t.Error("Allowed = true, want false") + } + if result.Reason != "malware" { + t.Errorf("Reason = %q, want %q", result.Reason, "malware") + } + if result.ScannerName != "clamav" { + t.Errorf("ScannerName = %q, want %q", result.ScannerName, "clamav") + } + if result.InfraError { + t.Error("InfraError = true, want false — this is a genuine scanner verdict, not an infrastructure failure") + } +} + +func TestGroup_Scan_MonitorNeverBlocks(t *testing.T) { + g := newTestGroup([]entry{ + { + scanner: &fakeScanner{name: "trivy", result: Result{ + Allowed: false, + Reason: "cve found", + Findings: []Finding{{Severity: "medium", Title: "CVE-1234"}}, + }}, + mode: modeMonitor, + }, + }, false) + + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + if !result.Allowed { + t.Error("Allowed = false, want true — monitor mode must never block") + } + if len(result.Findings) != 1 || result.Findings[0].Title != "CVE-1234" { + t.Errorf("Findings = %+v, want the monitor scanner's finding folded in", result.Findings) + } +} + +func TestGroup_Scan_FirstBlockCancelsOthers(t *testing.T) { + var slowSawCancel bool + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "fast-block", result: Result{Allowed: false, Reason: "blocked"}}, mode: modeBlock}, + {scanner: &fakeScanner{name: "slow", delay: 2 * time.Second, cancelled: &slowSawCancel}, mode: modeBlock}, + }, false) + + start := time.Now() + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + elapsed := time.Since(start) + + if result.Allowed { + t.Error("Allowed = true, want false") + } + if elapsed >= 2*time.Second { + t.Errorf("Scan() took %v, want it to return promptly once the slow scanner's context was cancelled", elapsed) + } + if !slowSawCancel { + t.Error("slow scanner never observed context cancellation") + } + + if got := testutil.ToFloat64(metrics.ScanErrors.WithLabelValues("npm", "slow", "cancelled")); got != 1 { + t.Errorf("scan_errors{error_type=cancelled} = %v, want 1 — the slow scanner was aborted by a sibling's block, not by its own timeout", got) + } + if got := testutil.ToFloat64(metrics.ScanErrors.WithLabelValues("npm", "slow", "timeout")); got != 0 { + t.Errorf("scan_errors{error_type=timeout} = %v, want 0 — intra-group cancellation must not be mislabelled as a timeout", got) + } +} + +func TestGroup_Scan_WaitsForAllBlockScannersBeforeAllowing(t *testing.T) { + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "fast", result: Result{Allowed: true}}, mode: modeBlock}, + {scanner: &fakeScanner{name: "slow", delay: 30 * time.Millisecond, result: Result{Allowed: true}}, mode: modeBlock}, + }, false) + + start := time.Now() + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + elapsed := time.Since(start) + + if !result.Allowed { + t.Error("Allowed = false, want true") + } + if elapsed < 30*time.Millisecond { + t.Errorf("Scan() returned after %v, want it to wait for the slower block scanner", elapsed) + } +} + +func TestGroup_Scan_ErrorFailClosedByDefault(t *testing.T) { + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "flaky", err: errors.New("connection refused")}, mode: modeBlock}, + }, false) + + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + if result.Allowed { + t.Error("Allowed = true, want false — default posture is fail-closed on scanner error") + } + if !result.InfraError { + t.Error("InfraError = false, want true — the block came from a scanner call failure, not a verdict") + } + if !strings.Contains(result.Reason, "connection refused") { + t.Errorf("Reason = %q, want it to include the underlying error for server-side logging", result.Reason) + } +} + +func TestGroup_Scan_ErrorFailOpen(t *testing.T) { + g := newTestGroup([]entry{ + {scanner: &fakeScanner{name: "flaky", err: errors.New("connection refused")}, mode: modeBlock}, + }, true) + + result := g.Scan(context.Background(), Request{Ecosystem: "npm"}) + if !result.Allowed { + t.Error("Allowed = false, want true — FailOpen must treat scanner errors as allowed") + } +} diff --git a/internal/scanner/http.go b/internal/scanner/http.go new file mode 100644 index 0000000..90a4e5e --- /dev/null +++ b/internal/scanner/http.go @@ -0,0 +1,97 @@ +package scanner + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" +) + +// HTTPScanner adapts an external HTTP scanning service to the Scanner +// interface. It POSTs a small JSON notification describing the staged +// artifact, including a signed fetch URL; the external service is +// responsible for GETting that URL itself, running the real scan against +// those bytes, and replying with a verdict before the request's deadline. +// +// Request body: +// +// { +// "ecosystem": "npm", "name": "left-pad", "version": "1.0.0", +// "filename": "left-pad-1.0.0.tgz", "purl": "pkg:npm/left-pad@1.0.0", +// "content_type": "application/octet-stream", "size": 1234, +// "fetch_url": "https://proxy.internal/_internal/scan-fetch?..." +// } +// +// Response body: +// +// { +// "allowed": true, "reason": "", +// "findings": [{"severity": "high", "title": "...", "description": "..."}] +// } +// +// Any compliant adapter — a trivy wrapper, a clamav-rest bridge, a Wiz +// connector, or an in-house service — need only implement this contract. +type HTTPScanner struct { + name string + url string + headers map[string]string + client *http.Client +} + +// NewHTTPScanner creates an HTTPScanner named name that notifies url of +// staged artifacts, attaching headers to every request (e.g. for auth). +// If client is nil, http.DefaultClient is used. +func NewHTTPScanner(name, url string, headers map[string]string, client *http.Client) *HTTPScanner { + if client == nil { + client = http.DefaultClient + } + return &HTTPScanner{name: name, url: url, headers: headers, client: client} +} + +// Name returns the scanner's configured name. +func (s *HTTPScanner) Name() string { return s.name } + +type httpScanResponse struct { + Allowed bool `json:"allowed"` + Reason string `json:"reason"` + Findings []Finding `json:"findings"` +} + +// Scan notifies the configured URL of req and waits for a verdict. +func (s *HTTPScanner) Scan(ctx context.Context, req Request) (Result, error) { + body, err := json.Marshal(req) + if err != nil { + return Result{}, fmt.Errorf("marshal scan request: %w", err) + } + + httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, s.url, bytes.NewReader(body)) + if err != nil { + return Result{}, fmt.Errorf("build scan request: %w", err) + } + httpReq.Header.Set("Content-Type", "application/json") + for k, v := range s.headers { + httpReq.Header.Set(k, v) + } + + resp, err := s.client.Do(httpReq) + if err != nil { + return Result{}, fmt.Errorf("calling scanner %q: %w", s.name, err) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusOK { + return Result{}, fmt.Errorf("scanner %q returned status %d", s.name, resp.StatusCode) + } + + var out httpScanResponse + if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { + return Result{}, fmt.Errorf("decoding scanner %q response: %w", s.name, err) + } + + return Result{ + Allowed: out.Allowed, + Reason: out.Reason, + Findings: out.Findings, + }, nil +} diff --git a/internal/scanner/http_test.go b/internal/scanner/http_test.go new file mode 100644 index 0000000..7597826 --- /dev/null +++ b/internal/scanner/http_test.go @@ -0,0 +1,119 @@ +package scanner + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + "time" +) + +func TestHTTPScanner_Scan(t *testing.T) { + tests := []struct { + name string + respStatus int + respBody string + wantAllowed bool + wantReason string + wantErr bool + }{ + { + name: "allowed", + respStatus: http.StatusOK, + respBody: `{"allowed": true}`, + wantAllowed: true, + }, + { + name: "blocked with reason and findings", + respStatus: http.StatusOK, + respBody: `{"allowed": false, "reason": "malware detected", "findings": [{"severity": "high", "title": "EICAR", "description": "test signature"}]}`, + wantAllowed: false, + wantReason: "malware detected", + }, + { + name: "non-200 status is an error", + respStatus: http.StatusInternalServerError, + respBody: `{}`, + wantErr: true, + }, + { + name: "malformed JSON is an error", + respStatus: http.StatusOK, + respBody: `not json`, + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var got Request + if err := json.NewDecoder(r.Body).Decode(&got); err != nil { + t.Errorf("decode request body: %v", err) + } + if got.Ecosystem != "npm" || got.Name != "left-pad" || got.FetchURL == "" { + t.Errorf("unexpected request body: %+v", got) + } + if r.Header.Get("Authorization") != "Bearer secret" { + t.Errorf("missing/incorrect Authorization header: %q", r.Header.Get("Authorization")) + } + + w.WriteHeader(tt.respStatus) + _, _ = w.Write([]byte(tt.respBody)) + })) + defer srv.Close() + + s := NewHTTPScanner("test", srv.URL, map[string]string{"Authorization": "Bearer secret"}, nil) + + result, err := s.Scan(context.Background(), Request{ + Ecosystem: "npm", + Name: "left-pad", + Version: "1.0.0", + FetchURL: srv.URL + "/fetch", + }) + + if tt.wantErr { + if err == nil { + t.Fatalf("Scan() error = nil, want error") + } + return + } + if err != nil { + t.Fatalf("Scan() unexpected error: %v", err) + } + if result.Allowed != tt.wantAllowed { + t.Errorf("Allowed = %v, want %v", result.Allowed, tt.wantAllowed) + } + if result.Reason != tt.wantReason { + t.Errorf("Reason = %q, want %q", result.Reason, tt.wantReason) + } + }) + } +} + +func TestHTTPScanner_Scan_ContextTimeout(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + time.Sleep(50 * time.Millisecond) + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"allowed": true}`)) + })) + defer srv.Close() + + s := NewHTTPScanner("slow", srv.URL, nil, nil) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Millisecond) + defer cancel() + + _, err := s.Scan(ctx, Request{Ecosystem: "npm", Name: "left-pad"}) + if err == nil { + t.Fatal("Scan() error = nil, want timeout error") + } +} + +func TestHTTPScanner_Name(t *testing.T) { + s := NewHTTPScanner("clamav", "http://example.invalid", nil, nil) + if got := s.Name(); got != "clamav" { + t.Errorf("Name() = %q, want %q", got, "clamav") + } +} diff --git a/internal/scanner/scanner.go b/internal/scanner/scanner.go new file mode 100644 index 0000000..380bb4d --- /dev/null +++ b/internal/scanner/scanner.go @@ -0,0 +1,65 @@ +// Package scanner provides pluggable pre-cache artifact scanning. +// +// A Scanner inspects an artifact staged in the proxy's own storage before +// it becomes visible to clients, and returns a verdict on whether it may +// be cached. The proxy never uploads artifact bytes to a scanner directly: +// it hands the scanner a short-lived signed URL and the scanner pulls the +// bytes itself. See HTTPScanner for the built-in adapter that implements +// this over a small HTTP/JSON contract, letting trivy, ClamAV, Wiz, or any +// custom service integrate without the proxy needing built-in knowledge of +// any specific tool. +package scanner + +import "context" + +// Request describes a staged artifact awaiting a scan verdict. +type Request struct { + Ecosystem string `json:"ecosystem"` + Name string `json:"name"` + Version string `json:"version"` + Filename string `json:"filename"` + PURL string `json:"purl"` + ContentType string `json:"content_type"` + Size int64 `json:"size"` + + // FetchURL is a short-lived signed URL the scanner must GET itself to + // retrieve the exact bytes staged in the proxy's storage. + FetchURL string `json:"fetch_url"` +} + +// Finding describes a single issue reported by a scanner. +type Finding struct { + Severity string + Title string + Description string +} + +// Result is a scanner's verdict for a Request. +type Result struct { + Allowed bool + Reason string + Findings []Finding + + // ScannerName identifies which scanner produced this result. Set by + // Group, not by individual Scanner implementations. + ScannerName string + + // InfraError reports whether Allowed: false was forced by a scanner + // call failing (network error, timeout, bad response) rather than an + // actual verdict from the scanner. Set by Group. Callers that surface + // Reason to untrusted clients must not do so when this is true: it may + // contain raw connection errors (internal hostnames, ports) instead of + // a verdict meant to be shown outside the proxy. + InfraError bool +} + +// Scanner is the extension point for pluggable pre-cache scanning. +type Scanner interface { + // Name identifies this scanner in logs and metrics. + Name() string + + // Scan requests a verdict for req. Implementations must respect ctx + // cancellation: Group cancels in-flight scans once a blocking verdict + // has already been decided by another scanner. + Scan(ctx context.Context, req Request) (Result, error) +} diff --git a/internal/server/server.go b/internal/server/server.go index 6fd56bd..0eb5194 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -49,6 +49,7 @@ package server import ( + "cmp" "context" "database/sql" "encoding/json" @@ -73,6 +74,7 @@ import ( "github.com/git-pkgs/proxy/internal/metrics" "github.com/git-pkgs/proxy/internal/mirror" "github.com/git-pkgs/proxy/internal/packageurl" + "github.com/git-pkgs/proxy/internal/scanner" "github.com/git-pkgs/proxy/internal/storage" "github.com/git-pkgs/registries/fetch" "github.com/git-pkgs/registries/safehttp" @@ -229,6 +231,10 @@ func (s *Server) serve(listener net.Listener) error { proxy.HTTPClient = &metadataClient proxy.AuthForURL = s.authForURL proxy.Cooldown = cd + scanGroup, err := configureScanning(proxy, s.cfg.Scanning, s.cfg.BaseURL, s.logger) + if err != nil { + return fmt.Errorf("configuring scanners: %w", err) + } proxy.CacheMetadata = s.cfg.CacheMetadata proxy.MetadataTTL = s.cfg.ParseMetadataTTL() proxy.MetadataMaxSize = s.cfg.ParseMetadataMaxSize() @@ -258,6 +264,100 @@ func (s *Server) serve(listener net.Listener) error { }) // Mount protocol handlers + s.mountProtocolHandlers(r, proxy) + + // Health, stats, and metrics endpoints + r.Get("/health", s.handleHealth) + r.Get("/stats", s.handleStats) + r.Get("/openapi.json", s.handleOpenAPIJSON) + r.Get("/metrics", func(w http.ResponseWriter, r *http.Request) { + // Breaker state is only held in the fetcher, so publish it on scrape. + s.breakers.snapshot() + metrics.Handler().ServeHTTP(w, r) + }) + + // Internal route used by external scanners to pull staged artifact + // bytes before they're committed to the cache. Only wired up when + // scanning is actually configured, so there's no unauthenticated path + // to storage objects sitting in the router when the feature is unused. + // Restrict this to internal-network access only at the + // ingress/network-policy layer. + if scanGroup.Enabled() && len(proxy.ScanSigningKey) > 0 { + r.Get("/_internal/scan-fetch", proxy.ServeScanFetch) + } + + // Web UI. Mounted under /ui so a reverse proxy can apply different + // access rules to it than to the package endpoints above (#123). + r.Route("/ui", func(ui chi.Router) { + ui.Mount("/static", http.StripPrefix("/ui/static/", staticHandler())) + ui.Get("/", s.handleRoot) + ui.Get("/install", s.handleInstall) + ui.Get("/search", s.handleSearch) + ui.Get("/packages", s.handlePackagesList) + ui.Get("/package/{ecosystem}/*", s.handlePackagePath) + ui.Get("/api/browse/{ecosystem}/*", s.handleBrowsePath) + ui.Get("/api/compare/{ecosystem}/*", s.handleComparePath) + }) + r.Get("/", func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, "/ui/", http.StatusFound) + }) + + // API endpoints for enrichment data + enrichSvc := enrichment.New(s.logger) + apiHandler := NewAPIHandler(enrichSvc, s.db) + + r.Get("/api/package/{ecosystem}/*", apiHandler.HandlePackagePath) + r.Get("/api/vulns/{ecosystem}/*", apiHandler.HandleVulnsPath) + r.Post("/api/outdated", apiHandler.HandleOutdated) + r.Post("/api/bulk", apiHandler.HandleBulkLookup) + r.Get("/api/search", apiHandler.HandleSearch) + r.Get("/api/packages", apiHandler.HandlePackagesList) + + // Start background context (used by mirror jobs and cleanup) + bgCtx, bgCancel := context.WithCancel(context.Background()) + s.cancel = bgCancel + s.startGradleBuildCacheEviction(bgCtx) + + // Mirror API endpoints (opt-in via mirror_api config or PROXY_MIRROR_API env) + if s.cfg.MirrorAPI { + mirrorSvc := mirror.New(proxy, s.db, s.storage, s.logger, 4) //nolint:mnd // default concurrency + jobStore := mirror.NewJobStore(bgCtx, mirrorSvc) + mirrorAPI := NewMirrorAPIHandler(jobStore) + r.Post("/api/mirror", mirrorAPI.HandleCreate) + r.Get("/api/mirror/{id}", mirrorAPI.HandleGet) + r.Delete("/api/mirror/{id}", mirrorAPI.HandleCancel) + go jobStore.StartCleanup(bgCtx) + } + + s.http = &http.Server{ + Addr: s.cfg.Listen, + Handler: r, + ReadTimeout: serverReadTimeout, + WriteTimeout: serverWriteTimeout, // Large artifacts need time + IdleTimeout: serverIdleTimeout, + } + + s.logger.Info("starting server", + "listen", s.cfg.Listen, + "base_url", s.cfg.BaseURL, + "ui_url", s.cfg.UIBaseURL, + "storage", s.storage.URL(), + "database", s.cfg.Database.String()) + go s.updateCacheStatsMetrics() + go s.startEvictionLoop(bgCtx) + + if listener != nil { + return s.http.Serve(listener) + } + return s.http.ListenAndServe() +} + +// configureScanning builds the scanner group from cfg and wires it into +// proxy, returning the group so the caller can decide whether to mount the +// internal scan-fetch route. +// mountProtocolHandlers constructs every ecosystem handler and mounts it on +// r under its protocol prefix. +func (s *Server) mountProtocolHandlers(r chi.Router, proxy *handler.Proxy) { npmHandler := handler.NewNPMHandler(proxy, s.cfg.BaseURL, s.cfg.Upstream.NPM) cargoHandler := handler.NewCargoHandler( proxy, @@ -336,81 +436,17 @@ func (s *Server) serve(listener net.Listener) error { r.Mount("/apk", http.StripPrefix("/apk", apkHandler.Routes())) r.Mount("/debian", http.StripPrefix("/debian", debianHandler.Routes())) r.Mount("/rpm", http.StripPrefix("/rpm", rpmHandler.Routes())) +} - // Health, stats, and metrics endpoints - r.Get("/health", s.handleHealth) - r.Get("/stats", s.handleStats) - r.Get("/openapi.json", s.handleOpenAPIJSON) - r.Get("/metrics", func(w http.ResponseWriter, r *http.Request) { - // Breaker state is only held in the fetcher, so publish it on scrape. - s.breakers.snapshot() - metrics.Handler().ServeHTTP(w, r) - }) - - // Web UI. Mounted under /ui so a reverse proxy can apply different - // access rules to it than to the package endpoints above (#123). - r.Route("/ui", func(ui chi.Router) { - ui.Mount("/static", http.StripPrefix("/ui/static/", staticHandler())) - ui.Get("/", s.handleRoot) - ui.Get("/install", s.handleInstall) - ui.Get("/search", s.handleSearch) - ui.Get("/packages", s.handlePackagesList) - ui.Get("/package/{ecosystem}/*", s.handlePackagePath) - ui.Get("/api/browse/{ecosystem}/*", s.handleBrowsePath) - ui.Get("/api/compare/{ecosystem}/*", s.handleComparePath) - }) - r.Get("/", func(w http.ResponseWriter, r *http.Request) { - http.Redirect(w, r, "/ui/", http.StatusFound) - }) - - // API endpoints for enrichment data - enrichSvc := enrichment.New(s.logger) - apiHandler := NewAPIHandler(enrichSvc, s.db) - - r.Get("/api/package/{ecosystem}/*", apiHandler.HandlePackagePath) - r.Get("/api/vulns/{ecosystem}/*", apiHandler.HandleVulnsPath) - r.Post("/api/outdated", apiHandler.HandleOutdated) - r.Post("/api/bulk", apiHandler.HandleBulkLookup) - r.Get("/api/search", apiHandler.HandleSearch) - r.Get("/api/packages", apiHandler.HandlePackagesList) - - // Start background context (used by mirror jobs and cleanup) - bgCtx, bgCancel := context.WithCancel(context.Background()) - s.cancel = bgCancel - s.startGradleBuildCacheEviction(bgCtx) - - // Mirror API endpoints (opt-in via mirror_api config or PROXY_MIRROR_API env) - if s.cfg.MirrorAPI { - mirrorSvc := mirror.New(proxy, s.db, s.storage, s.logger, 4) //nolint:mnd // default concurrency - jobStore := mirror.NewJobStore(bgCtx, mirrorSvc) - mirrorAPI := NewMirrorAPIHandler(jobStore) - r.Post("/api/mirror", mirrorAPI.HandleCreate) - r.Get("/api/mirror/{id}", mirrorAPI.HandleGet) - r.Delete("/api/mirror/{id}", mirrorAPI.HandleCancel) - go jobStore.StartCleanup(bgCtx) - } - - s.http = &http.Server{ - Addr: s.cfg.Listen, - Handler: r, - ReadTimeout: serverReadTimeout, - WriteTimeout: serverWriteTimeout, // Large artifacts need time - IdleTimeout: serverIdleTimeout, - } - - s.logger.Info("starting server", - "listen", s.cfg.Listen, - "base_url", s.cfg.BaseURL, - "ui_url", s.cfg.UIBaseURL, - "storage", s.storage.URL(), - "database", s.cfg.Database.String()) - go s.updateCacheStatsMetrics() - go s.startEvictionLoop(bgCtx) - - if listener != nil { - return s.http.Serve(listener) +func configureScanning(proxy *handler.Proxy, cfg config.ScanningConfig, baseURL string, logger *slog.Logger) (*scanner.Group, error) { + scanGroup, err := scanner.NewGroup(cfg, logger) + if err != nil { + return nil, err } - return s.http.ListenAndServe() + proxy.Scanners = scanGroup + proxy.ScanSigningKey = []byte(cfg.SigningKeyExpanded()) + proxy.ScanFetchBaseURL = cmp.Or(cfg.FetchBaseURL, baseURL) + return scanGroup, nil } func upstreamSafeHTTPOptions(upstream config.UpstreamConfig) safehttp.Options { diff --git a/internal/server/server_test.go b/internal/server/server_test.go index 87142df..2f195dd 100644 --- a/internal/server/server_test.go +++ b/internal/server/server_test.go @@ -284,6 +284,69 @@ func testStartUsesConfiguredLoopbackUpstreams(t *testing.T) { } } +// TestScanFetchRouteNotMountedWhenScanningDisabled verifies the internal +// scan-fetch route is absent (404), not merely unauthenticated, when +// scanning is disabled: mounting it unconditionally would expose an +// unauthenticated way to pull arbitrary storage objects by path. +func TestScanFetchRouteNotMountedWhenScanningDisabled(t *testing.T) { + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("reserving proxy address: %v", err) + } + t.Cleanup(func() { _ = listener.Close() }) + listenAddress := listener.Addr().String() + + tempDir := t.TempDir() + cfg := config.Default() + cfg.Listen = listenAddress + cfg.BaseURL = "http://" + listenAddress + cfg.Database.Path = filepath.Join(tempDir, "proxy.db") + cfg.Storage.URL = "file://" + filepath.Join(tempDir, "artifacts") + if err := cfg.Validate(); err != nil { + t.Fatalf("validating config: %v", err) + } + + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + proxyServer, err := New(cfg, logger, BuildInfo{Version: "test", Commit: "test"}) + if err != nil { + t.Fatalf("creating server: %v", err) + } + startErr := make(chan error, 1) + go func() { + startErr <- proxyServer.Start(listener) + }() + defer func() { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := proxyServer.Shutdown(ctx); err != nil { + t.Errorf("shutting down server: %v", err) + } + if err := <-startErr; !errors.Is(err, http.ErrServerClosed) { + t.Errorf("Start() error = %v, want %v", err, http.ErrServerClosed) + } + }() + + client := &http.Client{Timeout: 250 * time.Millisecond} + deadline := time.Now().Add(5 * time.Second) + var resp *http.Response + for { + var requestErr error + resp, requestErr = client.Get(cfg.BaseURL + "/_internal/scan-fetch") + if requestErr == nil { + break + } + if time.Now().After(deadline) { + t.Fatalf("proxy did not start: %v", requestErr) + } + time.Sleep(10 * time.Millisecond) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusNotFound { + t.Errorf("scan-fetch status = %d, want 404 when scanning is disabled", resp.StatusCode) + } +} + // seedTestPackage creates a package, version, and artifact in the database for testing // page rendering. The package is created under the npm ecosystem with version 1.0.0. func seedTestPackage(t *testing.T, db *database.DB, name string) { diff --git a/internal/server/swagger_gen.go b/internal/server/swagger_gen.go index f72f32b..03eae74 100644 --- a/internal/server/swagger_gen.go +++ b/internal/server/swagger_gen.go @@ -1,3 +1,3 @@ -//go:generate swag init -g ../../cmd/proxy/main.go -o ../../docs/swagger --outputTypes go,json --parseInternal +//go:generate swag init -g ../../cmd/proxy/main.go -d .,../handler -o ../../docs/swagger --outputTypes go,json --parseInternal package server