Skip to content

refactor(vulnfeeds): migrate debian-copyright-mirror to Go - #5885

Open
jess-lowe wants to merge 7 commits into
google:masterfrom
jess-lowe:refactor/debian-copyright-mirror
Open

refactor(vulnfeeds): migrate debian-copyright-mirror to Go#5885
jess-lowe wants to merge 7 commits into
google:masterfrom
jess-lowe:refactor/debian-copyright-mirror

Conversation

@jess-lowe

Copy link
Copy Markdown
Contributor

Migrates the debian-copyright-mirror service from Python to Go and extracts reusable networking, archive, and decompression utilities for other vulnfeeds tools.

Key Changes

  • Go Migration: Replaced debian-copyright-mirror.py and debian-copyright-mirror.sh with a native Go service (cmd/mirrors/debian-copyright-mirror/main.go).
  • Streaming Pipeline: Streams filelist.yaml.xz over HTTP through an xz -dc pipe directly into a custom streaming line parser, extracting ~44k package paths without disk writes.
  • Worker Pool: Added a concurrent worker pool with HTTP/2 keep-alive connection pooling, per-request timeouts, retry logic, and atomic failure rate tracking (-max-failure-rate).
  • Native Archiving & GCS Upload: Built-in POSIX .tar packaging and direct streaming to Google Cloud Storage (cloud.google.com/go/storage).
  • Shared Utilities:
    • utility/decompress.go: Generic XZ streaming decompression helpers.
    • utility/download.go: Resilient atomic file downloader with exponential backoff.
    • utility/archive.go: Tar archive creation and extraction with traversal/bomb protections.
    • gcs-tools/gcs.go: Added ParseGCSPath URI helper.
  • Dockerfile & Deployment: Switched to a multi-stage Go build on a minimal alpine:3.21 image with the Go binary as direct ENTRYPOINT, removing the need for google-cloud-cli and wrapper shell scripts.

Testing Done

  • go test ./... in vulnfeeds (all tests passing).
  • golangci-lint run (0 issues).
  • Verified full download and local .tar generation with live Debian changelog metadata.

Comment thread vulnfeeds/utility/archive.go Fixed
}

for {
hdr, err := tr.Next()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants