Fix memory plateau after daily virus-database reload

[eldadfux]

The RSS chart (flat for hours, then a near-vertical jump that does not come back down) matches a daily daily.cvd hot-reload, not a leak on the scan path.

Default DEFENDER_UPDATE_INTERVAL_SECS is 3600. When ClamAV publishes a new daily database (typically once a day), Defender:

1. Downloads the new CVD
2. Compiles a new Engine while the previous one stays live (zero-downtime)
3. Atomically swaps ArcSwap

Peak RSS is therefore about two engines plus unpack/compile scratch (~4.5 GiB from a ~2 GiB baseline). After the swap, glibc malloc commonly does not return the old arenas to the OS, so RSS plateaus ~1.5× above the original baseline (the 3.55 GiB level in the screenshot).

Changes

• Stream CVD I/O: version checks read only the 512-byte header; downloads go straight to disk; verify and unpack stream from the file so the gzip body and previous tar members are not held while compiling.
• Skip unused members: bytecode / YARA / leftover .tmp files are drained, not retained. PUA files are skipped when DEFENDER_LOAD_PUA is off.
• Bounded automaton: Aho-Corasick is pinned to ContiguousNFA. The default heuristic can pick a DFA that is many times larger and is a major contributor to the two-engine peak.
• Return pages after swap: Linux builds use jemalloc; after compile and after ArcSwap, unused dirty pages are purged. Reload logs rss_before / rss_compiled / rss_after.

A brief spike while both engines exist is still expected for zero-downtime updates. RSS should return near the pre-reload baseline once in-flight scans release the old Arc.

Verification

cargo test --locked --all-targets

On a running instance, watch the reload line after a daily publish:

scan engine swapped atomically rss_before=... rss_compiled=... rss_after=...

rss_after should be close to rss_before, not stuck near rss_compiled.

  

[greptile-apps]

Greptile Summary

The PR reduces peak and retained RSS during virus-database reloads by streaming CVD processing, limiting the Aho-Corasick representation, and using jemalloc reclamation on Linux.

• Streams downloads, verification, and CVD member ingestion rather than retaining complete archives.
• Filters unsupported and disabled-PUA members before allocating their contents.
• Uses a contiguous NFA and shrinks signature collections before constructing an engine.
• Adds allocator initialization, explicit page reclamation, reload RSS metrics, and regression coverage.

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking defect identified.

The changed download, verification, ingestion, swap, and allocator paths preserve existing signature support and installation ordering while reducing transient and retained memory.

Important Files Changed

Filename	Overview
src/updater/mod.rs	Streams downloads into temporary files, verifies before atomic installation, and logs memory around engine replacement.
src/engine/mod.rs	Streams verified CVD members into the builder, filters irrelevant files, compacts collections, and pins the prefilter to a contiguous NFA.
src/cvd/verify.rs	Replaces whole-file buffering with incremental checksum and signature verification.
src/cvd/unpack.rs	Adds bounded member-by-member archive processing while preserving the existing full-unpack API.
src/alloc.rs	Configures Linux jemalloc reclamation and provides RSS measurement and formatting helpers.
src/signatures/mod.rs	Introduces a signature-file allowlist consistent with every format currently handled by the engine.
Cargo.toml	Adds Linux-specific jemalloc dependencies and background-thread support.
tests/memory.rs	Extends swap-memory coverage with explicit allocator reclamation.
tests/pipeline.rs	Verifies that temporary and unknown files do not interfere with directory loading.

_{Reviews (1): Last reviewed commit: "Fix RSS step-up after daily CVD hot-relo..." | Re-trigger Greptile}