feat(perf): retain bounded optional provider metrics around waves - #23
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #13.
Adds optional
--metrics-urlsnapshots before and after each wave, outside measured dispatch/settlement. Retains bounded raw evidence and a plan/wave-bound companion; offline comparison verifies and exposes selected series, counter deltas and compatible acceptance ratios. The allowlist contains the two vLLM draft/accepted counters and running/waiting gauges. Identities include complete canonical labels; missing/reset/inconsistent counters never become a numeric ratio.Bounds: 2s per scrape, 1MiB body, 64KiB lines, 256 selected series, 16 labels and 4096 encoded label bytes per series; 30s whole-run telemetry admission allowance and 16MiB retained raw data. Completed overhead is charged to the remaining budget and reconstructed from verified receipts on resume. Budget exhaustion records skipped snapshots without calls. Filesystem/scheduler overruns cannot be physically bounded and are disclosed.
No model-auth forwarding, redirects, retries, quiescence wait loop or performance-eligibility changes. Optional HTTP/parser/transport failures remain diagnostics, while missing/tampered promised evidence remains an integrity error. Opt-out preserves historical plan/wave/request shapes; differing metrics policies cannot silently form a matched comparison.
Scope: server-wide counters are not automatically workload-attributed. Scrapes can perturb server state or cadence even outside the measured interval. Nondecreasing counters do not prove no restart. Parsed counters use f64, so values above 2^53 cannot preserve all unit increments; raw evidence remains available. Hashes establish file bindings, not authenticated timing or provider execution. Raw labels/comments need privacy review before sharing.
Verified: independent SIX and HELP initially blocked three defects (unselected-series interference, impossible failure metadata, HTTP 600 aborting collection). Each defect was reproduced by a real CLI regression, fixed, and approved in both follow-up reviews at 88db35d. Integrated fmt, Clippy -D warnings, 153 workspace tests passed/1 ignored, release build and actual loopback CLI smoke including 4 metrics scrapes. No live model/provider request was made; server-specific metric availability and attribution remain unqualified.