perf(sipflow): scope query bucket enumeration to the requested window - #271
Merged
Merged
Conversation
Every recording/flow query walked ALL historical buckets — one fresh sqlite connection plus one (almost always empty) SELECT per bucket — so per-query read IO grew linearly with retention (13 buckets = ~77 SELECTs per recording at 3 query types). Queries now enumerate only buckets overlapping the requested window widened by one bucket on each side (absorbs flush lag and writer/ querier clock skew), keeping per-query read IO constant. The legacy whole-tree scan stays available via StorageManager::with_scan_out_ of_range(true) for replayed or mis-bucketed data; diagnostics use it, as does the existing cross-day-bucket test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Every recording/flow query enumerated all historical buckets: one
fresh sqlite connection plus one — almost always empty — SELECT per
bucket (
discover_data_dirsdeliberately appends out-of-range buckets,"scanning extra directories is safe"). With 13 accumulated hour buckets
and 3 query types per call, that measured ~77 SELECT statements per
recording (194,530 selects / 2,527 recordings, via the pipeline
metrics from #270), and per-query read IO grew linearly with retention.
What this PR changes
StorageManager::get_folders_in_range— the query hot path — now usesdiscover_data_dirs_in_range, which enumerates only buckets overlappingthe requested window, widened by one bucket on each side to absorb
flush lag and writer/querier clock skew. Ingest buckets by wall-clock
capture time, so per-query read IO is now constant instead of growing
with history.
The legacy whole-tree scan is preserved behind
StorageManager::with_scan_out_of_range(true)for replayed ormis-bucketed data (e.g. a bucket written by a
subdirs = nonewriterrooted elsewhere). Diagnostics (
run_diag) opt in, as does the existingtest_daily_dir_containing_other_days_data_is_still_queryable.discover_data_dirsitself is unchanged.Verification
bucket is never touched) and daily scoping.
— correctness intact.
2.7 MB/s / 29 rIOPS (~8×). Write path untouched (~211 wIOPS).