Repository navigation
fix(services): bound scorer memory for long raw windows - #93
Merged
Merged
Conversation
Raw windows default to 2048 packets, about 0.3 GiB of ONNX activations per window. A micro-batch of 64 would request ~19 GiB and get the 1 GiB score pod OOM-killed. Split the micro-batch of a raw-byte model into ONNX runs whose estimate fits a budget (half the cgroup memory limit, or CORVUS_SCORE_MEMORY_BUDGET_MB). The estimate is the inference formula of corvus_ml.membudget for the default architecture. At load the scorer logs the estimate and the rows per run, exports corvus_score_model_estimated_window_bytes and logs an error when a single window exceeds the budget, but keeps serving. Raise the score pod memory to 1Gi request / 2Gi limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since 0.5.0 raw windows default to
max_packets2048. PR #92 measured about 0.26 GiB of ONNXinference memory per window at that size. The scorer batches up to
score_batch_max(64) windowsinto one ONNX run, so a 2048-packet raw model in shadow would request ~16 GiB per batch and get the
1 GiB score pods OOM-killed, taking all scoring down.
Fix
budget / per-window estimaterows (at least 1); the scores are concatenated in window order. Feature models are unaffected.
corvus_ml.membudget, evaluated for the defaultarchitecture with
max_packets,max_tokens_per_packetandmacro_windowsfrom the feature spec(the spec does not carry the layer sizes; larger models need more).
CORVUS_SCORE_MEMORY_FRACTION(0.5) of the cgroup memory limit, orCORVUS_SCORE_MEMORY_BUDGET_MB.corvus_score_model_estimated_window_bytesand
corvus_score_model_max_rows, and logs an ERROR when one window exceeds the budget. The modelis still served.
values.yaml.Rows per run for the default model:
Tests
New
services/tests/scoring/test_score_memory.py(fakes only, no ONNX model, no sleeps); ruff,mypy, full services pytest,
helm lintandhelm unittestpass.