Repository navigation
fix(ml): train on few players without crashing on empty splits - #90
Merged
Merged
Conversation
With one player per label stratum the val and test splits are empty, and the runner scored the fitted candidate on them (sklearn: 0 samples). - skip scoring, metrics and confusion on an empty split; report why (test_skipped / val_skipped) and leave the metrics out - run ONNX parity and benchmark on the first non-empty split (test, val, train) and record it as parity_on - train still registers the model; the job result carries warnings - name every skip and parity reason when no candidate is usable
split_by "session" groups the train/val/test split by session instead of player, so a smoke test with one or two players gets a test split. It is leakage-prone (a player lands in train and test) and is recorded in the report and job result, which carry a warning. Not combinable with tune.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
trainjob on a snapshot with a single player failed withFound array with 0 sample(s) (shape=(0, 12)) while a minimum of 1 is required by IsolationForest. The split is by player; with one player per label stratum the val and test splits are empty, andrun_candidate(ml/src/corvus_ml/eval/runner.py) called_eval_metrics->cand.calibrated_scoreon the empty test split.Changes
fix(ml): nothing is scored on an empty split.test/valmetrics arenullwithtest_skipped/val_skipped(test split empty: needs at least 2 players per label stratum); the confusion table and learning curve are left out. ONNX parity and the benchmark run on the first non-empty split (test, val, train), recorded asonnx.parity_on, so the model can still be registered.trainregisters it and the job result carrieswarnings. When no candidate is usable, theJobErrornames every skip/parity reason. The report renders a "No test metrics" section.feat(ml): newsplit_by(playerdefault,session) for jobs andrun.split_byineval.toml. Leakage-prone, for smoke tests with few players; recorded in the report (banner) and job result (warning). Refused together withtune.docs/reference/ml-jobs.md,docs/how-to/evaluate-models.md.A candidate that crashes in
fiton very few windows (not an empty training split) is not turned into a skip; only an empty/one-class training split is, as before.Tests
tests/test_eval_few_players.pyand new cases intests/test_job_models.py(single player: evaluate, train+register+warning, sequence candidate, no usable candidate,split_by). Synthetic data, fixed seeds, no network.