EE/QDN: Reproducible Model-Training Pipeline Configuration (Sprint 1) - #976
Merged
Conversation
Adds patterns for the upcoming pipeline port (*.pth, *.onnx, outputs/, .cache/) plus src/prototypes/data_files/ and src/prototypes/spectrograms/, which were untracked, ungitignored, and ~922MB - one `git add -A` away from an accidental large commit.
Ported from .delete/archive/Prototypes/engine/torch_impl/ into its permanent home so Kiernan's Hydra config has runnable code behind it. Two pre-existing bugs fixed while porting: - main.py: typo cfg.run.cKLDivLossheckpoint_path -> cfg.run.checkpoint_path (broke the run.test=true checkpoint-loading branch) - train.py: missing OmegaConf import (NameError in save_checkpoint/ load_checkpoint, currently dead code but a latent trap) main.py also now seeds numpy and random alongside torch, since augment.py's masking uses random.randint/random.random and would otherwise stay non-reproducible despite training.seed being set. dataset.py ported with 5 dead imports removed (librosa, numpy, hashlib, diskcache, soundfile - confirmed unused, real audio loading uses torchaudio + an LMDB cache). model/ (8 files: __init__, effv2, ghost_effv2, panns_cnn14, panns_mobilenetv1, panns_mobilenetv2, quant, utils) ported verbatim - no issues found.
The archived augment.py's SpecAugment(p, freq_mask_param, time_mask_param, n_freq_mask, n_time_mask) doesn't match 4 of the 5 augmentation presets already committed in config/augmentation/ - those pass freq_mask_ratio/time_mask_ratio/max_total_time_ratio/mask_value, none of which existed on the old constructor. Every preset except none.yaml raised TypeError at hydra.utils.instantiate() time. Rewritten to support both APIs so no preset YAML needs to change: - freq_mask_param/time_mask_param (legacy pixel width, used only by original_unfixed_reference.yaml) - freq_mask_ratio/time_mask_ratio (fraction of axis size, used by default/light/heavy.yaml) Exactly one of each pair must be given (ValueError otherwise, naming the offending pair). max_total_time_ratio caps cumulative time-strip width across all time strips combined; with ratio 1.0 the cap can never bind, reproducing the "no safety cap" behaviour through the same code path as the capped presets rather than a special-cased branch. Verified against all 5 presets plus the cap's cumulative behaviour (500-trial stress test) before wiring into the pipeline.
Hydra config set for the SpecAugment validation experiment work (config.yaml, config/model/*.yaml [7 variants], config/teacher_model/ *.yaml [6 variants], config/augmentation/*.yaml [5 presets], config/local/cpu.yaml), used per team agreement as the shared baseline for Kiernan/Nolan/Praveen rather than a separate config setup. Attribution comment already present at the top of config.yaml. One additive fix: 4 of the 7 model presets (ghost_efficientnet_v2, panns_cnn14, panns_mobilenetv1, panns_mobilenetv2) were missing the norm_choice key that model/__init__.py reads unconditionally - all four crashed immediately on model construction. Added norm_choice: freeze_bn to each, matching what efficientnet_v2.yaml/efficientnet_v2_qat.yaml already used. Flagging for Kiernan/Praveen to confirm freeze_bn is the intended choice per architecture (panns_mobilenetv2_qat.yaml uses swap_rms_norm for its QAT variant, so it isn't automatically the same choice for every model) - see README.md Known Limitations. The 6 config/teacher_model/*.yaml presets have the same missing-key gap, left unfixed since the distillation path is deferred to Sprint 2. Nothing else in any of these files was changed or removed.
New sibling to Kiernan's config/local/cpu.yaml, same opt-in pattern (+local=local_data_files, @Package _global_, zero effect unless selected). Points system.audio_data_directory at the real dataset already checked out in this repo at src/prototypes/data_files/ (128 species, untracked - see .gitignore), which cpu.yaml doesn't cover since it points at a different path from Kiernan's own machine. Composes with cpu.yaml: python main.py +local=cpu +local=local_data_files Path is 2 levels up from this folder to reach src/prototypes/data_files/, not 4 like cpu.yaml's own example (which points outside the repo entirely) - verified with os.path.relpath and a directory listing before committing to it, since the depth isn't the same for both targets.
No pyproject.toml existed at the destination - the archive's version was kitchen-sink (ONNX/TFLite/gradio/tensorflow entries for code staying in the archive) and conflicted with the repo-root requirements.txt files on Python/TF versions. This is a deliberately separate, scoped-down uv-managed environment for src/prototypes/engine/augmentation/ only. Trimmed to what main.py/train.py/dataset.py/model/*/augment.py actually import, with scikit-learn and tensorboard promoted from transitive to direct dependencies (train.py uses sklearn.metrics and torch.utils.tensorboard.SummaryWriter directly). uv.lock generated via `uv lock` - 86 packages resolved. Verified with `uv sync` (installs cleanly) and a real end-to-end training run.
src/tests/pipeline/engine_training/smoke_test/test_train_smoke.py, following the repo's one existing test convention (src/tests/integration/engine_backend/integration_harness/): stdlib unittest, no third-party packages needed to run the test file itself (synthetic audio written with the stdlib wave module). Runs the real main.py CLI as a subprocess against a synthetic 3-class dataset (alternating clip lengths to exercise both the random-crop and pad-by-repeat branches in dataset.py, plus synthetic background-noise files so the default preset's AddBackgroundNoise is actually exercised). Asserts exit code 0, a best_*.pth checkpoint, a TensorBoard event file, and a correct class_names.txt. 6 files per synthetic class, not fewer - found during development that <5 files/class makes the default val_split=0.2 round down to 0 samples, crashing train.py's _evaluate with ZeroDivisionError. Documented in this test's README and the pipeline's own README. Verified passing: `Ran 1 test in 34.053s - OK`.
README.md replaces the deleted placeholder (attribution, file layout, setup/execution instructions with example commands, unresolved- dependency list, baseline configuration record, and known limitations - including that this is a new baseline built on Kiernan's config, not a reproduction of any previously-trained model's actual training run, per team decision). docs/training.md ported near-verbatim from the archive (already accurate) with one new section documenting the augment.py rewrite. docs/baseline_smoke_training_log.txt: full stdout/stderr from a real, successful run of the smoke test's exact scenario (3 synthetic classes, ghost_efficientnet_v2, 2 epochs, CPU) - Training complete, best model saved from epoch 1, return code 0.
…eyond Sprint 1 scope) While reviewing how the "selected model" was previously trained for the Sprint 1 config task, found the question doesn't have a single answer in this repo. Wrote these up as standalone findings docs rather than folding them into the training-pipeline task, since they concern the wider Engine team, not just this sprint's deliverable. Model_Provenance_Audit.md: - Engine.Dockerfile deploys echo_engine_iot.py, which defaults (ACTIVE_INFERENCE_MODEL: efficientnetv2_tflite in echo_engine.json) to a PyTorch-trained TFLite model - not the TF-Serving echo_model that docs/architecture/Engine_Documentation.md and src/production/README.md describe as "the" model. - The committed echo_model/1/ weights have no traceable training run - commit 573e4e3's own message says they were force-copied from "upstream" to unblock CI, explicitly not from a local training run. - optimised_engine_pipeline.ipynb (the documented training notebook) sets no global seed beyond an initial file-listing shuffle, and its test split is ~1% of the data (16 files). Species_Class_List_Audit.md: - class_names.json (21 species, used elsewhere in the system) and the TFLite model's own class_mapping.json (123 species) only overlap on 8 species even case-insensitively - roughly 60% of class_names.json doesn't correspond to any class the deployed TFLite model can output. - 5 species in the local dataset use underscore-separated names (e.g. Spilopelia_chinensis) that silently fail to match the space-separated form used everywhere else. - 5 entries in the TFLite class list (brant, jabwar, sheowl, spodov, wiltur) don't look like species names, though real audio folders exist for at least two of them. Findings only - no production files changed. Each doc ends with recommended next steps requiring team/original-author input before acting on them.
…enchmarking ignore entry
Mangon3
approved these changes
Aug 10, 2026
Mangon3
left a comment
Mangon3
left a comment
Collaborator
There was a problem hiding this comment.
Looks good, I'll approve but there are some possible improvements:
train.py - bf16 mixed precision looks disabled. use_amp = (self.dtype == torch.float16) and "cuda" in ..., and autocast is enabled=self.use_amp. So with the default dtype: bfloat16, autocast never turns on and training runs in fp32. bf16 autocast doesn't need the GradScaler, so it could be enabled independently of use_amp.
train.py - save_checkpoint/load_checkpoint are unused. train() saves with torch.save(self.model.state_dict(), ...) directly and nothing calls load_checkpoint, so the richer checkpoint (optimizer/epoch/cfg) and resume path are dead code. Try wiring the code together.
quant.py Duplicate import torch, both as quant and as quantization, and numpy is unused. Also input_size=(1,3,32,32) won't match the real spectrogram shape.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main.py,train.py,dataset.py,model/— 8 files) from.delete/archive/Prototypes/engine/torch_impl/into its permanent home atsrc/prototypes/engine/augmentation/, so that Kiernan Nguyen's shared Hydra config has runnable code behind it.main.py(cfg.run.cKLDivLossheckpoint_pathshould becfg.run.checkpoint_path, which was breaking therun.test=truecheckpoint-loading branch), and a missingOmegaConfimport intrain.pythat caused aNameErrorinsave_checkpoint/load_checkpoint.numpyandrandomalongsidetorchinmain.py, because augmentation masking usesrandom.randint/random.random, so the pipeline wasn't actually reproducible before, only the model init and dataloader shuffling were.augment.py'sSpecAugmentclass, because the archived version's constructor didn't match 4 of the 5 committed augmentation presets (default/light/heavy/original_unfixed_reference), so they would all raise aTypeErroronhydra.utils.instantiate. The new version supports both the legacy pixel API and the ratio-based API, and I didn't need to change any preset YAML to make this work.config.yaml, 7model/presets, 6teacher_model/presets, 5augmentation/presets,local/cpu.yaml) as the shared baseline, with one additive fix on top: I addednorm_choice: freeze_bnto 4 model presets (ghost_efficientnet_v2,panns_cnn14,panns_mobilenetv1,panns_mobilenetv2) that were missing this required key and crashed on model construction. I didn't remove or rename anything.config/local/local_data_files.yaml, a new opt-in local-dev config (following the same pattern ascpu.yaml) that points at the real local dataset insrc/prototypes/data_files/.pyproject.tomlanduv.lock(86 packages resolved) for this pipeline's dependencies, kept separate from the repo-root TF-basedrequirements.txtfiles.src/tests/pipeline/engine_training/smoke_test/..gitignorepatterns for pipeline artifacts (*.pth,*.onnx,outputs/,.cache/), and also to stop the untracked ~922MBsrc/prototypes/data_files//src/prototypes/spectrograms/local dataset from being accidentally committed.src/prototypes/engine/augmentation/README.md(setup and execution instructions, unresolved-dependency list, baseline configuration record, known limitations) and indocs/training.md(ported content plus a new SpecAugment section), and I captured a real run indocs/baseline_smoke_training_log.txt.Test plan
python src/tests/pipeline/engine_training/smoke_test/test_train_smoke.pyand gotRan 1 test in 34.053s — OK.ghost_efficientnet_v2, 3 real species fromsrc/prototypes/data_files/, 1 epoch, CPU. It completed successfully and saved a checkpoint.config/augmentation/*.yaml) instantiate viahydra.utils.instantiate, and each one ran a dummy tensor throughforward()without error.max_total_time_ratiocumulative-cap behaviour over 500 random trials, and it stayed within the cap every time.uv syncinstalls cleanly (86 packages) fromsrc/prototypes/engine/augmentation/pyproject.toml.Notes for reviewers
norm_choicemissing from 4 model preset YAMLs,augment.py's API mismatch with the committed presets, aval_splitedge case (each class needs at least 5 files, or the validation split rounds down to 0 and crashestrain.py's_evaluatewith aZeroDivisionError), and the seeding/reproducibility gap. I've fixed all of these and detailed them inREADME.mdunder "Known Limitations".norm_choice: freeze_bnthe right default forghost_efficientnet_v2and thepanns_*models? Sincepanns_mobilenetv2_qat.yamlalready usesswap_rms_normfor its QAT variant, this might not be uniform across every architecture. I made this choice as a judgment call to unblock the pipeline, not as a design decision on the team's behalf, so I'd appreciate a second opinion.config/teacher_model/*.yamlpresets have the same missing-norm_choicegap. I've left this unfixed for now, since the distillation path is deferred to Sprint 2.run.test=truecheckpoint-loading path, and a full (non-synthetic) training run against the real ~922MB local dataset to Sprint 2.