docs(tests): spec for the oracle-vetting test framework#368
Open
sameeul wants to merge 4 commits into
Open
Conversation
Specification only — changes nothing in the test tree. Defines:
* the core principle (vetting is a property of a feature x config x oracle
assertion, not of a feature name; regression tests never claim vetting)
* the four test kinds (oracle / regression / invariant / mechanics)
* a git-tracked coverage registry (one row per assertion) with the headline
"vetted by >=1 oracle" rollup as the metric we drive to 100%
* the oracle-tool catalog (pyradiomics, radiomicsj, mirp, matlab, cellprofiler,
mitk, feature2djava, wndcharm, imea, imagej, fraclac, ibsi, analytic)
* config-matching recipes and the config-matrix -> valid/invalid triage that
derives the set of tests (keep valid, snapshot production-only, drop invalid
with a recorded reason)
* naming conventions for tests and benchmark data
* tolerance policy and an authoring checklist
Registry seeding, recipes, matrices, generators, and test renames are follow-up
work gated on approval + merge of this spec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sameeul
force-pushed
the
spec/oracle-vetting-framework
branch
from
89f573e to
f84b25e
Compare
Companion to SPEC.md: how to stand up each oracle tool locally (Docker or Python
venv; MATLAB excluded). Per-tool setup, feasibility, coverage-by-family, and a
suggested rollout order, from one research pass per tool.
Highlights / corrections found during research:
* pyradiomics -> Docker (radiomics/pyradiomics = 3.0.1); mirp/imea -> pure-pip
venv; feature2djava/cellprofiler -> prebuilt Docker; wndcharm -> custom Py2.7
Docker; radiomicsj -> Maven uber-jar/Docker.
* radiomicsj 2.1.2 does not exist (Maven Central has 2.1.3..2.1.18) -> pin 2.1.3
or 2.1.18.
* FracLac plugin is GUI-bound; its shifting-grid box-count is reproduced headless
by the existing shiftgrid_boxcount.ijm macro (a from-method reimplementation).
FRACT_DIM_BOXCOUNT also has radiomicsj + imea as headless oracles.
* imea 3D is heightmap-based (not voxel) -> 2D-morphology oracle only.
* MITK: no prebuilt image, but a build-once Docker using the ClassificationCmdApps
CLI-only config (~2-3 h, Qt from apt) yields a reusable pinned image -> feasibility
med; deferred unless a 4th IBSI opinion is wanted.
Coverage: nearly every Nyxus family has >=1 headless oracle; genuine gaps are
neighbor features and image-quality (analytic only) and thin raw/central/Hu moments.
Tool-run mechanics remain the deferred decision in SPEC.md sec.9; this doc informs it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sameeul
force-pushed
the
spec/oracle-vetting-framework
branch
from
9015839 to
6469466
Compare
Companions to the framework spec (SPEC.md / TOOLS.md); data + generators, no test-tree changes. audit/ — baseline coverage snapshot of the current merged tree (seeds the SPEC registry): vetting_report.csv (758 features), vetting_pivot.csv, gap_*.csv, TEST_VETTING_REPORT.md, and the repo-relative regeneration scripts (extract_features -> scan_tests -> merge -> report) + raw audit_*.txt inputs. Baseline predates the GLCM fix and histogram/GLCM oracle tests still on their own branches; the registry is maintained forward from here. oracles/fraclac/ — headless ImageJ macro reproducing FracLac's shifting-grid box-counting (the FracLac plugin is GUI-only) + a numpy cross-check, preserved from the fractal-dim work. README.md — indexes SPEC.md / TOOLS.md / audit/ / oracles/ and the baseline caveat. Hardcoded absolute paths in the audit scripts were made repo-relative so the pipeline regenerates from any checkout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sameeul
force-pushed
the
spec/oracle-vetting-framework
branch
from
6469466 to
d8b7f6c
Compare
… not analytic-only Correct an over-conservative claim: the neighbor family DOES have a tool oracle. CellProfiler MeasureObjectNeighbors vets CLOSEST_NEIGHBOR*_DIST (1:1, centroid- Euclidean) and NUM_NEIGHBORS / PERCENT_TOUCHING (with-caveat: disk-dilation vs Euclidean contour distance, perimeter-set differences). A scipy.cKDTree boundary-pixel recipe reproduces Nyxus's definitions exactly and is the primary oracle for the angle features + a bit-exact backstop for the rest. Only the neighbor ANGLE stats stay analytic-only: CellProfiler's AngleBetweenNeighbors is a different quantity (interior subtended angle of the two closest, 0-180) than Nyxus ANG_BW_NEIGHBORS_* (population stats of all-neighbor bearings, 0-360). CellProfiler config to match Nyxus PERCENT_TOUCHING: "Within a specified distance" = radius-based touch (pre-PolusAI#359); "Adjacent" = 8-connected touch (post-PolusAI#359). Updates the coverage-by-family table + adds a neighbor reconciliation subsection to TOOLS.md, and adds MeasureObjectNeighbors to the SPEC.md tool catalog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A specification for a Nyxus oracle-vetting test framework — a single doc,
tests/vetting/SPEC.md. Changes nothing in the test tree. It exists to be reviewed/approved before any test modification begins.Why
The current tests mix two jobs (drift-guard vs. correctness) and, in places, label the same feature "vetted" in one file and "unvetted" in another depending on config — which reads as a contradiction. The spec fixes the model, not just the labels.
Core idea
Vetting is a property of a
(feature × config × oracle)assertion, not of a feature name.What the spec defines
oracle/regression/invariant/mechanics.oracle_coverage.csv), one row per assertion, with the headline metric "features vetted by ≥1 oracle" (plus a ≥2-oracle redundancy count and the remaining gaps) as the number we drive to 100%.test_ibsi_*,test_compat_3d_*,test_glcm.h).Settled in the spec
Home dir
tests/vetting/; CSV registry; naming + renames approved. Tool-run mechanics (how each tool is invoked to generate goldens) are deferred, decided per tool before first use.Not in this PR
Registry seeding, recipes, matrices, generators, and all test moves/renames — follow-up work gated on merge of this spec.
🤖 Generated with Claude Code