Found reviewing #241.
scripts/run_trait_graph_audit.py:125-146 walks the manifest and asserts each ok row's output path exists. That catches the failure it was built for — 342 rows pointing at artifacts lost when research/ was gitignored. Three adjacent failure modes still pass silently.
1. A sidecar can vanish without the check noticing
Every report has a <report>.md.citations.md sidecar, and --verify never looks at one. The manifest's output column names only the report. Delete all 353 sidecars and just trait-graph-sweep --verify still exits 0.
This is already observable: research/traits/metabolism/cellulolysis-deep-research-codex.md is tracked and has no sidecar — the only report in the tree without one.
2. Existence is not non-emptiness
.exists() passes for a zero-byte file. A truncated write, a killed process mid-flush, or a provider returning an empty body all yield an artifact that satisfies the check and contains nothing. The current corpus is fine — smallest report is 20 KB, smallest sidecar 72 lines — so a floor can be set now without picking a number that fails on real data.
3. The check only runs manifest to disk, never disk to manifest
A file on disk with no ok row is invisible to --verify, and because resume is keyed on file existence (output_path() + the pending filter at run_trait_graph_audit.py:112-117), such a file suppresses a call that was never paid for or recorded. cellulolysis-deep-research-codex.md is exactly this shape: tracked, no manifest row anywhere in 714 rows. It is harmless today only because its -codex suffix is not the -falcon name resume looks for.
Fix
In --verify: assert the sidecar exists alongside each ok artifact, assert both are above a small non-zero byte floor, and walk research/traits/** reporting any report file with no ok row. Keep it credential-free so CI and a fresh clone can run it.
Related: #242 (the CURIE scan belongs in the same pass), #245 (the stray codex artifact).
Found reviewing #241.
scripts/run_trait_graph_audit.py:125-146walks the manifest and asserts eachokrow'soutputpath exists. That catches the failure it was built for — 342 rows pointing at artifacts lost whenresearch/was gitignored. Three adjacent failure modes still pass silently.1. A sidecar can vanish without the check noticing
Every report has a
<report>.md.citations.mdsidecar, and--verifynever looks at one. The manifest'soutputcolumn names only the report. Delete all 353 sidecars andjust trait-graph-sweep --verifystill exits 0.This is already observable:
research/traits/metabolism/cellulolysis-deep-research-codex.mdis tracked and has no sidecar — the only report in the tree without one.2. Existence is not non-emptiness
.exists()passes for a zero-byte file. A truncated write, a killed process mid-flush, or a provider returning an empty body all yield an artifact that satisfies the check and contains nothing. The current corpus is fine — smallest report is 20 KB, smallest sidecar 72 lines — so a floor can be set now without picking a number that fails on real data.3. The check only runs manifest to disk, never disk to manifest
A file on disk with no
okrow is invisible to--verify, and because resume is keyed on file existence (output_path()+ thependingfilter atrun_trait_graph_audit.py:112-117), such a file suppresses a call that was never paid for or recorded.cellulolysis-deep-research-codex.mdis exactly this shape: tracked, no manifest row anywhere in 714 rows. It is harmless today only because its-codexsuffix is not the-falconname resume looks for.Fix
In
--verify: assert the sidecar exists alongside eachokartifact, assert both are above a small non-zero byte floor, and walkresearch/traits/**reporting any report file with nookrow. Keep it credential-free so CI and a fresh clone can run it.Related: #242 (the CURIE scan belongs in the same pass), #245 (the stray codex artifact).