Skip to content

Phase 2C.2: evaluate lightweight teaching-image routes - #31

Draft
xueyang-dev wants to merge 3 commits into
mainfrom
codex/phase2c2-model-evaluation
Draft

Phase 2C.2: evaluate lightweight teaching-image routes#31
xueyang-dev wants to merge 3 commits into
mainfrom
codex/phase2c2-model-evaluation

Conversation

@xueyang-dev

Copy link
Copy Markdown
Owner

Summary

This draft PR adds the first controlled evaluation loop for lightweight local teaching-image routes. It does not change the production Model Package, fixed Workflow Pack, Provider Hub defaults, teacher UI, or any generated production asset.

What changed

  • Added a versioned candidate audit for SSD-1B and Sana 0.6B with pinned revisions, selected fp16 component sizes, SHA-256 identities, license/platform notes, and a fail-closed Sana decision.
  • Added a deterministic, project-authored 12-scene SVG/Pillow flat-cartoon dataset generator with captions, hashes, and rights evidence. Dataset outputs remain under ignored runtime/.
  • Added a fixed six-case × two-seed evaluation contract for base and base+LoRA variants, with resumable/idempotent execution, retry isolation, identity invalidation, PNG verification, provenance, Asset Manifest binding, and a teacher review export whose scores remain pending.
  • Added the official Diffusers SDXL LoRA training wrapper and documented the real MPS pilot.
  • Added a complete evaluation report and decision document.

Real evidence

  • SSD-1B segmind/SSD-1B revision 60987f37e94cd59c36b1cba832b9f97b57395a10, Apache-2.0 model card, 4,465,653,694 selected fp16 bytes. Local component SHA-256 values are recorded in benchmarks/phase2c2/candidate-audit.v1.json.
  • Original LoRA training completed on the 12-scene set in 96.382 s with 12 steps, rank 4, fp16, gradient checkpointing, batch 1, accumulation 2; final weight is 10,910,072 bytes. The hardlink aliases used by the official trainer do not duplicate model bytes.
  • SSD pilot: 24/24 real MPS images succeeded (six cases × two seeds × base/LoRA), with no final near-solid output, OOM, crash, or timeout. Base mean generation was 11.480 s; LoRA mean was 15.220 s. Maximum observed MPS driver allocations were 8,250,933,248 and 8,284,487,680 bytes respectively.
  • The LoRA clearly shifted style toward flat cartoon, but count/action/spatial failures remained. Teacher review is still pending_review for all 24 records; no teacher conclusion is inferred.
  • Sana Efficient-Large-Model/Sana_600M_512px_diffusers revision 2defc07f5fb66d0c53ace051585e9a2cb83f8c15 was not downloaded or run: the official NVIDIA license restricts the checkpoint to NVIDIA Processors and the current host is Apple Silicon. Its official ComfyUI path also requires custom nodes, outside this experiment boundary.

Validation

  • npm run test:api — 651 passed, 3 skipped.
  • npm run build:web — TypeScript and Vite build passed.
  • npm run test:web-state — passed.
  • npm run test:e2e — 14 passed.
  • Focused Phase 2C.2 tests — 4 passed.
  • Ruff and git diff --check passed.

Generated images, model weights, LoRA weights, dataset files, caches, virtualenv, reports, and runtime state are ignored and are not in this PR. The PR intentionally remains Draft pending teacher review and any later product decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant