Add Trace the Ace mastery-event experiments

[heathsanchez]

Primary objective

Optimize for minimum log loss on genuinely unseen/private evaluation data. Public leaderboard movement, AUC, novelty, and write-up appeal are secondary. A change is retained only when it improves robust out-of-sample probability quality or adds stable orthogonal value in a strictly OOF ensemble.

The detailed promotion/validation order is now frozen in competitions/trace_the_ace/PLAN_UNSEEN_LOGLOSS.md.

Current stack

• v71_mastery_events.py: extracts tutor-question → student-answer → tutor-feedback episodes, applies conservative speaker-role repair, builds objective-conditioned mastery features, and evaluates grouped OOF log loss.
• v72_supervision_audit.py: measures multi-objective same-session supervision, opposite-label contrastive pairs, objective sparsity, and micro-assessment density using aggregate-only outputs.
• v73_contrastive_mastery.py: uses same-session opposite-label pairs to suppress generic session ability and learn objective-specific mastery residuals.
• v74_semantic_objective_prior.py: models objective difficulty hierarchically so rare/unseen objectives can borrow strength from semantically related skills.
• .github/workflows/trace-ace-mastery.yml: runs Python 3.12 self-tests on public Actions and supports a full private-data experiment through a secret URL without committing or logging competition data.

Priority change

Before larger pretrained-model experiments, the next implementation priority is V75 canonical student-state trajectory preprocessing:

1. conservative role repair with confidence;
2. interaction episode segmentation;
3. student-vs-tutor evidence separation;
4. low-information turn down-weighting rather than blind deletion;
5. objective-conditioned relevance;
6. explicit assistance/independence state;
7. chronology and terminal-state emphasis;
8. safe math-surface normalization while preserving raw text;
9. multiple retained views: raw, student-only, objective-local, canonical event sequence, terminal mastery window.

The governing rule is: remove nuisance variation, not educational variation.

Promotion rule

A candidate is promoted in this order:

1. lower aggregate session-cold log loss;
2. lower or non-inferior hard/rare/objective-cold loss;
3. lower worst-fold/tail risk;
4. safer calibration / fewer high-confidence errors;
5. orthogonal residual value in a strictly OOF ensemble.

A model that looks clever but worsens expected unseen log loss is rejected.

Validation

V71, V72 and V73 self-tests have passed in GitHub Actions. V74 is wired into the same workflow. Full-corpus transcript execution remains gated on private TRACE_ACE_DATA_URL transport so raw competition data never enters the public repository or logs.