Skip to content

Calibrate real TinyDev Mean detector evidence - #31

Merged
byte271 merged 7 commits into
mainfrom
feat/real-tiny-dev-detector-evidence
Aug 18, 2026
Merged

byte271 merged 7 commits into
mainfrom
feat/real-tiny-dev-detector-evidence

Conversation

@byte271

@byte271 byte271 commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Moves the real TinyDev corpus into the next M6 prerequisite: source-bound detector evidence on the original generation-token track.

  • reconstructs Hugging Face SynthID observations from exact generated continuation token IDs, never silently substituting decoded/re-encoded text
  • scores both Mean and Weighted Mean using the pinned open adapter compatibility path
  • calibrates the frozen headline FPRs at 5% and 1% from exactly 100 unwatermarked threshold-calibration rows; 1% is the frozen primary estimand
  • evaluates the eight attack-development pristine samples, including four watermarked positives and four negative controls
  • records calibrated decisions, standardized margins, exact Clopper-Pearson calibration/attack-negative intervals, and the 80% pristine-TPR interpretability gate
  • keeps the artifact explicitly DEVELOPMENT_ONLY; this is not M6 readiness and not confirmatory evidence
  • adds a pinned real workflow that first reproduces the 208-sample GPT-2/SynthID TinyDev corpus with Torch 2.13.0, then calibrates/evaluates the detectors and uploads both content-addressed artifacts
  • adds self-hash/tamper and population-contract regressions using a full synthetic 104-pair corpus without requiring Torch in normal CI

The frozen spec names 5% and 1% as headline FPRs and 1% FPR as the primary TPR estimand. This PR does not attempt a 0.1% headline because TinyDev has only 100 calibration negatives.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

Important

Review available on request

  • 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7269cb79-590a-43e3-b29e-a385df8c3f05


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@byte271
byte271 merged commit e30109a into main Aug 18, 2026
5 checks passed
@github-actions
github-actions Bot deleted the feat/real-tiny-dev-detector-evidence branch August 22, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant