Research: is TEE hardware attestation necessary, or does a reproducible-backtest checksum approach cover the real threat?

[JSONbored]

Problem

Once the advisory backtest check is live and has a real track record, the same question applies to Loopover: if hosted ORB (per the current ~1-2-months-out roadmap) ever uses a backtest result to justify an automated decision to a customer, "trust the maintainer's report" is a weaker claim than "here's a cryptographically-attested execution anyone can verify." Before investing in TEE-specific infrastructure research, decide whether a cheaper mechanism already covers the real threat.

Area

Research spike. No code changes expected as the primary deliverable (a small proof-of-concept is acceptable if genuinely cheap, not required to close this issue).

Proposal

Threat model first. Who is the actual verifier, and what are they trying to verify — that the ground-truth corpus wasn't tampered with, that the score computation ran the real published code, or that the reported number wasn't fabricated outright? Different threats need different mechanisms: a checksum solves tampering; attestation solves "did the real code actually run"; neither solves "I don't trust the maintainer's infrastructure at all." Name which threat is actually in scope before evaluating a mechanism against it.

Then: does a reproducible-backtest approach cover that threat? Publish the corpus snapshot's checksum (BacktestCorpusManifest.checksum), the exact freeze point, and the script version, and let a skeptical third party independently re-run scoreBacktest/compareBacktestScores themselves and compare checksums. This proves the reported score is real, not fabricated, without any new infrastructure — for corpora that are safe to publish. It does NOT cover a private/held-out corpus (the corpus that must stay undisclosed for the anti-overfitting split to mean anything) — that's the one case this approach structurally can't reach.

Produce a short written recommendation: the reproducibility approach is sufficient, TEE research (#TEE-2, filed alongside this issue) isn't needed; the held-out-corpus gap is real and worth the TEE research; or neither is worth pursuing given the actual threat model. Any of these three is an acceptable, complete outcome.

Deliverables

• A written recommendation (issue comment or a linked doc) naming the threat in scope and the final call.

Resources

• Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082 (parent epic)
• BacktestCorpusManifest.checksum (scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084) — the reproducibility-check building block
• The held-out corpus (calibration: deterministic seeded held-out/visible split of the backtest corpus #8087) — the concrete case a reproducibility check can't cover

Boundaries

No production code commitment beyond an optional cheap proof-of-concept. No hosted-product architecture decision made here beyond the written recommendation.

maintainer-only — research/strategy, not a build task for contributors.

[linear-code]

JSO-5607

[JSONbored]

Threat-model analysis + recommendation (2026-07-22) — Phase-A conclusion for this spike.

1. Who verifies, against which threat?

Three distinct threats hide inside "trust the maintainer's report":

#	Threat	What disproves it
T1	The corpus was tampered with (history cherry-picked/edited to flatter a rule change)	A versioned, checksummed corpus snapshot whose checksum is pinned at run time — BacktestCorpusManifest.checksum (#8084)
T2	The reported score didn't come from running the published code	Re-running the public scoring code over the same frozen corpus and the same head/base shas — scoreBacktest/compareBacktestScores ship publicly in @loopover/engine
T3	The maintainer's infrastructure is untrusted outright, ground truth included	Nothing that runs on maintainer infrastructure — including a TEE. The labels are human-override events recorded in the maintainer's own D1; attestation proves computation, not data provenance

Verifiers today: gittensor miners/contributors affected by advisory verdicts (stakes: advisory-only — #8105 is deliberately unresolved, nothing gates a merge). Verifiers later: hosted-ORB customers, if a backtest result ever justifies an automated decision to them.

2. Does the reproducible-backtest approach cover the in-scope threats?

Yes — T1 and T2, fully, for publishable corpora. The freeze point already ships end to end:

• scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084 exports the checksummed manifest; the Extend the backtest check to logic/regex changes via a dedicated CI job, using captured raw context #8139 CI runs persist metadata.corpusChecksum + head/base shas with every comparison (completed in PR feat(ci): backtest logic/regex detection changes against recorded history in a dedicated CI job #8147 specifically for this posture); the scoring code is public npm. A skeptic can re-export, verify the checksum, and re-run both sides of any recorded comparison with one command each.
• Reproduction requires corpus access. Post-calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129/calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130/Extend the backtest check to logic/regex changes via a dedicated CI job, using captured raw context #8139, corpora embed raw evaluated content (issue text, diffs, model responses). For the repos ORB reviews today — public gittensor repos — that content is already public, so the corpus is publishable and reproducibility closes the gap outright.
• The residual gap is exactly the held-out/private corpus: hosted ORB reviewing private customer repos cannot publish that history for third-party re-runs. That narrow case is Research: survey viable TEE/confidential-compute options on Cloudflare Workers/Containers #8137's scope — and it is the only case where TEE buys anything.
• T3 is covered by neither mechanism. A perfectly attested execution proves "this exact code ran over data with this checksum" — not that the recorded human-override history is honest. Closing T3 would need provenance at event-capture time (e.g. receipts verifiable by the humans whose overrides they record). Different mechanism, different issue, and only worth filing if hosted customers actually demand it.

3. Recommendation

Reproducibility covers the real, current threat. TEE is not necessary now.

• Advisory-only stakes + publishable corpora + an already-shipped freeze point = closed trust gap for the current product.
• Revisit trigger: hosted ORB using a backtest over a private corpus to justify a customer-facing automated decision. Evaluate against Research: survey viable TEE/confidential-compute options on Cloudflare Workers/Containers #8137's survey at that point, not before.
• Cheap opportunistic hardening (not filed as a requirement): for public-repo corpora, publish the corpus snapshot (or a pointer) alongside the run so third-party re-runs don't require asking the maintainer for the export.

Phase A answer: reproducible-checksum covers it today; TEE only ever matters for the private-corpus subset — see #8137.