[feat][evaluation] Centralized experiment scheduling: OSS seams, quota accounting, and reclaim paths

[xueyizheng]

Summary
Today each experiment drives its own item dispatch loop, filling up to its configured concurrency. That works in isolation but has no notion of global ordering or of a shared resource budget: when many experiments run in one deployment, whoever ticks first wins, and nothing prevents the sum of all experiments from exceeding the resources actually available.
This PR adds the seams and bookkeeping needed for a centralized scheduler — one that decides across experiments, by priority and by declared resource consumption — while keeping the existing per-experiment path byte-for-byte unchanged for experiments that do not opt in.
The scheduler implementation itself is pluggable and is not part of this PR; what lands here is the open-source side: schema, entities, narrow ports, projections, quota accounting, and the reclaim paths that keep the ledger honest.
What's in it
Opt-in, frozen at creation. An experiment carries a dispatch mode that is decided once at creation and never re-derived. Retries and re-runs read the frozen column, so changing configuration after the fact cannot silently move a running experiment between modes. Only experiments whose trigger source opts in are enrolled; everything else behaves exactly as before.
Declared consumption, expanded into constraints. An experiment declares what a single item costs (sandbox:default=1, model:some-model=1000, …). Each declaration expands into layered ledger keys — the concrete resource, its cross-source total, and the category-level total — so a per-resource cap and a per-category cap can both be enforced, and adding a new source cannot be used to slip past a total.
Two drivers cannot both dispatch. For enrolled experiments the legacy per-experiment tick keeps every other duty (archiving finished items, timeout handling, terminal wrap-up, next tick) but drops item submission. Letting both dispatch would overrun the budget immediately, since the legacy path neither consults global priority nor touches the ledger.
A reservation projection on the run log. Item run logs carry a reservation state so "reserved but not yet consumed" is distinguishable from "queued" and from "running". The consumer redeems the reservation when it takes execution rights, which is what makes at-most-once dispatch verifiable rather than hopeful.
Reclaim paths — the bulk of the fixes here. Quota that is taken must come back on every exit, and most of this PR's bug fixes are exactly that:
terminal wrap-up releases quota for items that never finished, judged by item state rather
than turn state (the two diverge when an external execution process dies);
deleting an experiment releases its quota (previously leaked, irreversibly);
retry releases the superseded run's reservations before the run id is rewritten — afterwards
the ledger keys can no longer be reconstructed;
terminal wrap-up also drives the run log to a terminal state, not just the main table: the
release predicate, the reconciliation view and the timeout sweep all read the run log, so
leaving it behind hands all three a false fact;
items whose reservation has gone missing are returned to the queue instead of being dropped,
which previously left them stuck in Processing forever.
A distinguishable failure for impossible declarations. If a single item declares more of a dimension than the deployment's registered ceiling, no amount of waiting will ever admit it. Previously this was indistinguishable from ordinary contention — the same "nothing granted" line every tick, with the experiment simply never progressing. There is now a dedicated error code carrying the dimension, the declared amount and the ceiling, surfaced on the item so the user can see which dimension to change and to what. A registered ceiling of zero is deliberately not treated this way: that is an operator throttle, and experiments are expected to queue until it is lifted.
Statistics stay consistent. Two daemon terminal paths were racing the main-table status write and a counter could drift; enrolled experiments now record statistics when execution rights are taken, and items failed for an impossible declaration adjust the queued/failed buckets directly (they never entered execution, so no consumer will do it for them).
Compatibility
Default behaviour is unchanged: without opt-in, dispatch, statistics and error surfaces are
identical to before.
The quota gate ships with a no-op implementation registered in the OSS wiring, so a build with
no scheduler configured behaves exactly as it does today.
Schema changes are additive columns plus one narrowed index; no column is dropped or retyped.

[codecov]

Codecov Report

❌ Patch coverage is 82.86674% with 153 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...luation/domain/service/expt_run_item_event_impl.go	74.86%	34 Missing and 14 partials ⚠️
...ules/evaluation/domain/service/expt_manage_impl.go	40.00%	39 Missing and 3 partials ⚠️
...on/domain/service/expt_run_scheduler_event_impl.go	62.68%	20 Missing and 5 partials ⚠️
...ation/domain/service/expt_manage_execution_impl.go	87.36%	8 Missing and 4 partials ⚠️
...aluation/infra/repo/experiment/expt_item_result.go	0.00%	8 Missing ⚠️
...valuation/application/convertor/experiment/expt.go	89.47%	2 Missing and 2 partials ⚠️
...d/modules/evaluation/application/experiment_app.go	93.93%	2 Missing and 2 partials ⚠️
...tion/domain/component/central_reservation_guard.go	80.00%	4 Missing ⚠️
...modules/evaluation/application/eval_openapi_app.go	88.46%	2 Missing and 1 partial ⚠️
backend/modules/evaluation/pkg/conf/configer.go	0.00%	3 Missing ⚠️

@@           Coverage Diff           @@
##             main     #646   +/-   ##
=======================================
  Coverage        ?   78.52%           
=======================================
  Files           ?      703           
  Lines           ?    86846           
  Branches        ?        0           
=======================================
  Hits            ?    68195           
  Misses          ?    14685           
  Partials        ?     3966           

Flag	Coverage Δ
unittests	78.52% <82.86%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...uation/application/convertor/experiment/openapi.go	92.37% <100.00%> (ø)
backend/modules/evaluation/domain/entity/event.go	84.61% <100.00%> (ø)
backend/modules/evaluation/domain/entity/expt.go	85.50% <100.00%> (ø)
...les/evaluation/domain/entity/expt_dispatch_mode.go	100.00% <100.00%> (ø)
...evaluation/domain/entity/expt_quota_consumption.go	100.00% <100.00%> (ø)
...nd/modules/evaluation/domain/entity/expt_result.go	64.28% <100.00%> (ø)
...ckend/modules/evaluation/domain/entity/expt_run.go	98.72% <100.00%> (ø)
...luation/domain/entity/expt_scheduling_privilege.go	100.00% <100.00%> (ø)
backend/modules/evaluation/domain/entity/param.go	87.71% <ø> (ø)
backend/modules/evaluation/domain/entity/target.go	100.00% <100.00%> (ø)
... and 21 more

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d570c8...f6cad49. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.