Feat/lever protocol and ledger - #26
Open
contactacefleetdev wants to merge 8 commits into
Open
Conversation
…BYTES_PER_TOKEN Adds `ace.sidecar.levers`: the public contract optimization modules are written against, containing no optimizations itself. Implementations ship separately and register through the `ace.sidecar.levers` entry-point group, so this repository stays a measurement product that gains levers when a package providing them is installed, and an absent lever package is the ordinary case rather than an error. Three properties the design turns on: * One lever, every agent. Levers read the corpus shape `_scan`, `_scan_antigravity` and `_scan_codex` already agree on, never a provider's wire format. Supporting a fourth coding agent is a scanner, not a lever rewrite. * Measurement is universal, actuation is not. Scoring runs off transcripts and so works for every agent the sidecar can read. Rewriting bytes needs a write path and only some agents have one; `Lever.requires_content` declares which half a lever needs and the registry refuses the mismatch rather than degrading. * Levers propose, the ledger prices. No lever returns a dollar figure. Pricing happens once, where the tokenizer and rate catalog live, which keeps a saving auditable and keeps provider-specific cache economics out of a lever that is meant to be provider-neutral. The ledger prices nothing it cannot count exactly: without the text in hand it reports `unmeasurable` rather than deriving tokens from a byte ratio. It nets the cache-write penalty against gross saving and reports a break-even turn, because an edit to content the cache has already stored invalidates the prefix from that point and the next turn re-writes the remainder at a premium. On a 10-turn fixture the same result trimmed at creation nets +$0.024, while expiring it at turn 5 nets -$0.019 and needs 17 turns to repay itself; a gross-savings-only ledger reports the second as a win. `LedgerReport` ranks and deliberately exposes no total, because levers scored alone can claim the same bytes. Also corrects BYTES_PER_TOKEN, 4.0 -> 2.8. Against 4,512 text-only single-call results from the local corpus the observed characters-per-token distribution is p10 1.51 / median 2.16 / p90 2.64 / p95 2.82 / p99 4.34 — 4.0 sat at the 99th percentile, the prose figure applied to code, JSON, logs and diffs. The value chosen is the upper tail rather than the median because the derivation is contaminated in the direction that lowers the ratio, and because `score` divides by this constant, so the correction raises every rail figure by ~1.43x. Left alongside it is `_check_image_bridge`, which warns once if `insights._CHARS_PER_TOKEN` ever stops aliasing this name: the two are one quantity — `_measure` multiplies an image's token count by it and `score` divides by it — so images round-trip only while they agree, and are silently mispriced by their ratio when they do not.
contactacefleetdev
force-pushed
the
feat/lever-protocol-and-ledger
branch
from
August 27, 2026 18:51
92a9a20 to
9caa5e0
Compare
Puts the measured half of the lever rail on the page beside the simulated half,
and — more importantly — makes the page say which of the two it is showing.
`levers/rail.py` sits between the scanner and the renderer so neither has to know
about levers: `insights._build_payload` calls `rail_payload` and passes the result
through, `dashboard_render` reads it. The import is lazy and wrapped, because
discovery loads third-party distributions and nothing a stranger's package does at
import time may take this dashboard down; a failure there costs the live column and
leaves every measured figure on the page intact.
Five states, only one of which may print a dollar figure:
no_package nothing registers against the entry-point group — the ordinary
state for the OSS sidecar on its own
all_off levers installed, all resolving to `off`; presence is not consent
no_counter no exact token counter, so the ledger priced nothing. A byte-ratio
fallback would produce a number here, which is why there is none
no_content levers ran, but the scan carries sizes and hashes rather than text
measured real edits, exactly counted, net of the cache-write penalty
`no_content` is the state that matters for this page. The transcript scan holds no
tool-result bytes by design, so a lever can be installed, enabled, and still price
nothing here: `measured` is unreachable from transcripts alone and needs the proxy
or a hook to supply the text. That is a property of the measurement path rather
than a bug, and it is reported distinctly from `no_counter` because the two look
identical on a page and mean opposite things about whether a lever is worth
enabling.
The live block renders separately from the headroom rows rather than merging into
them. A headroom row is a byte-turn simulation of what a lever would be worth; a
live row is a lever that ran and was priced exactly. Merging them into one number
is the most misleading thing this page could do. A live row can also read negative
— an edit to already-cached content pays a re-write premium the saving has to earn
back — and showing that is the point.
`_lever_note` now reads the actual state instead of asserting "none are wired",
which was a hardcoded truth of Phase 0 and stops being true the moment a lever
package is installed.
…esult Completes the chain the ledger was built for. The rail could price a lever but had nothing real to price: transcripts carry hashes and sizes, and an exact token delta needs the bytes. - counter.py: an exact counter built from the credential in flight. Under `no_key: true` the OAuth token the proxy relays is the only credential that can reach /v1/messages/count_tokens. No preflight probe -- the first count settles it; a 401/403 latches and keeps the reason so the rail can say why rather than going blank. A 429 deliberately does not latch. - shadow.py: adapts a /v1/messages body to the same typed model transcripts produce, builds the counterfactual copy-on-write, and prices the delta against the turn's own billed buckets, newest first. Runs after the response on a worker thread; never sees the relayed bytes. - lever_turns: a sibling table to `turns`. A counterfactual and a real charge must not share a record. Per-lever, never totalled. Third-party diagnostics are scrubbed to numeric values only. - the rail READS the store rather than recomputing, refreshed outside the payload cache so the one live number is not frozen. Rendering folds into the existing rail as rows, not a second panel: one scoreboard a reader can compare, with measured rows tagged and scaled separately so they can never be added to the simulated ones. Also fixes two pre-existing bugs found on the way: - usage_to_request_log imported ace.observability.telemetry, absent from this distribution. It raised on every turn into the except that protects the response, so ~/.ace/telemetry.db was silently empty and the dashboard reported zero spend on a working sidecar. - rail_payload priced with a ctx leaked from a loop, attributing one lever's settings to every other lever's pricing.
A bar has to be scaled against something and both options were wrong. Scaled against the simulated rows, a measured $0.75 renders as a 240th of a simulated $179 -- but those are different claims, not a ratio. Scaled within the measured group, which is what shipped, the largest measured lever drew FULL WIDTH directly beneath the largest simulated one, so a $0.75 measurement was visually indistinguishable from $179 of headroom. The dollar figure and the MEASURED tag carry the row. Nothing is implied by length.
A truncation's saving is countable and this codebase already counts it. Its COST is a re-read that may never happen, and reporting the saving alone quotes a gross figure as if it were net. That input was the one number the ledger could not derive, and it decided the whole lever: break-even sits at a 41% re-read rate, so the difference between 10% and 50% is +$468 and -$118 on the same corpus. Measurable in shadow, without changing a prompt. An agent request carries the whole conversation, so when a lever proposes cutting the result at position i, that same body already shows whether the agent later went back to that target. No applied edit, no state kept across requests. The distinction that decides the number: a later Read at a HIGHER offset is pagination, not damage. It fetches fresh bytes from the file and would have happened whether or not the earlier result was trimmed. Counting it condemns the lever for behaviour it did not cause -- and it is most of the signal. On this machine's 144 transcripts, "returned to the target at all" is 28.3%; true re-reads are 20.0%. Measured, per tool: all tools 20.0% -> net $272 of a $533 gross Bash only 0.1% -> net $176 of a $177 gross So the calibration question is answered: truncation is worth having, and scoping it to command output rather than file reads keeps essentially all of the saving while removing essentially all of the risk. Persisted per turn and surfaced on the measured rail row, because a rate that lives only in an analysis script is an anecdote. Old databases are migrated -- CREATE TABLE IF NOT EXISTS silently keeps the old shape, so a schema change alone never reaches a developer already running the sidecar.
…adow/prod preview
Two additions, one rule between them: nothing on this page may state a number
it cannot source.
REFERENCES. A figure with nothing beside it cannot be acted on -- "$0.06 per
turn" is not high or low until something says what else it could have been.
Each tile now carries a comparison point and the receipt for it:
SELF the developer's own distribution over the sessions in scope
PEER another agent on this same machine, same window (a controlled
comparison: one developer, one period, one corpus)
PUBLISHED a provider-stated constant, with the source URL and as_of that
ace.gateway.pricing already records
MEASURED a figure this repo derived, with the derivation attached
NO REF no grounded reference exists -- said plainly
That last one is the point. The most requested number here is the optimal
conversation-turn count for coding quality, and there is no figure for it
this process can verify. It renders as NO REF with what establishing one
would take (a replay eval: the same task at varying context lengths,
scored). An invented benchmark is worse than a blank one -- a reader cannot
tell them apart, and the blank one at least prompts the right question.
This repo already holds that line for model capability claims in
data/model_market/benchmarks.yaml, where an empty `source` means nobody has
pointed at the leaderboard yet.
MODE PREVIEW. A top-level off/shadow/prod control annotates the metrics the
enabled skills would move, with per-source attribution on hover. Client-side
and inert by design: switching to `prod` previews, it does not enable
anything. A lever is enabled in ~/.ace/config.json and nowhere else -- a
dashboard that could switch one on from a hover is a dashboard that can
silently start rewriting a developer's prompts.
The attribution never sums. Levers are scored alone so their shares overlap;
the headline takes the single largest contributor and the hover says so,
the same discipline the ledger and the rail already keep. Simulated and
measured sources are labelled separately and never merged.
…does not move
The toggle previously annotated three cost tiles. It now answers the question
for every headline metric, in that metric's own unit -- tokens where the
figure is tokens, dollars where it is dollars -- and attributes each to the
skill producing it.
tokens_in -365.5M tok (-8.4%) via bash truncate
tokens_per_commit -892k tok (-8.4%)
list_price_cost -$181.72 (-5.9%)
cost_per_turn -$0.0035 (-5.9%)
cost_per_session -$0.6310 (-5.9%)
cost_per_commit -$0.4432 (-5.9%)
Note the two rates. Volume falls 8.4% while cost falls 5.9%, and the gap is
the point: the removed tokens are disproportionately cache reads at 0.1x, so
a token saved is worth less than an average token. A single percentage
applied to both would have overstated the money by ~40%.
Half the value is the metrics reported as UNCHANGED, which are now computed
and stated rather than left blank:
tokens_out the model generates these; removing prompt content
does not shorten a reply
commits_per_session no context lever can write code
api_requests / only a loop guardrail moves these, and it has
conversation_turns detected no loop in this scope
A blank slot and "this skill cannot move this number" look identical to a
reader and are different claims. Claiming an output-token reduction for a
prompt-side lever would have been the most flattering error available here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Turns the lever rail from a simulation into something that measures, then makes the
dashboard say where every number came from.
Stacked on the two commits already on this branch (
9caa5e0,aa6b79f), whichdefined the lever protocol, the priced ledger, and the rail wiring.
What this adds
Levers now run for real on proxied turns.
levers/shadow.pyadapts a/v1/messagesbody into the same typed model transcripts produce, builds thecounterfactual copy-on-write, and prices the delta against the turn's own billed
buckets. It never sees the relayed bytes — only the throwaway
parsedcopy theroute already makes — and runs after the response on a worker thread, so it costs
the developer's turn nothing.
An exact counter, built from the credential in flight. Under
no_key: truethe OAuth token the proxy relays is the only credential that can reach
/v1/messages/count_tokens. There is no preflight probe: the first count settleswhether the endpoint accepts a subscription token, a 401/403 latches and keeps the
reason so the rail can say why rather than going blank, and a 429 deliberately
does not latch.
Results persist.
lever_turnsis a sibling table toturns— a counterfactualand a real charge must not share a record. Aggregated per lever and never totalled.
Third-party diagnostics are scrubbed to numeric values only, so a lever logging a
matched command cannot put session text in the database. Existing databases are
migrated.
The re-read rate is instrumented — the cost side of a truncation, and the one
input the ledger cannot derive. Break-even sits at a 41% re-read rate, so the
difference between 10% and 50% is +$468 and −$118 on the same corpus. Measured in
shadow without changing a prompt: a request carries the whole conversation, so the
body already shows whether the agent went back to a target a lever proposed to cut.
Measured on this machine's 144 transcripts: 20.0% all tools, 0.1% Bash only.
The distinction that decides it is pagination — a later
Readat a higher offsetfetches fresh bytes and would have happened anyway. Counting it as damage takes the
figure from 20.0% to 28.3%, across the break-even line, for behaviour the lever did
not cause.
Grounded references beside each metric. Every tile carries a comparison and its
provenance:
SELF(your own distribution),PEER(another agent, same machine,same window),
PUBLISHED(a provider constant with the source URL andas_ofalready in
pricing.py),MEASURED(derived here, derivation attached), orNO REF.That last one matters. There is no verifiable figure for the quality-optimal
conversation-turn count, so it renders as
NO REFwith what establishing one wouldtake — a replay eval — rather than an invented number. Same standard
data/model_market/benchmarks.yamlalready holds, where an emptysourcemeansnobody has pointed at the leaderboard yet.
An off/shadow/prod preview. Annotates every headline metric with its
counterfactual, in that metric's own unit, attributed to the skill producing it:
tokens_inlist_price_costcost_per_turncost_per_commitThe two rates differ on purpose. Volume falls 8.4% while cost falls 5.9%: the
removed tokens are disproportionately cache reads at 0.1×, so a saved token is
worth less than an average one. One percentage across both would overstate the
money by ~40%.
Four tiles render
NO CHANGEwith the reason —tokens_out(the model generatesthese; removing prompt content does not shorten a reply),
commits_per_session,api_requestsandconversation_turns. A blank slot and "this cannot move" lookidentical to a reader and are different claims.
prodpreviews only. A lever is enabled in~/.ace/config.jsonand nowhere else;a dashboard that could switch one on from a hover is one that can silently start
rewriting prompts.
Two pre-existing bugs fixed on the way
usage_to_request_logimportedace.observability.telemetry, which does notexist in this distribution. It raised on every turn into the
exceptthatprotects the response, so
~/.ace/telemetry.dbstayed empty and the dashboardreported zero spend on a working sidecar.
rail_payloadpriced with actxleaked from a loop, attributing one lever'ssettings to every other lever's pricing.
Testing
98 tests, 46 of them new. No live provider call anywhere — the upstream relay and
the counting endpoint are both driven through
httpx.MockTransport.Covered specifically: the fidelity invariant (a lever stripping 96% of the prompt
still leaves the upstream bytes identical), credential latching, the pagination
distinction in the re-read instrument, the numbers-only store invariant, and
migration of an older database.
Notes for review
origin/maininfcdde56;_nav_jsfrom53d3281survives and isemitted. Suite green after the merge.
feat/ace-skills-shared-packageinai-compute-efficiencyprovides levers that register against this contract. Neither branch needs the
other to build.
feat/code-quality-metricscontains an older duplicate of some of thesefiles, swept in by a parallel session. If that lands first expect conflicts; the
version here is newer.
Type of Change
Verification
Describe the tests and checks performed to verify changes:
pytestpasses cleanly.python -m buildandtwine check dist/*pass.scripts/e2e_test.pypasses.Checklist
CHANGELOG.mdwith my changes.