Skip to content

Commit da7283c

Browse files
Merge pull request #140 from CodeGraphContext/fix/provider-price-events
fix(observe): preserve provider prices on planner and failed node events
2 parents 41ea115 + 352b0e4 commit da7283c

6 files changed

Lines changed: 297 additions & 37 deletions

File tree

‎ROADMAP.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -454,7 +454,7 @@ allowances; the agent records document-driven denials and approval decisions.
454454
- [x] **10.4 — Cost attribution** per run, thread (session), node and explicit tenant. The
455455
price is now recorded, not guessed: every gateway publishes a
456456
`cost_usd` through the same `llm_output` envelope, the runtime's usage
457-
callback accumulates it per node and writes it onto the `end` event, and
457+
callback accumulates it per node and writes it onto the `end` or `error` event, and
458458
an `AgentNode` writes the per-call figure onto each `model` event. A
459459
backend that reports no price still falls back to a `RateCard` estimate,
460460
and `recorded_cost_usd` and `estimated_cost_usd` never mix — a recorded
@@ -467,9 +467,10 @@ allowances; the agent records document-driven denials and approval decisions.
467467
`grapharc cost --tenant NAME` select whole labelled runs, refuse conflicting
468468
labels, and leave old unlabelled runs unassigned. Planning tokens without
469469
a recorded price are estimated or marked unpriced, including in tenant
470-
totals. **Producer limit:** planner turns and failed node terminal events
471-
still omit the provider price ([#138](https://github.com/CodeGraphContext/GraphARC/issues/138));
472-
a rate-card estimate is not a recorded bill.
470+
totals. Planner turns retain the call-specific price and model from the
471+
returned message envelope, including parse failures. Every node terminal
472+
error path retains its own usage scope, excluding concurrent siblings'
473+
spend. A rate-card estimate is not a recorded bill.
473474
Tenant labels provide attribution, not authorization.
474475
- [ ] **10.2 — Rollback** and versioned graph/prompt configs.
475476
- [ ] **10.5 — Alerting** on budget, failure, and verifier-drift.

‎docs/deep-dive.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ OpenRouter also carries routing: model-level `fallback_models` chains, provider
9494

9595
**Cost ceilings are enforced, on both sides of a call.** A `SpendMeter` refuses before a call once the ceiling is reached, and after a call it charges the cost and *then* raises if that crossed the line — so overspend is bounded by the single call that crossed it rather than discovered a node later. The limit worth stating: a call whose cost the provider does not report cannot be charged, and those land in `unpriced_calls` rather than being guessed at. `unpriced_calls > 0` means the ceiling saw less than the whole bill.
9696

97-
Caveats each backend accepts openly. **Claude CLI:** `bind_tools` and `with_structured_output` raise `NotImplementedError` — the adapter implements neither, so what you get is LangChain's `BaseChatModel` default, and `claude -p` in the tool-free mode GraphARC drives it in offers nothing to implement them with. GraphARC also has no cache control on this path — the CLI decides and the usage envelope reports what it did — and calls spend subscription quota. **OpenRouter:** credit is reserved against `max_tokens`, so the default is deliberately modest. **OpenAI:** the API returns token counts and no price, so `cost_usd` is `None` and a dollar ceiling counts calls instead of enforcing — pass `price_per_million=` or price the trace afterwards with `observe.cost.RateCard`; token budgets are unaffected. **Ollama:** free by definition, so calls are charged `0.0` rather than counted as unpriced, and whether tool-calling works depends on the model you pulled rather than on the adapter. **All of them:** a reported per-call `cost_usd` is captured and budgeted against. Successful node events and agent model events carry it into the trace; planner turns and failed node terminal events currently omit it. See [ROADMAP.md](../ROADMAP.md) §10.4 for explicit tenant attribution and producer pricing limits.
97+
Caveats each backend accepts openly. **Claude CLI:** `bind_tools` and `with_structured_output` raise `NotImplementedError` — the adapter implements neither, so what you get is LangChain's `BaseChatModel` default, and `claude -p` in the tool-free mode GraphARC drives it in offers nothing to implement them with. GraphARC also has no cache control on this path — the CLI decides and the usage envelope reports what it did — and calls spend subscription quota. **OpenRouter:** credit is reserved against `max_tokens`, so the default is deliberately modest. **OpenAI:** the API returns token counts and no price, so `cost_usd` is `None` and a dollar ceiling counts calls instead of enforcing — pass `price_per_million=` or price the trace afterwards with `observe.cost.RateCard`; token budgets are unaffected. **Ollama:** free by definition, so calls are charged `0.0` rather than counted as unpriced, and whether tool-calling works depends on the model you pulled rather than on the adapter. **All of them:** a reported per-call `cost_usd` is captured and budgeted against. Successful and failed node terminal events, planner turns and agent model events carry it into the trace. See [ROADMAP.md](../ROADMAP.md) §10.4 for explicit tenant attribution and producer pricing limits.
9898

9999

100100
## Tools and the harness
@@ -248,15 +248,15 @@ A stable system is not one that claims to have no edges — it is one whose edge
248248
- **`run_command` is not confined.** Argv-only and never a shell, but the child is an ordinary process with your privileges.
249249
- **`interrupt()` suspends but cannot be resumed.** LangGraph's native interrupt stops the graph and shows on `get_state`, and there is no supported resume path — resuming means passing a `Command` as *input*, which is closed by design. Use the session layer's approval gate for human-in-the-loop.
250250
- **Still unwrapped from LangGraph:** `retry_policy`, `cache_policy`, `durability`, subgraphs. `.inner` reaches them, but execution entry points there fail closed, so `.inner` is an inspection escape hatch and not a way to run the graph.
251-
- **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Explicit tenants are supported by `attribute_tenant` and `grapharc cost`; legacy unlabelled runs are excluded from named tenant bills. Planner turns and failed node terminal events still omit the provider price, so their tokens need a rate card or remain unpriced.
251+
- **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Explicit tenants are supported by `attribute_tenant` and `grapharc cost`; legacy unlabelled runs are excluded from named tenant bills. Planner turns and all failed node terminal paths retain the provider price and model identity when reported; parallel node failures retain only their own token spend.
252252
- **A node's tokens are its own, not the run's movement while it ran.** Worth stating because it was the other way round: an `end` event carried the difference between two readings of the run's *shared* meter, so under fan-out the workers' windows overlapped and each was credited with its siblings' concurrent spend. Three workers costing 8 tokens each traced as 24/16/8, and `metrics` and `cost` agreed on 48 for 24 tokens of real work — doubling the estimated bill purely because the work ran in parallel. Attribution now comes from a per-node scope on the meter, so the same work costs the same serially and in parallel; a hand charge the usage callback never saw still lands on the node that made it.
253253
- **A planning round is an envelope, not a measurement.** A `round` event used to carry the planner's `tokens` and the round's `duration_ms`, both of which `metrics`, `cost` and `replay` add on top of node totals — and the planner's spend was already reported by its own `plan` event, so it was counted twice, and a round's duration encloses the plan plus every node it ran. Neither is on the event now; both are on its `state_delta` as `round_tokens` / `round_iterations` / `round_duration_ms`, where no reader sums them. `RoundRecord.iterations` also holds a figure now rather than always `0`.
254254
- **The Claude CLI backend is completion-only, and an agent node on it is *delegated* rather than governed.** The CLI has no tool-calling wire format, so GraphARC cannot run its own gated loop over it. Rather than refuse, `AgentNode` hands the whole loop to Claude Code's headless agent, in one of two named tiers. `allowlist`, the default, pre-approves exactly the Claude Code twins of the node's own registered tools (`read_file`→`Read`, … `run_command`→`Bash`), so one operator declaration governs both the governed loop and the delegated one — but the enforcement is Claude Code's own gating, not this graph's per-call policy, there are no per-tool trace events, and anything unlisted falls to headless default gating, which fails closed. `bypass`, explicit opt-in only, runs `bypassPermissions`: every tool Claude Code has, no checks at all. In either tier the calls are not confined by the sandbox executor and the token figure is the sub-agent's own rather than one GraphARC metered call by call; a `--max-tokens` the delegated path cannot enforce is refused rather than silently unapplied. The workspace boundary and the wall-clock ceiling still hold — the CLI runs in its own session, and the deadline kills the whole process group, not just the direct child. It warns on `DelegatedToolUseWarning` at construction, naming the tier, and marks every trace event `executor=delegated` with its `delegated_mode`; filter that warning to an error to get the old refusal back. Structured output still needs an OpenAI-wire backend: `openrouter`, `openai`, or a local `ollama`.
255255
- **A session turn is synchronous**, and a runner claim is a claim rather than a lease — a runner that dies holding a session does not release it by itself. `SessionManager.reclaim(session_id)` releases one deliberately, refusing unless the recorded runner pid is gone from this host; it is not automatic, because pid liveness is host-local and pids are recycled.
256256
- **`.env` and `grapharc.toml` follow the same discovery rule: the working directory, and nowhere else.** Neither searches parent directories — a run must not be governed by a file you did not know about, and must not be *billed* to one either. **This is a behaviour change:** the credential loader used to walk up to `/`, so a `.env` in an ancestor directory (a `$HOME` one on a shared box, a client project one above a demo checkout) was picked up silently. If you relied on that, move the file into the directory you run from, `export` the variable, or pass `env_file=` to name it explicitly. A real environment variable still beats any file.
257257
- **`grapharc run` has no budget unless you give it one.** Set any of `--max-tokens`, `--max-iterations`, `--max-seconds`, or `--max-concurrency`; without them each dimension is unlimited and the gate admits a topology of any worst-case cost.
258258

259-
**Verified this pass:** `pytest` → green, 2,354 selected and 13 deselected (the live ones); `ruff check .` clean; all eight `grapharc demo` stages green, plus the `trace` / `metrics` / `viz` / `replay` tour against a freshly recorded demo trace; the wheel builds and imports all submodules in a clean virtualenv with `[all]`, and `0.1.8` on PyPI is that wheel. The counts are a snapshot, not a property of the project — `pytest` re-derives them in one command, which is the only reason they are quoted, and `tests/test_deep_dive.py` fails this line rather than letting it drift.
259+
**Verified this pass:** `pytest` → green, 2,384 selected and 13 deselected (the live ones); `ruff check .` clean; all eight `grapharc demo` stages green, plus the `trace` / `metrics` / `viz` / `replay` tour against a freshly recorded demo trace; the wheel builds and imports all submodules in a clean virtualenv with `[all]`, and `0.1.8` on PyPI is that wheel. The counts are a snapshot, not a property of the project — `pytest` re-derives them in one command, which is the only reason they are quoted, and `tests/test_deep_dive.py` fails this line rather than letting it drift.
260260

261261
[ROADMAP.md](../ROADMAP.md) tracks what is built and what is not, item by item.
262262

‎grapharc/planner/proposal.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -690,10 +690,19 @@ def propose(
690690
error = ""
691691
proposal = self._stamp(proposal)
692692

693+
# Read this call's envelope, rather than a model's mutable last_usage:
694+
# the latter can describe another concurrent call, or an earlier call
695+
# when this one failed before producing a message.
696+
metadata = raw_message.response_metadata if raw_message is not None else {}
697+
cost = (metadata.get("token_usage") or {}).get("cost")
698+
if cost is None:
699+
cost = metadata.get("cost_usd")
693700
self._emit(
694701
ctx,
695702
duration_ms=(time.perf_counter() - started) * 1000,
696703
tokens=tokens,
704+
cost_usd=cost,
705+
model=metadata.get("model_name"),
697706
state_delta={
698707
"structured": structured,
699708
"nodes": proposal.node_count() if proposal else 0,

‎grapharc/runtime/graph.py‎

Lines changed: 22 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -855,6 +855,7 @@ def _leave(
855855
usage: Any = None,
856856
) -> Any:
857857
"""Everything both node wrappers do after the node body returns."""
858+
measured = _usage_fields(ctx, tokens_before, usage)
858859
delta: dict[str, Any] | None = None
859860
if result is not None:
860861
try:
@@ -864,7 +865,7 @@ def _leave(
864865
"error",
865866
duration_ms=duration_ms,
866867
error=str(err),
867-
tokens=ctx.meter.tokens - tokens_before,
868+
**measured,
868869
)
869870
raise
870871

@@ -881,31 +882,15 @@ def _leave(
881882
"error",
882883
duration_ms=duration_ms,
883884
error=f"budget: {exc.reason}",
884-
tokens=ctx.meter.tokens - tokens_before,
885+
**measured,
885886
)
886887
raise
887888

888-
# The provider's own price for every model call made inside this node,
889-
# collected by the usage callback. Left None when no backend reported
890-
# one, so `observe.cost` can keep a measured figure apart from a
891-
# rate-card estimate instead of recording a guess as a fact.
892-
models = getattr(usage, "models", ())
893-
# This node's own spend, from the meter's per-node scope, not the movement
894-
# of the run's shared total while the node ran. Under fan-out those differ:
895-
# the workers' windows overlap, so every worker used to be credited with
896-
# each sibling's concurrent spend — three workers costing 8 tokens each
897-
# traced as 24/16/8, and `metrics` and `cost` both reported 48 for 24
898-
# tokens of real work, doubling the estimated bill purely because the work
899-
# ran in parallel. Falls back to the difference when no scope was open, so
900-
# a caller driving `_leave` outside `charging()` still gets a figure.
901-
spent = getattr(usage, "tokens", None)
902889
emit(
903890
"end",
904891
state_delta=delta,
905892
duration_ms=duration_ms,
906-
tokens=ctx.meter.tokens - tokens_before if spent is None else spent,
907-
cost_usd=getattr(usage, "cost_usd", None),
908-
model=models[0] if len(models) == 1 else None,
893+
**measured,
909894
)
910895
return result
911896

@@ -935,7 +920,7 @@ async def awrapped(state: Any, config: RunnableConfig) -> Any:
935920
"error",
936921
duration_ms=(time.perf_counter() - t0) * 1000,
937922
error=repr(exc),
938-
tokens=ctx.meter.tokens - tokens_before,
923+
**_usage_fields(ctx, tokens_before, usage),
939924
)
940925
raise
941926
return self._leave(
@@ -974,7 +959,7 @@ def wrapped(state: Any, config: RunnableConfig) -> Any:
974959
"error",
975960
duration_ms=(time.perf_counter() - t0) * 1000,
976961
error=repr(exc),
977-
tokens=ctx.meter.tokens - tokens_before,
962+
**_usage_fields(ctx, tokens_before, usage),
978963
)
979964
raise
980965
return self._leave(
@@ -991,6 +976,22 @@ def wrapped(state: Any, config: RunnableConfig) -> Any:
991976
return wrapped
992977

993978

979+
def _usage_fields(ctx: RunContext, tokens_before: int, usage: Any) -> dict[str, Any]:
980+
"""The same measured spend on every terminal event, successful or failed.
981+
982+
A node's own scope excludes concurrent siblings' spend. The shared meter
983+
difference is only a fallback for callers outside `charging()`. A missing
984+
provider price stays missing; rate-card estimates belong to trace readers.
985+
"""
986+
spent = getattr(usage, "tokens", None)
987+
models = getattr(usage, "models", ())
988+
return {
989+
"tokens": ctx.meter.tokens - tokens_before if spent is None else spent,
990+
"cost_usd": getattr(usage, "cost_usd", None),
991+
"model": models[0] if len(models) == 1 else None,
992+
}
993+
994+
994995
def topology_delta(arc: GraphARC) -> dict[str, Any]:
995996
"""The declared shape of a graph, as one trace-event `state_delta`.
996997

0 commit comments

Comments
 (0)