Skip to content

Commit 10839fe

Browse files
committed
chore(release): 0.22.2 — status writer + websocket extra + max_output_tokens reserve
## Summary Patch release, no protocol change. Three unrelated bugs closed in one shipping window: a status field that structurally could not report the failure it existed to report (`backend_reachable` initialised to `None` and never written), a packaging extra that the SDK's own `ImportError` told users to install but that `pyproject.toml` never declared, and a pre-flight that priced at the input rate and so silently under-reserved every real LLM call. None is a wire change — all three are SDK-side. The status and packaging fixes are behaviourally invisible to a healthy configuration; the reserve fix is visible to anyone whose `max_output_tokens` is currently being ignored. ### Fixed - **DEF-STATUS-WRITER-01** — `NullRunStatus.backend_reachable` was initialised to `None` in `__init__` and read in `status()` — and never assigned anywhere in the SDK. The comment above the initialisation claimed the pair was "set in `_authenticate` and updated on every successful / failed backend call thereafter"; no such writer had ever existed. Two silent consequences: the DEGRADED headline was gated on `is False` so it never fired (pinned to `None`), and the MISCONFIGURED arm was gated on `_last_backend_attempt_at is not None` so it was dead for the same reason. Fix: one writer `_note_backend_attempt(ok)` called from the four sites that actually learn whether the backend answered. "Ok" means "the backend answered", NOT "the call succeeded": a 402/403/429 and a 401 are the backend replying, and only a transport-level failure sets it False. Collapsing those would brand a healthy server that correctly refused a call as unreachable and drive DEGRADED off a credential problem. The pair is read under a new lock because gate calls arrive on the tool thread while `_authenticate` runs on the caller's thread. (`b076643`, +100/-8; test files `test_status_backend_reachable_has_a_writer.py`, `test_status_ws_connected_reads_real_attribute.py`) - **DEF-SDK-16 (packaging: declare the `websocket` extra)** — `WebSocketConnection.connect` raises `websockets library not available. Install with: pip install nullrun[websocket]`, but `nullrun` had no `websocket` extra. `[project.optional-dependencies]` declared exactly two — the opentelemetry pair and `dev` — and `websockets` appeared nowhere in `pyproject.toml`. Two declarations, not one: `websocket = ["websockets>=14.0,<17.0"]` and `dev` also gains `websockets` (so `tests/test_ws_push.py` resolves through a declared edge, not through `langchain-core -> langsmith -> websockets` which nobody here controls). A new source pin asserts the general property: every `nullrun[<extra>]` install hint appearing anywhere under `src/` must name an extra `pyproject.toml` actually declares. (`a8cb823`, +24/-1; `tests/test_websocket_extra_is_declared.py`) ### Added - **DEF-MOT-01 (reserve: forward caller-declared `max_output_tokens` to `/gate`)** — per ADR-016 §2.1 the backend prices an output reserve from TWO independent signals (the caller's declared cap and its own per-model catalog cap) and refuses to reserve from either one alone. The SDK's half of that contract is now narrow and specific: when the caller set a cap it is sent, when the caller did not the key is ABSENT (not present-and-null — a null is a value the backend has to defend against, not an honest "I don't know"), and a cap of zero or less is treated as "not declared" rather than silently under-reserving. New ContextVar `_call_max_output_tokens_var`, reader `get_call_max_output_tokens()`, and a `max_output_tokens=` parameter on `set_call_context(...)`. The via-edge box's `token_split()` falls back to the declared cap before the one-token estimate; every step down errs toward charging MORE, never less. (`f095c46`, +251/-9; test file `tests/test_max_output_tokens_reserve.py`) ### Verification | Check | Result | |---|---| | `ruff check src tests` | All checks passed | | `mypy src/nullrun` | Success: no issues found in 39 source files | | `pytest -q` | **1913 passed, 12 skipped** in 156.34s (vs baseline 1849 / 12 — **+64 new tests** from `test_status_backend_reachable_has_a_writer`, `test_status_ws_connected_reads_real_attribute`, `test_websocket_extra_is_declared`, `test_max_output_tokens_reserve`) | | Scratch diff | clean (`dist_local/`, `*.defect*` absent) | | `nullrun.__version__` | `0.22.2` | | Wire-format compatibility | unchanged from 0.22.1 (no protocol change; the `max_output_tokens` key is new-but-OPTIONAL, the backend ignores its absence as before) | ### Commits included ``` b076643 fix(status): backend_reachable was structurally pinned to None a8cb823 fix(packaging): DEF-SDK-16 — declare the `websocket` extra the ImportError names f095c46 feat(reserve): forward caller-declared max_output_tokens to /gate (DEF-MOT-01) e56241b fix errors ```
1 parent 87cd7e8 commit 10839fe

14 files changed

Lines changed: 910 additions & 21 deletions

‎CHANGELOG.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,131 @@
1+
## [0.22.2] - 2026-10-16
2+
3+
Patch release, no protocol change. Three unrelated bugs closed in one
4+
shipping window: a status field that structurally could not report
5+
the failure it existed to report (`backend_reachable` initialised to
6+
`None` and never written), a packaging extra that the SDK's own
7+
`ImportError` told users to install but that `pyproject.toml` never
8+
declared, and a pre-flight that priced at the input rate and so
9+
silently under-reserved every real LLM call. None is a wire
10+
change — all three are SDK-side. The status and packaging fixes are
11+
behaviourally invisible to a healthy configuration; the reserve fix
12+
is visible to anyone whose `max_output_tokens` is currently being
13+
ignored.
14+
15+
### Migration
16+
17+
Three things differ from 0.22.1. None is silent on a healthy
18+
configuration, but the third is a new opt-in signal that the wire
19+
contract was always supposed to carry:
20+
21+
1. **`NullRunStatus` reflects a backend that has actually been
22+
attempted.** A backend unreachable since `init()` used to read
23+
`backend_reachable=None` and so the DEGRADED headline stayed
24+
`STATE_OK` — a monitoring surface that structurally could not
25+
report the failure it exists to report. The writer is now
26+
called from every site that learns whether the backend answered
27+
(both `/gate` paths in `check_workflow_budget`, both
28+
`/auth/verify` outcomes); "answered" means transport-level, not
29+
HTTP-2xx — a 402/403/429 is the backend replying and stays
30+
reachable, only DNS / refused / TLS / timeout sets it False.
31+
2. **`pip install nullrun[websocket]` actually installs `websockets`.**
32+
The error message named an extra `pyproject.toml` never declared
33+
(only the opentelemetry pair and `dev` were listed). The failure
34+
mode was worse than a missing feature: a user told precisely how
35+
to install it typed it, and got a *second* error. `websockets`
36+
was in `uv.lock` only as a transitive of `langsmith`, so the
37+
WS wire test's coverage depended on a dependency edge nobody
38+
here controls. Now declared with floor `14.0` (the `additional_headers`
39+
spelling replaced `extra_headers` in the asyncio client in
40+
14.0; an older install would `TypeError` at connect time,
41+
later and in production traffic, instead of at the
42+
`ImportError` the user was told to fix).
43+
3. **`set_call_context(max_output_tokens=…)` reaches `/gate`.**
44+
The pre-flight has always sent `estimated_tokens=1`, which
45+
prices at the INPUT rate and covers no output at all. Output
46+
bills at the completion rate (5× input on the live catalog),
47+
so a real 20k-token Opus response costs 50¢ against a 1¢
48+
reserve: `/track` then rejects the row with
49+
`CONSUME_OVERBUDGET` and the period counter under-counts the
50+
spend. Now the caller's declared cap is sent when set, the key
51+
is ABSENT (not present-and-null) when not, and a cap of zero
52+
or less is treated as "not declared" rather than silently
53+
under-reserving. The via-edge box's `token_split()` falls back
54+
to the declared cap before the one-token estimate, never the
55+
other way.
56+
57+
### Fixed
58+
59+
- **DEF-STATUS-WRITER-01** — `NullRunStatus.backend_reachable` was
60+
initialised to `None` in `__init__` and read in `status()` — and
61+
never assigned anywhere in the SDK. The comment above the
62+
initialisation claimed the pair was "set in `_authenticate` and
63+
updated on every successful / failed backend call thereafter";
64+
no such writer had ever existed. Two silent consequences: the
65+
DEGRADED headline was gated on `is False` so it never fired
66+
(pinned to `None`), and the MISCONFIGURED arm was gated on
67+
`_last_backend_attempt_at is not None` so it was dead for the
68+
same reason. Fix: one writer `_note_backend_attempt(ok)` called
69+
from the four sites that actually learn whether the backend
70+
answered. "Ok" means "the backend answered", NOT "the call
71+
succeeded": a 402/403/429 and a 401 are the backend replying,
72+
and only a transport-level failure sets it False. Collapsing
73+
those would brand a healthy server that correctly refused a
74+
call as unreachable and drive DEGRADED off a credential
75+
problem. The pair is read under a new lock because gate calls
76+
arrive on the tool thread while `_authenticate` runs on the
77+
caller's thread. (`b076643`, +100/-8; test files
78+
`test_status_backend_reachable_has_a_writer.py`,
79+
`test_status_ws_connected_reads_real_attribute.py`)
80+
81+
- **DEF-SDK-16 (packaging: declare the `websocket` extra)** —
82+
`WebSocketConnection.connect` raises
83+
`websockets library not available. Install with: pip install
84+
nullrun[websocket]`, but `nullrun` had no `websocket` extra.
85+
`[project.optional-dependencies]` declared exactly two — the
86+
opentelemetry pair and `dev` — and `websockets` appeared
87+
nowhere in `pyproject.toml`. Two declarations, not one:
88+
`websocket = ["websockets>=14.0,<17.0"]` and `dev` also gains
89+
`websockets` (so `tests/test_ws_push.py` resolves through a
90+
declared edge, not through `langchain-core -> langsmith ->
91+
websockets` which nobody here controls). A new source pin
92+
asserts the general property: every `nullrun[<extra>]` install
93+
hint appearing anywhere under `src/` must name an extra
94+
`pyproject.toml` actually declares. (`a8cb823`, +24/-1;
95+
`tests/test_websocket_extra_is_declared.py`)
96+
97+
### Added
98+
99+
- **DEF-MOT-01 (reserve: forward caller-declared
100+
`max_output_tokens` to `/gate`)** — per ADR-016 §2.1 the
101+
backend prices an output reserve from TWO independent signals
102+
(the caller's declared cap and its own per-model catalog cap)
103+
and refuses to reserve from either one alone. The SDK's half
104+
of that contract is now narrow and specific: when the caller
105+
set a cap it is sent, when the caller did not the key is
106+
ABSENT (not present-and-null — a null is a value the backend
107+
has to defend against, not an honest "I don't know"), and a
108+
cap of zero or less is treated as "not declared" rather than
109+
silently under-reserving. New ContextVar
110+
`_call_max_output_tokens_var`, reader
111+
`get_call_max_output_tokens()`, and a `max_output_tokens=`
112+
parameter on `set_call_context(...)`. The via-edge box's
113+
`token_split()` falls back to the declared cap before the
114+
one-token estimate; every step down errs toward charging
115+
MORE, never less. (`f095c46`, +251/-9; test file
116+
`tests/test_max_output_tokens_reserve.py`)
117+
118+
### Verification
119+
120+
| Check | Result |
121+
|---|---|
122+
| `ruff check src tests` | All checks passed |
123+
| `mypy src/nullrun` | Success: no issues found in 39 source files |
124+
| `pytest -q` | **1913 passed, 12 skipped** in 156.34s (vs baseline 1849 / 12 — **+64 new tests** from `test_status_backend_reachable_has_a_writer`, `test_status_ws_connected_reads_real_attribute`, `test_websocket_extra_is_declared`, `test_max_output_tokens_reserve`) |
125+
| Scratch diff | clean (`dist_local/`, `*.defect*` absent) |
126+
| `nullrun.__version__` | `0.22.2` |
127+
| Wire-format compatibility | unchanged from 0.22.1 (no protocol change; the `max_output_tokens` key is new-but-OPTIONAL, the backend ignores its absence as before) |
128+
1129
## [0.22.1] - 2026-10-15
2130

3131
Patch release, no protocol change. The SDK's public exception surface

‎pyproject.toml‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
66
name = "nullrun"
77
# Full release history lives in CHANGELOG.md; only the current version
88
# is pinned here.
9-
version = "0.22.1"
9+
version = "0.22.2"
1010
# Kept under the 200-char preview threshold so the full line is visible
1111
# without an "expand" click. The headline is the canonical §1 statement
1212
# from positioning.md — "runtime decision layer for tool-using AI agents"
@@ -77,7 +77,31 @@ opentelemetry = [
7777
"opentelemetry-api>=1.26.0,<2.0",
7878
"opentelemetry-sdk>=1.26.0,<2.0",
7979
]
80+
# DEF-SDK-16 (RUN_ID 20261015T1809). ``transport_websocket.connect``
81+
# raises ``ImportError("… Install with: pip install nullrun[websocket]")``
82+
# when the ``websockets`` import fails — but no ``websocket`` extra was
83+
# ever declared, so the command in the error message did not exist and
84+
# pip rejected it. The extra below makes the message true.
85+
#
86+
# The floor is 14.0, not "any version": the transport calls
87+
# ``websockets.connect(url, additional_headers=...)``. The
88+
# ``additional_headers`` spelling belongs to the asyncio client that
89+
# became the default ``websockets.connect`` in 14.0; earlier releases
90+
# spelled the same argument ``extra_headers``, so a lower floor
91+
# resolves and then fails at connect time with a TypeError instead of
92+
# at import time.
93+
websocket = [
94+
"websockets>=14.0,<17.0",
95+
]
8096
dev = [
97+
# ``tests/test_ws_push.py`` imports ``websockets`` at module scope
98+
# and spins up a real local WS server. It used to pass only because
99+
# ``langchain-core`` -> ``langsmith`` -> ``websockets`` happened to
100+
# pull it in transitively; a langchain/langsmith release that drops
101+
# that edge would turn the whole module into a collection error.
102+
# Declared directly so the test suite's WS coverage depends on the
103+
# websocket extra existing, not on someone else's dependency graph.
104+
"websockets>=14.0,<17.0",
81105
"pytest>=8.0",
82106
"pytest-asyncio>=0.23",
83107
# Sprint 0 (coverage): ``pytest-rerunfailures`` is used on a
@@ -135,6 +159,15 @@ dev = [
135159
# bound as the `[fastapi]` extra so CI and end users agree on the
136160
# minimum. `httpx` (already a core dep) covers `TestClient`.
137161
"fastapi>=0.100,<1.0",
162+
# `tests/test_websocket_extra_is_declared.py` parses
163+
# `pyproject.toml` at collection time to pin the general
164+
# "every `nullrun[<extra>]` install hint under `src/` names
165+
# an extra `pyproject.toml` actually declares" property.
166+
# `tomllib` is stdlib from 3.11; on 3.10 the test must fall
167+
# back to `tomli`, and the fallback is only safe if the dep
168+
# is declared. Conditional on `<3.11` so the upper-bound
169+
# environments do not pull a redundant package.
170+
"tomli>=2.0; python_version < '3.11'",
138171
]
139172

140173
[project.scripts]

‎src/nullrun/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,5 @@
55
string and the SDK_MIN_VERSION constant.
66
"""
77

8-
__version__ = "0.22.1"
8+
__version__ = "0.22.2"
99
__platform_version__ = "1.0.0"

‎src/nullrun/context.py‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,23 @@
4242
# enforced on /track only).
4343
_call_model_var: ContextVar[str | None] = ContextVar("call_model", default=None)
4444
_call_tools_var: ContextVar[tuple[str, ...]] = ContextVar("call_tools", default=())
45+
# The `max_output_tokens` the caller intends to request from the
46+
# provider — Anthropic/OpenAI's `max_tokens` / `max_output_tokens`.
47+
#
48+
# The pre-flight has always sent `estimated_tokens=1`, which prices at
49+
# the INPUT rate and covers no output at all. Output is billed at the
50+
# completion rate (5× the input rate on the live catalog), so a real
51+
# 20k-token Opus response costs 50¢ against a 1¢ reserve: /track then
52+
# rejects the row with CONSUME_OVERBUDGET and the period counter
53+
# under-counts the spend.
54+
#
55+
# `None` means "the caller didn't say", and the backend keeps the old
56+
# arithmetic — a missing signal must not invent a reserve. The backend
57+
# bounds whatever arrives here by its own per-model catalog cap, so a
58+
# client can only ever LOWER the reserve, never inflate it.
59+
_call_max_output_tokens_var: ContextVar[int | None] = ContextVar(
60+
"call_max_output_tokens", default=None
61+
)
4562
# Per-call MCP tool class + annotations. Set via the
4663
# ``set_mcp_tool_context`` helper when the SDK recognises an MCP
4764
# server. The gate honors `tool_class` over its own
@@ -142,6 +159,18 @@ def get_call_tools() -> tuple[str, ...]:
142159
return _call_tools_var.get()
143160

144161

162+
def get_call_max_output_tokens() -> int | None:
163+
"""The declared output cap for the next pre-flight, or ``None``.
164+
165+
``None`` is the honest default: the SDK cannot invent an output
166+
size for a call it does not make. The backend treats the pair
167+
(declared, catalog cap) as the only inputs it will price an output
168+
reserve from — see ADR-016 §2.1 — and refuses to reserve from
169+
either signal alone.
170+
"""
171+
return _call_max_output_tokens_var.get()
172+
173+
145174
def get_call_mcp_class() -> str | None:
146175
"""Canonical tool class for the next ``/check`` call.
147176
@@ -802,9 +831,10 @@ def create_child_span_with_id(parent: SpanContext, span_id: str) -> SpanContext:
802831
def set_call_context(
803832
model: str | None = None,
804833
tools: list[str] | tuple[str, ...] | None = None,
834+
max_output_tokens: int | None = None,
805835
) -> None:
806-
"""Set per-call context (model name, tool list) for the next /gate
807-
pre-flight check.
836+
"""Set per-call context (model name, tool list, output cap) for the
837+
next /gate pre-flight check.
808838
809839
Replaces the previous fake ``model="budget-precheck"`` and
810840
``estimated_tokens=1`` always-default / always-empty pre-flight.
@@ -821,11 +851,25 @@ def set_call_context(
821851
``blocked_tools`` aggregate and returns block on any
822852
match. Pass ``None`` to leave the current value
823853
unchanged, ``[]`` to clear.
854+
max_output_tokens: The output cap you are about to send the
855+
provider — Anthropic's ``max_tokens``, OpenAI's
856+
``max_output_tokens``. Sent only when you pass a positive
857+
integer; omitting it keeps the pre-flight exactly as it
858+
was, which under-reserves output rather than guessing at
859+
a number. The backend clamps this to its own per-model
860+
catalog cap, so a value larger than the model can emit is
861+
not a way to raise your own reserve.
824862
"""
825863
if model is not None:
826864
_call_model_var.set(model)
827865
if tools is not None:
828866
_call_tools_var.set(tuple(tools))
867+
if max_output_tokens is not None:
868+
# A cap of zero or less is not a cap; forwarding it would let a
869+
# caller declare "no output" and reserve nothing at all. Treat
870+
# it as "not declared" instead of silently under-reserving.
871+
if max_output_tokens > 0:
872+
_call_max_output_tokens_var.set(max_output_tokens)
829873

830874

831875
def set_mcp_tool_context(

‎src/nullrun/edge.py‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -386,14 +386,23 @@ def edge_transport_from_env(env: dict[str, str] | None = None) -> EdgeTransport
386386
def token_split(check_request: dict[str, Any]) -> tuple[int, int]:
387387
"""The token counts this call is reported with.
388388
389-
Explicit `edge_input_tokens` / `edge_output_tokens` win. Absent
390-
them, `estimated_tokens` is reported as OUTPUT tokens — output is
391-
what the price is dominated by for most models, and reporting the
392-
whole estimate as output errs toward charging more rather than
393-
less. Under-reporting is the one direction this cannot take, since
394-
the box cannot check the number and will simply believe it.
389+
Precedence for output: explicit `edge_output_tokens`, then the
390+
declared `max_output_tokens`, then `estimated_tokens`. Each step
391+
down is a weaker answer to "how big will the output be" — a known
392+
size, then the upper bound the caller is about to ask the provider
393+
for, then a blob the runtime only ever sets to 1.
394+
395+
Every step errs toward charging MORE, never less. That is
396+
deliberate and load-bearing: the box cannot check the number, it
397+
will believe it, and output is what the price is dominated by for
398+
most models (5× the input rate on the live catalog). Reporting the
399+
declared cap when it is the only real signal costs a little
400+
over-reserve on calls that finish short; reporting `1` instead
401+
under-reserves every real call.
395402
"""
396403
out = check_request.get("edge_output_tokens")
404+
if out is None:
405+
out = check_request.get("max_output_tokens")
397406
if out is None:
398407
out = check_request.get("estimated_tokens") or 0
399408
inp = check_request.get("edge_input_tokens") or 0

0 commit comments

Comments
 (0)