Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,14 @@ specific replay and wire shaping remain later request projections.

## Model Context Reconstruction

An **Ordered Tool History Projection** preserves the contributions carried by
provider history together with their order, actor, item identity, and text or
media payload. Complete Responses output and partial streaming sidecars have
different coverage: a sidecar cannot replace ordinary message content or prove
its position without recorded replay positions. Folding and replay share source
selection and generated-image interpretation. Native replay policy remains in
request preparation; the projection never mutates checkpoint messages.

A **Session Log** is the append-only source of persisted conversation events.
It retains message, summary, and lifecycle history without becoming a second
mutable message state.
Expand Down
26 changes: 26 additions & 0 deletions docs/eager-tool-readiness-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Eager tool readiness parsing

On Node v24.16.0 / macOS, compared main `945119d4584ca2ee7bfac9f50b68eb5995b3a21a` with the seal-first readiness change. The same opt-in benchmark test was copied into the baseline checkout. Four separate Jest processes ran sequentially in candidate / baseline / baseline / candidate order after the other local checks completed.

Each process reports the median of seven samples after two warmups. Each sample streams deterministic JSON arguments and a subsequent tool index through the real `ChatModelStreamHandler`, exercising argument reconciliation, readiness, and eager dispatch. Graph bookkeeping and external dispatch use the existing test fixtures. No model, network, production provider parsing, or full LibreChat request is measured. Each sample asserts that the first call prestarted.

Ranges below are the two process medians, in milliseconds. Input sizes refer to SQL string content before its JSON envelope.

| Input | Chunk size | Baseline elapsed | Candidate elapsed | Baseline CPU | Candidate CPU |
| --- | ---: | ---: | ---: | ---: | ---: |
| 128 B | 64 B | 0.213–0.293 | 0.172–0.192 | 0.381–0.445 | 0.338–0.402 |
| 128 B | 256 B | 0.081–0.124 | 0.076–0.085 | 0.083–0.155 | 0.077–0.094 |
| 8 KiB | 64 B | 3.488–3.552 | 2.151–2.226 | 4.483–4.625 | 3.195–3.488 |
| 8 KiB | 256 B | 0.927–0.930 | 0.559–0.668 | 0.932–0.935 | 0.559–0.686 |
| 48 KiB | 64 B | 46.579–47.605 | 23.229–25.351 | 53.479–54.837 | 33.493–37.595 |
| 48 KiB | 256 B | 11.864–12.156 | 6.106–6.679 | 11.934–12.243 | 6.221–6.753 |

The larger fixtures show about 28–51% lower handler elapsed time and 22–49% lower process CPU across these ranges. Tiny cases are dominated by timing variability. These results exceed the proposed 10% local-handler improvement gate, but do not establish an end-to-end user latency percentage, concurrent-load performance, or a retained-memory reduction. Process CPU includes runtime/GC work.

Reproduce on each checkout with the identical benchmark fixture:

```sh
BENCH_EAGER_READINESS=1 npx jest src/__tests__/stream.eagerArgsDivergence.test.ts --runInBand -t 'benchmarks tool stream'
```

The behavioral regression test separately asserts that readiness does not parse unsealed arguments and parses a sealed argument string once before dispatch. Provider fragment reconciliation still parses its own buffers; that compatibility-sensitive behavior is unchanged.
100 changes: 100 additions & 0 deletions docs/tool-approval-replay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Tool approval replay contract

An approval belongs to an execution, not to a JavaScript object or the current
hook registry. Reconstructing a `Run` must not turn a rejected action into an
allowed action or repeat a sibling whose completed result was checkpointed.

## Ownership and lifecycle

1. The ToolNode resolves execution arguments and evaluates current policy.
2. An `ask` records the exact tool-call identity, arguments and decision
allowlist. The originating execution owns that review. Parent nodes carry
child evidence through the trusted subagent adapter but do not consume it.
3. Before publishing a pause, the direct batch waits for its started siblings
to settle. Completed results, hook context, reference content and completion
delivery state accompany the interrupt in the checkpoint. Only the active
batch is included; another thread's in-flight results are not copied.
4. `Run.resume` obtains private evidence from the checkpoint, not host-supplied
configurable fields. Public interrupts do not expose the private records.
5. A fresh ToolNode restores completed siblings and consumes the reviewed
decision for pending calls. Current denial still wins. Changed proposals or
disallowed decisions fail closed. Removing a hook cannot remove a pending
review; a valid approval uses the arguments actually reviewed.
6. A child checkpoint fork receives a new execution scope. Only the trusted
subagent adapter rebinds evidence from the manifest's proven source scope
into that destination, including durable pending writes that have not yet
been consumed. Each child restores its own pending evidence. Unrelated child
and parent scopes are unchanged. A foreign owner is accepted only as transit
through a trusted child tool, a matching parent batch record, and a validated
descendant manifest; changing tool-call IDs cannot bypass ownership checks.
7. Completed batches release their process-local acceleration cache. That cache
is not the durable source of truth: selecting a checkpoint replaces any
newer local result for that batch, including when the checkpoint is empty.
8. The checkpoint also retains per-tool usage counters and active-call turns.
Restored outputs and pending calls keep their original execution and
completion indices, with or without a hook registry. Historical per-call
turn entries are not copied into the active batch's checkpoint.
9. Batch identity binds the assistant message and tool proposals, not transcript
length: additional resume messages cannot repeat completed work. The
pre-batch output-reference snapshot is a frozen input view, distinct from
the graph's shared live registry. Replay retains its original batch turn
without lowering the shared counter or discarding concurrent outputs.
Cached messages are rebound to the active reference scope; completed siblings
cannot change what pending siblings originally resolved. Current reference
size limits also apply to the frozen view.
10. Direct calls without IDs use their original proposal position in the private
completion record. Filtering completed named calls cannot shift that identity;
successful ID-less outputs survive pauses without repeating their side effects.

The private checkpoint record is versioned. Its codec preserves LangChain
messages; Command outputs are reconstructed before graph execution. Malformed
records fail closed. The replay module has no HTTP, database or host UI dependency.

## Recovery guarantees and host obligations

This protocol guarantees replay of **checkpointed completed work** when resuming
an approval pause. Rebuilding a Run/ToolNode with the same checkpointer is a
supported operation; retaining the old hooks or instances is not required.

Standalone ToolNode callers must supply a stable `configurable.thread_id` (or
the SDK-managed approval execution scope) to identify a replay. Anonymous
invocations are independent operations and never share completed-result state.

This is not a distributed exactly-once transaction for arbitrary external tools.
A process can die after an external side effect but before a checkpoint commits.
That outcome is ambiguous: absence of a saved result does not prove the tool did
not run. Hosts must serialize ownership of a running thread, and tools requiring
retry safety must use a durable operation/idempotency key or reconcile with their
external system. A deliberate checkpoint fork is a new branch, not an ordinary
duplicate submission. The SDK must not advertise arbitrary shell commands or
third-party tools as exactly-once across that crash window.

## Compatibility and rollout

- New consumers accept legacy approval payloads without the private record.
Reviewed rejection and allowlist restrictions still apply. Unreviewed direct
siblings fail closed when their prior policy cannot be reconstructed.
- Existing public approval payload and resume-decision shapes are unchanged.
- Old SDK consumers cannot restore the new completion record. Do not route a
paused run between SDK versions indiscriminately: drain or version-pin paused
runs during rollout and before rollback. A durable checkpointer alone is not
an execution-owner lease.

## Focused acceptance coverage

- Fresh Run and ToolNode: direct and event approval with rewritten arguments,
with the former hook registry absent.
- Fresh runtime with a different Run ID: mixed direct/event pause, preserving
one side effect, its returned result, and one completion event.
- Repeated terminal resume: no additional execution.
- Nested checkpoint forks: rejection remains effective without the former
hooks, and parent siblings remain independent of child approval evidence.
- Codec and validation: messages, Commands, nested ownership, absent legacy
state and malformed records. Restored results require a proposal binding;
custom payloads that resemble private wrappers preserve their public shape.
- Older-checkpoint selection overrides a newer local cache; empty execution
scopes remain isolated anonymous invocations.

External-side-effect crash reconciliation and concurrent independent host
ownership require the host/tool contract above; these tests do not prove those
external guarantees.
50 changes: 50 additions & 0 deletions docs/tool-history-projection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Ordered Tool History Projection

Model Context Reconstruction owns source evidence separately from the serving
provider's replay policy. A preparation-scoped, copy-on-write cache shares the
Responses source selection and ordered contributions between tool-less folding
and sealed-turn replay. Complete output takes precedence over streaming sidecars;
streaming positions order sidecars when text mapping is unambiguous.

The projection does not select the serving provider or model. Primary and fallback
preparations each select native replay versus portable folding using the actual
serving model and invocation options. The cache is not persisted or shared between
runs. Plain messages without Responses evidence allocate no projection objects.

Fallback preparation starts from the pruned history before any primary wire
shaping. Each fallback applies its own input limits, thinking normalization,
folding, and measured request projection. A model's call-specific API-mode answer
is authoritative, including `false`; a Responses default cannot override an
explicit Chat selection. Graph regressions exercise both directions and a failed
intermediate fallback, including source provenance and native media.

Primary and fallback paths reuse the serving-policy, artifact projection, tail
cache, and synthetic-context compaction helpers. Fallbacks also restore legacy
formatting and sanitize orphan tool pairs before adding cache markers. Artifact
expansion is retried without the artifact when necessary; oversized synthetic
context is compacted against the final, serving-specific measured payload before
rejecting a fallback. No content is appended after the final budget check.

Regression coverage checks source identity through fallback folding before origin
tracking, mixed model/tool provenance, completed media, mixed native and parsed
calls, bounded nested traversal, per-value argument limits, and source immutability.
This is the first shared consumer slice; compaction and budgeting retain their
existing interfaces rather than changing checkpoint formats in this PR.

## Benchmark

Run `npx tsx src/scripts/bench-tool-history-projection.ts` from the repository root.
The benchmark compares a shared preparation with separate preparation caches for
folding and native replay. Results are medians of five alternating samples, each
containing 100 preparations. It is a cache comparison, not a baseline-release
speedup claim or a general allocation profiler.

Local sample (milliseconds per 100 preparations):

| History | Separate caches | Shared cache | Projection objects per preparation |
| --- | ---: | ---: | ---: |
| 500 plain-text messages | 2.56 | 2.40 | 0 |
| 100 tool-heavy Responses messages | 1150.27 | 1105.02 | 100 |

The plain-text fold also asserts input-array identity. Timings are diagnostic,
not CI thresholds; machine load and runtime warmup affect these small differences.
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@librechat/agents",
"version": "3.8.0",
"version": "3.8.2",
"reova": {
"enabled": true,
"endpoint": "https://telemetry.reo.dev/data"
Expand Down
92 changes: 91 additions & 1 deletion src/__tests__/stream.eagerArgsDivergence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ import {
StreamLimitExceededError,
} from '@/llm/streamLimits';
import { GraphEvents, Providers, StepTypes } from '@/common';
import * as eagerArgs from '@/tools/eagerEventExecution';
import { ChatModelStreamHandler } from '@/stream';
import { ToolNode } from '@/tools/ToolNode';
import { HandlerRegistry } from '@/events';
Expand Down Expand Up @@ -259,6 +260,93 @@ function canonicalToolCall(
}

describe('eager args divergence (LibreChat#14371)', () => {
it('defers readiness parsing until a seal and reuses the parsed request', async () => {
const graph = createGraph();
const { toolExecuteCalls } = installToolExecuteResponder();
const handler = new ChatModelStreamHandler();
const metadata = { langgraph_node: 'agent' };
const parse = jest.spyOn(eagerArgs, 'coerceRecordArgs');
await streamChunks({
handler,
graph,
metadata,
toolCallChunks: toToolCallChunks('call_1', 'db_query', [
'{"sql":',
'"select 1"}',
]),
});
expect(toolExecuteCalls).toHaveLength(0);
expect(parse).not.toHaveBeenCalled();
await streamNextToolIndex({ handler, graph, metadata, callId: 'call_2' });
expect(toolExecuteCalls).toHaveLength(1);
expect(toolExecuteCalls[0].toolCalls[0].args).toEqual({ sql: 'select 1' });
expect(
parse.mock.calls.filter(([value]) => typeof value === 'string')
).toEqual([['{"sql":"select 1"}']]);
});

const benchmark = process.env.BENCH_EAGER_READINESS === '1' ? it : it.skip;
benchmark(
'benchmarks tool stream handling with deterministic arguments',
async () => {
installToolExecuteResponder();
for (const size of [128, 8192, 49152]) {
const payload = JSON.stringify({
sql: Array.from(
{ length: Math.ceil(size / 16) },
(_, i) => `column_${i.toString(16).padStart(8, '0')} `
)
.join('')
.slice(0, size),
});
for (const chunkSize of [64, 256]) {
const fragments: string[] = [];
for (let i = 0; i < payload.length; i += chunkSize) {
fragments.push(payload.slice(i, i + chunkSize));
}
const samples: number[] = [];
const cpuSamples: number[] = [];
for (let round = 0; round < 9; round++) {
const graph = createGraph();
const handler = new ChatModelStreamHandler();
const metadata = { langgraph_node: 'agent' };
const chunks = toToolCallChunks('call_1', 'db_query', fragments);
const cpuStart = process.cpuUsage();
const start = performance.now();
await streamChunks({
handler,
graph,
metadata,
toolCallChunks: chunks,
});
await streamNextToolIndex({
handler,
graph,
metadata,
callId: 'call_2',
});
const elapsed = performance.now() - start;
const cpu = process.cpuUsage(cpuStart);
if (round >= 2) {
samples.push(elapsed);
cpuSamples.push((cpu.user + cpu.system) / 1000);
}
expect(graph.eagerEventToolExecutions.has('call_1')).toBe(true);
jest.clearAllMocks();
}
process.stdout.write(
JSON.stringify({
size,
chunkSize,
elapsedMs: samples.sort((a, b) => a - b)[3],
cpuMs: cpuSamples.sort((a, b) => a - b)[3],
}) + '\n'
);
}
}
}
);

afterEach(() => {
jest.restoreAllMocks();
});
Expand Down Expand Up @@ -511,7 +599,9 @@ describe('eager args divergence (LibreChat#14371)', () => {
{
chunk: {
content: '',
tool_calls: [{ id: 'call_1', name: 'db_query', args: { sql: 'SELECT 1;' } }],
tool_calls: [
{ id: 'call_1', name: 'db_query', args: { sql: 'SELECT 1;' } },
],
response_metadata: { finish_reason: 'tool_calls' },
} as unknown as t.StreamChunk,
},
Expand Down
Loading
Loading