Drive-to-green #494: remove processStream / run / runTurn / replay waivers - #495
Merged
Merged
Conversation
Add a canonical provider stream classifier (process_stream_events) in the OpenAI processor and a provider-agnostic reconciler (core/streaming.py) that computes partialText/requiresReconciliation/completionCommitted from a classified StreamChunk sequence. Register the real Processor.processStream vector adapter and delete its waiver. Conformance: 147 passed, 33 skipped (2 processStream vectors now green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…tor waivers Implements TurnConformance.run (agent loop, 28 vectors) and TurnConformance.runTurn (snapshot/portability turn engine, 5 vectors) as provider-agnostic engines in prompty/core, driven by abstract invoke_model/dispatch_tool/permission callbacks. Registers real adapters and deletes the last VECTOR_WAIVERS entries so Python conformance has zero waivers across processStream/run/runTurn/replay. Adds direct unit tests for both engines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Drives the 5 TurnConformance.runTurn vectors through the canonical prompty::engine::TurnEngine via scripted ports, removing the runTurn waiver. No turn logic reimplemented in the adapter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Ports the verified Python core/agent_loop.py to owned Rust runtime code (src/engine/agent_loop.rs): a provider-agnostic run engine driven by invoke_model/dispatch_tool callbacks. Wires all 28 TurnConformance.run vectors through it via scripted callbacks in the model harness adapter, removing the run waiver. No loop logic in the adapter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add provider-agnostic prompty::streaming::reconcile_stream (mirrors Python core/streaming.py) and wire the Processor.processStream vectors through the real OpenAI classifier (prompty_openai::processor::process_stream, added as a dev-dependency cycle) plus the new reconciler. Removes the processStream waiver; both vectors now run through the real adapter (721 passed). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Port the Python agent-loop and turn engines into PromptyModel so TurnConformance.run (28 vectors) and TurnConformance.runTurn (5 vectors) run directly in the model-only harness with no waiver. Register real adapters with projection normalizers and remove the run/runTurn waivers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add provider-agnostic reconcileStream in PromptyModel and extend the OpenAI stream classifier to emit determinate (refusal) and indeterminate (sse transport) failure chunks. Drive both processStream vectors through the real OpenAIProcessor via the SDK-level ProcessStreamVectorTests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…pters Port the verified Python reference engines to @prompty/core and wire real TypeScript @vector adapters for all three previously-waived contracts: - agent-loop-engine.ts -> TurnConformance.run (28 vectors) - turn-engine.ts -> TurnConformance.runTurn (5 vectors) - stream-reconcile.ts -> Processor.processStream reducer (2 vectors) processStream drives the REAL @prompty/openai stream classifier and reduces via the provider-agnostic reconcileStream. Removes all three TypeScript vector waivers; the model conformance harness now satisfies them directly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…tream classifier Wire real C# conformance adapters for the three drive-to-green contracts: - run (28 vectors) via AgentLoopEngine - runTurn (5 vectors) via SnapshotTurnEngine - replay (5 vectors) via ReferenceTurnRunner journal normalization Remove their VECTOR_WAIVERS entries (now driven in-harness). processStream (2 vectors): add real OpenAIProcessor.ClassifyStreamEvents + drive green at provider layer in SpecVectorStreamTests (Core.Tests cannot reference the OpenAI provider), mirroring the process/toRequest precedent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Comment on lines
+346
to
+350
| foreach (var m in messages) | ||
| { | ||
| if (m["content"] is JsonValue value && value.TryGetValue<string>(out var content)) | ||
| total += content.Length; | ||
| } |
Comment on lines
+371
to
+379
| foreach (var m in droppedUsers) | ||
| { | ||
| if (m["content"] is JsonValue value && value.TryGetValue<string>(out var content)) | ||
| { | ||
| var trimmed = content.Trim(); | ||
| if (trimmed.Length > 0) | ||
| topics.Add(trimmed); | ||
| } | ||
| } |
Comment on lines
+724
to
+726
| foreach (var annotation in new[] { "notes", "summary_contains", "rust_expected_error" }) | ||
| if (expected.ContainsKey(annotation)) | ||
| observed[annotation] = expected[annotation]?.DeepClone(); |
Comment on lines
+829
to
+831
| foreach (var d in denyTools) | ||
| if ((d as JsonValue)?.GetValue<string>() is { } name) | ||
| deny.Add(name); |
Comment on lines
+862
to
+864
| foreach (var m in conversation) | ||
| if (predicate(m)) | ||
| return m; |
Comment on lines
+941
to
+943
| foreach (var d in dt) | ||
| if ((d as JsonValue)?.GetValue<string>() is { } name) | ||
| denyTools.Add(name); |
Comment on lines
+1107
to
+1169
| foreach (var line in File.ReadAllLines(journalPath)) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(line)) | ||
| continue; | ||
| var record = JsonNode.Parse(line) as JsonObject ?? new JsonObject(); | ||
| var kind = (record["kind"] as JsonValue)?.GetValue<string>(); | ||
| if (kind == "summary") | ||
| { | ||
| var summary = record["summary"] as JsonObject ?? new JsonObject(); | ||
| normalized.Add( | ||
| $"summary:{Str(summary["sessionId"])}:{Str(summary["status"])}:" | ||
| + $"turns={Str(summary["turns"])}:checkpoints={Str(summary["checkpoints"])}"); | ||
| continue; | ||
| } | ||
|
|
||
| var ev = record["event"] as JsonObject ?? new JsonObject(); | ||
| var type = (ev["type"] as JsonValue)?.GetValue<string>() ?? string.Empty; | ||
| if (kind == "session") | ||
| { | ||
| if (type == "session_end") | ||
| { | ||
| var payload = ev["payload"] as JsonObject ?? new JsonObject(); | ||
| normalized.Add($"session:{type}:{Str(ev["sessionId"])}:{Str(ev["turnId"])}:{Str(payload["status"])}"); | ||
| } | ||
| else | ||
| { | ||
| normalized.Add($"session:{type}:{Str(ev["sessionId"])}:{Str(ev["turnId"])}"); | ||
| } | ||
| continue; | ||
| } | ||
|
|
||
| var pl = ev["payload"] as JsonObject ?? new JsonObject(); | ||
| var iteration = Str(ev["iteration"]); | ||
| switch (type) | ||
| { | ||
| case "permission_requested": | ||
| normalized.Add($"turn:{type}:{iteration}:{Str(pl["requestId"])}"); | ||
| break; | ||
| case "permission_completed": | ||
| normalized.Add($"turn:{type}:{iteration}:{Str(pl["approved"])}"); | ||
| break; | ||
| case "tool_execution_start": | ||
| normalized.Add($"turn:{type}:{iteration}:{Str(pl["toolName"])}"); | ||
| break; | ||
| case "tool_execution_complete": | ||
| case "tool_result": | ||
| var value = $"turn:{type}:{iteration}:{Str(pl["toolName"])}:{Str(pl["success"])}"; | ||
| var errorKind = Str(pl["errorKind"]); | ||
| if (!string.IsNullOrEmpty(errorKind)) | ||
| value = $"{value}:{errorKind}"; | ||
| normalized.Add(value); | ||
| break; | ||
| case "error": | ||
| normalized.Add($"turn:{type}:{iteration}:{Str(pl["errorKind"])}"); | ||
| break; | ||
| case "turn_end": | ||
| normalized.Add($"turn:{type}:{iteration}:{Str(pl["status"])}"); | ||
| break; | ||
| default: | ||
| normalized.Add($"turn:{type}:{iteration}"); | ||
| break; | ||
| } | ||
| } |
Comment on lines
+41
to
+49
| foreach (var env in doc.RootElement.GetProperty("vectors").EnumerateArray()) | ||
| { | ||
| if (env.GetProperty("operation").GetString() != "processStream") continue; | ||
| var vec = env.GetProperty("vector"); | ||
| result.Add(( | ||
| vec.GetProperty("name").GetString()!, | ||
| vec.GetProperty("input").Clone(), | ||
| vec.GetProperty("expected").Clone())); | ||
| } |
Comment on lines
+648
to
+655
| foreach (var item in steeringMsgs) | ||
| { | ||
| var it = item as JsonObject ?? new JsonObject(); | ||
| steering.Add(new AgentSteeringMessage( | ||
| (it["inject_before_iteration"] as JsonValue)?.GetValue<int>() ?? 0, | ||
| (it["role"] as JsonValue)?.GetValue<string>() ?? "user", | ||
| (it["text"] as JsonValue)?.GetValue<string>() ?? string.Empty)); | ||
| } |
Add provider-agnostic agent loop, snapshot turn, and stream reconcile engines to the Go model package and drive the four previously-waived vector contracts (Processor.processStream x2, TurnConformance.run x28, TurnConformance.runTurn x5, TurnConformance.replay x5). Only the 5 absent-pipeline waivers remain (load/render/parse/wire/process). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Register real, provider-agnostic Java adapters for the four remaining waived contracts and remove their waiver entries: - Processor.processStream: classify provider/transportError SSE events into text/failure chunks and reconcile partial text + completion state - TurnConformance.run: scripted provider-agnostic agent loop (steering, guardrails, cancellation, context trimming, tool dispatch) - TurnConformance.runTurn: drive the real main-source TurnEngine - TurnConformance.replay: drive the real ReferenceTurnRunner + journal normalization Only the 7 out-of-scope waivers (load/render/parse/process/toRequest/ enrich/mapModel) remain, covered by dedicated Java driver tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
Drive-to-green #494 — statusJava (the final runtime) now registers real, provider-agnostic adapters for the target contracts and removes their waivers. The 3 core target contracts (
¹ Legitimate package-layering pointer, not a contract gap. For Swift/C#, streaming-wire classification lives in the provider/SDK assemblies ( ² Rust Design notes
Out-of-scope waivers (never part of this epic; each has an explicit reasoned message, no silent skips)
Verification
Left as draft — not marking ready-for-review or merging; that's for the human. |
| } | ||
|
|
||
| private static final class RecordingPostCommit implements Ports.PostCommitPort { | ||
| private final List<String> effectIds = new ArrayList<>(); |
Comment on lines
+950
to
+996
| foreach (var raw in events) | ||
| { | ||
| if (raw is not JsonObject evt) | ||
| continue; | ||
| var kind = (evt["kind"] as JsonValue)?.GetValue<string>(); | ||
| if (kind == "provider") | ||
| { | ||
| if (evt["value"] is not JsonObject value | ||
| || value["choices"] is not JsonArray choices | ||
| || choices.Count == 0 | ||
| || choices[0] is not JsonObject choice | ||
| || choice["delta"] is not JsonObject delta) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| if (delta["content"] is JsonValue content && content.TryGetValue<string>(out var text)) | ||
| chunks.Add(new TextChunk { Value = text }); | ||
|
|
||
| if (delta["refusal"] is JsonValue refusal && refusal.TryGetValue<string>(out var reason)) | ||
| { | ||
| chunks.Add(new FailureChunk | ||
| { | ||
| Failure = new StreamFailure | ||
| { | ||
| Outcome = StreamFailureOutcome.Determinate, | ||
| Message = $"Model refused: {reason}", | ||
| }, | ||
| }); | ||
| } | ||
| } | ||
| else if (kind == "transportError") | ||
| { | ||
| chunks.Add(new FailureChunk | ||
| { | ||
| Failure = new StreamFailure | ||
| { | ||
| Outcome = StreamFailureOutcome.Indeterminate, | ||
| Message = (evt["message"] as JsonValue)?.GetValue<string>() ?? string.Empty, | ||
| }, | ||
| }); | ||
| } | ||
| else | ||
| { | ||
| throw new InvalidOperationException($"unsupported stream event kind: {kind}"); | ||
| } | ||
| } |
The replay vectors are fully provider-agnostic (scripted model by scenario name, add/fail host tools, permission allow/deny, normalized journal). Rust already ships the complete ReferenceTurnRunner session machinery and a passing golden-replay test (tests/harness_turn_runner.rs). The replay waiver claimed a dependency on the provider wire+process layer, which is false. Wire a real replay_impl in the vector adapter that drives ReferenceTurnRunner (CollectingEventSink, JsonlEventJournalWriter, InMemoryCheckpointStore, FunctionHostToolExecutor add/fail, scenario permission resolver, scripted model callback), reads back the JSONL journal, and normalizes each record to the observable event strings the vectors assert. Removes the TurnConformance.replay waiver. All 5 replay vectors pass (114-118); cargo fmt + clippy clean.
Member
Author
Drive-to-green status — target contracts now REAL in all 7 runtimesThe four epic target contracts — Waivers removed this pass (were wrong / closeable)
Each removal is paired with a registered adapter, so a removed-but-unimplemented waiver fails hard (XPASS/no-adapter) — green means genuinely covered. Target-contract coverage
Remaining waivers — all legitimate, out of scope for #494These are honest absent-layer or package-layering gaps, not target contracts. Closing them means building entire SDK/provider layers.
Verification (changed runtimes this pass)
Draft — leaving ready/merge to a human. |
5 tasks
…integer-conversion) The replay adapter narrowed a platform-width int to int32 without a range check, which CodeQL flags as a potential overflow. Clamp to [MinInt32, MaxInt32] before converting. Behavior is unchanged for the small maxIterations values the vectors carry; the guard just makes the narrowing explicit and safe.
CodeQL's go/incorrect-integer-conversion data-flow did not accept the clamp-and-reassign idiom. Use the canonical guarded form: perform the int32 conversion only inside an \ >= MinInt32 && n <= MaxInt32\ check. Behavior is identical for the small maxIterations values the vectors carry.
Member
Author
CI fix — CodeQL greenThe only failing check was CodeQL: one Fixed by performing the conversion only inside a both-bounds guard ( CodeQL gate now passes with zero open code-scanning alerts on the PR; |
This was referenced Aug 22, 2026
Seth Juarez (sethjuarez)
marked this pull request as ready for review
August 23, 2026 02:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drive-to-green #494 — eliminate the 3 remaining
@vectorwaiversTracking: #494. Removing the last waived conformance contracts (
Processor.processStream,TurnConformance.run,TurnConformance.runTurn, plus JavaTurnConformance.replay) by implementing real adapters, not waivers.Draft — do not merge. Pushing incrementally; a running status of which waivers are gone will be maintained in a comment below.
Progress so far
Processor.processStream— real classifier + reconciler, waiver deleted. Conformance 147 passed / 33 skipped; full suite 1599 passed; ruff clean.TurnConformance.run(28 vectors)TurnConformance.runTurn(5 vectors) — assessing whether it needs a net-new snapshot/portability engineApproach note
Building the reference implementation in Python first (fast verify loop), then porting to the priority runtimes. Deviation from strict Rust-first order is a deliberate verifiability choice; will be reconciled before ready-for-review.