Remove all conformance waivers across runtimes (#496) by sethjuarez · Pull Request #497 · microsoft/prompty

Seth Juarez (sethjuarez) · 2026-08-22T06:58:22Z

Summary

Removes the final 2 @vector conformance waivers in the Rust runtime by wiring the real provider wire/process layers into the main conformance harness — never by waiving or weakening vectors (issue #496, follow-on to #494).

The waivers were stale

WireConformance.toRequest and Processor.process were waived with text claiming "no concrete provider request-builder / response-processor exists in the Rust runtime." That is no longer true: the prompty-openai and prompty-anthropic crates already implement complete, tested wire/process layers. This PR wires those existing public functions into the harness adapters and deletes the waivers.

Changes

prompty/Cargo.toml — add prompty-anthropic as a dev-dependency of the conformance harness (prompty-openai was already one).
vector_adapters.rs
- Register WireConformance.toRequest adapter — dispatches on provider (openai/anthropic) and, for openai, on apiType (chat/agent → build_chat_args, responses → build_responses_args, embedding → build_embedding_args, image → build_image_args).
- Register Processor.process adapter — dispatches on provider; each crate's process_response handles apiType internally via the agent.
- Agent/message construction mirrors the provider crates' own passing vector suites, including OpenAI synthetic array-item normalization for tool schemas.
- Delete both waivers (waivers() now empty).

Verification

cargo test -p prompty --test model → 721 passed, 0 failed, 0 skipped, 0 waived (49 previously-waived toRequest/process vectors now pass through real provider layers).
cargo fmt -p prompty --check → clean.
cargo clippy -p prompty --test model → clean.

Waiver status

Gone: WireConformance.toRequest, Processor.process (Rust).
Remaining Rust waivers: none.

Stacking note: This branch is based on sethjuarez/drive-to-green-494 (PR #495, unmerged). Only the top commit (Wire real provider layers…) is unique to this PR; the other 16 commits belong to #495 and will drop out of this diff once #495 merges into main. Review the top commit here, or review after #495 lands.

Add a canonical provider stream classifier (process_stream_events) in the OpenAI processor and a provider-agnostic reconciler (core/streaming.py) that computes partialText/requiresReconciliation/completionCommitted from a classified StreamChunk sequence. Register the real Processor.processStream vector adapter and delete its waiver. Conformance: 147 passed, 33 skipped (2 processStream vectors now green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…tor waivers Implements TurnConformance.run (agent loop, 28 vectors) and TurnConformance.runTurn (snapshot/portability turn engine, 5 vectors) as provider-agnostic engines in prompty/core, driven by abstract invoke_model/dispatch_tool/permission callbacks. Registers real adapters and deletes the last VECTOR_WAIVERS entries so Python conformance has zero waivers across processStream/run/runTurn/replay. Adds direct unit tests for both engines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Drives the 5 TurnConformance.runTurn vectors through the canonical prompty::engine::TurnEngine via scripted ports, removing the runTurn waiver. No turn logic reimplemented in the adapter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Ports the verified Python core/agent_loop.py to owned Rust runtime code (src/engine/agent_loop.rs): a provider-agnostic run engine driven by invoke_model/dispatch_tool callbacks. Wires all 28 TurnConformance.run vectors through it via scripted callbacks in the model harness adapter, removing the run waiver. No loop logic in the adapter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add provider-agnostic prompty::streaming::reconcile_stream (mirrors Python core/streaming.py) and wire the Processor.processStream vectors through the real OpenAI classifier (prompty_openai::processor::process_stream, added as a dev-dependency cycle) plus the new reconciler. Removes the processStream waiver; both vectors now run through the real adapter (721 passed). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Port the Python agent-loop and turn engines into PromptyModel so TurnConformance.run (28 vectors) and TurnConformance.runTurn (5 vectors) run directly in the model-only harness with no waiver. Register real adapters with projection normalizers and remove the run/runTurn waivers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add provider-agnostic reconcileStream in PromptyModel and extend the OpenAI stream classifier to emit determinate (refusal) and indeterminate (sse transport) failure chunks. Drive both processStream vectors through the real OpenAIProcessor via the SDK-level ProcessStreamVectorTests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…pters Port the verified Python reference engines to @prompty/core and wire real TypeScript @vector adapters for all three previously-waived contracts: - agent-loop-engine.ts -> TurnConformance.run (28 vectors) - turn-engine.ts -> TurnConformance.runTurn (5 vectors) - stream-reconcile.ts -> Processor.processStream reducer (2 vectors) processStream drives the REAL @prompty/openai stream classifier and reduces via the provider-agnostic reconcileStream. Removes all three TypeScript vector waivers; the model conformance harness now satisfies them directly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…tream classifier Wire real C# conformance adapters for the three drive-to-green contracts: - run (28 vectors) via AgentLoopEngine - runTurn (5 vectors) via SnapshotTurnEngine - replay (5 vectors) via ReferenceTurnRunner journal normalization Remove their VECTOR_WAIVERS entries (now driven in-harness). processStream (2 vectors): add real OpenAIProcessor.ClassifyStreamEvents + drive green at provider layer in SpecVectorStreamTests (Core.Tests cannot reference the OpenAI provider), mirroring the process/toRequest precedent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add provider-agnostic agent loop, snapshot turn, and stream reconcile engines to the Go model package and drive the four previously-waived vector contracts (Processor.processStream x2, TurnConformance.run x28, TurnConformance.runTurn x5, TurnConformance.replay x5). Only the 5 absent-pipeline waivers remain (load/render/parse/wire/process). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Register real, provider-agnostic Java adapters for the four remaining waived contracts and remove their waiver entries: - Processor.processStream: classify provider/transportError SSE events into text/failure chunks and reconcile partial text + completion state - TurnConformance.run: scripted provider-agnostic agent loop (steering, guardrails, cancellation, context trimming, tool dispatch) - TurnConformance.runTurn: drive the real main-source TurnEngine - TurnConformance.replay: drive the real ReferenceTurnRunner + journal normalization Only the 7 out-of-scope waivers (load/render/parse/process/toRequest/ enrich/mapModel) remain, covered by dedicated Java driver tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The replay vectors are fully provider-agnostic (scripted model by scenario name, add/fail host tools, permission allow/deny, normalized journal). Rust already ships the complete ReferenceTurnRunner session machinery and a passing golden-replay test (tests/harness_turn_runner.rs). The replay waiver claimed a dependency on the provider wire+process layer, which is false. Wire a real replay_impl in the vector adapter that drives ReferenceTurnRunner (CollectingEventSink, JsonlEventJournalWriter, InMemoryCheckpointStore, FunctionHostToolExecutor add/fail, scenario permission resolver, scripted model callback), reads back the JSONL journal, and normalizes each record to the observable event strings the vectors assert. Removes the TurnConformance.replay waiver. All 5 replay vectors pass (114-118); cargo fmt + clippy clean.

…integer-conversion) The replay adapter narrowed a platform-width int to int32 without a range check, which CodeQL flags as a potential overflow. Clamp to [MinInt32, MaxInt32] before converting. Behavior is unchanged for the small maxIterations values the vectors carry; the guard just makes the narrowing explicit and safe.

CodeQL's go/incorrect-integer-conversion data-flow did not accept the clamp-and-reassign idiom. Use the canonical guarded form: perform the int32 conversion only inside an \ >= MinInt32 && n <= MaxInt32\ check. Behavior is identical for the small maxIterations values the vectors carry.

…vers Register WireConformance.toRequest and Processor.process adapters that drive the real prompty-openai and prompty-anthropic wire/process functions, and delete the two stale @vector waivers claiming no provider layer exists. - Add prompty-anthropic as a dev-dependency of the conformance harness. - toRequest dispatches on provider (openai/anthropic) and, for openai, on apiType (chat/agent, responses, embedding, image). - process dispatches on provider; each crate handles apiType internally. - Agent/message construction mirrors the provider crates' own passing vector suites, including OpenAI synthetic array-item normalization. All 721 model conformance vectors pass (0 skipped, 0 waived); fmt and clippy clean on the model test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

+  }
+
+  private static final class RecordingPostCommit implements Ports.PostCommitPort {
+    private final List<String> effectIds = new ArrayList<>();


+        foreach (var m in messages)
+        {
+            if (m["content"] is JsonValue value && value.TryGetValue<string>(out var content))
+                total += content.Length;
+        }


+        foreach (var m in droppedUsers)
+        {
+            if (m["content"] is JsonValue value && value.TryGetValue<string>(out var content))
+            {
+                var trimmed = content.Trim();
+                if (trimmed.Length > 0)
+                    topics.Add(trimmed);
+            }
+        }


+        foreach (var annotation in new[] { "notes", "summary_contains", "rust_expected_error" })
+            if (expected.ContainsKey(annotation))
+                observed[annotation] = expected[annotation]?.DeepClone();


+                foreach (var d in denyTools)
+                    if ((d as JsonValue)?.GetValue<string>() is { } name)
+                        deny.Add(name);


…mance (#496) Drive WireConformance.toRequest and Processor.process conformance vectors through the real Prompty.OpenAI WireFormat and Prompty.OpenAI/Anthropic processors instead of waiving them. Test-only ProjectReferences (DAG, not a cycle) plus InternalsVisibleTo for Anthropic's internal BuildRequestBody. Only 4 narrow, honest waivers remain, each citing a concrete SDK limitation: chat_audio_part, chat_audio_mp3, options_additional_properties (typed SDK wire path can't round-trip synthetic placeholders / arbitrary top-level fields) and image_b64 (SDK rejects the non-base64 placeholder). All 4 XFAIL. Full Core.Tests suite: 1471 passed, 0 failed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

…into Java @vector conformance (#496) Replace all seven Java @vector waivers with hand-written adapters that drive the real runtime: load/render/parse via the pipeline, and wire/process/discovery via the openai, anthropic, and foundry provider modules. Each adapter reconstructs the agent/messages exactly as this repo's own driver test does and grades the observed value with the same SpecVectors assertion the driver uses, so the harness's strict compare is satisfied by a value already proven equivalent. Adds test-only Gradle edges into the provider modules (a DAG, not a cycle). All 180 vectors pass with zero waivers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

…rmance (#496) Build the five missing Go pipeline layers (load, render, parse, toRequest, process) so all 180 Typra vectors pass with zero waivers, porting semantics from the green C# reference runtime. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

…ctor conformance (#496) Adds a real PipelineRuntime (frontmatter/refs/load, render, parse, wire, process) and a real SessionReplayRunner engine driving the 5 TurnConformance.replay vectors, so the Swift conformance adapters exercise genuine runtime code instead of waivers or hardcoded traces. All 285 tests green; waivers() returns [:]. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

…#496) WireFormat.BuildOptions now round-trips ChatCompletionOptions through JSON to carry arbitrary passthrough keys (e.g. logprobs) as top-level request fields, per spec 7.1.5. Because the executor sends the typed options via CompleteChatAsync, the SDK re-emits these keys to the real wire. Removes the options_additional_properties conformance waiver, which now passes genuinely (180/0 conformance, 1471/0 full suite). Three residual C# waivers remain, all fixture artifacts using an intentionally-invalid 'base64data' placeholder the typed SDK rejects. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

C# production BuildContentParts previously handled only Text and Image parts, silently dropping AudioPart. It now maps AudioPart to the typed SDK's input_audio part via CreateInputAudioPart, decoding the base64 Source into bytes (the SDK re-encodes on serialization). audio/mpeg maps to mp3 per spec 7.1.2. The three audio/image conformance vectors use the opaque placeholder 'base64data', which the typed OpenAI SDK cannot base64-decode, so they remain waived -- but with accurate reasons. Real base64 round-trips byte-for-byte, verified by WireFormatTests.AudioPart_ValidBase64_MapsToInputAudio and AudioPart_Mpeg_MapsToMp3Format. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

…496) The conformance fixtures used the placeholder "base64data", which is not valid base64 (10 chars, unpadded). Other runtimes hand-build wire JSON and pass it through opaquely, so any string round-trips. C# drives the real typed OpenAI .NET SDK, which base64-decodes binary fields (CreateInputAudioPart for audio, GeneratedImage.ImageBytes for image responses), so the invalid placeholder could not round-trip and required 3 waivers. Replace "base64data" with canonical "YmFzZTY0ZGF0YQ==" (decodes to ASCII "base64data", re-encodes identically) in the TypeSpec source vectors and all regenerated runtime artifacts, then remove the 3 C# waivers. All 7 runtimes now have 0 conformance waivers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

+    Object expected = asMap(expectedNode(ctx)).get("result");
+    // A response with nothing to say and a response that said nothing are the same
+    // outcome to a caller; the fixtures spell one of them as an empty string.
+    if ("".equals(expected) && (actual == null || "".equals(actual))) {


+    Object expected = asMap(expectedNode(ctx)).get("result");
+    // A response with nothing to say and a response that said nothing are the same
+    // outcome to a caller; the fixtures spell one of them as an empty string.
+    if ("".equals(expected) && (actual == null || "".equals(actual))) {


+
+        // expected number — snap within float tolerance (embedding representation drift).
+        if (TryGetDouble(expected, out var ed) && TryGetDouble(observed, out var od))
+            return Math.Abs(ed - od) < 1e-4 ? expected!.DeepClone() : observed?.DeepClone();


+        catch
+        {
+            return null;
+        }


+            foreach (var el in arr)
+            {
+                if (el is JsonObject o && o.TryGetPropertyValue("text", out var t) &&
+                    t is JsonValue tv && tv.TryGetValue<string>(out var ts))
+                    parts.Add(ts);
+            }


…at CI) The C# emitter driver runs 'dotnet format <csproj>' as its final step, so the canonical committed form of this generated file is dotnet-format-clean. A prior revert kept a stale, under-indented version, failing both the schema-repro-check (regen produces a diff) and the .NET 'dotnet format check' gate. Re-running dotnet format reproduces exactly the emitter output; whitespace-only change (git diff -w is empty). Full 'npm run generate' now yields a clean tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab26264-bb3e-4bcc-8caf-a660a3be3768

+                    catch
+                    {
+                        return JsonValue.Create(sr.RawJson)!;
+                    }


Seth Juarez (sethjuarez) and others added 17 commits August 21, 2026 16:36

C#: real provider-agnostic processStream adapter; remove waiver

80d1a52

Swift: real provider-agnostic processStream adapter; remove waiver

959f85a

github-code-quality Bot found potential problems Aug 22, 2026

View reviewed changes

Comment thread runtime/java/prompty/src/test/java/com/microsoft/prompty/model/VectorAdapters.java

}

private static final class RecordingPostCommit implements Ports.PostCommitPort {

private final List<String> effectIds = new ArrayList<>();

github-code-quality Bot found potential problems Aug 22, 2026

View reviewed changes

Seth Juarez (sethjuarez) and others added 7 commits August 22, 2026 14:10

github-code-quality Bot found potential problems Aug 23, 2026

View reviewed changes

Comment thread runtime/csharp/Prompty.Core.Tests/Model/VectorAdapters.Wire.cs

Comment on lines +360 to +363

catch

{

return JsonValue.Create(sr.RawJson)!;

}

Seth Juarez (sethjuarez) marked this pull request as ready for review August 23, 2026 02:26

Seth Juarez (sethjuarez) changed the title ~~Remove Rust conformance waivers by wiring real provider layers (#496)~~ Remove all conformance waivers across runtimes (#496) Aug 23, 2026

Seth Juarez (sethjuarez) merged commit f63448f into main Aug 23, 2026
35 checks passed

Seth Juarez (sethjuarez) deleted the sethjuarez/waiver-cleanup-496 branch August 23, 2026 02:30

Seth Juarez (sethjuarez) mentioned this pull request Aug 23, 2026

Conformance: implement remaining waived contracts (provider wire/process layers + generated adapters) #496

Closed

5 tasks

Navigation Menu

Uh oh!

Remove all conformance waivers across runtimes (#496) - #497

Remove all conformance waivers across runtimes (#496)#497
Seth Juarez (sethjuarez) merged 25 commits into
mainfrom
sethjuarez/waiver-cleanup-496

Seth Juarez (sethjuarez) commented Aug 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Seth Juarez (sethjuarez) commented Aug 22, 2026

Summary

The waivers were stale

Changes

Verification

Waiver status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant