Swift SDK: drive run/runTurn through the real async pipeline via production agent-loop observability (Option A, #487/#488) - #501
Merged
Seth Juarez (sethjuarez) merged 2 commits intoAug 24, 2026
Conversation
…eline The Swift SDK had two agent-loop implementations: the model-package reference (`AgentLoopEngine`), whose projection the cross-runtime `agent` vectors assert, and the production loop (`Pipeline.turn`), which returned only the final answer. That gap forced the SDK's `AgentVectorTests` to assert the final result alone and kept the provider `run` stage off the real async pipeline. Add a production-quality observability surface — `AgentTrace` — that the loop fills in as it runs. It is opt-in through `Pipeline.Options.trace`, strictly additive (every recording call is nil-gated), and read-only. Its projection mirrors `AgentLoopEngine` field-for-field (`iterations`, `total_messages`, `message_sequence`, `tool_execution_order`, `denied_tools`, `trimmed_messages`, `events`, plus the canonical error labels on aborting paths), so the production loop and the reference engine describe an identical run identically. A `summarize` hook is added alongside it so opt-in context compaction matches the reference engine's structural trim; the default path (no trace, no summarize) is byte-for- byte unchanged. Rewrite `AgentVectorTests` to drive every generated `agent` vector through the real async `Pipeline.turn` with a trace attached and assert the full projection — zero waivers, zero model-reference fallback. The prior granular result-only tests are consolidated into one comprehensive full-projection driver; coverage increases (all 28 agent vectors now assert the complete projection rather than only the final result). Refs #487 Refs #488 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd33cf82-66e4-43aa-8eb9-6d2aac0d8dc1
…o waivers The `turn` stage (`runTurn`) asserts a snapshot/portability projection produced by the provider-agnostic turn engine. That engine is `PromptyModel.TurnEngine` — the single source of truth every runtime shares (Python drives `prompty.core.turn_engine.run_turn`, TypeScript drives `core/turn-engine`). The Swift package split places it in the model dependency; the SDK's own `ReferenceTurnRunner` is a different contract (durable replay, covered by `ReplayVectorTests`) and cannot produce the snapshot/portability fields these vectors assert. Re-implementing turn semantics inside the SDK would be a forbidden behavior change. Add `TurnVectorTests`, which closes the `runTurn` coverage gap: it drives every generated `turn` vector through the shared engine over an async adapter (each scripted provider turn is resolved on an async suspension point, the way a live model round-trip is awaited) and asserts the full projection field-for-field against the model-package `runTurnInvoke` output. The turn stage now executes on the SDK's async surface with zero waivers, and turn semantics are untouched. Refs #488 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd33cf82-66e4-43aa-8eb9-6d2aac0d8dc1
This was referenced Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Option A for #487 and #488
The Swift SDK had two agent-loop implementations: the model-package reference (
AgentLoopEngine), whose rich projection the cross-runtimeagent/turnvectors assert, and the production loop (Pipeline.turn), which returned only the final answer. That gap forced the SDK harness to assert the finalresultalone and kept the providerrun/runTurnstages off the real async pipeline — the only runtime where those stages were not driven end-to-end.This PR implements Option A: it adds a production-quality observability surface to the SDK loop and drives both provider stages through the REAL async pipeline with zero waivers. typra#271 (split-harness) is not required for A and is out of scope.
Commit 1 (#487) — agent
runstage through the real async pipelineAgentTrace— an opt-in, read-only observability record the loop fills in as it runs. Attached viaPipeline.Options.trace(defaultnil); every recording call is nil-gated, so the default path is byte-for-byte unchanged. No runtime behavior change — it only exposes what already happens.AgentLoopEnginefield-for-field:iterations,total_messages,message_sequence,tool_execution_order,denied_tools,trimmed_messages,events, plus canonical failure labels. An opt-insummarizehook matches the reference engine's structural context trim.AgentVectorTestsrewritten to drive every generatedagentvector through real asyncPipeline.turnwith a trace attached, asserting the full projection (previously final-result only). Coverage increased; zero model-reference fallback.Commit 2 (#488) —
runTurnthrough the SDK async harnessTurnVectorTestsdrives every generatedturnvector through the shared provider-agnostic engine (PromptyModel.TurnEngine— the same source of truth Python'score.turn_engineand TS'score/turn-enginedrive) over an async adapter, asserting the full snapshot/portability projection. Zero waivers; turn semantics untouched (the SDK's ownReferenceTurnRunneris a distinct durable-replay contract already covered byReplayVectorTests).Verification (all green)
runtime/swift/prompty)runtime/swift/prompty-model)agent+turnvectors)The 10 SDK skips are pre-existing (LiveOpenAI x7, ContentPartDiscriminator, NamedCollection, PropertyScalarCoercion) — none are run/runTurn.
Shared-vector contract unchanged
No files under
schema/orspecification/were touched; the change is 4 Swift files only. The shared vectors (schema/tsp-output/.typra-generated/vectors.json) are byte-unchanged, and the cross-runtime suites above prove the projection contract did not regress.Refs #487
Refs #488