Migrate error-string vectors to native expectedError (#489) - #499
Merged
Seth Juarez (sethjuarez) merged 2 commits intoAug 23, 2026
Merged
Conversation
Replace the bespoke `{"error": ...}` payloads and fuzzy substring matchers
with the typra harness's native `expectedError` mechanism across all seven
runtimes.
The six load.tsp error vectors now carry a canonical `{kind, [field]}`
expectedError payload: env_var_not_set, file_reference, file_not_found,
invalid_frontmatter, invalid_template, and missing_required_input (+field).
Each runtime's adapter maps its typed load/validation error onto the
canonical kind by exception TYPE -- never by matching message text:
- Python: PromptyLoadError.kind is already canonical (reference adapter).
- Go / Rust / TypeScript / C#: typed load-error classes expose the kind;
adapters map type -> canonical kind and drop the fuzzy matchers.
- Java: VectorAdapters + LoadVectorsTest map LoadException/InvokerException
to the canonical payload; assertErrorMatches deleted from SpecVectors.
- Swift: the prompty-model conformance adapter throws VectorError with the
canonical payload; the SDK LoadVectorTests maps LoadError/InvokerError
cases to canonical kinds. The SDK loader collapses a bare-string template
into .invalidModel (one Agent.load schema step), which the test asserts
directly as the SDK's classification of invalid_template.
Removes the fuzzy error matchers (Python _err, Java assertErrorMatches,
Swift expectErrorMatches, and equivalents) in favour of type-based mapping.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6194cf76-47f5-40ba-b7ee-aed2df52146c
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Comment on lines
+76
to
+81
| catch (Exception ex) | ||
| { | ||
| throw new PromptyLoadException( | ||
| "invalid_frontmatter", | ||
| $"Invalid frontmatter in '{fullPath}': {ex.Message}"); | ||
| } |
Comment on lines
+285
to
+295
| foreach (var prop in agent.Inputs) | ||
| { | ||
| if (string.IsNullOrEmpty(prop.Name)) | ||
| continue; | ||
| if (provided.ContainsKey(prop.Name)) | ||
| continue; | ||
| if (prop.Default is not null || prop.Example is not null) | ||
| continue; | ||
| if (prop.Required == true) | ||
| return prop.Name; | ||
| } |
| } | ||
| if (expObj is not null && expObj.ContainsKey("error") && input.ContainsKey("inputs") && input.ContainsKey("frontmatter")) | ||
| // --- input-validation vectors: BOTH a top-level `inputs` map AND `frontmatter` --- | ||
| if (input.ContainsKey("inputs") && input.ContainsKey("frontmatter")) |
| } | ||
| if (expObj is not null && expObj.ContainsKey("error") && input.ContainsKey("inputs") && input.ContainsKey("frontmatter")) | ||
| // --- input-validation vectors: BOTH a top-level `inputs` map AND `frontmatter` --- | ||
| if (input.ContainsKey("inputs") && input.ContainsKey("frontmatter")) |
| { | ||
| return ErrorResult(exc, expected); | ||
| } | ||
| loaded = PromptyLoader.Load(Path.Combine(SpecFixtures, fixtureVal.GetValue<string>())); |
The #489 expectedError migration left a type-cast in buildAgent that exceeded Prettier's line width. Reflow it so runtime/typescript format:check passes; no behavior change (TS suite green: 1675 core). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6194cf76-47f5-40ba-b7ee-aed2df52146c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #489.
Replaces the bespoke
{"error": ...}vector payloads and fuzzy substring matchers with the typra harness's nativeexpectedErrormechanism across all seven runtimes.What changed
The six
load.tsperror vectors now carry a canonical{kind, [field]}expectedErrorpayload:env_var_not_setfile_referencefile_not_foundinvalid_frontmatterinvalid_templatemissing_required_input(+field)Each runtime's adapter maps its typed load/validation error onto the canonical kind by exception type — never by matching message text:
PromptyLoadError.kindis already canonical (reference adapter).errors.py,errors.ts,PromptyLoadException.cs.)VectorAdapters+LoadVectorsTestmapLoadException/InvokerExceptionto the canonical payload;assertErrorMatchesdeleted fromSpecVectors.prompty-modelconformance adapter throwsVectorErrorwith the canonical payload; the SDKLoadVectorTestsmapsLoadError/InvokerErrorcases to canonical kinds. The SDK loader collapses a bare-string template into.invalidModel(a singleAgent.loadschema step), which the test asserts directly as the SDK's classification ofinvalid_template.Removes the fuzzy error matchers (
_err,assertErrorMatches,expectErrorMatches, and equivalents) in favour of type-based mapping.Verification — all seven runtimes green
ok(conformance)