Stop unresolved templates before they reach a tool (#153)

[jeremymanning]

Closes #153.

Measured before

$ orchestrator run bad_template.yaml
"success": true, "error": null

$ cat out/bad.txt
value = {{ nonexistent_step.result.field }}

Literal Jinja source on disk, in a file a user would read as output, from a run that exited reporting no error. validate refused the same document — so this is the mirror image of #241, with the validator right and the executor wrong.

Measured after

UnresolvedTemplateError - Parameter 'content' of tool 'filesystem' still contains
unresolved template references: '{{ nonexistent_step.result.field }}'.
The step was stopped before it could act on them.

PipelineExecutionError: Pipeline execution failed: Task 'write_it' failed
$ cat out/bad.txt
(no file written)

Exit 1, no artifact, the reference named.

Where the fix goes, and why not where the bug is

TemplateManager catches UndefinedError, logs a warning, and returns the original template text (template_manager.py:516). That fallback is deliberate and has to stay — resolution runs in several passes, and a reference that cannot resolve yet may resolve later. Re-raising there would break legitimate deferred resolution.

What was missing is a point where the fallback stops being acceptable. That is resolve_before_tool_execution, which already promised "tool parameters with all templates resolved" and is the last place the promise can be kept: past it the value is written to a file or sent to a model. It is also the single funnel — Tool.execute routes every tool through it (tools/base.py:40), so this is one edit rather than one per output-writing tool.

The check now enforces what the docstring already claimed.

Survival, not presence

A first version compared only the rendered output and flagged any {{ }} it found. That rejected escaped Jinja: {{ '{{' }} renders to a literal {{, so writing documentation about templates became impossible. Its own test caught it.

The rule is now: flag a marker only if it appeared in the input and came back byte-identical. A marker present in the output but not the input is content — escaped delimiters, or a step result that legitimately contains template-looking text (a scraped page, a code sample). Both cases have tests.

One bad reference takes the whole string down

StrictUndefined aborts the entire render, so in "{{ here }} but not {{ missing }}" the resolvable {{ here }} does not survive either. Both markers are reported.

I initially asserted only {{ missing }} would be reported, and the test failed — the code was right and my expectation was wrong. Reporting just the undefined reference would imply the others had been substituted. This behaviour is now pinned and documented, because a user hitting it would otherwise find it baffling.

Coverage

tests/test_unresolved_templates.py — 21 tests:

• end to end through the CLI: non-zero exit, no artifact written, the reference named in the output
• a resolvable template still passes (no false positives)
• the error names the parameter, the tool, and the surviving markers
• {% if %} blocks caught, not just {{ }}
• references buried in nested lists/dicts caught
• _-prefixed plumbing parameters not inspected
• escaped delimiters and template-looking step results both pass
• the marker scanner, parametrised over strings, dicts, dict keys, nested lists, tuples, non-strings, and multi-line blocks

Numbers

	before	after
Blocking suite	435 passed	456 passed, 13 skipped, 0 failed, 0 xfailed
Collection	3230	3251, 0 warnings
uv lock --check / ruff on src / git diff --check	—	clean

ADR 0001 now states this contract and the action registry from #241; neither was written down.

Note on scope

This closes the reported defect at the tool boundary. It does not make an unresolved reference a compile-time error — validate already rejects this pipeline, so the two now agree, but a reference that only becomes resolvable at runtime is still legal by design. Tightening that further is a schema decision, not a bug fix.