Make globals part of the shared pipeline language (#104) - #450
Merged
Conversation
#449 gave the compiler, validator and runtime one filter registry. Globals were left out, and they are the other half of the language: the runtime offers eight (`now`, `file_exists`, `include_file` and five loop helpers), the compiler and both validators offered none. So a pipeline writing `{{ now() }}` drew four errors from two validators: Undefined variable: 'now' Undefined variable: 'file_exists' Undefined task reference: 'now()' Undefined task reference: 'file_exists('' ...and then ran correctly and wrote the right file. The data-flow validator splits a reference on `.`, which leaves the call syntax attached, so it looked `now()` up as a *task id*. Same false-positive class as #448. The obvious symmetry -- copy the globals the way the filters are copied -- is wrong, and the e2e test added here is what says so. Filters transform a value the caller already holds; every one of our globals answers a question about the state of a *run*: - id: make_it # writes ./artifact - id: check_it # content: "{{ file_exists('artifact') }}" At run time `check_it` writes `exists=True`, because by then `make_it` has run. With the globals copied into the compiler's environment it renders at compile time instead and writes `exists=False` -- no error, just a wrong answer in a file. Leaving them unregistered is what makes the compiler keep the template for the runtime. The rule is therefore split: every environment knows the *names*, only the runtime holds the implementations. `pipeline_global_names()` states the names once, derived from what the runtime registers rather than hand-listed, because a hand-kept list drifts and drift is what this module exists to stop. Measured: catalogue validating 18 -> 20 of 117; blocking suite 602 -> 616 passed, 0 failed. Five mutations, all killed: - copy globals into every environment -> 2 tests, incl. the e2e writing `exists=False` - remove the template-validator skip -> 9 tests - remove the data-flow-validator skip -> 9 tests - derive no globals at all -> 10 tests - stop subtracting Jinja's own globals -> 3 tests Two of my own tests were weak and were fixed before the mutations above could speak: the e2e case originally used `./a.txt`, which the compiler skips for an unrelated reason (its runtime-reference heuristic bails on a dot), so it passed under the very mutation it existed to catch; and the parametrized case interpolated the bare call, so the pipeline carried the literal string `now()` and was never a template at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#449 gave the compiler, validator and runtime one filter registry. Globals were left out, and they are the other half of the language.
The defect
A pipeline writing
{{ now() }}drew four errors from two validators:...and then ran correctly and wrote the right file. The data-flow validator splits a reference on
., which leaves the call syntax attached, so it lookednow()up as a task id. Same false-positive class as #448.Why the obvious fix is wrong
The symmetry — copy the globals the way the filters are copied — is what the review recommended, and it corrupts output. Filters transform a value the caller already holds. Every one of our globals answers a question about the state of a run:
At run time
check_itwritesexists=True, because by thenmake_ithas run. With the globals copied into the compiler's environment it renders at compile time instead and writesexists=False— no error, just a wrong answer in a file. Leaving them unregistered is precisely what makes the compiler keep the template for the runtime.So the rule is split: every environment knows the names, only the runtime holds the implementations.
pipeline_global_names()states the names once, derived from what the runtime registers rather than hand-listed.Measured
Mutations — five, all killed
exists=FalseTwo of my own tests were weak
Both were fixed before the mutations above could speak, and both are the same pattern flagged in the last review:
./a.txt, which the compiler skips for an unrelated reason (its runtime-reference heuristic bails on a dot), so it passed under the very mutation it existed to catch;now()and was never a template at all.Not in this PR
now()is still nondeterministic within a run — two steps rendering it get different timestamps. That is fixed byexecution.timestamp, which needs the typed RuntimeContext. Groundwork found while investigating: seven separate sites construct anexecutiondict (orchestrator.py ×3, control_system.py, hybrid_control_system.py ×2, declarative_engine.py), and they disagree — the CLI path rendersexecution.timestampas2026-08-02T19:59:53.199477while orchestrator.py:306 formats%Y-%m-%d-%H:%M:%S.🤖 Generated with Claude Code