Skip to content

Loop variables validate in one validator and are refused by the other #469

Description

@jeremymanning

Summary

{{ item }} and {{ index }} inside a for_each step are bound by the runtime and rejected by the template validator. The data-flow validator accepts them. Two validators disagree about the same names, and the one that rejects is wrong.

This is the false-positive class removed in #448 (slugify), #450 (now), #454 (execution.timestamp) and #459 (pipeline_id): a name the runtime provides, reported as undefined.

Evidence

data-flow validator          template validator        runtime
item        -> valid         Undefined variable: 'item'    provides 'item'
item.name   -> valid                                       provides '$item'
index       -> valid         Undefined variable: 'index'   provides 'index'

The runtime's available-variable list, from a real run:

['timestamp', 'debug_mode', ..., '$item', '$index', 'item', 'index',
 'iteration', '$iteration', '$is_first', '$is_last', '$loop_state', ...]

DataFlowValidator already declares them:

LOOP_VARIABLES = frozenset({"item", "index", "loop", "iteration", "is_first", "is_last"})

TemplateValidator does not know them at all.

Reproduction

examples/enhanced/fact_checker_enhanced.yaml, whose verify_sources step declares for_each and whose nested steps use the loop variable:

- id: verify_sources
  for_each: <AUTO>list of sources to verify</AUTO>
  steps:
    - id: verify_source
      parameters:
        prompt: "Name: {{ item.name }}\nURL: {{ item.url }}"
$ orchestrator validate examples/enhanced/fact_checker_enhanced.yaml
Undefined variable: 'item'
Undefined variable: 'index'

Scale

From the catalogue triage of the 25 Undefined variable failures: item appears in 4 files and index in 2. The same triage found the remaining distribution:

count name
14 output_path
5 current_timestamp
4 item
2 index
2 each current_date, and several pipeline-specific names

output_path is a genuinely undeclared parameter and a different problem. current_timestamp / current_date are names the runtime does not provide and are correctly refused (verified in test_runtime_context.py).

Why this needs care rather than an allowlist

Adding item/index to a global known-names set would accept {{ item }} outside any loop, where nothing binds it — trading a false positive for a false negative, which is the trade rejected in #461.

The fix wants loop scope: the names are bound inside a step that declares for_each / foreach / create_parallel_queue / action_loop, and inside that step's nested steps:, and nowhere else. core/template_scope.py already tracks Jinja-level scope; this is the pipeline-level equivalent and should be declared once and shared, the way template_references now is — not implemented separately in each validator.

Not to do

Do not make DataFlowValidator reject them to match. It is the one that is right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions