Pipeline validates, then cannot run: inferred dependencies are computed and discarded

[jeremymanning]

Summary

A pipeline whose step references an undeclared output of another step passes orchestrator validate and then fails at run time. The data-flow validator computes the dependency correctly, and the compiler discards it.

This came out of writing the four unknown-output integration cases requested in review.

Reproduction

id: uo
name: UO
steps:
  - id: make
    tool: filesystem
    action: write
    parameters:
      path: "./out_a.txt"
      content: "A ran"
  - id: use
    tool: filesystem
    action: write
    parameters:
      path: "./out_b.txt"
      content: "make.path={{ make.path }}"

$ orchestrator validate p.yaml
✓ p.yaml is valid
  pipeline: uo
  tasks: 2
    - make
    - use

$ orchestrator run p.yaml
PipelineExecutionError: Task 'use' failed and policy is 'fail':
Parameter 'content' of tool 'filesystem' still contains unresolved
template references: '{{ make.path }}'.

What each layer believes

The data-flow validator does infer the dependency:

DataFlowValidator().validate_pipeline_data_flow(...)
#   valid    : True
#   warnings : [('undefined_output', "Task 'make' does not declare its outputs, ...")]
#   graph    : {'make': set(), 'use': {'make'}}      <-- the edge is here

The compiled pipeline does not:

YAMLCompiler().compile(...)
#   make: dependencies=[]
#   use:  dependencies=[]                            <-- the edge is gone

So use is scheduled in the same level as make, make's result is not in
context yet, the render fails, and the step is stopped. The runtime behaves
correctly here (it fails closed rather than writing an empty value — #445);
the problem is that validation promised the pipeline was fine.

Second, smaller defect: the warning is invisible

The warning that hints at this never reaches stdout. It is emitted through the
logger, so orchestrator validate prints ✓ ... is valid and a script
capturing stdout sees nothing at all. A warning nobody can see is not a
warning.

The design fork

Two defensible resolutions, and they are not equivalent:

A. Infer the dependency and schedule on it. Makes the compiler agree with
the graph the validator already builds. Convenient, matches what an author
plainly meant, and no existing pipeline breaks. Risk: ordering becomes
implicit, and a cycle introduced by a template reference is now possible.

B. Refuse at validation. Require an explicit dependencies: entry when a
step references another step's output. Explicit, no hidden ordering. Risk: it
will reject pipelines that currently run, wherever the reference happens to be
in a later level anyway.

I recommend A, because the validator already computes the edge and using
it costs nothing, whereas B converts a class of working pipelines into errors.
But this changes scheduling, so it wants a decision rather than my assumption.

Not to do

Do not "fix" this by making the undeclared-output case an error. It is
deliberately a warning: when a task does not declare its outputs, the
validator has no basis for rejecting a field name, and asserting otherwise is
the false-positive class removed in #448/#450/#461.

[jeremymanning]

Correction to the reproduction in this issue, and confirmation of the underlying defect.

The pipeline I filed uses {{ make.path }}. That is not a field the filesystem tool returns — it returns {'result': {'action': ..., 'path': ..., 'size': ...}, 'success': True, 'error': None}, so the correct reference is {{ make.result.path }}.

My reproduction therefore conflated two separate problems, and would have failed at run time even with the dependency bug fixed. I should have checked the producer's actual output shape before filing.

The underlying defect is real and confirmed independently. With the correct reference:

                          run exit    out_b.txt
before (main @ 9615f61)      1        MISSING
after  (canonical graph)     0        made=./out_a.txt

use references make and declares no dependencies:. Before the fix the edge was computed by three different mechanisms and given to none of them, so both steps landed in the same execution level, make's result was not in context, and the render failed.

A further finding while mapping the mechanisms: DependencyValidator._get_task_dependencies never scanned parameters at all — only for_each, condition, while, until and create_parallel_queue.on. Its _extract_template_dependencies also matched only six hard-coded suffixes (result|output|data|content|status|metadata), so make.path was not even recognised as a task reference. The commonest reference in the language was invisible to the graph that checks for cycles.

The {{ make.path }} case is now its own test, asserting the opposite: it must still fail, because ordering a step correctly does not invent a field the producer never returned. What is wrong there is only that validate stays silent — the data-flow validator warns, and the warning never reaches stdout. That is the second half of this issue and is being addressed separately.