Implement structured output variables (vars field)

[jeremymanning]

Problem

Issue #307 specifies structured output variables that are missing from the current implementation:

steps:
  - id: analyze_data
    vars:
      - summary: "Brief summary of analysis results"
      - statistics: {"mean": 0.0, "std": 0.0, "count": 0}
      - recommendations: ["List of recommended actions"]
      - confidence_score: None  # Default format - save as string

Solution Required

Implement structured variable system with LangChain integration:

1. YAML Specification Support

• Parse vars field in step definitions
• Support format descriptions and example objects
• Variable access via pipeline.step_name.var_name syntax

2. LangChain Structured Outputs

• Generate Pydantic models from variable definitions
• Enforce output structure during model execution
• Type validation and conversion

3. Variable Management Integration

• Store structured outputs in existing VariableManager
• Template resolution with structured data access
• Cross-step variable dependencies

Implementation Details

Variable Definition Formats

vars:
  - simple_string: None                    # String output (default)
  - formatted_text: "Markdown formatted result"  # Format hint
  - json_object: {"key": "value"}        # JSON structure example
  - typed_list: ["item1", "item2"]      # List format example
  - complex_structure:                     # Nested structure
      name: "string"
      value: 0
      items: []

LangChain Integration

• Auto-generate Pydantic models from variable definitions
• Use structured output parsers during model execution
• Handle parsing errors and validation failures

Acceptance Criteria

• vars field parsing in YAML compiler
• Pydantic model generation from variable definitions
• LangChain structured output integration
• Variable access via dot notation
• Template resolution with structured data
• Type validation and error handling
• Documentation with examples

Files to Modify

• src/orchestrator/compiler/yaml_compiler.py - YAML parsing
• src/orchestrator/execution/variable_manager.py - Structured storage
• src/orchestrator/models/ - Structured output integration
• src/orchestrator/core/pipeline.py - Variable access API

Priority: High - Core data management requirement from #307

[jeremymanning]

No longer relevant after Claude Skills refactor (issue #426)

[jeremymanning]

Re-verified: still missing.

grep -rn '"vars"\|'\''vars'\''' src --include='*.py' → 0 hits. step.variables is read at execution/engine.py:765 but is a different, unrelated mechanism.

Scope note: the original title says "with LangChain integration". LangChain is no longer on the canonical path, so the LangChain coupling is dropped — the ask is structured output variables, by whatever means fits the current architecture.

---

Reopened during a repository-wide issue audit. This issue was closed on 2025-10-18 with "No longer relevant after Claude Skills refactor (issue #426)", but that refactor never shipped a working replacement — its compilers are orphaned and unreachable from orchestrator run (see the audit comment on #426). The gap described here is in the canonical YAMLCompiler path, which that refactor never touched. Evidence above was re-verified against main today.