Skip to content

Add examples/supported/: examples tested as product behaviour (#104) - #445

Open
jeremymanning wants to merge 1 commit into
mainfrom
feat/supported-examples
Open

Add examples/supported/: examples tested as product behaviour (#104)#445
jeremymanning wants to merge 1 commit into
mainfrom
feat/supported-examples

Conversation

@jeremymanning

Copy link
Copy Markdown
Member

Step 3 of the sequence. Partially addresses #104.

examples/ holds 111 pipelines of which 3 compile. They are the project's primary documentation surface, so a reader has no way to tell which parts of the syntax are real.

examples/supported/ is the answer: a small set tested as product behaviour. Every file must

  1. compile under orchestrator validate
  2. execute through the CLI
  3. execute through the Python API
  4. produce the same normalised result document from both surfaces
  5. produce the artifacts and declared outputs its expectations record

tests/test_supported_examples.py enforces all five and fails if a file is added without an entry declaring what it should do — so the set cannot grow past its coverage. 26 tests over 5 examples.

Example Demonstrates
01_hello_filesystem typed parameters, templating, dependencies, declared outputs
02_parallel_fanout_fanin independent steps sharing an execution level, joined
04_conditions condition:, on_false routing, skipped steps
06_failure_policy timeout, bounded retry, on_failure: continue, honest exit code
07_templates_and_outputs step-result references and declared outputs

Each exercises semantics pinned in #441#444, which is why they can be real specs rather than aspirations.

06_failure_policy exits 1 on purpose. Not every supported example is a success case, and the expectations table records the intended exit code rather than assuming zero — a pipeline containing a failing step is supposed to say so.

Writing 04_conditions found two validator defects

Both would hit any user writing the documented conditional syntax:

{{ content | length > 10 }} was reported as an unknown filter called "length > 10". The filter name was taken as everything after the pipe up to a parenthesis, so any operator following a filter became part of its name. It is now the leading identifier.

A condition referencing a pipeline parameter was reported as a dependency on a non-existent task. The dependency validators only ever received the task list, so they could not know content was a declared parameter rather than a missing step. Declared parameter, input and loop-variable names are now passed in.

I found these only because I wrote an example using the syntax the docs describe — which is the argument for this directory existing.

What these fixes did not do

Neither moved the existing catalog: still 108 failing, now out of 116. Those failures are elsewhere. I am stating this explicitly because "fixed two validator bugs" invites the assumption that the catalog improved, and it did not.

Deliberate absences

The numbering follows the planned set, so gaps read as intentional:

  • 03_data_etl, 05_loop — the tools and for_each/while exist but are not yet pinned by contract tests.
  • 08, 09, 11 — model pipelines; they cannot run hermetically in the blocking layer and belong in the live job.
  • 10_model_fallback — absent on purpose. There is no model fallback: select_model raises rather than substituting a model the pipeline did not ask for (Pin timeout and retry semantics; establish that fallback fails closed #444). An example must not be written for behaviour that does not exist, and writing one would have quietly created pressure to implement silent model substitution — the exact thing the Dartmouth cost-control work exists to prevent.
  • 1214 — checkpoint/resume, sub-pipelines, web research; each needs its own acceptance job first.

Not in this PR

The other 108 examples are untouched. Moving them to experimental/ and archive/ is mechanical but touches every docs link that references them, and it is a separate change from establishing the tested set. This PR makes the supported set exist; quarantining the rest is the follow-up.

Numbers

before after
Blocking suite 515 passed 541 passed, 13 skipped, 0 failed, 0 xfailed
Collection 3310 3336
Examples validating 3 / 111 8 / 116 (5 new, 3 pre-existing)
ruff / uv lock --check / git diff --check / docs drift clean

`examples/` holds 111 pipelines of which 3 compile. They are the
project's primary documentation surface, so a reader has no way to tell
which parts of the syntax are real.

examples/supported/ is the answer: a small set that is tested as product
behaviour. Every file in it must validate, execute through the CLI,
execute through the Python API, produce the same normalised result
document from both surfaces, and produce the artifacts and declared
outputs its expectations record. tests/test_supported_examples.py
enforces all five and fails if a file is added without an entry
declaring what it should do, so the set cannot grow past its coverage.

Five to start, each exercising semantics pinned in the preceding PRs:
typed parameters and declared outputs; parallel fan-out with a dependent
join; `condition:` with `on_false` routing and skipped steps; timeout,
bounded retry and `on_failure: continue`; step-result references.

06_failure_policy exits 1 on purpose. Not every supported example is a
success case, and the expectations table records the intended exit code
rather than assuming zero -- a pipeline containing a failing step is
supposed to say so.

Writing 04_conditions found two validator defects, both of which would
have hit any user writing the documented conditional syntax:

  - `{{ content | length > 10 }}` was reported as an unknown filter
    called "length > 10". The filter name was taken as everything after
    the pipe up to a parenthesis, so any operator following a filter
    became part of its name. It is now the leading identifier.

  - A condition referencing a pipeline *parameter* was reported as a
    dependency on a non-existent task. The dependency validators only
    ever received the task list, so they could not know that `content`
    was a declared parameter rather than a missing step. Declared
    parameter, input and loop-variable names are now passed in.

Neither fix moved the existing catalog: still 108 failing, now out of
116. Those failures are elsewhere, and this PR does not claim otherwise.

10_model_fallback from the planned set is deliberately absent. There is
no model fallback -- select_model raises rather than substituting a
model the pipeline did not ask for -- and an example must not be written
for behaviour that does not exist. The README records why, along with
the other planned-but-absent numbers, so the gaps read as deliberate.

Blocking suite 515 -> 541 passed, 13 skipped, 0 failed, 0 xfailed.
Collection 3310 -> 3336.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jeremymanning

Copy link
Copy Markdown
Member Author

CI 9/9 green. The legacy tally moved by +3, which is above the ±2 noise floor I had established, so I checked rather than dismissing it. It is the same flakiness family.

main this PR
failed 471 474
passed 1849 1846
errors 189 189
deselected 614 640

deselected +26 is this PR's new tests. The three newly-failing:

tests/orchestrator/quality/reporting/test_analytics.py::TestQualityInsight::test_quality_insight_creation
tests/orchestrator/quality/reporting/test_metrics.py::TestMetricsSnapshot::test_snapshot_creation
tests/orchestrator/quality/reporting/test_metrics.py::TestQualityMetric::test_quality_metric_creation

Evidence they are flaky rather than caused by this change:

Revising the noise floor upwards, again

On #443 I put the run-to-run swing at "at least ±2", based on two tests flipping. It is at least ±3, and the cluster is larger than the individual tests I had named — quality/reporting looks like a single unstable module rather than a handful of unrelated flaky tests.

That makes the aggregate legacy count less useful than I have been treating it. The named -rfE diff is doing all the real work; the tally is only a prompt to run it. Quarantining or fixing tests/orchestrator/quality/reporting/ would make the number meaningful again, and is worth its own issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant