Return a typed PipelineResult with an execution trace (#332)

[jeremymanning]

Implements #332's result contract. The result.qc() half of that issue is not here — see the end.

Correcting something I told you

I said declared outputs were "parsed, never resolved". That was wrong — _extract_outputs renders them, and has all along.

The actual defect is worse and less visible: declared outputs changed the shape of the return value.

# pipeline without outputs:
{"write_note": {...}, "read_back": {...}}

# same method, pipeline with outputs:
{"steps": {...}, "outputs": {...}}

Two shapes from one method, distinguishable only by inspecting keys — and a pipeline with a step called outputs collided with the second outright.

Mostly recovering discarded state, not new machinery

Field	Already existed on	Was
per-step status / error	Task.status / .error	discarded
step timing	Task.started_at / .completed_at	discarded
retries	Task.retry_count	discarded
dependency + execution order	Pipeline.get_execution_levels()	discarded
run identity + start	executor context	discarded

All of it was computed, then dropped at return final_result.

PipelineResult is a Mapping, so result["step_id"] returns the raw value exactly as before — the several thousand assertions that index results directly are untouched. The trace arrives as attributes.

status and success are different questions — and conflating them was a live bug

status records whether the task finished; success whether it worked. A tool returning {"success": False} without raising does finish.

My first implementation selected failures on status == FAILED, and the golden exit-code test went green when it should have been red — a pipeline with a failing step exiting 0. That is precisely the failure mode this project keeps having to re-fix, and it reappeared inside the change meant to make failures legible.

StepResult.success now accounts for both, and the CLI's exit code consults it.

Serialisation

The CLI emitted json.dumps(results, default=str). That default= silently stringified anything unserialisable, so the output was neither stable nor round-trippable — CLI/API equivalence was impossible to assert, not merely unasserted.

It now serialises to_dict(), which needs no coercion. normalized() drops the execution id and wall-clock times — the only fields that legitimately differ between runs — and the golden test compares whole normalised documents from both surfaces, as you asked, rather than one nested value.

Test expectations that changed, and why none is a weakening

1. Golden/CLI tests read the typed document instead of the old flat map. The contract changed by request.
2. My own assertion that a failed step has status == "failed" contradicted the status/success distinction this PR introduces. The code was right; my expectation was wrong.
3. A len(stdout.splitlines()) < 30 ceiling stood in for "no log noise" while the payload was compact. The typed document is legitimately longer, so that bound had started measuring the document rather than the noise. Round-tripping the whole of stdout already proves there is no interleaved logging; an explicit check for DEBUG/INFO/WARNING/Traceback replaces the magic number.

Numbers

	before	after
Blocking suite	476 passed	490 passed, 13 skipped, 0 failed, 0 xfailed
Collection	3271	3285
ruff / uv lock --check / git diff --check / action-docs drift	—	clean

Not in this PR

result.qc() — #332 also asks for an orchestrator model to grade output quality. That is a quality-evaluation feature, not a result contract, and it cannot sit inside a deterministic acceptance contract; your own quality section says an LLM judge "may supplement — but never replace" deterministic scoring. It wants its own issue.

Fallback events. retries is a count, and skips are recorded, but nothing in the runtime currently records why a fallback happened — there is no emission point to read. Adding one is a change to the execution path rather than to the result type, so it belongs with whatever introduces fallback policy.

provider is read from the step envelope and is currently None for every model step, because the envelope records model_used and no provider. The field is in the contract and the plumbing is one line at the point the envelope is built — but that line belongs with the model-selection work, not smuggled in here.

[jeremymanning]

CI 9/9 green, and the legacy backlog improved.

Via the annotation route (repos/OWNER/REPO/check-runs/<job-id>/annotations), comparing against the post-merge main run rather than a PR-branch one:

	main @ post-#441	this PR
failed	475	473
passed	1845	1847
errors	189	189
skipped	200	200
total red	664	662

Two legacy tests recovered, errors unchanged, nothing regressed. deselected moves 575 → 589 (+14), which is this PR's new contract/e2e-marked tests being excluded from the legacy selection by construction.

The two recoveries are plausibly tests that asserted on a failing step and previously got the error handler's retry policy where the reason belonged — but I have not attributed them individually, and I am not going to claim it without the per-test diff.