[lstein]

Summary

A session could deadlock when an If branch, or the frames an If runs in, depended on an Iterate or a For. next() returned None, is_complete() stayed False, and no error was raised, so the queue item hung. The simplest case is #9484's src → iterate → body → collect → if.true_input with a literal condition. In a related case, a per-item If ran a For on its rejected branch, and the error from that For was recorded.

This PR builds on #9843. It keeps #9843's frame-cardinality and axis rules: an If runs once per item of every iterator open at it, whichever branch it selects. Within those rules:

• The rejected branch is skipped. A literal If waits only for the ancestors of its selected branch and for the iterators that give it its frames, so an Iterate or For inside the rejected branch no longer blocks it.
• A For on a branch is gated together with its loop body. This includes a For nested in a gated For's body. A per-item If therefore runs none of that loop for items that reject the branch.
• Frames are found through For outputs. Iterators that reach an If through a For's final output count as frame sources for gating, empty-frame detection and validation.
• Empty frames finish. An If whose innermost frame Iterate is empty produces no execution, and downstream Collects get [].
• A computed condition waits for the If's frame iterators, so the If can no longer close with zero executions before they run.
• Shared nested axes are supported. A condition outside the branches' nested axes runs over the innermost axis that both branches share.
• Unsupported shapes fail before anything runs, raising UnsupportedIfIterationAxesError. These include unrelated open axes arriving through different If inputs, a root condition choosing between branches with different deeper axes, and a literal If pairing an axis reopened after a Collect whose length can differ from the closed axis. A reopen is accepted only when the Iterate reads directly from the closing Collect (or a chain of Collects), that Collect takes one item input over one nested chain of axes, and nothing is appended through its collection input. An unrelated axis that reaches a literal If only through the unselected branch's For output adds no frames and is allowed.

per-item If over ["", "x"], true ← Collect(For.output_collection), false ← fallback
before: For and its body run for item 0 too; an error there is recorded
after:  item 0 → "fallback" with nothing run on the branch; item 1 → the For's results

Related Issues / Discussions

Closes #9484
Closes #9917

Builds on #9843, whose description lists #9818–#9821 as addressed. Semantics follow the maintainer's decision in this PR's discussion: keep the existing If cardinality, and reject unsupported axis relationships explicitly.

QA Instructions

• uv run --no-sync pytest -n 4 tests → 14526 passed, 211 skipped, 7 xfailed. The graph tests also pass under several PYTHONHASHSEED values.
• New regression tests in tests/app/services/shared/test_execution_engine_differential.py assert completion plus exact frames and values, or the rejection error before invocation. All of these shapes run on the compatibility scheduler, and each test asserts which scheduler ran. Every test fails on main except two: the cross-product and the closed-then-reopened axis tests guard against over-rejecting shapes main already runs.
• Each production change was reverted on its own, and at least one named test fails each time.
• About 240 additional If/Iterate/Collect/For graph shapes were compared against main on both schedulers:
  • No shape that main completes produces a different value.
  • No shape is rejected that main completes.
  • No Collect runs ahead of an If that never ran.
  • Only two shapes still stall, and main stalls on them too (see Review, bug(graph): computed-condition If stalls when its branch reaches an empty Iterate through a For #9918).
• -m slow tests/app/services/shared/test_for_scheduler_performance.py, medians of 3 runs interleaved with main: all within 1% (for example, 15.02s against 15.00s, and 5.26s against 5.24s).
• ruff check and ruff format --check pass.

Review

• Fixed during review: the For-output blocker, including the variant where a For is nested in the body. Also fixed:
  • Stalls on Ifs inside For bodies and on Iterate → For → If that this branch had introduced.
  • Over-rejection of cross products carried by one input.
  • Collect-closed axes being treated as related.
  • An If that closed empty before its inner iterators ran.
• Known limitations, all present on main:
  • A per-item If fed directly by For.output_collection, with no Collect, still runs the For for items that reject that branch. The If can only be prepared once that For has finished.
  • With a computed condition, an If whose branch reaches an empty Iterate only through a For's final output stalls (bug(graph): computed-condition If stalls when its branch reaches an empty Iterate through a For #9918).
  • A root-level node that feeds only a rejected For's body still runs. This wastes work but changes no results.
  • A computed per-item condition whose branch is a Collect of a Collect keeping axes deeper than the condition is accepted and stalls (bug(graph): computed-condition If stalls on a Collect-of-Collect branch deeper than its condition #9922).
• Rejected although main completes them correctly: reopens with a node between the Collect and the Iterate (its length cannot be checked statically), reopens of a Collect whose item crosses unrelated axes, and reopens through more Collects than the item has axes. Each of these can drop or misgroup items in a closely related shape, so they fail explicitly.

Compatibility / Rollout

No persisted-state or API changes. Graphs whose behaviour changes previously hung, ran work on rejected branches, or produced wrong values. Unsupported axis relationships now raise UnsupportedIfIterationAxesError before any node runs.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

[JPPhoto]

Merge blockers:

• invokeai/app/services/shared/graph_materializer.py:1597-1611,1825-1830: with an open Iterate feeding an If branch, the materializer rejects the If while activation gates the iterator on that unresolved If. The new direct-input test and both literal-branch cases stall. The PR description lists this shape as unsupported, but the new tests require completion. Effect: these validated graphs never finish. Likelihood: deterministic for the test graphs. Recovery: rewrite the graph and retry, or implement support. Test: the focused module reports 765 passed, 6 failed; the direct-input case stalls with trace=[].

• invokeai/app/services/shared/graph_materializer.py:1550-1557: per-item If executions cannot attach root-frame branch outputs because attachment requires the source frame to equal the target frame. The new root-source test stalls after conditions and iterate nodes. Effect: the graph never finishes or collects its selected root values. Likelihood: deterministic for per-item conditions with root-frame branch sources. Recovery: rewrite the graph and retry, or support selecting root-frame values for per-item Ifs. Test: both scheduler-mode cases fail in the focused module.

Other findings/issues:

• tests/app/services/shared/test_execution_engine_differential.py:3552-3563: this test passes when the scheduler stalls before preparing any If; the sibling case does so with an empty trace. Effect: it can pass without exercising the behavior named by the test. Likelihood: the current graph shape. Test: assert meaningful progress or the expected blocked state alongside the frame assertion.

Suggestions:

• Consider parameterizing the literal-branch cardinality test over both scheduler modes; it currently checks only the default mode.

[lstein]

@JPPhoto thanks, all three findings were real. Fixed in e27f6fd; your new tests pass unchanged on both schedulers.

Open Iterate feeding an If branch (graph_materializer.py:1597-1611,1825-1830). You were right that the PR's tests demanded a shape it declared unsupported. That shape is now supported. When an If's frames depend on the branch it selects, its branch sources are admitted or rejected from the condition value (a literal, or the executed condition source) instead of from an If execution that can't exist yet. The iterator runs, and the If is then created in the selected branch's frames. Your direct-input test and both literal-branch cardinality cases pass. The cardinality test now also runs on the compatibility scheduler, as you suggested.

Per-item If with root-frame branch sources (graph_materializer.py:1550-1557). This one had two parts:

• A root source is admitted when any item frame selects it, and rejected once every item's condition is known and none does.
• Attachment now allows a value prepared in a parent frame to feed a per-item If.

test_per_item_if_can_select_values_from_root_frame passes in both modes.

test_if_is_not_materialized_in_parent_frame_while_branch_frame_is_open passing on a stall. All 8 cases were in fact stalling. It is replaced by test_if_takes_frames_from_selected_branch_left_open, which asserts completion and the exact per-frame sink values. When the frameless branch wins, the If correctly runs once at (), so the old "never at ()" assertion was wrong for those cases.

Probing beyond these turned up more shapes, all fixed with regression tests. The updated description lists them. Two are worth calling out:

• An If whose condition lies outside its iterator frames could be created with no condition edge at all, so it silently read False.
• A per-item condition with a nested open Iterate created each If execution four times.

Each of these shapes also failed on main (all 6 of your new tests stall there too). The remaining limitations are all broken on main as well and are filed as #9818, #9819, #9820 and #9821.

[JPPhoto]

Merge blockers:

• invokeai/app/services/shared/graph_if_activation.py:147-151: For.output_collection already loses the For frame in _get_edge_depth(), then Collect.item subtracts another frame. With a per-item condition [False, True] and a true branch containing For -> Collect(For.output_collection), the branch runs for the false item too: I observed body executions at (0,0) and (0,1) while that item’s sink used the fallback. An error in that inactive body is recorded despite the false condition. Effect: inactive branch work can consume resources or fail a valid workflow. Likelihood: reproducible for mixed per-item conditions and this supported branch shape. Recovery: cancel and retry after correction; completed work may remain. Test: extend the For-output regression with [False, True]; assert no For/body execution for (0,), including an injected body failure.

Other findings/issues:

• tests/app/services/shared/test_execution_engine_differential.py:3471-3487: the for-output and collection-input cases report generic_allowed=False; force_compatibility_scheduler=False therefore uses _ExecutionScheduler, same as True. Effect: these tests do not verify the claimed generic-scheduler coverage. Likelihood: confirmed for both cases. Test: assert the selected scheduler, or use a graph that actually qualifies for the generic scheduler.

[lstein]

@JPPhoto thanks for the re-review. Both findings are real. I'm fixing the For.output_collection double-subtract (with the [False, True] regression and an injected body failure) and the scheduler-coverage assertions.

Merging #9843 into this branch exposed a semantic conflict I'd like you to settle before I go further, because both sides are your tests.

A. If cardinality when only the unselected branch is per-item. Shape: source[0,1] -> Iterate -> per-item node -> If.true_input, root -> If.false_input, literal condition=False, If.value -> Collect.item.

• test_literal_if_result_uses_selected_branch_frame_cardinality (your e8ff5df on this PR) expects the If to run once at (), giving ["root-value"].
• test_literal_if_selecting_a_root_branch_skips_the_per_item_branch (Fix graph execution edge cases #9843, now on main) expects trace.count("if") == 2, giving ["root", "root"].

B. A branch that runs per inner item under a per-outer-item condition. Shape: outer Iterate -> collection -> inner Iterate -> body -> If.true_input, condition per outer item, a root value on false_input, If -> Collect.

• On main, test_if_branch_deeper_than_its_condition_completes_or_is_rejected[nested-branch-root-false] expects UnsupportedIfIterationAxesError before anything runs.
• On this PR, test_per_outer_item_if_keeps_nested_iterate_frame_open_on_selected_branch (6 cases) runs it and keeps the inner-item frames.

With a draft merge, every main test passes. All 24 failures are PR cases that follow the PR's model, where the If takes its frames from the branch it selects. Main's model takes them from every iterator above the If, so the result count doesn't depend on the condition.

Which model do you want?

• Selected branch: I change the two main tests above and finish the merge.
• Static frames: I drop or rewrite the PR's tests for A and B and the dependent cases. The PR shrinks to whatever deadlock fixes main still lacks. On unmodified main, 35 of the PR's 80 test cases fail, many as stalls. I haven't yet separated which of those stalls depend on A.

[JPPhoto]

@lstein, I would preserve the current cardinality behavior for A in this PR. When the literal condition selects the root input, I expect the If to retain the surrounding per-item cardinality while leaving the unselected per-item computation inactive. Changing that to one root result is coherent, but it changes the output shape of a graph that already has defined behavior. I would decide that separately and specify how downstream grouping and empty branches behave.

For B, I would keep the explicit error for this asymmetric case, where only the selected branch has deeper open axes. I would preserve the existing supported case where both branches share those deeper axes. Broadcasting the outer condition into a deeper branch while the fallback remains at the outer frame needs a clear output cardinality contract before we support it.

I do not mean that If frames must always be static: completed Collect branches already preserve their selected frame groups. My recommendation is to keep the currently supported behavior by shape, make unsupported shapes fail clearly, and handle any broader selected-branch cardinality change as a separate semantics decision.

[lstein]

@JPPhoto thanks. I merged main with your rule: If cardinality stays as it is, and unsupported axis shapes fail explicitly. Both of your #9843 tests (A and B) pass unchanged. I deleted this PR's tests that assumed the If takes its frames from the selected branch, and dropped the code behind them.

Your findings:

• For-output blocker: fixed. The depth arithmetic was only half the cause. The For and its loop body weren't counted as part of the branch, so nothing gated them. A For on a branch is now gated together with its body, including a For nested in that body. There are regressions for the [False, True] shape with an injected body failure, and for the nested-For variant.
• Scheduler coverage: you were right that those shapes only run on the compatibility scheduler. I dropped the fake generic variants, and each test now asserts the scheduler that actually ran.

New rejections: a literal If now also gets the pre-invocation check, and the check runs through For final outputs. UnsupportedIfIterationAxesError is raised for unrelated open axes that reach the If through different inputs, and for a root condition choosing between branches with different deeper axes. A cross product carried by a single input is still allowed. I compared about 240 graph shapes against main: nothing main completes is rejected or changes value.

One decision for you: a literal If that selects an axis reopened after a Collect gives the same wrong groups as main when the reopened collection has a different length. In g_closed_reopen3_T, t:x is dropped and group [2] comes out empty. One narrow rule would reject that shape instead: allow the reopen only when the new Iterate reads straight from the closing Collect. That turns a silently wrong result into an error, but it changes the result of a graph main completes, so I left it out. Do you want it?

Known limitations, all shared with main:

• A For whose final output feeds a per-item If directly still runs for items that reject that branch.
• With a computed condition, an If whose branch reaches an empty Iterate only through a For output stalls.
• The reopened-axis wrong groups described above.

[JPPhoto]

Some fixable items:

1. invokeai/app/services/shared/graph_if_runtime.py:37,100-115 rejects a previously supported literal-If graph. The unchanged test_graph_static_nested_ifs_ignore_unselected_loop_branch_context selects a_for.output_collection with condition=False, but validation also sees the unrelated axis from the unselected b_for branch and raises UnsupportedIfIterationAxesError. The expected output has groups (0,) = ["a0", "a1"] and (1,) = ["a2"]. Linux and macOS CI fail on this test. Preserve this behavior and keep the regression test.

2. The PR description still says an If runs in the selected branch’s frames and describes tests for that cardinality model. The latest decision and current implementation keep cardinality static and reject some axis shapes. Update the description to match.

@lstein To answer your question, yes, I would reject unsafe reopened-axis shapes because they can silently drop items. For example, a workflow can collect processed images, append a reference through Collect.collection, then iterate the enlarged collection while an If pairs items against the earlier axis. The graph can complete with an empty group and a missing item. This is plausible batch composition, so I would not dismiss it as negligible. The proposed direct Collect-to-Iterate rule needs one more constraint: that Collect must not also append items through its collection input, or it does not guarantee matching cardinality.

[lstein]

@JPPhoto thanks. Both items are addressed.

1. Over-rejection of test_graph_static_nested_ifs_ignore_unselected_loop_branch_context: fixed; the test passes unchanged. For a literal If, an unrelated axis that reaches it only through the unselected branch's For output adds no frames, so it is no longer treated as a cross product. The full suite passes locally (14526 passed, after merging your latest main merge).
2. Description: rewritten for the static-cardinality model and the explicit rejections.

Reopened axes: done as you suggested, so this now closes #9917. A literal If may pair an axis reopened after a Collect only when:

• the Iterate reads directly from the closing Collect (chains of Collects are followed);
• that Collect has one item input and nothing through collection, as you pointed out;
• the item's axes form one nested chain. A Collect over a cross product keeps one axis open and groups by it, so a reopen of it does not have the closed axis's length.

Anything else raises UnsupportedIfIterationAxesError before a node runs. Only the downstream direction pairs: selecting the closed axis over its reopening is rejected.

Against main, the rejected shapes split into two groups. Some silently drop or misgroup items on main, like your batch example. The rest complete correctly on main but cannot be told apart statically from a broken neighbour: a length-preserving node between the Collect and the Iterate, a Collect over a cross product, and more Collects than the item has axes. Validation results don't depend on hash seed, and the Collect walk is iterative and cached, so deep or lattice-shaped Collect graphs validate in linear time.

[JPPhoto]

Approved.