AIP-104: Task Iteration and Dynamic Task Batching - #62922

Open

dabla wants to merge 120 commits into

apache:mainfrom

dabla:feature/dynamic-task-iteration

Commits on May 23, 2026

refactor: Implemented Dynamic Task Iteration
dabla
committed
875d8a5
refactor: Renamed apply_upstream_relationship to register_with_dag
dabla
committed
968aeb6

Commits on May 26, 2026

refactor: Fixed MappedOperator serialization tests
dabla
committed
dff7b43
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
0b62a5b
refactor: Make update of env variable thread-safe in _execute_task method of task runner module
dabla
committed
c1627eb
refactor: Refactored _run_tasks so it uses a while loop instead of recursive function calls
dabla
committed
ce134e6
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
fd63862
fix: Preserve partition_size through serialization for partitioned tasks
dabla
committed
2cd47b7
refactor: Updated Uv.lock file
dabla
committed
995ae33

Commits on May 27, 2026

refactor: Reformatted TestPartitionedOperator
dabla
committed
b88afe9

Commits on May 28, 2026

Commits on May 29, 2026

refactor: Clone the context before each indexed task execution to have thread safety
dabla
committed
d5bed18
refactor: Make sure MappedIterableOperator doesn't retry itself
dabla
committed
85dee45
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
a07a359
refactor: Extracted clone_context from IterableOperator as it should be part of the context module
dabla
committed
c80e2ae
refactor: Fixed timeout resolution on IterableOperator
dabla
committed
50e9ed9
refactor: Re-organized imports of expandinput
dabla
committed
705ef4f
refactor: Updated uv.lock file
dabla
committed
cea8a25
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
234d228
refactor: If DateTimeTrigger isn't available, then failed tasks won't be deferred and will be executed immediately
dabla
committed
01c4ad8
refactor: Reformatted TestCurrentContext
dabla
committed
8204f49
refactor: Reformatted TestIterableOperator
dabla
committed
651238b
refactor: Fixed assertions TestExpandInput
dabla
committed
501d058
refactor: Fixed TestPartitionedOperator
dabla
committed
5c8bfc9
refactor: Updated uv.lock file
dabla
committed
ac7c530
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
5453f06

Commits on May 30, 2026

Commits on May 31, 2026

refactor: Fixed typing of optional DateTimeTrigger import in IterableOperator
dabla
committed
1616fb4

Commits on Jun 1, 2026

refactor: Improve iterable task execution with AsyncAwareExecutor and streaming map
dabla
committed
b118943

Commits on Jun 2, 2026

Merge branch 'main' into feature/dynamic-task-iteration
dabla
committed
39af3b8
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
933da36
Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
fcfb3f4

Commits on Jun 4, 2026

Commits on Jun 6, 2026

Commits on Jun 8, 2026

Commits on Jun 9, 2026

Commits on Jun 10, 2026

refactor: Reformatted _run_tasks method in IterableOperator
dabla
committed
bb5663a

Commits on Jun 12, 2026

Commits on Jun 13, 2026

refactor: Use suppress instead of try/except in shutdown method of AsyncAwareExecutor
dabla
committed
2bbc479

Commits on Jun 19, 2026

refactor: Fixed calculation of reschedule_date in IterableOperator
dabla
committed
a588578

Commits on Jun 23, 2026

Commits on Jun 25, 2026

Commits on Jun 30, 2026

Commits on Jul 1, 2026

refactor: Remove _length from attrs in ExpandInput to avoid serialization as this is a runtime property which we don't want to be serialized
dabla
committed
ce93835

Commits on Jul 2, 2026

Commits on Jul 3, 2026

Commits on Jul 6, 2026

Commits on Jul 9, 2026

Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
0976ee9

Commits on Jul 16, 2026

Commits on Jul 22, 2026

Commits on Jul 23, 2026

Merge branch 'main' into feature/dynamic-task-iteration
dabla
authored
532e5d4
Update uv.lock
dabla
authored
9905a1c