[Feature]: Add durable terminal-scope API for cleanup and compensation

[zhongkechen]

What would you like?

Add an SDK-supported durable terminal-scope API for registering cleanup and compensation actions that run when a scope reaches a true terminal outcome, but do not run when the current invocation merely suspends.

The exact API name and shape should be decided through an RFC. An illustrative Python API is:

def review_with_microvm(
    scope_context: DurableContext,
    terminal: DurableTerminalActions,
) -> ReviewResult:
    vm = scope_context.step(
        lambda step_context: launch_microvm(),
        name="launch-microvm",
    )

    terminal.cleanup(
        lambda cleanup_context: terminate_microvm(vm["microvm_id"]),
        name="terminate-microvm",
    )

    terminal.compensate(
        lambda compensation_context: cancel_review(vm["microvm_id"]),
        name="cancel-review",
    )

    callback = scope_context.create_callback(name="review-complete")
    scope_context.step(
        lambda step_context: dispatch_review(vm, callback.callback_id),
        name="dispatch-review",
    )
    return callback.result()


result = context.terminal_scope(
    review_with_microvm,
    name="review-with-microvm",
)

The important capability is the lifecycle contract, not the proposed names:

Scope outcome	Compensation actions	Cleanup actions
Success	Do not run	Run
Application/operation failure	Run in reverse registration order	Run
Explicit cancellation/termination	Policy-controlled; preferably enabled by default	Run when the runtime can execute terminal work
Durable suspension/replay boundary	Do not run	Do not run

Problem

Python normally encourages cleanup through try/finally, context managers, and contextlib.ExitStack. Those models are unsafe around durable operations because the SDK currently suspends by raising SuspendExecution, which derives from BaseException and unwinds the synchronous call stack.

A normal except Exception correctly does not intercept suspension, but Python still executes:

• active finally blocks;
• context-manager __exit__ methods;
• callbacks registered with ExitStack;
• generator-based context-manager cleanup.

This pattern can release a resource while the durable execution is still logically using it:

vm = context.step(launch_microvm_step, name="launch-microvm")

try:
    callback = context.create_callback(name="review-complete")
    context.step(
        lambda step_context: dispatch_review(vm, callback.callback_id),
        name="dispatch-review",
    )
    return callback.result()
finally:
    context.step(
        lambda step_context: terminate_microvm(vm["microvm_id"]),
        name="terminate-microvm",
    )

When callback.result() suspends, finally runs and terminates the MicroVM. The same issue applies to any suspending operation, including wait, wait_for_condition, invokes, retry delays, and suspension within map or parallel branches. It is not callback-specific.

Users can manually duplicate cleanup after the success path and in except Exception:

try:
    result = perform_durable_work_that_may_suspend(context)
except Exception:
    context.step(cleanup_step, name="cleanup")
    raise

context.step(cleanup_step, name="cleanup")
return result

This works because SuspendExecution derives from BaseException, not Exception. However, the workaround:

• duplicates orchestration code;
• becomes difficult to maintain with multiple acquired resources;
• makes reverse-order compensation cumbersome;
• is easy to implement inconsistently across nested, map, and parallel scopes;
• does not provide a clear cancellation policy;
• cannot use familiar Python resource-management constructs safely;
• encourages users to catch BaseException or use finally, either of which can mishandle SDK control flow;
• cannot express the intent as clearly as an SDK lifecycle primitive.

Goals

• Provide an idiomatic way to express work that must occur on logical completion/failure rather than invocation exit.
• Make suspension a distinct non-terminal outcome and guarantee that terminal actions do not run because of it.
• Support general orchestration scopes, not only callbacks or resource acquisition.
• Support both unconditional terminal cleanup and failure-only compensation.
• Preserve deterministic replay and stable durable-operation identity.
• Ensure terminal actions are themselves durable, retryable, observable, and replay-safe.
• Work inside top-level handlers and isolated child contexts used by map/parallel operations.
• Provide precise typing for scope bodies and terminal actions.
• Leave room for a language-neutral lifecycle contract that other Durable Execution SDKs can expose idiomatically.

Non-goals

• Guarantee that cleanup runs after infrastructure-level hard termination when no invocation is available to execute it.
• Replace external leases, TTLs, or reapers for resources that must eventually be reclaimed under every failure mode.
• Provide exactly-once external side effects. Cleanup and compensation steps still require normal durable-step idempotency.
• Treat suspension as failure, cancellation, or scope completion.
• Make ordinary context managers suspension-aware. Python context managers are tied to lexical stack exit, not durable terminal state.
• Expose SuspendExecution as application control flow that users should catch.

Possible Implementation

1. Scope and registration model

terminal_scope could execute a deterministic callable against an isolated child durable context and a terminal-action registry:

class DurableTerminalActions(Protocol):
    def cleanup(
        self,
        func: Callable[[StepContext], None],
        *,
        name: str | None = None,
        config: StepConfig | None = None,
    ) -> None: ...

    def compensate(
        self,
        func: Callable[[StepContext], None],
        *,
        name: str | None = None,
        config: StepConfig | None = None,
    ) -> None: ...


class DurableContext(Protocol):
    def terminal_scope(
        self,
        func: Callable[
            [DurableContext, DurableTerminalActions],
            T,
        ],
        *,
        name: str | None = None,
        config: TerminalScopeConfig | None = None,
    ) -> T: ...

The API should align with existing step and run_in_child_context argument ordering and naming conventions after review; the signatures above are illustrative.

Registrations should be deterministic declarations. On every replay, execution reruns the scope body and reconstructs the same registrations before reaching the same suspension or terminal path. Registration names and ordering must remain stable for a given checkpoint history.

2. Suspension handling

With the current synchronous Python execution model, the scope implementation can distinguish internal suspension from application failures:

try:
    result = body(child_context, actions)
except SuspendExecution:
    # Suspension is not terminal. Run no compensation or cleanup.
    raise
except Exception:
    run_compensation(actions)
    run_cleanup(actions)
    raise
else:
    run_cleanup(actions)
    return result

This is conceptual pseudocode. The implementation must preserve TimedSuspendExecution, fatal SDK control-flow exceptions, KeyboardInterrupt, SystemExit, cancellation signals, and other BaseException subclasses appropriately. The public API should not require user code to import or catch internal suspension exceptions.

The implementation should centralize outcome classification rather than relying on arbitrary broad except BaseException handling.

3. Durable identity and replay

The scope should own stable namespaces for:

• body operations;
• each registered compensation;
• each registered cleanup;
• terminal phase/progress if explicit checkpointing is required.

Terminal actions must execute through durable child contexts or equivalent SDK-managed operations. Completed actions must consume checkpoints on replay and must not repeat their bodies.

The design must define behavior when:

• the body suspends before all registrations are reached;
• the body fails after acquiring several resources;
• a compensation action suspends;
• a cleanup action suspends;
• a compensation or cleanup action fails and is retried;
• replay observes some terminal actions completed and later actions pending;
• a scope is nested inside another terminal scope;
• a scope is used within each item of a map or branch of a parallel operation.

Registration should not rely only on ephemeral in-memory state after a terminal phase begins. Either deterministic replay must reconstruct the complete registration set before resuming terminal work, or the SDK must durably record sufficient scope metadata to resume it safely.

Captured objects used by terminal-action callables must be derived from deterministic inputs or checkpointed results. Documentation should make clear that registration does not make arbitrary captured mutable state durable.

4. Ordering

Suggested defaults:

• compensations run in reverse registration order, matching the usual acquisition/rollback model;
• cleanups run in reverse registration order so resources unwind like a stack;
• cleanup still runs if a compensation fails, where execution policy permits;
• multiple failures are preserved through a documented primary/secondary or aggregate-error model.

Configuration could later permit forward or parallel execution, but the initial API should favor deterministic sequential behavior.

5. Error semantics

The RFC should specify:

• whether all terminal actions are attempted after one fails;
• how the original body failure and traceback are preserved;
• how cleanup/compensation failures are chained or aggregated;
• which retry configuration applies to each action;
• whether a failed cleanup changes an otherwise successful scope into a failed scope;
• what happens if terminal actions exhaust retries;
• how cancellation arriving during terminal processing is handled.

A reasonable initial policy is:

• on body failure, preserve the body failure as primary;
• attempt remaining compensation and cleanup actions;
• attach terminal-action failures through explicit SDK error fields, exception chaining, or ExceptionGroup as appropriate;
• on body success, a cleanup failure fails the scope;
• allow per-action retry configuration using existing step retry semantics.

The choice of ExceptionGroup requires care: application exceptions, SDK errors, and internal BaseException control flow must not be combined in a way that changes suspension handling.

6. Cancellation and hard termination

Cancellation must be defined separately from suspension.

The API could expose a policy such as:

TerminalScopeConfig(
    compensate_on_cancellation=True,
    cleanup_on_cancellation=True,
)

The contract must acknowledge that terminal actions can run only when the Durable Execution service schedules code to process the terminal transition. For externally terminated compute or lost invocations, users still need resource-native leases, TTLs, idempotent deletion, or a reaper as a safety net.

7. Relationship to context managers

A context-manager-shaped public API would be attractive:

with context.terminal_scope("review") as terminal:
    ...

However, a normal __exit__ is invoked on SuspendExecution, so it cannot by itself distinguish lexical exit from durable terminal completion unless the SDK context manager explicitly recognizes and rethrows internal suspension without running terminal actions.

Even with that special handling, a callback-style terminal_scope(func, ...) may be safer because it can:

• provide an isolated child context;
• own the entire body/terminal operation namespace;
• classify outcomes in one SDK-controlled boundary;
• avoid suggesting that arbitrary Python context managers are durable;
• type the body result directly.

The RFC should compare both forms, but the lifecycle semantics must not depend on normal context-manager cleanup behavior.

8. Observability

Execution history and telemetry should make the lifecycle explicit:

• scope started;
• scope suspended without terminal actions;
• terminal outcome selected;
• compensation started/completed/failed;
• cleanup started/completed/failed;
• scope terminal processing completed.

This is important for diagnosing cleanup that is delayed, retried, or partially completed.

9. Compatibility and rollout

This can be introduced as an additive API, so it should not be a breaking change. Because it creates a new durable orchestration primitive and potentially a cross-SDK lifecycle contract, it should go through an RFC and conformance review.

If a new operation/checkpoint type is introduced, backward compatibility and older-runtime behavior must be specified. If implemented initially as SDK composition over run_in_child_context and existing steps, operation identity and upgrade behavior still require tests.

10. Testing and acceptance criteria

• Suspension from every suspending operation type runs no registered terminal action.
• TimedSuspendExecution is treated as suspension, not failure.
• Other internal BaseException control-flow/fatal signals retain their documented behavior.
• Replay reconstructs registrations deterministically.
• Success runs cleanup through durable/checkpointed semantics.
• Failure runs compensation and cleanup in the documented order.
• Cancellation follows an explicit, tested policy.
• A terminal action can itself suspend and resume.
• Completed terminal actions are skipped on replay.
• Terminal-action retries do not repeat already completed actions.
• Nested terminal scopes behave deterministically.
• Map/parallel branches use isolated scope contexts and do not interfere.
• Multiple body/compensation/cleanup failures preserve documented errors and tracebacks.
• Type checking covers generic results and valid terminal-action signatures.
• API docs warn that cleanup side effects remain at-least-once and should be idempotent.
• Documentation distinguishes terminal scopes from finally, with, and ExitStack.
• Examples include the MicroVM lifecycle use case.
• Language-neutral requirements are proposed if this becomes a cross-SDK capability.

11. Open design questions

• Should cleanup run on success and failure while compensate runs only on failure/cancellation, or should outcome-specific hooks be exposed directly?
• Is registration-before-acquisition required, or may a resource be acquired and then registered?
• Must registrations be checkpointed when declared, or is deterministic reconstruction sufficient?
• Should terminal actions receive the body result or failure?
• Should actions be captured callables reconstructed during replay or named descriptors persisted by the SDK?
• Should terminal processing be a new service-visible operation type or SDK composition over existing primitives?
• What is the cancellation contract supported by the service today?
• How should cleanup behave when a map uses early completion and abandons unfinished branches?
• Should compensation be included in the first version or added after a cleanup-only terminal scope?
• Should the API be callback-based, context-manager-based with special suspension handling, or support both?
• What naming best distinguishes durable terminal lifecycle from ordinary Python lexical scope?

Is this a breaking change?

No. The proposal is an additive API.

Does this require an RFC?

Yes. It introduces new lifecycle semantics, replay rules, error aggregation, cancellation policy, and cross-SDK considerations.

Additional Context

The proposed terminal_scope shape is a synthesis for Durable Execution rather than a direct copy of another SDK. Related established patterns include:

• the Saga pattern and reverse-order compensation;
• Temporal Java's Saga.addCompensation(...) / compensate();
• Cadence Java's similar Saga helper;
• workflow-engine compensation handlers such as BPMN compensation;
• asynchronous resource APIs such as Reactor usingWhen, which distinguish completion, error, and cancellation cleanup but are not durable/replay-aware;
• Python's ExitStack, which provides registration and reverse-order cleanup but is lexical and therefore runs during suspension unwinding.

The durable-specific requirement is to distinguish suspension from all terminal outcomes. A normal invocation boundary must not trigger cleanup or compensation.

Related issues:

• Corresponding Java SDK feature proposal: [Feature]: Add durable terminal-scope API for cleanup and compensation aws-durable-execution-sdk-java#646
• Java documentation for the current finally hazard: [Docs]: Document finally cleanup behavior during durable suspension aws-durable-execution-sdk-java#645
• Cross-language documentation pattern: Add pattern: Cleanup on completion or failure (not on suspension) aws-durable-execution-docs#131
• Detailed cross-language discussion: Add pattern: Cleanup on completion or failure (not on suspension) aws-durable-execution-docs#131 (comment)