Skip to content

Paradigm completion: fault tolerance and recovery on the Flink runtime #6

Description

@Ugbot

The bar: state as a materialized view over the runtime own ordered log plus checkpoints — a separate durable event log is not required.

Two runtimes clear it. agentic-pekko uses real Pekko Persistence (EventSourcedBehavior + journal) and has a test proving the transcript survives entity restart while brain invocations stay flat. agentic-clj uses real Datomic with working as-of time travel.

The flagship Flink runtime does not. enableCheckpointing, CheckpointingMode, setStateBackend and CheckpointedFunction appear zero times in the framework — only in a build-excluded demo, and there with the non-durable HashMapStateBackend.

Supporting gaps:

  • PollingSource splits are explicitly stateless (serialize() returns new byte[0]), so every non-Kafka channel is at-most-once across a restart.
  • No sink implements SupportsCommitter/TwoPhaseCommittingSink.
  • Savepoint restore of an A2A bridge stream NPEs (snapshot restored with a null serializer arg).
  • Both Flink-state vector memories call getMapState() from non-keyed ProcessFunctions, so they throw in open() and are unreachable through the shipped corpus/retrieval API.

Deliverable: checkpointing configured (or explicitly documented as a caller responsibility), a durable state backend option, sources that actually snapshot offsets, and a test that restarts from a checkpoint and asserts the conversation view is intact.


Roadmap epic. Detailed findings, file:line citations and sub-tasks live in the AI-grind tracker (project AGS), which is the source of truth for this work. Local dev — build, test, tracking, profiling — is managed with AI-grind.


Tracked as AGS-6 in the devtools-mcp tracker.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions