feat: add filesystem-backed SerDes stage

[zhongkechen]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Closes #463.

Implements Approach A from ADR-005 and provides the SerDes context requested in #509.

Description

• Extend SerDes with then(SerDesStage), returning the SerDes interface so fluent callers do not depend on the concrete pipeline implementation.
• Define a pipeline as exactly one root SerDes value codec followed by zero or more reversible String → String SerDesStage components. Serialization runs in declaration order and deserialization runs in reverse order.
• Require stages to use self-identifying formats: deserialization reverses recognized valid input, rejects recognized malformed or unsupported input, and passes unrecognized input through unchanged. This lets raw external payloads reach the root value codec without pipeline-specific bypass control flow.
• Keep root SerDes value codecs context-free and pass SerDesContext explicitly only to SerDesStage and nested BinarySerDesStage calls. Serialization-stage context exposes the root input object through originalValue(); deserialization-stage context has a null originalValue(). No SerDes-specific thread-local storage is used.
• Make CloudDurableTestRunner and LocalDurableTestRunner serialize initial invocation input with a separate context-free value codec. By default, they use the persisted SerDes when it is a plain value codec or the root value codec when it is composable; withInputSerDes(...) provides an explicit value-codec override. Persisted pipeline stages are not used to encode initial input.
• Add BinarySerDesStage for reversible byte[] → byte[] transformations and ComposableBinarySerDesStage to expose an ordered binary chain as one top-level string stage.
• Make both binary boundaries customizable through the same StringBinaryCodec interface. The processing-order builder uses startWith(codec), zero or more then(binaryStage) calls, endWith(codec), and build(). The outer stage adds a reserved versioned frame for reliable format recognition.
• Add RetryableSerDesException, RetrySerDesStage, and RetryBinarySerDesStage. The wrappers retry explicitly retryable failures using existing bounded RetryStrategy implementations and pass the same stage context to every attempt.
• Add FileSystemSerDesStage to the core SDK as a reversible SerDesStage for filesystem-backed payload storage. The stage, storage/path configuration, and preview APIs live in the dedicated software.amazon.lambda.durable.serde.filesystem Java package without adding a Maven module.
• Support string payloads, versioned envelopes, ALWAYS/OVERFLOW modes, immutable content-hashed files, attempt-aware operation identities, and pass-through of unrecognized input.
• Publish every offloaded payload to a unique path with one CREATE_NEW write. Publication does not use hard links or renames, so the implementation works on both EFS and S3 Files without overwriting data referenced by an earlier checkpoint.
• Add structured preview parity with the Python and TypeScript SDKs: include-all/exclude-all modes, include/exclude/mask selectors, anywhere/exact-path matching, configurable mask text, a default 4 KB budget, and a standalone preview builder. Custom preview callbacks receive both the preceding stage string and SerDesContext, including originalValue().
• Allow FileSystemSerDesStage to be followed by other stages that transform its inline or file-reference envelope. Filesystem overflow and preview-size checks apply at the filesystem stage boundary; later stages are responsible for their own size expansion.
• Harden filesystem references with producer execution/entity ownership, content-hash validation, filesystem-provider-correct path resolution, path containment, and symlink rejection.
• Carry stage context through durable runtime and persisted-result testing paths, including operation metadata, replay attempts, chained invoke results, executor-backed processing, binary substages, and retry attempts.
• Update the ADR, design, configuration, and SerDes guides for pipeline composition, stage-only context, serialization originalValue, binary stages, retry wrappers, filesystem stages, executor behavior, core SDK packaging, stage format recognition, structured previews, S3 Files compatibility, and independent test-runner input codecs.
• Document a binary-only top-level pipeline as a rejected alternative: it simplifies the type model and can benefit binary-heavy pipelines, but adds full-payload UTF-8 conversion and allocation to the common direct Jackson string path.

Demo/Screenshots

Not applicable. Configuration examples are included in the SerDes guide.

Checklist

• I have filled out every section of the PR template
• I have thoroughly tested this change

Testing

Coverage includes:

• Root SerDes plus SerDesStage pipeline creation, immutable appending, ordering, reverse processing, pass-through of unrecognized input, malformed recognized input, nulls, and failure metadata.
• API validation that only SerDesStage components can be appended after the root value codec and that fluent composition returns the SerDes interface.
• Naming and package audit confirming every named built-in stage implementation ends in Stage and filesystem-specific APIs live under software.amazon.lambda.durable.serde.filesystem.
• Explicit SerDesContext propagation through top-level stages, binary substages, executor-backed calls, and every retry attempt.
• Serialization originalValue() propagation to string stages, binary substages, retry wrappers, and filesystem preview callbacks; deserialization verifies that it is absent.
• Ordered binary serialization and reverse deserialization with custom boundary codecs and versioned format recognition.
• Inline and executor-backed execution, stage context propagation, caching, and executor isolation.
• String and binary retry selection, exhaustion, delay handling, interruption, and context propagation.
• Filesystem stage behavior, stages after filesystem storage, storage modes, envelope recognition and validation, immutable unique publication, providers without hard-link support, ownership, content-hash validation, provider-specific paths, path containment, and symlink rejection.
• Structured preview include/exclude/mask behavior, exact-path and anywhere matching, nested objects and arrays, custom masks, byte budgets, JSON pipeline integration, original-object access, and malformed input.
• Root input/output, step replay and retry attempts, child contexts, callbacks, invoke boundaries, and local/cloud runner behavior.
• Default and custom test-runner input value codecs, persisted-pipeline isolation, composable-input rejection, and filesystem integration without explicit input configuration.

Validated with:

• The full eight-module Maven reactor on Java 17
• Focused filesystem, pipeline, binary-stage, retry, and structured preview tests
• SDK Javadoc generation
• mvn spotless:check
• git diff --check

Cloud integration tests remain opt-in through -Dtest.cloud.enabled=true.

Examples

The SerDes guide, ADR, and design documentation include configuration examples.

[zhongkechen]

Addressed the remaining review findings and follow-up changes in c9d4390, 97f0e10, 6a813eb, 4164e46, fd92ee9, and 7f09c77:

• Preserved RetryableSerDesException from filesystem preview generation and added retry coverage.
• Added callback/invoke regression coverage for external errors with a missing errorType.
• Filesystem envelopes now include payloadDigest; deserialization verifies inline/file content integrity and requires content-addressed file paths to use the same digest.
• Split filesystem E2E deployment into one persistent shared VPC/EFS infrastructure stack and persistent Java-version-specific Lambda stacks. A prerequisite CI job updates the shared stack once; every Java matrix job imports the same subnet, security group, and access-point outputs and updates its Lambda stack in place.
• Added generator regression tests covering the ordinary, filesystem Lambda, and persistent infrastructure templates.
• Confirmed the older TIMED_OUT/STOPPED nested invoke finding was already addressed in e3bf9e4 with replay coverage.

Validation: targeted SDK tests (55 tests), examples test compilation, generator unit tests, all SAM/CloudFormation template-generation modes, Actionlint, Spotless, YAML parsing, and git diff --check.

[github-actions]

Codex AI review

[P1] Do not let payload bytes select the persisted SerDes. This frame is only a prefix, so any caller able to supply it bypasses the configured input codec without callee opt-in or trusted provenance. This can bypass validation/decryption and make filesystem envelopes execute under the cross-execution INPUT rule. Carry this signal in trusted backend metadata or authenticate it with explicit callee opt-in; otherwise continue using getInputSerDes(). Add spoof and prefix-collision tests.

[github-actions]

Codex AI review

[P2] Avoid writing files for serialization results that will not be checkpointed. ChildContextOperation normalizes results before checking isVirtual or whether its parent already completed, so FLAT map/parallel branches and late branches publish UUID files whose envelopes are discarded. High-cardinality workflows therefore leak files and storage I/O. Move that skip decision before persisted serialization or provide a normalization-only path that runs the value codec without storage stages, with FLAT/early-completion file-count tests.

[github-actions]

Codex AI review

[P3] Do not classify every IOException as transient. Secure traversal failures for symlink components, non-directories, and permission errors also arrive as IOException; wrapping them as retryable makes RetrySerDesStage repeat permanently invalid paths for the full backoff. Convert structural/path-security failures to SerDesException and reserve RetryableSerDesException for genuinely transient storage failures. Apply the same distinction to the read catch and add retry-wrapper symlink tests.

[github-actions]

Codex AI review

Found one high-severity input-boundary issue and two filesystem lifecycle/error-handling defects. Cloud EFS behavior remains covered only by opt-in tests.

Reviewed commit 0b38e868f9ab7797c80274554aa064ed028a7628. Workflow run