perf: add hybrid S3 writes and shared lease keepalive

[zTz01]

PR #50 Overview / PR #50 概述

• Pull request: Tele-AI/Fluxon#50
• Title: perf: add hybrid S3 writes and shared lease keepalive
• Branch: perf/s3-hybrid-kv-writes → main
• Snapshot date: 2026-08-04
• Change scope: 34 files, +12,657 / -1,267

English

Summary

PR #50 upgrades FluxonFS S3 object I/O from a mostly sequential, chunk-by-chunk RPC path to a size-aware bounded pipeline. Small objects are completed with one RPC, while large objects use write sessions with bounded buffering, concurrent batch delivery, backpressure, and an explicit finalize barrier. Large payload batches use Fluxon KV by reference regardless of whether the Controller and target Agent share an Owner: KV uses shared memory locally and its Transfer Engine across Owners or machines. Raw data RPC remains an operational fallback.

The PR also strengthens the lifecycle around temporary KV keys, lease keepalive, background tasks, and FS-before-KV shutdown. On the read side, it adds bounded concurrent piece reads, holder-backed extraction of cached bytes, and TCP_NODELAY for small HTTP responses.

Motivation

The previous S3 write path performed parent-directory checks and creation, then sent object data through repeated write RPCs before truncating the destination to its final size. Fixed RPC overhead dominated small-object writes, while large-object writes paid for many sequential data calls and additional payload copies.

Large write sessions introduce a second requirement: temporary KV references and holder-backed frames must remain valid until all consumers finish, and KV memory must not be unmapped before FluxonFS releases every holder. This PR treats data flow, temporary-key cleanup, lease keepalive, and shutdown ordering as one end-to-end lifecycle.

Main changes

1. Hybrid S3 write path

• Objects smaller than 4 MiB remain buffered in the Gateway and are committed through one typed put_small_object RPC. The Agent validates permissions, creates missing parent directories, truncates the destination, and writes the full payload in that RPC.
• At exactly 4 MiB or above, the writer opens a long-lived write session. The threshold is evaluated while streaming, so the request does not need to declare its final size in advance.
• PutObject, each UploadPart, and the final assembly performed by CompleteMultipartUpload use the hybrid writer.
• The large-object path uses approximately 32 MiB submissions, frames of at most 8 MiB, batches of up to four frames, a default 128 MiB Controller inflight window, a 32 MiB Agent queue, and four sender tasks per target Agent.
• Queue admission applies byte-bounded backpressure. A data ACK means that frames were accepted or deduplicated by the Agent; finalize(expected_frames, final_size) is the barrier that waits for all writes and sets the final file length.

2. KV-backed payload path with raw-RPC fallback

KV-ref is the reference-based payload path used by a write session. Instead of carrying the batch payload in the Agent RPC, the Controller stores the batch under a temporary Fluxon KV key, then sends only the key, offset, frame boundaries, source-node generation, and sequence metadata. The Agent resolves that reference with ordinary kv_get and keeps the returned payload owner alive until the corresponding frames have been written.

• Object size is the only normal-path selector: objects below 4 MiB use put_small_object; objects at or above 4 MiB enter a write session whose batches first use KV-ref. FS does not select a different transport for cross-Owner placement.
• With the same Owner, KV can expose the shared mmap allocation locally. Across Owners or machines, kv_get invokes KV's internal Transfer Engine and returns payload accessible to the Agent. FluxonFS does not expose or directly select the Transfer API.
• PreferredSubCluster is only an optional source-local placement hint. If no eligible local preference exists, KV chooses placement; FluxonFS still uses KV-ref.
• The Agent validates the source generation and deterministic temporary-key identity, obtains the payload from KV, slices frames, and places them into the bounded write queue.
• A lease, kv_put, reference-RPC, kv_get, or reference-validation failure permanently downgrades that session to raw RPC. The retry uses the same sequence and offset, allowing the Agent to deduplicate a batch whose success ACK was lost.
• Export-scoped Agent snapshots are cached in the local Controller. Normal S3 requests choose an Agent and call it directly; the central FS Master registry is consulted only when the snapshot is stale or routing reports NodeNotFound.

3. Temporary-key cleanup and shared lease keepalive

• A Controller-owned cleanup actor records each temporary key before starting kv_put, making it the single final cleanup owner even when the put result is unknown or deletion fails.
• Committed keys are explicitly deleted with retry. Commit-unknown keys continue to be rechecked because a timed-out put may complete after an earlier delete attempt.
• The first write session lazily allocates a Controller-scoped 180s lease. Later sessions and batches reuse the same lease and generation, keeping lease and periodic keepalive overhead O(1) with respect to the number of sessions.
• FluxonFS and MQ reuse the generic bounded LeaseKeepaliveActor implementation, but they do not share one runtime actor or one lease. Generation checks prevent an old inflight keepalive from reviving an unregistered lease.

4. S3 read-path improvements

• GetObject maintains a sliding window of bounded concurrent piece reads while preserving offset order in the HTTP body.
• On a KV cache hit, FluxonFS validates the encoded FlatDict, locates the requested bytes field, and creates a holder-backed slice instead of materializing the full dictionary and copying the field into an intermediate buffer.
• The Axum listener enables TCP_NODELAY, reducing Nagle/delayed-ACK latency for small objects and small Range responses.

5. Retryable shutdown and ownership barriers

• Framework shutdown now has persistent phase state, atomic task admission, registered-task join barriers, and retryable progress after a failed shutdown attempt.
• FluxonFS registers as a KV pre-shutdown dependent. KV shutdown waits for FS to stop admission, drain sessions and background work, release holder-backed frames, and acknowledge completion before KV can unmap shared memory.
• Controller and target-Agent shutdown paths retain unfinished session/task authority on timeout so a later attempt can continue converging.
• Python/PyO3 uses one background shutdown owner for explicit close(), repeated close attempts, and Drop fallback, avoiding competing cleanup paths.

Data-path summary

Condition	Selected path	Completion contract
Object size < 4 MiB	Single put_small_object RPC	Parent creation, overwrite, and full write complete in one Agent operation
Object size >= 4 MiB	Bounded write session with KV-ref batches	finalize waits for every expected frame and applies the final length
Same Owner	KV local shared-memory path	Agent receives a DataRef and holder-backed slices
Different Owner or machine	KV internal Transfer Engine	Agent receives the same DataRef; FS does not carry the payload RPC
KV-ref operation fails	Raw data RPC fallback	Same sequence/offset is retried; the session remains in raw mode
S3 GET	Bounded concurrent piece reads	Results are emitted to HTTP in object-offset order

Compatibility and explicit boundaries

• The S3 HTTP contract, directory-listing behavior, KV-miss policy, and cache-refill semantics remain unchanged.
• No new public put_start / put_commit API or second FluxonFS-specific keepalive subsystem is introduced.
• Backends without both small-put and write-session support retain the existing chunked write/truncate fallback.
• KV-ref removes large payloads from FS DataRef RPCs, but it is not end-to-end zero copy. Local placement still copies the Controller batch into KV memory, cross-Owner placement additionally performs KV-managed transfer, and the current FS read interface eventually materializes a Vec<u8>.
• finalize waits for write_all completion and sets the final length; it does not call fsync or syncfs.
• Writes target the final path directly. abort releases session resources but does not roll back bytes already written to restore an older object version.

Performance and validation

The documentation added by this PR includes a single-node rclone v1.60.1 comparison between FluxonFS S3 and Alluxio S3 Proxy. Both systems were run three times across 2,000 × 4 KiB, 256 × 1 MiB, and 32 × 256 MiB objects at concurrency 1, 8, and 32, for 162 validated cases in total.

• In the reported results, FluxonFS led all 18 persisted-PUT and cold-read size/concurrency combinations: 36%–726% for persisted PUT and 7%–661% for cold reads.
• The strongest cold-read gains appeared for small and medium objects under concurrency. Large-object cold reads remained closer to the underlying NVMe limit.
• For hot reads, FluxonFS led 4 KiB object throughput by approximately 8%–43%; medium and large sequential-throughput results were generally close, except for a larger 1 MiB gain at concurrency 8.

The user-facing validation covers bucket checks, listing, upload, download, delete, Range GET, and Multipart Upload through rclone. At the snapshot date, the latest PR check rollup is green for wheel packaging, two-virtual-node CI, large-scale MQ CI, and documentation image construction. The PR remains open and is awaiting review approval.

Documentation included

• FluxonFS S3 object I/O design
• Turn a local folder into an S3 service with FluxonFS

---

中文

概述

PR #50 将 FluxonFS S3 对象 I/O 从以顺序分块 RPC 为主的链路，升级为按对象大小选择的有界流水线。小对象通过一次 RPC 完成，大对象进入带有界缓存、并发 batch 发送、背压和明确 finalize 屏障的 write-session。大 payload 无论同 Owner 还是跨 Owner 都通过 Fluxon KV 引用传递：KV 在本地复用共享内存，跨 Owner 或跨机时使用内部 Transfer Engine；Raw RPC 只作为运行失败兜底。

本 PR 同时强化了临时 KV key、lease keepalive、后台任务以及 FS-before-KV 关闭顺序的生命周期。在读取侧，新增有界并发 piece 读取、holder-backed 缓存数据提取，以及面向小型 HTTP 响应的 TCP_NODELAY 优化。

背景与目标

原 S3 写入路径会逐级检查和创建父目录，再通过多次 write RPC 发送对象数据，最后将目标文件截断到最终长度。对小对象来说，固定 RPC 开销占比很高；对大对象来说，顺序数据调用和额外 payload 复制限制了吞吐。

大对象 write-session 还带来另一项要求：临时 KV 引用和 holder-backed frame 必须存活到所有消费者完成，FluxonFS 释放全部 holder 之前，KV 共享内存不能被卸载。因此，本 PR 将数据流、临时 key 回收、lease keepalive 和关闭顺序作为同一个端到端生命周期处理。

核心改动

1. S3 混合写入链路

• 小于 4 MiB 的对象保留在 Gateway 内存中，通过一次 typed put_small_object RPC 提交。Agent 在同一个操作内校验权限、创建缺失父目录、截断目标文件并写入完整 payload。
• 对象累计大小恰好达到或超过 4 MiB 时，writer 打开长生命周期 write-session。阈值在流式接收过程中判断，不要求请求预先声明最终大小。
• PutObject、每个 UploadPart，以及 CompleteMultipartUpload 的最终对象组装都接入混合写入器。
• 大对象链路使用约 32 MiB submit、不超过 8 MiB 的 frame、每 batch 最多 4 个 frame、Controller 默认 128 MiB session 在途窗口、Agent 32 MiB 队列，以及每个目标 Agent 4 个 sender task。
• 队列接纳执行按字节计量的有界背压。数据 ACK 只表示 Agent 已接纳或去重 frame；finalize(expected_frames, final_size) 才是等待全部写入完成并设置最终文件长度的屏障。

2. KV-backed payload 路径与 Raw RPC 降级

KV-ref 是 write-session 使用的引用式 payload 路径。Controller 不把 batch payload 直接放进 Agent RPC，而是先以临时 key 将 batch 写入 Fluxon KV，再通过 RPC 发送 key、offset、frame 边界、source node generation 和 sequence 等元数据。Agent 使用普通 kv_get 解析该引用，并让返回的 payload owner 一直存活到对应 frame 写入完成。

• 常态路径只按对象大小选择：小于 4 MiB 使用 put_small_object；达到或超过 4 MiB 后进入 write-session，每个 batch 先走 KV-ref。FS 不因跨 Owner 改选其他传输方式。
• 同 Owner 时，KV 可直接暴露本地共享 mmap allocation；跨 Owner 或跨机时，kv_get 由 KV 内部 Transfer Engine 搬运并返回 Agent 可访问的 payload。FluxonFS 不暴露或直接选择 Transfer API。
• PreferredSubCluster 只作为可选的 source-local 放置提示；没有合适的本地提示时由 KV 自行选址，仍然使用 KV-ref。
• Agent 校验 source generation 和可重算的临时 key 身份，从 KV 取得 payload，切分 frame 后放入有界写队列。
• lease、kv_put、引用 RPC、kv_get 或引用校验失败时，session 会永久降级到 Raw RPC。重试沿用相同 sequence 和 offset，因此 Agent 可以去重“数据已接纳但成功 ACK 丢失”的 batch。
• 本地 Controller 按 export 缓存 Agent 快照。正常 S3 请求直接选取 Agent 并发起 RPC；只有快照过期或路由返回 NodeNotFound 时，才访问中央 FS Master registry 刷新快照。

3. 临时 key 回收与共享 lease keepalive

• Controller-owned cleanup actor 在 kv_put 开始前记录临时 key，即使 put 结果不确定或 delete 失败，它仍然是唯一最终回收者。
• 已确认提交的 key 通过显式 delete 和重试回收；commit 状态不确定的 key 会持续复查，因为超时的 put 可能在较早的 delete 之后才完成。
• 第一个 write-session 懒申请 Controller 级 180s lease；后续 session 和 batch 复用同一 lease 与 generation，使 lease 和周期 keepalive 开销相对 session 数保持 O(1)。
• FluxonFS 与 MQ 复用通用的有界 LeaseKeepaliveActor 实现，但不共享同一个运行时 actor 或 lease。generation 校验可阻止旧的在途 keepalive 复活已注销 lease。

4. S3 读取链路优化

• GetObject 使用有界并发 piece 滑动窗口，同时保证 HTTP Body 仍按对象 offset 顺序输出。
• KV 缓存命中时，FluxonFS 会校验编码后的 FlatDict、定位目标 bytes 字段，并创建 holder-backed slice，不再先物化完整字典并把字段复制到中间缓冲区。
• Axum listener 启用 TCP_NODELAY，降低小对象和小 Range 响应受到 Nagle/delayed-ACK 交互影响时的额外延迟。

5. 可重试关闭与资源所有权屏障

• Framework shutdown 新增持久阶段状态、原子 task admission、注册任务 join 屏障，以及关闭失败后的可重试推进能力。
• FluxonFS 作为 KV 的 pre-shutdown dependent 注册。KV shutdown 必须等待 FS 停止接纳、排空 session 与后台工作、释放 holder-backed frame，并确认完成后才能卸载共享内存。
• Controller 和目标 Agent 在关闭超时时保留未完成 session/task 的 authority，使后续尝试能够继续收敛。
• Python/PyO3 对显式 close()、失败后的再次 close 和 Drop fallback 使用同一个后台 shutdown owner，避免多条清理路径相互竞争。

数据路径摘要

条件	选择的路径	完成语义
对象大小 < 4 MiB	单次 put_small_object RPC	Agent 在一个操作内完成父目录创建、覆盖和完整写入
对象大小 >= 4 MiB	使用 KV-ref batch 的有界 write-session	finalize 等待所有预期 frame，并设置最终长度
同 Owner	KV 本地共享内存路径	Agent 接收 DataRef 并使用 holder-backed slice
跨 Owner 或跨机	KV 内部 Transfer Engine	Agent 接收同一种 DataRef，FS RPC 不携带 payload
KV-ref 运行失败	Raw data RPC 兜底	使用相同 sequence/offset 重试，session 后续保持 Raw 模式
S3 GET	有界并发 piece 读取	结果按对象 offset 顺序写入 HTTP Body

兼容性与明确边界

• S3 HTTP 契约、目录列举行为、KV miss 策略和缓存回填语义保持不变。
• 不新增公开的 put_start / put_commit API，也不引入第二套 FluxonFS 专用 keepalive 子系统。
• backend 未同时支持 small-put 和 write-session 时，继续使用原有的分块写入与 truncate 回退路径。
• KV-ref 将大 payload 移出 FS DataRef RPC，但不是端到端零拷贝：本地放置仍需把 Controller batch 复制到 KV 内存，跨 Owner 还会发生 KV 管理的远程传输，当前 FS 读取接口最终仍会物化 Vec<u8>。
• finalize 等待 write_all 完成并设置最终长度，但不调用 fsync 或 syncfs。
• 写入直接作用于最终路径；abort 只释放 session 资源，不会回滚已经写入的字节并恢复旧对象版本。

性能与验证

本 PR 新增文档包含 FluxonFS S3 与 Alluxio S3 Proxy 的单机 rclone v1.60.1 对比。两套服务分别重复运行 3 次，覆盖 2,000 × 4 KiB、256 × 1 MiB、32 × 256 MiB 三种规模和并发 1、8、32，共完成 162 个通过内容与磁盘 I/O 校验的案例。

• 报告结果中，FluxonFS 在持久化 PUT 与冷读的全部 18 组大小/并发组合中领先：持久化 PUT 为 36%–726%，冷读为 7%–661%。
• 小文件和中文件在并发冷读中的收益最明显；大文件冷读更接近底层 NVMe 吞吐上限。
• 热读场景下，FluxonFS 的 4 KiB 对象吞吐领先约 8%–43%；除 1 MiB、并发 8 的较明显优势外，中大文件顺序吞吐整体接近。

面向用户的验证通过 rclone 覆盖 bucket 检查、列举、上传、下载、删除、Range GET 和 Multipart Upload。截至快照日期，PR 最新检查汇总中 wheel 打包、双虚拟节点 CI、大规模 MQ CI 和文档镜像构建均为绿色。PR 当前仍处于开放状态，等待审阅批准。

随 PR 提供的文档

• FluxonFS S3 对象 I/O 设计
• 把一个本地文件夹变成 S3 服务：FluxonFS 的目录、对象与缓存链路

[ActivePeter]

发现 3 个需要在合并前修复的生命周期与资源回收问题，详见行内评论。