status`。
+2. **gc_ttl 语义**:Pull 路径中「`elapsed >= gc_ttl → OBJECT_HAS_LEASE`」的判定在 Push 下不再需要(对端写完即释放),已省略。
+3. **传输回退**:Push 仅在确认 transport 支持单边 WRITE 时启用;TCP/共享内存等应继续走 Pull。当前由 `MC_OFFLOAD_PUSH` 手动控制,尚未做 transport 能力自动探测。
+4. **构建/测试**:改动尚未在 Linux 目标上编译验证与端到端压测;Windows 开发机无法构建 Mooncake(依赖 RDMA/etcd 等)。
+```
diff --git a/docs/source/deployment/mooncake-store-deployment-guide.md b/docs/source/deployment/mooncake-store-deployment-guide.md
index 640aec3c2e..6b8f15620b 100644
--- a/docs/source/deployment/mooncake-store-deployment-guide.md
+++ b/docs/source/deployment/mooncake-store-deployment-guide.md
@@ -1062,6 +1062,9 @@ the deferred direct-mmap path is desired while the arena is otherwise enabled.
```bash
export MC_YLT_LOG_LEVEL=info
+export MC_YLT_LOG_PATH=logs/rpc.log
+export MC_YLT_LOG_MAX_FILE_SIZE=1048576000
+export MC_YLT_LOG_MAX_FILES=3
```
Available: `trace`, `debug`, `info`, `warn` (or `warning`), `error`, `critical`. When unset (or set to an unrecognized value), the level defaults to `warn`.
diff --git a/docs/source/design/distributed-metadata-plane-lightweight-coordinator.html b/docs/source/design/distributed-metadata-plane-lightweight-coordinator.html
new file mode 100644
index 0000000000..fb8d33464b
--- /dev/null
+++ b/docs/source/design/distributed-metadata-plane-lightweight-coordinator.html
@@ -0,0 +1,2066 @@
+
+
+
+ Mooncake 分布式元数据管理面轻量 Coordinator 架构设计
+
+
+
+ Mooncake 分布式元数据管理面轻量 Coordinator 架构设计
+
+ 1. 设计摘要
+
+ 推荐方案：
+ 采用 Redis Cluster 风格的固定 slot 分片和客户端直连路由，同时引入一个不在前台请求路径上的轻量 Coordinator。
+ Coordinator 只负责集群视图、slot 迁移计划、ownership version 与 fencing 发布、资源摘要聚合和后台任务预算调节；GetReplicaList、PutStart、PutEnd 等热路径请求仍由客户端直接访问 slot 所属 MasterGroup 的 current primary。
+
+
+ 1.1 术语与层次
+
+ 术语 本文中的唯一含义
+ MasterNode 一个可寻址的 Master 进程实例；是部署、租约和故障域单位。 MasterGroup 一个复制组；是日志提交、主备切换和一致性单位，包含一个 current primary 和若干 replica。 Slot 由 key hash 得到的固定路由编号。 SlotGroup 一组连续或离散 slot；是负载统计和组间迁移单位。 Slot owner 拥有 SlotGroup 的 MasterGroup，而不是某个永久固定的 MasterNode。 Group primary MasterGroup 当前接收强一致前台请求的 MasterNode，随 term 变化。
+
+
+ 下文若使用“Master 服务”，仅泛指 MasterNode 上运行的服务。路由、所有权或 failover 描述必须明确使用
+ MasterGroup、Group primary 或 MasterNode，避免“Master Shard”同时指进程、复制组和 slot 分片。
+
+
+ 1.2 Generation、Term、Revision 与 Fencing
+
+ 本设计不再把所有版本都称为 epoch，而是只区分三种语义：generation/term 是 fencing token，
+ 用于隔离旧 owner；revision 只用于快照发布、CAS 和增量 watch；sequence 只用于同一数据流内的样本排序。
+ 只有 generation/term 会决定请求是否有权改变状态，revision 和 sequence 都不能授予写权限。
+
+
+
+初始状态：SlotGroup S belongs to MasterGroup G1, assignment_generation = 7
+
+Client C1 cached: S -> G1, generation 7
+Coordinator migrates S: G1 -> G2, generation becomes 8
+
+C1 sends Put(key, owner=G1, assignment_generation=7)
+G1/G2 compares request generation with durable assignment:
+ 7 < 8 -> reject STALE_ASSIGNMENT or MOVED(S, G2, 8)
+
+Without assignment_generation:
+ delayed C1 or old G1 might continue modifying S after cutover,
+ creating two owners and divergent object metadata.
+
+
+
+ Generation/term 的作用不是发现故障，而是隔离旧状态。
+ lease/heartbeat 用于判断参与者可能失活；generation/term 用于在发生切换后证明请求仍属于当前一代。
+ 仅检测到旧 primary 断连是不够的，因为它可能仍在运行，只是发生了网络分区。
+
+
+
+
+
+ 字段
+ 类别
+ 作用域
+ 何时递增
+ 防止的问题
+ 旧值处理
+
+
+
+
+ view_revisionRevision
+ 整个 ClusterView
+ 任意已发布集群视图发生变化
+ 增量更新乱序、旧 view 覆盖新 view
+ 客户端忽略旧 view；watch 缺口时重新读取全量 view
+
+
+ node_incarnation_idIncarnation UUID
+ 一个 node_id 的进程 incarnation
+ MasterNode 每次启动生成新的 UUID
+ 重启前残留 lease/heartbeat 覆盖新进程状态
+ 拒绝旧 incarnation 的续租和上报
+
+
+ termFencing term
+ 一个 MasterGroup
+ 组内选出或授权新的 primary
+ 旧 primary 在网络分区后继续提交写
+ 返回 STALE_TERM 或 NOT_PRIMARY
+
+
+ assignment_generationFencing generation
+ 一个 SlotGroup
+ SlotGroup 从一个 MasterGroup 迁到另一个 MasterGroup
+ 迁移后 source group 或旧客户端继续修改对象元数据
+ 返回 STALE_ASSIGNMENT 或 MOVED
+
+
+ control_generationFencing generation
+ 一个 Storage Client 的控制归属
+ client control owner 从一个 MasterGroup 迁到另一个 group
+ 两个 group 同时接收 segment heartbeat、mount/unmount 或下发任务
+ 拒绝旧 heartbeat/控制写并返回新 control owner
+
+
+ segment_incarnationFencing incarnation
+ 一个物理 segment/allocator incarnation
+ segment 重新注册、重建 allocator 或不可兼容地 remount
+ 旧 reservation/descriptor 写入已经重建的物理空间
+ reservation 失败，重新查询 segment 并分配
+
+
+ resource_revisionRevision
+ 一版聚合 ResourceView
+ Coordinator 发布新的资源摘要
+ 资源视图乱序覆盖
+ 忽略旧摘要；它不单独提供空间分配正确性
+
+
+ sample_seqSequence
+ 一个 segment 的遥测流
+ 同一 segment_incarnation 每发布一个样本递增
+ 延迟到达的容量/带宽样本覆盖新样本
+ 丢弃旧样本并结合 timestamp 判断 freshness
+
+
+ budget_revisionRevision
+ 一版 BackgroundBudget
+ Coordinator 调整后台任务预算
+ 旧预算重新放大迁移或 offload 流量
+ 忽略旧预算；当前预算过期则使用保守默认值
+
+
+
+
+
+ 四个 fencing scope 是 group term、slot assignment generation、client control generation
+ 和 segment incarnation。前三者可共享 FencingToken{scope_id, generation} 的实现；segment 使用 UUID incarnation token。
+ 它们属于不同状态机，不能跨 scope 比较。
+ revision/sequence 使用无符号 64 位单调计数器；incarnation 使用启动或重建时生成的 UUID，避免依赖旧进程持久化计数器。
+
+
+// 通用比较规则，不表示存在一个全局 generation。
+struct FencingToken {
+ string scope_id;
+ uint64_t generation;
+}
+
+SnapshotRevision = uint64_t; // view/resource/budget，只用于排序、watch 和 CAS
+IncarnationId = UUID; // node/segment，每次重启或重建生成新值
+StreamSequence = uint64_t; // telemetry，在同一 incarnation 内单调递增
+
+
+ 1.2.1 Term 为什么单独命名
+
+ term 本质上是 MasterGroup 的领导权 generation，但保留 term 这个名称是为了和 Raft 等复制协议一致。
+ QUORUM 模式由组选举产生更高 term；ASYNC_STANDBY 模式由 Coordinator 在取得独占 fencing token 后授权更高 term。
+ 无论采用哪种模式，数据写入权威方都必须检查 term。只把新 primary 地址写入服务发现，而不在提交点检查 term，不能防止双主。
+
+
+ 1.2.2 Fencing 是什么
+
+ Fencing（隔离旧持有者）是“在最终写入点拒绝旧 token”的机制，generation/term/incarnation 是本文的 fencing token。
+ 检查必须发生在真正改变状态的地方，例如 MasterGroup 日志提交、segment allocator reserve/free、HA Backend CAS，
+ 而不能只在客户端或入口 RPC 检查一次。否则请求通过检查后发生切主，延迟请求仍可能写入。
+
+
+ 1.3 一致性与故障恢复基础概念
+
+ 概念 解释及本文中的用途
+ Lease有到期时间的临时所有权/存活声明，持有者必须续租。用于故障检测和资源自动回收，但不能代替 fencing。 CASCompare-And-Swap：仅当持久化值仍等于预期版本时才更新。用于保证两个 Coordinator 或迁移流程不能同时提交冲突 owner。 Quorum复制组的多数派。3 voter 中至少 2 个确认后提交，可容忍 1 个故障；失去多数派时宁可拒绝写，也不能产生两个合法 primary。 OpLog按顺序记录元数据状态变化的操作日志。replica 和迁移 target 先加载 snapshot，再 replay 后续 OpLog。 Snapshot某个一致性位置上的完整状态快照，用于避免从第一条 OpLog 开始恢复。snapshot 必须携带其对应的 log/commit position。 Replay lagtarget/replica 已应用位置与 source committed position 的差距。只有追到 cutover position 才能安全接管。 RPORecovery Point Objective，可接受的数据丢失范围。QUORUM 已提交写目标是 RPO=0；异步主备的 RPO 由尚未 replay 的日志决定。 RTORecovery Time Objective，从故障发生到恢复服务允许的时间；受故障检测、选主和 replay lag 影响。 Idempotency key同一逻辑操作重试时保持不变的 ID，使重复请求只产生一次状态变化，用于 reservation 和后台 Task。 ResourceView容量、带宽、IOPS 和健康状态的近实时只读快照，用于选候选；它可能过期，因此最终 reserve 仍需 allocator 原子确认。 Reservation带 TTL 的临时资源承诺。PutStart 预留、PutEnd 提交、PutRevoke/超时释放，以避免并发请求超卖。 Replica descriptor描述一个对象副本所在 segment、offset/length、介质类型和访问参数的元数据；它不是对象数据本身。 Failure domain共享故障风险的边界，例如进程、机器、机架或 AZ。副本跨域放置是为了避免一次故障同时损坏全部副本。 Locality调用方与 segment 之间的 NUMA、主机、机架、AZ 或网络 fabric 距离，用于估算访问时延和网络成本。 Rendezvous hash对“实体、候选节点”组合计算稳定分数并选最高者；候选集合变化时只移动较少实体。本文用于生成初始 control-owner 建议，最终映射仍由 Coordinator 显式提交。 Hard constraint不满足就绝不能选择的条件，例如容量不足、介质能力不符或副本处于同一故障域；与可权衡的软评分分开处理。 Dominant resource某候选节点上占用比例最高的资源维度。例如容量只占 20% 但带宽占 90%，其主导资源是带宽，不能因容量充足继续放置。 Hysteresis进入和退出某状态使用不同阈值或连续观测窗口，避免热点副本、预算和读路由在临界值附近频繁来回切换。 P99.999.9% 请求时延不超过的值，即每 1,000 个请求约有 1 个更慢；用于约束极端尾延迟，而不仅是平均性能。 MOVEDSlotGroup 已稳定归属于另一个 MasterGroup；客户端应更新本地长期路由并重试。 ASKSlotGroup 正在迁移，客户端仅对本次请求访问 importing group，不永久覆盖本地 owner。
+
+
+ 1.4 一次 Put 请求如何使用这些版本号
+
+1. Client 从 ClusterView(view_revision=100) 得到：
+ slot 42 -> SlotGroup S7(assignment_generation=8)
+ -> MasterGroup G2(term=12)
+ -> primary endpoint B
+
+2. Client 向 B 发送 PutStart：
+ {group=G2, term=12, slot=42, assignment_generation=8, request_id=R}
+
+3. B 在写入 PROCESSING 元数据前检查：
+ a. 自己仍是 G2 term 12 的 primary；否则 NOT_PRIMARY/STALE_TERM
+ b. slot 42 仍属于 G2 且 generation=8；否则 MOVED/STALE_ASSIGNMENT
+ c. request_id R 是否已处理；若是则返回相同结果
+
+4. B 使用本地 ResourceView 选择目标存储节点，并从本组的 DelegatedExtentPool
+ 分配已由后台租入的空间和资源预算。常态 Put 不同步访问 Coordinator 或远端 allocator。
+
+5. B 写入 PROCESSING 元数据，将变更异步复制给 standby，然后返回 replica descriptors。
+
+6. Client 写数据并发送 PutEnd(R)。B 再次检查 term、assignment_generation 和 request_id，
+ 然后把对象从 PROCESSING 转为 COMPLETE，并异步复制该变更。
+
+
+ view_revision 只说明客户端使用哪一版路由快照，不授予写权限；真正保护写正确性的是对应作用域的
+ term、assignment_generation、segment_incarnation 以及幂等 request_id。即使客户端 view 不是最新，
+ 服务端也能拒绝危险写，并通过错误码引导客户端刷新。
+
+
+
+ 和 Redis 的关系：
+ Redis Cluster 没有独立 Coordinator，slot ownership 和 failover 信息由节点 gossip 传播，reshard 通常由外部工具驱动。
+ Mooncake 的不同点在于 Master 还承担物理空间管理、冷热分级、租约、client liveness 和后台迁移任务。
+ 因此引入轻量 Coordinator 是为了管理 Mooncake 特有的全局控制问题，而不是替代 Redis Cluster 的去中心化热路径。
+
+
+
+ P99.9 原则：
+ Coordinator 不能成为 P99.9 热路径依赖。客户端在正常读写时不访问 Coordinator；Coordinator 故障只影响扩缩容、rebalance、跨组迁移、视图发布和预算调节，不影响已有稳定 group 的前台读写或 quorum 组内选主。
+
+
+ 2. 设计目标与非目标
+ 2.1 设计目标
+
+ 将对象元数据按固定 slot 拆分到多个 MasterGroup，解除单 MasterNode 的 CPU、锁和 RPC 瓶颈。
+ 保持前台请求路径短：客户端计算 slot，直接访问对应 MasterGroup 的 current primary。
+ 在扩缩容、slot 迁移、failover、冷热分级、资源压力下保持 P99.9 时延稳定。
+ 用轻量 Coordinator 管理 slot view、资源摘要和后台预算，但不代理元数据请求。
+ 支持兼容式演进：单 Master 先暴露全量 slot view，再逐步引入多 Master 和迁移能力。
+
+
+ 2.2 非目标
+
+ 不把 Coordinator 做成所有元数据 RPC 的代理。
+ 不要求跨 slot Batch 操作具备全局原子性；默认提供逐 key 成功/失败结果。
+ 不要求第一阶段实现完整自动 rebalance；可以先由管理命令触发迁移计划。
+ 不把物理数据迁移放入 Master 进程内同步执行；Master 只负责元数据状态机和任务编排。
+
+
+ 3. 顶层架构
+
++---------------------------+
+| Mooncake Client / SDK |
+| - slot(key) |
+| - slot map cache |
+| - MOVED/ASK retry |
+| - batch regroup |
++-------------+-------------+
+ |
+ | foreground metadata RPC
+ v
++-------------------+ +-------------------+ +-------------------+
+| Group G1 primary | | Group G2 primary | | Group G3 primary |
+| serves slot groups| | serves slot groups| | serves slot groups|
+| object metadata | | object metadata | | object metadata |
+| lease / tasks | | lease / tasks | | lease / tasks |
++---------+---------+ +---------+---------+ +---------+---------+
+ | | |
+ | resource summary, latency stats, migration state
+ v v v
++---------------------------------------------------------+
+| Lightweight Coordinator |
+| - cluster view publisher |
+| - slot ownership / generation |
+| - migration planner |
+| - group placement / fencing publisher |
+| - resource summary aggregator |
+| - background budget controller |
++---------------------------+-----------------------------+
+ |
+ | durable cluster metadata
+ v
++---------------------------------------------------------+
+| HA Backend |
+| etcd / Redis / K8s Lease / persistent catalog / OpLog |
++---------------------------------------------------------+
+
++---------------------------------------------------------+
+| Storage Clients / Segments |
+| - memory / local disk / NoF SSD / CXL |
+| - heartbeat / capacity / health |
+| - execute copy/offload/promotion tasks |
++---------------------------------------------------------+
+
+
+ 3.1 核心原则
+
+
+
+ 原则
+ 含义
+
+
+
+
+ 热路径去中心化
+ 客户端通过本地 slot map 直连 owner MasterGroup 的 current primary，不经过 Coordinator。
+
+
+ 控制路径集中但轻量
+ Coordinator 管理 view、跨组迁移、group placement、fencing 发布和预算，不保存每个对象的热路径状态。
+
+
+ 对象状态单 owner
+ 同一个 key 的 Put/Get/Evict/Offload/Promotion 只由当前 slot owner MasterGroup 提交。
+
+
+ 所有权带 fencing token
+ 组内 primary 用 term，slot 改属用 assignment_generation，segment 控制权改属用 control_generation；三者不能混用。
+
+
+ 后台任务可退让
+ 迁移、snapshot、eviction、offload、promotion 必须受预算控制，P99.9 超阈值时自动降速。
+
+
+
+
+ 4. 组件职责
+ 4.1 Mooncake Client / SDK
+
+ 维护本地 ClusterView 和 slot map 缓存。
+ 计算 slot = hash(tenant_id, key) % 16384。
+ 按 slot owner MasterGroup 解析 current primary endpoint 并直连。
+ 处理 MOVED、ASK、NOT_PRIMARY、STALE_TERM、TRYAGAIN、STALE_ASSIGNMENT。
+ 对 Batch 请求按 owner regroup，并按原始 key 顺序合并响应。
+ 设置 per-RPC deadline，避免单个 Master 慢请求拖垮 P99.9。
+
+
+ 4.2 MasterGroup 与其 current primary
+
+ MasterGroup 负责一组 SlotGroup 的对象元数据；current primary 对外服务，replica 按复制模式同步状态。
+ 执行对象生命周期状态机：PutStart、PutEnd、PutRevoke、Remove、Evict。
+ 维护对象租约、hard pin、soft pin 和访问热度统计。
+ 生成和消费对象级后台任务，例如 copy、move、offload、promotion。
+ 由 current primary 向 Coordinator 上报前台时延、队列深度、slot 负载、资源使用摘要和迁移状态。
+ 订阅 Coordinator 发布的 cluster view、resource view 和 background budget。
+
+
+ 4.3 Lightweight Coordinator
+
+ 维护 MasterNode incarnation、MasterGroup membership、slot ownership、assignment generation 和 view revision。
+ 生成扩缩容和 rebalance 计划。
+ 协调 slot group 迁移状态机，但不搬运前台请求。
+ 汇总 segment resource summary，发布近实时 ResourceView。
+ 根据 P99.9、队列深度、CPU、网络、replay lag 动态调节后台任务预算。
+ 观察组内选主结果并发布带 term 的新 primary endpoint；通过 HA Backend fencing 防止旧 primary 继续写。
+
+
+ 4.4 HA Backend
+
+ 持久化 cluster view、slot ownership、membership lease 和 coordinator lease。
+ 保存 MasterGroup 的 OpLog、snapshot catalog、commit position 或恢复指针。
+ 在 Coordinator 故障时支持新 Coordinator 接管。
+
+
+ 4.5 Storage Client / Segment
+
+ 向 Master 或 Resource Registry 上报 segment 容量、水位、健康状态。
+ 执行数据路径操作，例如 RDMA 写、SSD offload、本地 promotion、replica clear。
+ 通过 task heartbeat 获取 offload/promotion/copy/move 任务。
+
+
+ 4.6 Segment 与 MasterGroup 的归属关系
+
+ 结论：
+ key 的 slot owner 和 segment 的 control owner 是两套独立映射。
+ slot owner 管对象及其 replica 状态；segment control owner 管 segment 注册、心跳、健康状态和客户端级任务通道。
+ 一个对象可以被放到任意健康 segment 上，因此对象的 slot owner 不要求等于该 segment 的 control owner。
+
+
+
+ 推荐按 client_id 而不是按 segment_id 分配 segment control owner：同一个 Storage Client
+ 注册的 MEMORY、LOCAL_DISK、NOF_SSD 等全部 segment 由同一个 control-owner MasterGroup 管理，heartbeat 发往该组 current primary。
+ 这样 client liveness、一次 heartbeat 中携带的多个 segment 状态以及下发给该 client 的任务队列都落在同一处，
+ 不需要拆分一次 heartbeat。Coordinator 在 ClusterView 中持久化该映射；初次加入时可用
+ rendezvous hash 选择 control-owner MasterGroup，并在负载不均衡时显式迁移，而不能由各 MasterNode 独立计算后直接生效。
+
+
+
+对象控制面：hash(tenant_id, key) -> slot -> slot owner MasterGroup -> current primary
+Segment 控制面：client_id -> client control-owner MasterGroup -> current primary -> 全部 segments
+
+例：
+ Group G1 owns slots [0, 4095] and controls Client X -> [Segment X.mem, X.ssd]
+ Group G2 owns slots [4096, 8191] and controls Client Y -> [Segment Y.mem]
+ Group G3 owns slots [8192, 16383] and controls Client Z -> [Segment Z.mem, Z.nof]
+
+ key K belongs to a slot owned by G2
+ K replicas may be placed on X.mem and Z.nof
+ G2 owns K metadata and decides create/remove/offload/promotion
+ G1 controls X.mem health; G3 controls Z.nof health
+ Storage Client X/Z remains the authority that atomically reserves/frees physical extents
+
+
+
+
+
+ 状态或动作
+ 唯一写入者/权威方
+ 其他 Master 如何使用
+
+
+
+
+ key、replica descriptor、对象状态机
+ key 的 slot owner MasterGroup
+ 非 owner 返回 MOVED，不得修改
+
+
+ client lease、segment mount/unmount、health、watermark
+ client control-owner MasterGroup
+ 通过只读 ResourceView 获取摘要
+
+
+ 物理 extent 的 reserve/free
+ Storage Client 的 allocator，使用 lease/CAS 和幂等 request_id
+ slot owner 直接请求 allocator；control owner 不在每次 Put 的同步链路中
+
+
+ offload/promotion/copy/move 的对象决策
+ 对象 slot owner MasterGroup
+ 任务经 client control owner 的 heartbeat 通道投递，完成后回报 slot owner
+
+
+ segment control owner 映射
+ Coordinator + HA Backend CAS
+ 所有 Master 订阅同一版本化映射
+
+
+
+
+
+ 因而“G1 管理 Segment X”只表示 G1 是 X 所属 client 的控制面 owner，并不表示只有 G1 的 slot
+ 才能在 X 上放 replica。所有 slot-owner group 都可依据 ResourceView 选择 X，但最终空间分配必须由 X 所在
+ Storage Client 的 allocator 原子确认，避免多个 Master 根据过期容量摘要重复分配。
+
+
+ 4.6.1 Owner 选择、故障与迁移规则
+
+ 初始分配： Coordinator 对 client_id 和 ACTIVE MasterGroup 集合做 rendezvous hash，选择一个 control-owner group；组内 primary/replica 由该组成员关系决定。hash 可叠加连接数、segment 数和 heartbeat QPS 权重。稳定性： MasterNode 加入或退出时不自动重算全部 client；只有 Coordinator 生成并提交的新 group 映射才改变 owner，避免大规模心跳抖动。故障切换： current primary 失效后由组内复制协议选出新 primary 并递增 term；Coordinator 发布 endpoint。只有 client 从一个 control-owner group 迁到另一个 group 时才递增 control_generation。Storage Client 同时校验 term 和 control_generation，旧 primary 必须被 fencing。主动迁移： 先让 target group 导入 client/segment 快照并接收增量状态，再递增 control_generation 切换 heartbeat，最后清理 source group；迁移期间对象 slot ownership 不变。故障域： 同一 MasterGroup 的 replicas 尽量跨节点、机架或 AZ；同一 client 的 segment 不拆给多个 control-owner group，除非未来 heartbeat 协议支持按 segment 独立分流。
+
+
+ 不要混淆三种变化：
+ MasterGroup 组内切主只提升 term；SlotGroup 跨组迁移改变对象元数据归属并提升 assignment_generation；
+ client control owner 跨组迁移改变 segment 控制归属并提升 control_generation。任何一种变化都不应隐式触发另外两种。
+
+
+ 4.6.2 跨组任务投递
+
+ 当对象 slot owner 与目标 segment control owner 不同，不能依赖两个 group 的内存队列或分布式事务。
+ 对象 owner 先在本组 OpLog 中提交 TaskIntent，再以至少一次语义投递到 control-owner group；后者通过
+ Storage Client heartbeat 下发。完成通知可重复发送，由对象 owner 根据 task_id 幂等提交最终 replica 状态。
+
+
+TaskEnvelope {
+ UUID task_id;
+ string object_owner_group_id;
+ uint64_t object_group_term;
+ uint32_t slot_id;
+ uint64_t assignment_generation;
+ string client_control_group_id;
+ uint64_t control_generation;
+ UUID target_segment_id;
+ TaskType type;
+}
+
+
+ 任一 fencing generation/term 不匹配时不得盲目执行：投递方刷新 ClusterView 后重新路由，执行方使用
+ task_id 去重。这样组内切主、SlotGroup 迁移或 client control owner 迁移都不会造成任务丢失或重复修改对象状态。
+
+
+ 4.7 多 Master 的分组：MasterGroup 与 MasterNode
+
+ 推荐模型：
+ 用 MasterGroup（也可称 ReplicaSet）表达主备复制关系，用不同 MasterGroup 之间的 slot
+ 分配表达 shard 分担关系。MasterNode 只是进程/机器实例，不应被永久标记成全局 primary、standby 或 shard。
+
+
+
+ Cluster
+ |
+ +-----------------+-----------------+
+ | |
+ MasterGroup G1 MasterGroup G2
+ owns slot groups 0..3 owns slot groups 4..7
+ (shard relation with G2) (shard relation with G1)
+ | |
+ primary: Master A primary: Master C
+ standby: Master B standby: Master D
+ (A/B are replication peers) (C/D are replication peers)
+
+Client route:
+ key -> slot -> slot_group -> master_group -> current primary endpoint
+
+
+
+ 因此，“A 和 B 是主备”表示它们是同一个 MasterGroup 的复制成员；“A 和 C 分担 shard”更准确地说是
+ A 所在的 G1 与 C 所在的 G2 分别拥有不同 slot group。主备关系发生在组内，分片关系发生在组间，
+ 两者不应使用同一个 role 字段表达。
+
+
+
+
+
+ 抽象
+ 职责
+ 所有权/版本
+
+
+
+
+ MasterNode描述进程地址、故障域、容量、存活租约；可承载一个或多个 group replica
+ node_id + node_incarnation_id
+
+ MasterGroup对象元数据的一致性与故障切换单元，包含一个 primary 和若干 standby
+ group_id + term
+
+ GroupMember描述某个 node 在某个 group 内的 PRIMARY、VOTER、LEARNER 等角色
+ 角色只在 group 内有效
+
+
+ SlotGroupAssignment把一段 slot 映射到一个 MasterGroup；扩缩容时迁移该映射及其状态
+ assignment_generation
+
+
+
+ 4.7.1 两种部署方式
+ 方式一：节点独占 group，建议作为第一阶段。
+
+G1 = {A primary, B standby}
+G2 = {C primary, D standby}
+
+
+ 关系直观，故障和容量边界容易分析，运维简单。
+ standby 正常情况下资源利用率较低，可提供只读诊断，但不应承接强一致前台写。
+ 两副本只能实现异步或同步主备，无法在网络分区下形成安全多数派；需要 quorum 强一致时，每组至少放置 3 个跨故障域 voter。
+
+
+ 方式二：一个节点承载多个 group replica，用于提高利用率。
+
+G1 = {A primary, B standby, C standby}
+G2 = {B primary, C standby, A standby}
+G3 = {C primary, A standby, B standby}
+
+
+ A、B、C 都承担一部分 primary shard，同时又互为其他 group 的 replica；“standby”是非 primary replica 的统称，QUORUM 模式对应 VOTER，ASYNC_STANDBY 模式对应 FOLLOWER。
+ 调度必须限制同一 node 的 primary 数、总 QPS、metadata bytes、replay 带宽和故障恢复负载。
+ 同一 group 的两个 replica 不能落在同一进程或同一故障域；否则节点或机架故障会同时丢失主备。
+ 一个 node 故障后，不能让其所有 standby 同时提升到同一个剩余 node，需要预先校验 failover capacity。
+
+
+ 4.7.2 分组与迁移规则
+
+ 组内切主： 只改变 MasterGroup.term 和 primary member，不改变该组拥有的 slot；客户端刷新 endpoint 即可。组间迁移： 把 SlotGroup 从 source MasterGroup 复制到 target MasterGroup，完成 snapshot/replay 后递增 assignment_generation 并切换归属。成员变更： 采用 learner 加入、追平、转 voter、移除旧 member 的联合配置流程，不能直接覆盖成员列表。分组粒度： MasterGroup 数量应多于物理节点数，才能细粒度均衡；但每组都有 OpLog、snapshot 和 heartbeat 成本，应使用固定数量的虚拟 group，而不是每个 slot 一个复制组。Coordinator 边界： Coordinator 决定 group membership 和 slot-to-group assignment；组内复制协议决定日志提交和 primary，不让 Coordinator 进入每次元数据写的提交路径。Segment 控制归属： ClientControlOwner 也指向 MasterGroup，而不是裸 node；heartbeat 访问该组当前 primary，组内切主不改变 client 的逻辑归属。
+
+
+ 关键约束：
+ 如果没有真正的组内复制协议和提交法定人数，只能称为 primary/standby snapshot failover，不能宣称强一致 HA。
+ 此时 primary 故障可能丢失尚未同步的 OpLog，ClusterView 必须暴露相应的 RPO 状态。
+
+
+ 5. 数据模型
+ 5.1 ClusterView
+
+struct ClusterView {
+ string cluster_id;
+ uint64_t view_revision;
+ uint32_t slot_count; // default: 16384
+ vector<MasterNode> nodes;
+ vector<MasterGroup> master_groups;
+ vector<SlotGroupOwner> slot_groups;
+ vector<ClientControlOwner> client_control_owners;
+}
+
+struct MasterNode {
+ string node_id;
+ string rpc_address;
+ NodeStatus status;
+ FailureDomain failure_domain;
+ UUID node_incarnation_id;
+}
+
+struct MasterGroup {
+ string group_id;
+ uint64_t term;
+ vector<GroupMember> members;
+ string primary_node_id; // cached routing endpoint for current term
+ ReplicationMode mode; // QUORUM / ASYNC_STANDBY
+ CommitPosition committed;
+}
+
+struct GroupMember {
+ string node_id;
+ GroupRole role; // PRIMARY / VOTER / FOLLOWER / LEARNER
+ CommitPosition replayed;
+}
+
+struct SlotGroupOwner {
+ uint32_t slot_group_id;
+ uint32_t slot_begin;
+ uint32_t slot_end;
+ string owner_master_group_id;
+ uint64_t assignment_generation;
+ SlotGroupState state;
+}
+
+struct ClientControlOwner {
+ UUID client_id; // applies to all segments of this client
+ string owner_master_group_id;
+ uint64_t control_generation; // independent from slot assignment
+ ControlOwnerState state;
+}
+
+
+ 5.2 ResourceView
+
+struct ResourceView {
+ uint64_t resource_revision;
+ vector<SegmentResource> segments;
+ ClusterPressure pressure;
+}
+
+struct SegmentResource {
+ UUID segment_id;
+ UUID client_id;
+ uint64_t control_generation;
+ UUID segment_incarnation;
+ string segment_name;
+ ReplicaType type; // MEMORY / LOCAL_DISK / NOF_SSD / CXL
+ ResourceCapability capability;
+ ResourceTelemetry telemetry;
+ FailureDomain failure_domain;
+ NetworkLocality locality;
+ SegmentHealth health;
+}
+
+
+ 5.3 多维资源、工作负载与放置抽象
+
+ 设计原则：
+ 不再用单一 used_bytes / capacity_bytes 判断节点优劣，而把“节点能提供什么”、
+ “对象需要什么”和“当前还承诺了多少资源”分开建模。容量大但带宽小的 segment 可承载冷数据，
+ 容量小但带宽大的 segment 可承载热点副本；同一对象的持久放置与实时读流量也可以由不同策略处理。
+
+
+ 建议增加以下六类稳定抽象：
+
+
+
+ 抽象
+ 回答的问题
+ 更新频率与归属
+
+
+
+
+ ResourceCapabilitysegment 的容量、顺序/随机读写带宽、IOPS、基础时延、并发度和介质能力上限
+ 慢变化；注册或基准测试时由 Storage Client 上报
+
+
+ ResourceTelemetry当前空闲容量、可用带宽、队列深度、P99 时延、错误率和热度
+ 快变化；control owner 聚合后发布带时间戳的摘要
+
+
+ FailureDomain / NetworkLocality节点、机架、AZ、NUMA、RDMA fabric 与访问方之间的故障隔离和网络代价
+ 低频变化；Coordinator 维护
+
+
+ WorkloadIntent对象预期大小、读写比、目标吞吐/P99、热度、生命周期、持久性和副本约束
+ 写入时显式提供，缺省值由历史观测推断
+
+
+ ResourceReservation已被并发 Put 或流量调度承诺、但尚未反映到 telemetry 的 bytes、bandwidth 和 IOPS
+ 实时；allocator/traffic admission 以 lease + segment incarnation 管理
+
+
+ PlacementPolicy + ScoreBreakdown如何执行硬约束过滤、软目标打分，以及为什么选择某个 segment
+ 版本化策略；Master 本地执行并记录可解释结果
+
+
+
+
+
+struct ResourceVector {
+ uint64_t capacity_bytes;
+ double read_bandwidth_Bps;
+ double write_bandwidth_Bps;
+ double read_iops;
+ double write_iops;
+ double concurrency;
+}
+
+struct ResourceCapability {
+ ResourceVector limit; // calibrated sustainable limit, not link peak
+ LatencyProfile baseline; // p50 / p99 by operation and object-size bucket
+ set<Feature> features; // RDMA_READ / PERSISTENT / RANDOM_WRITE / ...
+}
+
+struct ResourceTelemetry {
+ ResourceVector used;
+ ResourceVector reserved;
+ LatencyProfile observed;
+ double queue_depth;
+ double error_rate;
+ uint64_t sample_timestamp_ms;
+ uint64_t sample_seq; // monotonic within one segment_incarnation
+}
+
+struct WorkloadIntent {
+ uint64_t object_size_bytes;
+ AccessClass access_class; // HOT / WARM / COLD, only a hint
+ double expected_read_Bps;
+ double expected_write_Bps;
+ double expected_iops;
+ uint64_t target_p99_us;
+ DurabilityClass durability;
+ uint32_t replica_num;
+ FailureDomainPolicy isolation;
+ optional<string> consumer_locality;
+}
+
+struct ResourceReservation {
+ UUID reservation_id;
+ UUID segment_id;
+ ResourceVector amount;
+ uint64_t expire_at_ms;
+ UUID segment_incarnation;
+ string idempotency_key;
+}
+
+struct ScoreBreakdown {
+ double capacity_headroom;
+ double bandwidth_headroom;
+ double latency_cost;
+ double locality_cost;
+ double failure_correlation_cost;
+ double migration_cost;
+ double total_score;
+ string policy_version;
+}
+
+
+ 5.3.1 放置和读路由必须解耦
+
+ Replica Placement： 决定数据存在哪些 segment，主要优化容量、持久性、写入代价和故障域。Replica Read Selection： 在已有 replica 中按实时带宽、队列、时延和调用方 locality 选择读源；不因为一次流量变化立即搬数据。Tiering / Rebalance： 根据较长时间窗口的热度和资源价格决定复制、迁移、offload 或 promotion，带 hysteresis 和冷却时间。
+
+ 例如，大容量低带宽节点可保存完整冷副本，小容量高带宽节点只保存工作集热点副本；读请求优先访问高带宽副本，
+ 高带宽层容量不足时淘汰热点副本不会删除低带宽层的持久副本。这样“容量归属”和“流量归属”不再被一个 placement 决策绑死。
+
+
+ 5.3.2 可演进的算法接口
+
+PlacementResult Place(PlacementContext ctx) {
+ candidates = FilterByHardConstraints(
+ ctx.resource_view, ctx.intent, ctx.excluded_failure_domains);
+
+ candidates = CheckMultiResourceFeasibility(
+ candidates, capability - telemetry.used - telemetry.reserved);
+
+ ranked = policy.Score(candidates, ctx.intent, ctx.policy_version);
+ reservation = TryReserve(ranked, bytes + bandwidth + iops, bounded_attempts);
+
+ return {reservation, ranked.winner.score_breakdown};
+}
+
+
+ 硬约束和软评分分离；容量不足、目标 P99 不满足、故障域冲突必须先过滤，不能靠低分表达。
+ 采用归一化 headroom 或 Dominant Resource Fairness 思路检查多维可行性，避免容量充足却把带宽耗尽。
+ PlacementPolicy 是版本化、无状态接口；后续可从加权打分演进为 bin-packing、min-cost flow 或学习型策略，而不改变 Put 状态机。策略输入只使用版本化快照，输出保存 policy_version 和 ScoreBreakdown，便于回放、灰度和比较新旧算法。
+ 在线请求只对 Top-K 候选做有界 reserve 尝试；复杂的全局优化在 Coordinator 后台生成 policy 参数或迁移计划，不能进入 Put 热路径。
+
+
+ 5.3.3 指标语义要求
+
+ 避免错误建模：
+ 带宽不能只保存一个静态数值。至少区分读/写、可持续能力/当前使用量/已预留量，并按对象大小桶记录有效吞吐和时延；
+ 否则 100 Gbit/s 网卡、SSD 顺序吞吐和 4 KiB 随机访问会被错误地视为同一种资源。
+
+
+ 5.4 BackgroundBudget
+
+struct BackgroundBudget {
+ uint64_t budget_revision;
+ uint32_t migration_max_inflight_groups;
+ uint32_t migration_qps_budget;
+ uint64_t snapshot_bytes_per_sec;
+ uint32_t eviction_keys_per_round;
+ uint32_t offload_tasks_per_heartbeat;
+ uint32_t promotion_tasks_per_heartbeat;
+}
+
+
+ 6. 热路径流程分析
+ 6.1 GetReplicaList
+
+Client
+ -> calculate slot(key)
+ -> find owner from local ClusterView
+ -> RPC GetReplicaList(key, tenant_id, master_group_id, group_term,
+ slot_id, assignment_generation)
+
+Owner MasterGroup current primary
+ -> validate master_group_id, term, slot ownership and assignment_generation
+ -> lookup object metadata in local slot group
+ -> refresh lease / hotness counter
+ -> rank readable replicas by current latency / bandwidth / caller locality
+ -> if promotion-on-hit eligible: enqueue promotion task only
+ -> return replica list
+
+Client
+ -> if OK: use replica descriptors for data path
+ -> if MOVED/STALE_ASSIGNMENT: refresh slot map and retry
+ -> if TRYAGAIN: short backoff within request deadline
+
+
+
+ P99.9 控制点：读路径只访问一个 MasterGroup primary；promotion 不在前台执行；slot view 或 group term 过期通过重定向快速修正；RPC 有 deadline。
+
+
+ 6.2 PutStart / PutEnd
+
+PutStart:
+Client -> owner MasterGroup current primary
+ -> validate group term and assignment generation
+ -> build WorkloadIntent from request policy and observed history
+ -> filter hard constraints and score candidates from local ResourceView
+ -> allocate from the local DelegatedExtentPool
+ -> write object PROCESSING metadata
+ -> enqueue asynchronous standby replication
+ -> return replica descriptors
+
+Data Path:
+Client writes object data to selected replicas
+
+PutEnd:
+Client -> owner MasterGroup current primary
+ -> validate group term and assignment generation
+ -> transition PROCESSING -> COMPLETE
+ -> enqueue asynchronous standby replication
+ -> optionally enqueue offload task
+ -> return OK
+
+
+
+ P99.9 控制点：PutStart 不向 Coordinator 或远端空间分配服务同步请求资源；ResourceView 和 DelegatedExtentPool 都在本地。
+ 本地 extent 不足时快速失败并触发后台补充，禁止把远端 refill 或长时间候选扫描退化到当前请求中。
+
+
+ 6.3 Batch 请求
+
+Client input keys: [k1, k2, k3, k4, k5]
+
+Regroup by owner:
+ Master A: [k1, k4]
+ Master B: [k2, k5]
+ Master C: [k3]
+
+Parallel RPC:
+ send sub-batches concurrently with per-shard deadline
+
+Merge:
+ output results in original key order
+
+
+
+ P99.9 控制点：子批次并行，单个慢 shard 不阻塞其他 shard 的结果；业务可选择 fail-fast、partial success 或 bounded retry。
+
+
+ 6.4 时序图与当前 Mooncake 对比
+
+ 本节只统计正常稳定状态，不把首次建连、ClusterView watch、MOVED/STALE_TERM 重试和故障恢复计入常态热路径。
+ 当前实现以代码中的单个全局 MasterService 为基线：客户端 Get 调用一次
+ GetReplicaList；Put 调用 PutStart、执行数据传输、再调用 PutEnd；
+ segment allocator 由 Master 进程内的 segment_manager_ 访问。当前 1024 个 metadata shard 是进程内锁分片，
+ 不是分布式 MasterGroup。对应代码入口为 client_service.cpp::Get/Put、
+ master_client.cpp::GetReplicaList/PutStart/PutEnd 和 master_service.cpp::GetReplicaList/PutStart/PutEnd。
+ 图中的当前基线是默认非 HA 模式；当前 HA 仍是单个 active Master，客户端 RPC 形状不变，leader watch 在后台执行。
+ 若某种 HA 配置把 OpLog 外部持久化同步放入请求路径，应把该耗时作为单独基线实测，不能与默认模式混合比较。
+
+
+ 本节时序图使用 Mermaid 11 sequenceDiagram 绘制；HTML 通过官方推荐的 ESM 方式从 jsDelivr 加载。
+ 无网络环境仍会保留 Mermaid 源码，可在文档构建阶段改为 vendored Mermaid 或预渲染 SVG。
+
+
+ 6.4.1 计数口径
+
+ 符号 含义
+ G一个 Batch 中涉及的 owner MasterGroup 数量。 RPC 次数 一次 request/response 算一个 RPC；若统计单向网络消息，会在表中单独说明。 串行网络轮次 关键路径上必须前后等待的网络 round trip 数。多个并行 RPC 虽然增加消息数，但只增加一个串行轮次。 数据传输 Transfer Engine/RDMA/NoF 写读，不等同于 metadata RPC；单独计数。
+
+
+ 6.4.2 Get：当前实现
+
+sequenceDiagram
+ autonumber
+ actor App as Application
+ participant Client as Mooncake Client
+ participant Master as Global Master
+ participant Segment as Storage Segment
+ App->>Client: Get(key)
+ Client->>Master: GetReplicaList(key)
+ activate Master
+ Master->>Master: local metadata lookup
+
+ 6.4.3 Get：新设计
+
+sequenceDiagram
+ autonumber
+ actor App as Application
+ participant Client as Routed Client
+ participant Primary as Owner Group Primary
+ participant Segment as Selected Segment
+ participant Async as Async Aggregator
+ App->>Client: Get(key)
+ Client->>Client: slot hash + cached ClusterView lookup
+ Client->>Primary: GetReplicaList(term, assignment_generation)
+ activate Primary
+ Primary->>Primary: validate routing + local lookup
+ Primary-->>Client: ranked replica descriptors
+ Primary-)Async: aggregate lease and hotness update
+ deactivate Primary
+ Client->>Segment: TransferRead / RDMA read
+ Segment-->>Client: value bytes
+ Client-->>App: value
+ Note over Client,Primary: Metadata RPC = 1, Coordinator RPC = 0, quorum RTT = 0
+
+
+
+ 正常 Get 的通信次数与当前实现相同，新增 slot hash、view lookup、term/generation 比较均为本地 CPU 操作。
+ 但当前 Get 会刷新对象 read lease；如果新设计把每次 lease/hotness 更新同步复制到 quorum，Get 将额外增加一个 group commit RTT，
+ 形成明显劣化。推荐把访问热度和 lease heartbeat 做本地聚合、异步批量复制，并规定新 primary 在提升后至少一个
+ max_read_lease_ttl 窗口内禁止回收旧 replica。这样 failover 期间保守占用空间，但正常 Get 不增加网络轮次。
+
+
+ 6.4.4 Put：当前实现
+
+sequenceDiagram
+ autonumber
+ actor App as Application
+ participant Client as Mooncake Client
+ participant Master as Global Master
+ participant Storage as Target Storage Nodes
+ App->>Client: Put(key, value)
+ Client->>Master: PutStart
+ activate Master
+ Master->>Master: local placement + local allocation
+ Master-->>Client: replica descriptors
+ deactivate Master
+ Client->>Storage: TransferWrite to configured replicas
+ Storage-->>Client: transfer completion
+ Client->>Master: PutEnd
+ activate Master
+ Master->>Master: mark COMPLETE
+ Master-->>Client: OK
+ deactivate Master
+ Client-->>App: OK
+ Note over Client,Master: Metadata RPC = 2, allocator RPC = 0, group commit RTT = 0
+
+
+ 6.4.5 Put：新设计快路径
+
+ Put 只采用这一条前台路径。为避免每次 PutStart 同步访问空间分配服务，增加 DelegatedExtentLease：Storage Client allocator
+ 预先把互不重叠的 extent range、bytes/bandwidth/IOPS budget 租给 MasterGroup。Group primary 在租约范围内本地切分，
+ 后台低水位时批量续租。lease 必须携带 segment_incarnation + control_generation + owner_group_id + group_term + expire_at，
+ Storage Client 不得把同一 extent 同时租给两个 group。
+
+
+sequenceDiagram
+ autonumber
+ actor App as Application
+ participant Client as Mooncake Client
+ participant Primary as Owner Group Primary
+ participant LocalPool as Local Delegated Extent Pool
+ participant Standby as Standby Replicas
+ participant Storage as Target Storage Nodes
+ participant Allocator as Background Allocation Service
+ rect rgb(239, 246, 255)
+ Note over Primary,Allocator: Background refill, outside per-object critical path
+ Primary-)Allocator: LeaseExtentBatch at low watermark
+ Allocator-)Primary: delegated ranges + budgets
+ end
+ rect rgb(236, 253, 245)
+ Note over App,Storage: Per-object low-latency path
+ App->>Client: Put(key, value)
+ Client->>Primary: PutStart(term, assignment_generation)
+ Primary->>LocalPool: local allocate
+ LocalPool-->>Primary: descriptors
+ Primary-)Standby: async replicate PROCESSING
+ Primary-->>Client: replica descriptors
+ Client->>Storage: TransferWrite to configured replicas
+ Storage-->>Client: transfer completion
+ Client->>Primary: PutEnd
+ Primary-)Standby: async replicate COMPLETE
+ Primary-->>Client: OK
+ Client-->>App: OK
+ end
+ Note over Client,Primary: Same synchronous metadata RPC count as current = 2
+ Note over Primary,Allocator: Normal PutStart allocator RPC = 0, synchronous replication RTT = 0
+
+
+ 该快路径在同步通信形状上与当前实现一致：2 次 metadata RPC、按配置副本数执行数据写、0 次逐对象 allocator RPC、0 次同步复制 RTT。
+ 代价是 PROCESSING/COMPLETE 在 follower 确认前已经向客户端返回，因此 ASYNC_STANDBY 存在非零 RPO。
+ 因而本设计的 Put 语义必须明确公布这一 RPO，不能把异步复制描述为 RPO=0 的强一致提交。
+
+
+ primary 切换后，新 term 不得继续本地切分旧 term 的 delegated range；旧 range 进入 quarantine，等待 lease TTL
+ 到期或 allocator 明确回收，新 primary 获取新 range。这样即使旧 primary 暂时仍可访问数据网络，也只可能在隔离范围内产生孤儿写，
+ 不会与新 primary 重复分配同一 extent。该策略用临时容量占用换取正常 PutStart 的零 allocator RTT。
+
+
+ 6.4.6 Batch：当前实现与新设计
+
+sequenceDiagram
+ autonumber
+ participant Client
+ participant Current as Current Global Master
+ participant G1 as Group G1 Primary
+ participant G2 as Group G2 Primary
+ participant G3 as Group G3 Primary
+ rect rgb(249, 250, 251)
+ Note over Client,Current: Current implementation
+ Client->>Current: BatchGet / BatchPutStart(all keys)
+ Current->>Current: process keys in one Master process
+ Current-->>Client: per-key results
+ end
+ rect rgb(239, 246, 255)
+ Note over Client,G3: New design
+ Client->>Client: local regroup by owner group
+ par one parallel network round
+ Client->>G1: sub-batch 1
+ G1-->>Client: results 1
+ and
+ Client->>G2: sub-batch 2
+ G2-->>Client: results 2
+ and
+ Client->>G3: sub-batch 3
+ G3-->>Client: results 3
+ end
+ Client->>Client: merge in original key order
+ end
+
+
+ 当前跨任意 key 的 Batch 只发 1 个 Master RPC；新设计涉及 G 个 owner 时发送 G 个并行 RPC。
+ 消息数从 2 个单向消息增加为 2G，但理想情况下仍是 1 个串行网络轮次。它提高总吞吐和故障隔离，
+ 但整批完成时延变为所有 sub-batch 的最大值；因此必须支持 per-group deadline、partial result，以及限制最大并发 group 数。
+
+
+ 6.4.7 Storage heartbeat 与后台任务
+
+sequenceDiagram
+ autonumber
+ participant Storage as Storage Client
+ participant Current as Current Global Master
+ participant Object as Object-owner Group
+ participant Control as Control-owner Group
+ rect rgb(249, 250, 251)
+ Note over Storage,Current: Current implementation
+ Storage->>Current: Ping / Offload / Promotion heartbeat
+ Current-->>Storage: status / tasks
+ end
+ rect rgb(239, 246, 255)
+ Note over Storage,Control: New design
+ Object-)Control: batched TaskEnvelope
+ Storage->>Control: heartbeat(term, control_generation)
+ Control-->>Storage: task batch
+ Storage-)Control: batched completion
+ Control-)Object: idempotent completion batch
+ end
+ Note over Storage,Control: Per heartbeat RPC count unchanged, cross-group messages are asynchronous
+
+
+ 每类 heartbeat 的前台 RPC 次数不变，Coordinator 不参与。新增的跨组 TaskEnvelope/Completion 是后台批量通信，
+ 不进入应用 Get/Put 的同步关键路径；如果实现成每对象同步转发，则会放大 RPC，违反本设计。
+
+
+ 6.4.8 通信次数与性能结论
+
+
+
+ 操作
+ 当前实现
+ 新设计常态路径
+ 串行网络轮次变化
+ 性能判断
+
+
+
+
+ Get metadata
+ 1 client-Master RPC
+ 1 client-group-primary RPC；本地路由；无 Coordinator
+ 0
+ 基本不劣化；多 group 可降低锁和 CPU 排队。禁止每 Get 同步复制 lease/hotness。
+
+
+ Get data
+ 1 replica data read
+ 1 replica data read
+ 0
+ 不劣化；动态 read selection 可能改善 locality 和尾延迟。
+
+
+ Put metadata（新设计快路径）
+ PutStart + PutEnd = 2 RPC
+ 2 client RPC；delegated extent 本地分配；replication 异步
+ 0
+ 通信形状与当前一致，目标为基本不劣化；代价是明确的非零 RPO。
+
+
+ Put data
+ 按配置副本数写入目标存储节点
+ 按相同副本数写入目标存储节点
+ 0
+ 不因元数据分片增加；placement/locality 可能改善或恶化，需基准验证。
+
+
+ 跨组 Batch metadata
+ 1 Batch RPC，2 个单向消息
+ G 个并行 sub-RPC，2G 个单向消息
+ 理想值 0；仍为 1 round
+ 消息数增加；吞吐可扩展，但整批 P99.9 受最慢 group 影响。
+
+
+ Storage heartbeat
+ 每类 heartbeat 1 RPC
+ 每类 heartbeat 1 RPC 到 control-owner primary
+ 0
+ 不劣化；跨组任务必须批量异步。
+
+
+ 稳定状态 Coordinator
+ HA client watch backend
+ 周期/watch 更新 ClusterView 和 budget
+ 0（不在请求路径）
+ 不影响单请求时延，但增加低频控制面流量。
+
+
+
+
+
+ 结论：
+ Put 采用 delegated extent 和 ASYNC replication 的唯一快路径，并配合并行 sub-batch、异步 lease/hotness 和后台任务批量化后，
+ Get、单 key Put 和 heartbeat 的同步 RPC/网络轮次可以与当前实现保持一致，因此设计目标可以设为“基本不劣化”。
+ 该 Put 路径采用 ASYNC_STANDBY，允许非零 RPO；接口、监控和验收结果必须明确披露这一语义。
+
+
+ 6.4.9 必须通过的性能验收
+
+ 相同硬件、对象大小、replica 数和并发度下，对比当前单 Master 与采用 ASYNC_STANDBY 的新设计快路径。
+ 分别测试 4 KiB、64 KiB、1 MiB、16 MiB；报告 metadata-only 与 end-to-end 的吞吐、P50、P99、P99.9。
+ Get：单副本 MasterGroup 相比当前实现在无 CPU 饱和时 P99.9 增幅目标不超过 5%；超出则必须给出 route/serialization/queue breakdown。
+ Put：ASYNC + delegated extent 快路径相比当前实现在无 CPU 饱和时 metadata P99.9 增幅目标不超过 5%；分别报告 local allocation、async enqueue 和 data transfer。
+ Batch：固定总 key 数，扫描 G=1/2/4/8/16；报告消息数、最慢 group latency 和 partial-result 时间。
+ 验证 Coordinator 停止 60 秒时稳定 Get/Put 的 RPC 数和时延分布不发生变化。
+ 验证 lease/hotness 更新、TaskEnvelope 和 ResourceView 上报不出现在应用请求 trace 的同步 critical path。
+
+
+ 7. 控制路径流程分析
+ 7.1 MasterNode 加入集群
+
+1. New MasterNode starts with node_id and rpc_address.
+2. It registers membership lease in HA Backend.
+3. Coordinator observes the new node and marks it JOINING.
+4. Coordinator publishes a new ClusterView including the node but assigns no MasterGroup replica yet.
+5. After health check and warmup pass, node becomes ACTIVE.
+6. Coordinator adds it to selected MasterGroups as LEARNER.
+7. After snapshot/replay catches up, each group promotes the learner to VOTER or PRIMARY candidate.
+8. Coordinator may then generate a plan to move SlotGroups between MasterGroups.
+
+
+
+ 加入过程不影响已有 slot 的前台读写。只有当 slot group 迁移进入 cutover 时，相关 slot 的客户端会看到短暂 MOVED/ASK。
+
+
+ 7.2 MasterNode 优雅下线
+
+1. Operator marks MasterNode DRAINING.
+2. Coordinator stops placing new group replicas or primaries on this node.
+3. For every affected MasterGroup, add and catch up a replacement learner.
+4. Transfer primary leadership away, then remove this node from group membership.
+5. Only if group-level load remains imbalanced, move SlotGroups between groups in small batches.
+6. When the node carries no group replica, mark it INACTIVE and stop the process.
+
+
+
+ P99.9 控制点：drain 以小批量迁移执行；当前台 P99.9 超阈值时迁移自动暂停；不做全节点一次性冻结。
+
+
+ 7.3 扩容 Rebalance
+
+1. Coordinator collects load:
+ - per slot group QPS
+ - key count / metadata bytes
+ - foreground P99.9
+ - migration backlog
+ - resource pressure
+
+2. Coordinator selects candidate slot groups:
+ - avoid hottest slot groups first
+ - prefer large imbalance but low active QPS groups
+ - limit concurrent groups
+
+3. Coordinator starts migration:
+ source -> snapshot copy -> incremental replay -> short cutover -> cleanup
+
+4. Coordinator observes P99.9 guard:
+ if P99.9 or queue depth exceeds threshold: reduce budget or pause migration
+
+5. Repeat until target balance reached.
+
+
+ 7.4 缩容 Rebalance
+
+1. Operator marks nodes to remove as DRAINING.
+2. Coordinator computes target placement excluding DRAINING nodes.
+3. Slot groups migrate away in priority order:
+ - cold / low QPS slot groups first
+ - hot slot groups during low-traffic windows
+4. If remaining capacity is insufficient, Coordinator rejects shrink plan.
+5. After migration completes, DRAINING node is removed.
+
+
+
+ 缩容比扩容更容易造成尾延迟，因为目标节点承接更多负载。必须先做容量校验和 P99.9 预算校验，不能只看 slot 数是否均衡。
+
+
+ 7.5 Slot Group 迁移
+
+Plan:
+ Coordinator chooses source MasterGroup, target MasterGroup, slot_group_id, migration_budget.
+
+Prepare:
+ Target group primary creates empty slot group in IMPORTING state.
+ Source group primary marks slot group MIGRATING_SOURCE and records start_oplog_offset.
+
+Snapshot Copy:
+ Source scans metadata in bounded batches.
+ Target group replicates and commits imported batches but does not serve normal requests yet.
+
+Incremental Replay:
+ Source streams OpLog after start_oplog_offset.
+ Target group commits replay and reports committed replay_lag.
+
+Cutover:
+ Source briefly blocks writes for the slot group.
+ Source commits the final OpLog tail and emits a cutover position.
+ Target proves that the cutover position is committed by its replication mode.
+ Coordinator CAS-updates owner_master_group_id and increments assignment_generation with a fencing token.
+ Source group returns MOVED for new requests.
+ Target group serves requests with the new assignment generation.
+
+Cleanup:
+ Source keeps tombstone for late requests, then releases old metadata.
+
+
+ 7.6 MasterNode 故障与 MasterGroup Failover
+
+1. HA Backend lease expires or health checks fail.
+2. Coordinator marks node SUSPECT, then UNAVAILABLE after quorum/lease confirmation.
+3. QUORUM group elects a replica holding the committed prefix; ASYNC_STANDBY group requires Coordinator-authorized promotion and reports possible RPO loss.
+4. New primary proves its commit/replay position and obtains a higher group term/fencing token.
+5. Coordinator publishes the new primary endpoint and term; slot owner and assignment_generation remain unchanged.
+6. Clients receive NOT_PRIMARY/STALE_TERM or refresh view after RPC deadline.
+7. New primary serves foreground traffic; groups without quorum remain unavailable instead of accepting split-brain writes.
+
+
+
+ P99.9 控制点：故障检测不等待长 RPC；客户端使用短 deadline；failover 以 MasterGroup 为粒度并可并行；replica 持续 replay，避免切换后大规模恢复。组内切主不递增 assignment_generation，只有 SlotGroup 改属另一个 MasterGroup 时才递增。
+
+
+ 7.7 Coordinator 故障
+
+1. Active Coordinator lease expires in HA Backend.
+2. Standby Coordinator competes for coordinator lease.
+3. New Coordinator reads persisted ClusterView and migration states.
+4. It reconciles Master reports.
+5. Incomplete migration is either resumed or rolled back.
+6. Foreground metadata requests continue using last stable ClusterView.
+
+
+
+ Coordinator 故障不应影响稳定 slot 的读写，也不阻止具备 quorum 的 MasterGroup 完成组内选主。受影响的是新的扩缩容计划、跨组迁移、primary endpoint 发布和后台预算更新。
+
+
+ 7.8 资源压力与 Admission Control
+
+1. Storage clients report segment usage and health.
+2. MasterGroup primaries report allocation failures and queue depth.
+3. Coordinator computes cluster pressure level.
+4. Coordinator publishes ResourceView and BackgroundBudget.
+5. MasterGroup primaries adjust placement, eviction/offload speed, and admission policy.
+6. If resources remain insufficient, PutStart fails fast with a precise error.
+
+
+
+ P99.9 控制点：资源不足时不能让 PutStart 在 Master 内反复扫描和等待；应快速失败、降级 replica 数，或交给上层重试。
+
+
+ 7.9 冷热分级任务
+
+Get path:
+ GetReplicaList observes LOCAL_DISK-only hot key
+ -> enqueue promotion task
+ -> return current readable replica immediately
+
+Background path:
+ Storage client heartbeat pulls promotion/offload tasks
+ -> executes data movement
+ -> NotifyPromotionSuccess / NotifyOffloadSuccess
+ -> slot owner commits metadata transition
+
+
+
+ P99.9 控制点：冷热任务必须异步化；前台读不等待 promotion 完成；Coordinator 只调节任务预算，不参与单对象决策。
+
+
+ 8. P99.9 稳定性设计
+
+
+
+ 设计点
+ 要求
+ 原因
+
+
+
+
+ Coordinator 非热路径
+ 正常 Get/Put 不访问 Coordinator
+ 避免把 Coordinator 变成新的尾延迟来源
+
+
+ 前台 RPC 优先级
+ MasterNode 按 group 区分 foreground queue 和 background queue
+ 迁移和扫描不能挤占客户请求
+
+
+ 短锁和分片锁
+ 对象状态锁限定在 slot group 或 key 粒度
+ 避免全 shard 锁导致 P99.9 抖动
+
+
+ 迁移预算
+ 限制同时迁移 slot group、snapshot 带宽和 replay QPS
+ 扩缩容期间稳定前台时延
+
+
+ 快速失败
+ 资源不足、fencing token 过期、owner 错误时快速返回明确错误
+ 避免请求排队等待不可满足条件
+
+
+ 客户端 deadline
+ 每个 Master RPC 设置 deadline 和 bounded retry
+ 控制跨 shard Batch 的尾部等待
+
+
+ 自动保护闭环
+ 当 P99.9 超阈值时自动降低后台预算
+ 把客户时延优先级置于扩缩容速度之上
+
+
+
+
+ 9. 4+1 视图
+
+ 9.1 逻辑视图
+
+ 逻辑视图描述系统的核心抽象和职责边界。
+
+
+Object Key Space
+ -> Slot
+ -> Slot Group
+ -> Owner MasterGroup
+ -> Current Primary MasterNode
+
+MasterGroup owns:
+ - ObjectMetadata
+ - LeaseState
+ - ReplicaState
+ - ObjectTaskState
+ - HotnessState
+ - SlotGroupOpLog
+
+Coordinator owns:
+ - ClusterView
+ - SlotOwnership
+ - MigrationPlan
+ - ResourceSummary
+ - BackgroundBudget
+ - GroupPlacement / SlotAssignment
+ - FencingPublication
+
+HA Backend persists:
+ - ClusterView records
+ - Membership leases
+ - Slot assignment generation records
+ - Snapshot catalog
+ - OpLog pointers
+
+
+ 9.2 开发视图
+
+ 开发视图描述代码模块拆分建议。
+
+
+
+
+ 模块
+ 职责
+ 可能落点
+
+
+
+
+ ClusterViewClient
+ 客户端获取、缓存、刷新 slot map
+ mooncake-store/include/master_client.h 附近新增路由层
+
+ ShardRouter
+ 计算 slot，处理 MOVED/ASK，Batch regroup
+ MasterClient 内部组件
+
+
+ MasterNodeService
+ 承载一个或多个 MasterGroup replica 的服务进程
+ 从现有 MasterService 演进
+
+
+ SlotOwnershipManager
+ 校验 master_group_id、term、slot ownership 和 assignment_generation
+ MasterNode 内部
+
+
+ CoordinatorService
+ 集群视图、跨组迁移、fencing 发布、预算控制
+ 新增 coordinator 模块
+
+
+ ResourceRegistry
+ 汇总 segment 容量、水位、健康状态
+ Coordinator 子模块或独立库
+
+
+ MigrationManager
+ slot group snapshot、replay、cutover
+ Coordinator + source/target MasterGroup 协作
+
+
+ LatencyGuard
+ 根据 P99.9 和队列深度调节后台预算
+ Coordinator 子模块
+
+
+
+
+ 9.3 进程视图
+
+ 进程视图描述运行时并发关系和通信路径。
+
+
+Client Process
+ - foreground application threads
+ - MasterClient routing cache
+ - RPC pools per MasterGroup primary
+
+MasterNode Process
+ - foreground RPC workers
+ - background task workers
+ - one or more MasterGroup replicas
+ - per-group slot state store
+ - per-group OpLog append/replay worker
+ - metrics reporter
+ - migration sender/receiver
+
+Coordinator Process
+ - membership watcher
+ - cluster view publisher
+ - migration planner
+ - group placement / fencing publisher
+ - resource aggregator
+ - latency guard loop
+
+HA Backend
+ - lease/session service
+ - durable key-value records
+ - watch/notify stream
+
+
+
+ 前台通信路径：Client -> owner MasterGroup current primary。控制通信路径：MasterNode -> Coordinator -> HA Backend，或 Coordinator -> MasterNode。
+ 两者必须使用独立 RPC 队列或至少独立优先级，防止控制面流量影响客户请求。
+
+
+ 9.4 物理视图
+
+ 物理视图描述部署拓扑。
+
+
+Rack / AZ A:
+ MasterNode A: G1 primary, G2 voter, G3 voter
+ Storage Clients
+
+Rack / AZ B:
+ MasterNode B: G2 primary, G1 voter, G3 voter
+ Storage Clients
+
+Rack / AZ C:
+ MasterNode C: G3 primary, G1 voter, G2 voter
+ Storage Clients
+
+Coordinator:
+ active + standby deployment
+ backed by HA Backend lease
+
+HA Backend:
+ odd-number quorum deployment if using etcd-like backend
+
+
+
+ 同一 MasterGroup 的 replicas 应跨故障域部署，并为任一节点故障后的 primary 提升预留容量。Coordinator active/standby 也应跨节点部署，但 Coordinator 不承载热路径，因此其资源规格可以小于 MasterNode。
+
+
+ 9.5 场景视图
+
+ 场景视图对应架构的 +1，用关键用例验证其他四个视图是否闭合。
+
+
+
+
+ 场景
+ 参与组件
+ 关键判断
+
+
+
+
+ 正常 Get
+ Client、owner MasterGroup primary
+ 不访问 Coordinator；单 group primary 完成并按复制模式提交。
+
+
+ 正常 Put
+ Client、owner MasterGroup primary、Storage Client
+ Master 使用本地 ResourceView；不阻塞等待 Coordinator。
+
+
+ 扩容
+ Coordinator、Source Group、Target Group、Client
+ 迁移按预算执行；cutover 通过 assignment generation 和 MOVED 完成。
+
+
+ 缩容
+ Coordinator、Draining MasterNode、affected MasterGroups
+ 先容量校验，再低 QPS slot group 优先迁移。
+
+
+ MasterNode 故障
+ affected MasterGroups、Coordinator、HA Backend、Client
+ 各组独立提升 term 和 primary；slot owner/assignment_generation 不变。
+
+
+ Coordinator 故障
+ HA Backend、Standby Coordinator、MasterGroups
+ 稳定 slot 前台读写继续；迁移编排暂停后恢复。
+
+
+ P99.9 抖动
+ MasterGroups、Coordinator
+ LatencyGuard 降低后台任务预算，必要时暂停迁移。
+
+
+
+
+ 10. API 与协议建议
+ 10.1 客户端路由 API
+
+GetClusterView() -> ClusterView
+WatchClusterView(view_revision) -> ClusterViewDelta
+
+RequestContext {
+ string tenant_id;
+ uint32_t slot_id;
+ string master_group_id;
+ uint64_t group_term;
+ uint64_t assignment_generation;
+ string request_id;
+}
+
+
+ 10.2 MasterNode / MasterGroup 上报 API
+
+ReportShardStats(ShardStats) -> OK
+ReportMigrationState(MigrationState) -> OK
+ReportResourceUsage(ResourceUsage) -> OK
+
+
+ 10.3 Coordinator 控制 API
+
+SubmitRebalancePlan(RebalancePlan) -> PlanId
+PauseMigration(PlanId) -> OK
+ResumeMigration(PlanId) -> OK
+DrainNode(node_id) -> OK
+UpdateBackgroundBudget(BackgroundBudget) -> OK
+
+
+ 10.4 路由错误码
+
+MOVED(slot_id, target_group_id, target_endpoint, assignment_generation, group_term)
+ASK(slot_id, target_group_id, target_endpoint, assignment_generation, group_term)
+NOT_PRIMARY(master_group_id, primary_endpoint, current_term)
+STALE_TERM(master_group_id, current_term)
+TRYAGAIN(slot_id, reason)
+STALE_ASSIGNMENT(slot_id, current_generation)
+SLOT_NOT_OWNED(slot_id, owner_master_group_id)
+MASTER_READONLY(node_id)
+
+
+ 11. 新开发特性与独立验收清单
+
+ 本节是可直接进入需求和测试系统的 feature backlog。每个特性必须能单独部署或使用 fake/mock 依赖进行验收；
+ “代码已合入”“接口已定义”不算验收完成。除特别说明外，所有故障用例都必须有自动化测试，所有状态变化都必须暴露指标和结构化日志。
+
+
+
+
+
+ ID / 特性
+ 交付边界
+ 独立验收标准
+
+
+
+
+ F01定义 tenant-aware hash、16384 slots、跨语言测试向量和版本号；不包含远程路由。
+ 至少 C++/Python 对 10,000 个固定输入产生完全一致的 slot；进程重启、编译优化级别变化后结果不变；非法 hash version 明确报错。
+
+
+ F02序列化 MasterNode、MasterGroup、SlotGroupOwner、ClientControlOwner 及各自 fencing 字段；提供内存和 HA Backend adapter。
+ view 可 round-trip；CAS 拒绝旧 view_revision；进程重启恢复结果一致；未知字段可前向兼容；损坏记录不被静默接受。
+
+
+ F03提供全量读取、增量 watch、断线续传和 compact 后回退全量读取。
+ 连续提交 1,000 次 view 更新，watcher 不丢失且顺序一致；任意位置断线重连后收敛到最终 view；慢 watcher 不阻塞发布者。
+
+
+ F04实现 key -> slot -> SlotGroup -> MasterGroup -> primary endpoint，校验 group term 和 assignment_generation。
+ 使用 fake view 将全量 slot 路由到预期 endpoint；收到 NOT_PRIMARY、STALE_TERM、MOVED 后在 bounded retry 内刷新并成功；Coordinator 不出现在正常 Get/Put 调用链。
+
+
+ F05按 MasterGroup 并行拆分 Batch，并按输入顺序合并逐 key 结果。
+ 混合至少 3 个 group、重复 key 和部分失败时，输出顺序及错误一一对应；一个 group 超时不取消已完成 group；总等待不超过配置 deadline。
+
+
+ F06实现 node register、lease renew、SUSPECT、UNAVAILABLE、DRAINING、INACTIVE 状态机。
+ fake clock 下验证每个合法转换；lease 到期在规定窗口内标记 UNAVAILABLE；旧 node_incarnation_id 无法续租或覆盖重启后的新实例；非法跳转被拒绝并记录原因。
+
+
+ F07一个 MasterNode 承载多个逻辑 group；每组隔离元数据、OpLog、snapshot、指标和 RPC queue。
+ 单进程启动至少 32 个 group 并写入相同 key，各组数据互不串扰；单组 snapshot/restart 恢复不影响其他组；可按 group 查询 CPU、队列和 metadata bytes。
+
+
+ F08所有写请求携带 master_group_id + group_term；旧 primary 无法提交写。
+ 将 term 从 N 提升到 N+1 后，term=N 的 Put/Remove/任务完成通知全部返回 STALE_TERM；并发压力下旧 term 成功提交数为 0；读写错误包含 current term。
+
+
+ F09实现 3+ voter 的日志复制、commit index、leader election 和 learner catch-up。
+ 3 节点组在任意 1 节点故障后已确认写不丢失并恢复服务；隔离旧 primary 后不能形成双写；失去多数派时拒绝写；learner 追平后状态 hash 与 leader 一致。
+
+
+ F10实现异步 OpLog replay、replay position、显式提升和 RPO 暴露；不宣称 quorum 语义。
+ 注入 replay lag 后指标准确反映差值；提升时返回可能丢失的 position/bytes；未取得 fencing token 的 follower 不可提升；文档/API 明确返回当前 ReplicationMode。
+
+
+ F11Coordinator 依据故障域和容量放置 replicas，采用 learner-add/catch-up/promote/remove 流程。
+ 输入多机架拓扑时同组 replicas 不共故障域；目标不满足约束时计划被拒绝；成员变更中断后可恢复或回滚；任一时刻不产生两个有效 primary。
+
+
+ F12把 16384 slots 完整且无重叠地分配给多个 MasterGroup，并在服务端强制校验归属。
+ 覆盖检查证明每个 slot 恰好一个 owner；非 owner 请求返回包含目标 group 的 MOVED；旧 assignment_generation 返回 STALE_ASSIGNMENT；组内切主不改变 assignment_generation。
+
+
+ F13实现 IMPORTING、snapshot copy、committed replay、cutover fencing、MOVED 和 source cleanup。
+ 持续并发 Put/Get 时迁移一个 SlotGroup，cutover 后 source/target 对象状态 hash 一致、已确认写零丢失、同一 key 无双 owner；任一阶段 kill/restart 均能恢复或安全回滚。
+
+
+ F14按 client_id 将同一 Storage Client 的全部 segments 绑定到一个 MasterGroup，支持 control_generation 和 heartbeat redirect。
+ 同一 client 的不同介质 segment 始终落到同组；组内切主只改变 endpoint/term；跨组迁移只提升 control_generation；旧 control owner 无法接受新 heartbeat 或控制写。
+
+
+ F15对象 owner 提交 TaskIntent，至少一次投递到 segment control-owner group，并以 task_id 幂等完成。
+ 在投递前、投递后、执行后分别注入崩溃，最终任务恰好产生一次对象状态变化；重复消息不重复分配/释放；任一 term/generation 过期时刷新路由而不是执行。
+
+
+ F16上报容量、读写带宽、IOPS、并发度、延迟桶、实时使用量、reserved、队列及采样时间。
+ fake Storage Client 上报后 ResourceView 在时限内可见；同一 segment_incarnation 下较小 sample_seq 的样本不会覆盖新样本；超过 freshness 阈值的 segment 被标为 stale；读/写及不同对象大小桶不被合并。
+
+
+ F17以 lease/CAS 原子预留 bytes、bandwidth、IOPS，支持 TTL、幂等键、commit 和 revoke。
+ 100 个并发请求不会使任一资源维度超卖；相同幂等键只产生一份 reservation；TTL 后资源可回收；commit/revoke 重复调用结果稳定。
+
+
+ F18实现硬约束过滤、可插拔评分、Top-K 有界 reserve 和 ScoreBreakdown；首版使用确定性加权评分。
+ 容量不足、故障域冲突和能力不匹配的候选永不入选；固定快照与 policy version 输出完全可重放；每次决策可解释各分项；reserve 尝试不超过配置 K。
+
+
+ F24Storage Client allocator 批量租出互不重叠的 extent range 和多维预算；MasterGroup 在租约内本地分配、后台续租和归还。
+ 并发向两个 group 发放租约时 extent 零重叠；旧 segment_incarnation/control_generation 或过期 lease 无法 commit；正常 PutStart 不产生 allocator RPC；耗尽时有界触发后台批量 refill，当前请求快速失败而不无限等待。
+
+
+ F19在已有 replicas 中根据健康、实时队列、带宽、时延和 locality 排序，不改变持久 placement。
+ 注入慢副本后新读流量在观测窗口内转移，恢复后按 hysteresis 回切；无健康副本时返回明确错误；选择变化不产生数据迁移或 metadata ownership 变化。
+
+
+ F20分别限制迁移、snapshot、offload、promotion、eviction；根据前台 P99.9 和队列闭环降速/恢复。
+ 压测使 P99.9 超阈值后一个控制窗口内降低预算，连续健康窗口后渐进恢复；前台流量不经过 Coordinator；预算过期时采用保守默认值。
+
+
+ F21active/standby lease、fencing token、持久计划恢复；不承担 MasterGroup 日志提交。
+ 双实例竞争时最多一个可提交 view；kill active 后 standby 恢复未完成迁移状态；切换期间稳定 group 的 Get/Put 持续成功；旧 Coordinator CAS 全部失败。
+
+
+ F22提供 per-node/group/slot/client/segment 指标、结构化事件和只读审计 API。
+ 自动审计能检测 slot 重叠/缺口、同 group 双 primary、replica 跨域违规、stale fencing token 和 reservation 泄漏；健康集群误报为 0；每个异常可定位到实体 ID 和 view revision。
+
+
+ F23可注入进程退出、网络分区、延迟、丢包、磁盘错误、时钟推进和旧版本客户端；输出确定性报告。
+ CI 可独立复现指定 seed；至少覆盖旧 primary 隔离、迁移中断、watch 断线、重复任务、过期 ResourceView；失败报告包含 seed、时间线和最终不变量检查。
+
+
+
+
+ 11.1 通用 Definition of Done
+
+ 每项特性有独立 feature flag 或明确的启用条件，关闭时不改变现有单 Master 行为。
+ 新增持久化结构必须包含 schema/version，并至少验证一次旧版本读取新版本中已知字段的兼容路径。
+ 所有重试均有 deadline、最大次数和幂等键；验收不得依赖无限等待或人工查看日志。
+ 性能相关特性必须同时报告吞吐、P50、P99、P99.9 和资源占用，并保存可比较的基线。
+ 验收报告必须记录代码版本、配置、拓扑、随机 seed、开始/结束时间和不变量检查结果。
+
+
+ 11.2 依赖关系与并行开发边界
+
+基础契约： F01 -> F02 -> F03
+客户端： F01 + F02 -> F04 -> F05
+Group： F02 -> F07 -> F08 -> {F09 | F10} -> F11
+Slot： F04 + F07 -> F12 -> F13
+Segment： F02 + F07 -> F14 -> F15
+资源调度： F16 -> F17 -> {F18, F24}; F18 + F24 -> F19
+保护闭环： F16 -> F20
+系统 HA： F03 + F11 + F13 -> F21
+质量体系： F22、F23 从第一阶段开始，并为其他特性提供验收 harness
+
+
+ 箭头表示生产集成依赖，不表示验收必须串行。例如 F04 可用 fake ClusterView 验收，F15 可用两个内存 MasterGroup
+ 和 fake heartbeat 验收，F18 可使用固定 ResourceView 快照验收。这样各团队可以并行开发，同时保持每项交付可独立判定成功或失败。
+
+
+ 12. 演进计划
+
+
+
+ 阶段
+ 内容
+ 收益
+
+
+
+
+ 阶段一
+ 单 Master 暴露 16384 slot 的 ClusterView；客户端实现 slot 计算和路由缓存。
+ 不改变部署即可验证客户端路由逻辑。
+
+
+ 阶段二
+ 多个单副本 MasterGroup 静态分配 SlotGroup；Batch regroup；MOVED/STALE_ASSIGNMENT。
+ 核心元数据请求横向扩展。
+
+
+ 阶段三
+ 引入轻量 Coordinator，管理 view、resource summary 和后台预算。
+ 为扩缩容、P99.9 保护和资源协同打基础。
+
+
+ 阶段四
+ 实现 slot group 在线迁移、snapshot copy、incremental replay、ASK/MOVED cutover。
+ 支持在线扩缩容和 rebalance。
+
+
+ 阶段五
+ 为 MasterGroup 增加 replica、term、组内选主与 fencing；实现 Coordinator active/standby。
+ 降低故障影响范围，提高可用性。
+
+
+
+
+ 13. 风险与约束
+
+
+
+ 风险
+ 影响
+ 缓解措施
+
+
+
+
+ Coordinator 被误用到热路径
+ 成为新的 P99.9 瓶颈
+ 接口约束：Get/Put 不调用 Coordinator；客户端只周期性或按错误刷新 view。
+
+
+ ResourceView 过期
+ PutStart 选择到高水位或不可用 segment
+ allocator lease/CAS 二次确认；失败快速换候选或返回错误。
+
+
+ 迁移 replay lag 过大
+ cutover 窗口变长
+ lag 超阈值时暂停新迁移，降低前台写入冻结时间。
+
+
+ 跨 slot Batch 尾延迟
+ 最慢 shard 拉高整批响应
+ per-shard deadline、partial result、bounded retry。
+
+
+ 热点 slot group
+ 单 shard 局部热点
+ rebalance 避免迁移最热 group；必要时支持 hash tag 约束和 key-level split。
+
+
+ Coordinator split-brain
+ 生成冲突 view 或迁移计划
+ Coordinator lease + fencing token；所有 view 更新通过 HA Backend CAS。
+
+
+ term、assignment_generation、control_generation 混用
+ 普通切主触发无谓 slot 迁移，或旧 primary/旧 control owner 未被正确隔离
+ 分别校验组内领导权、SlotGroup 归属和 client 控制归属；协议字段与错误码独立。
+
+
+ 多个 group replicas 共置于同一故障域
+ 单节点或单机架故障同时失去 quorum
+ group placement 使用反亲和约束，并在部署前做 N-1 failover capacity 校验。
+
+
+ 异步主备被当作强一致复制
+ 故障提升后丢失已确认写或产生分叉
+ ClusterView 明确暴露 ReplicationMode、commit/replay position 和可承诺的 RPO；需要 RPO=0 时使用 quorum 提交。
+
+
+
+
+ 14. 最终建议
+
+
+ 轻量 Coordinator 路线是 Mooncake 分布式元数据面的推荐落地方案：它保留 Redis Cluster 风格的客户端直连和 slot redirect，避免中心化热路径；同时补足 Mooncake 相比 Redis 多出来的物理资源管理、冷热分级、slot 迁移限速和 P99.9 自动保护能力。
+
+
+ 实施时应坚持四条红线：Coordinator 不代理前台请求；slot owner 是 MasterGroup 而不是裸 MasterNode；
+ term、assignment_generation、control_generation 分别保护组内领导权、slot 归属和 segment 控制归属；segment_incarnation 隔离重建前后的物理空间；所有迁移动作都必须带 fencing 和可观测的 P99.9 保护阈值。
+ 只要这些边界成立，该设计既能横向扩展元数据吞吐，也能在扩缩容和故障场景下保持尾延迟可控。
+
+
+
+
diff --git a/docs/source/design/ssd-balance-allocation.md b/docs/source/design/ssd-balance-allocation.md
new file mode 100644
index 0000000000..5b314c8563
--- /dev/null
+++ b/docs/source/design/ssd-balance-allocation.md
@@ -0,0 +1,264 @@
+# SSD负载均衡分配策略设计文档
+
+## 1. 概述
+
+### 1.1 问题背景
+
+现有 `FreeRatioFirstAllocationStrategy` 在选择segment时只考虑DDR空闲比例，忽略了SSD水位。这导致以下问题：
+
+- 一个segment的DDR空闲但SSD已满时，数据仍被分配到该segment
+- 后续eviction时无法offload到SSD（因为SSD已满），DDR产生backpressure
+- 最终DDR被填满，整个节点无法接受新写入
+
+### 1.2 解决方案
+
+新增 `SsdBalanceAllocationStrategy`，按SSD空闲比例做负载均衡：
+
+- 默认只看SSD水位（alpha=0），优先选择SSD空闲的节点
+- SSD达到高水位时禁止向该节点写入，但不驱逐SSD数据
+- DDR达到驱逐水位时临时禁止写入，水位下降后自动恢复
+
+### 1.3 适用场景
+
+多节点集群中每个节点有DDR+本地SSD的分层存储环境。
+
+## 2. 设计目标
+
+| 目标 | 说明 |
+|------|------|
+| SSD比例均衡 | 按SSD空闲比例选择segment，优先写入SSD空闲的节点 |
+| SSD驱逐保护 | SSD达到高水位时禁止写入，绝不驱逐SSD数据（避免数据丢失） |
+| DDR准入控制 | 每个segment的DDR达到准入水位时禁止向该segment分配，自动fallback到其他segment |
+| 全满暂停 | 所有节点DDR都满时暂停所有put，返回 DDR_ADMISSION_REJECTED（-201），不触发eviction |
+
+## 3. 核心算法
+
+### 3.1 SSD比例计算
+
+```
+ssd_free_ratio = (ssd_total_capacity - ssd_used_bytes) / ssd_total_capacity
+```
+
+- 无SSD信息的segment：`ssd_free_ratio = 1.0`（不约束）
+- `ssd_used_bytes` 通过 `std::atomic` 跟踪，在offload成功时递增，磁盘驱逐时递减
+
+### 3.2 候选采样与排序
+
+```
+1. 采样 min(6 * replica_num, total_segments) 个候选segment
+2. 排除SSD使用率 >= ssd_high_watermark_ratio 的segment
+3. 按ssd_free_ratio降序排序
+4. 从top-N候选中尝试分配
+5. 如果replica_num未满足，fallback到随机分配
+```
+
+### 3.3 SSD高水位保护
+
+当segment的SSD使用率 >= `ssd_high_watermark_ratio`（默认0.90）时：
+
+- **禁止**向该segment分配新数据
+- **绝不驱逐**SSD上的已有数据（驱逐意味着数据不可恢复丢失）
+- SSD数据只能通过以下方式释放：
+ - 正常promotion（访问命中后提升回DDR）
+ - TTL过期（软pin到期后自动清理）
+- SSD水位下降后，节点自动恢复可写状态
+
+### 3.4 DDR写入准入控制（per-segment）
+
+通过 `--ddr_admission_watermark_ratio`（默认 0.0，即禁用）设定每个 segment 的 DDR 准入水位。
+当 segment 的 DDR 使用率 >= 该水位时：
+
+- 分配策略**跳过**该 segment，尝试分配到其他 segment
+- 所有 segment 都被跳过时，返回 `DDR_ADMISSION_REJECTED`（-201）
+- **不设置** `need_mem_eviction_`（避免触发 eviction 驱逐已有数据，DDR 数据零丢失）
+- 其他 segment DDR 下降（eviction 释放空间或 offload 完成）后自动恢复
+
+与 eviction 的关系：
+- `ddr_admission_watermark_ratio`（如 0.90）应设得**低于** `eviction_high_watermark_ratio`（0.95）
+- 准入阻写先于 eviction 驱逐发生，保护 DDR 数据不被驱逐
+- 如果所有 segment 都超过准入水位也无 eviction 触发，put 暂停直到有 segment 释放空间
+
+使用方式：
+
+```bash
+./mooncake_master --allocation_strategy=ssd_balance \
+ --ddr_admission_watermark_ratio=0.90
+```
+
+## 4. 决策流程
+
+### 4.1 AllocateAndInsertMetadata流程
+
+```
+AllocateAndInsertMetadata()
+│
+├── 获取AllocatorManager和SsdMetricsProvider
+│
+├── 调用 SsdBalanceAllocationStrategy::Allocate()
+│ │
+│ ├── 处理preferred segments
+│ │ ├── 检查SSD水位，跳过高水位segment
+│ │ └── 检查DDR准入水位，跳过超标segment
+│ │
+│ ├── 候选采样 + SSD比例排序
+│ │ ├── 排除excluded/used segments
+│ │ ├── 排除SSD高水位segments
+│ │ ├── 排除DDR准入水位超标的segments
+│ │ └── 按ssd_free_ratio降序排序，取top-N
+│ │
+│ ├── Fallback随机分配
+│ │ └── 同样排除SSD高水位和DDR准入超标segments
+│ │
+│ └── 返回结果
+│ ├── 有可用segment → replicas
+│ ├── 被DDR准入拒绝 → DDR_ADMISSION_REJECTED（-201）
+│ │ └── 不设need_mem_eviction_，保护DDR数据
+│ └── 其他原因失败 → NO_AVAILABLE_HANDLE（-200）
+│ └── 设need_mem_eviction_，触发eviction释放空间
+│
+└── 返回结果给客户端
+```
+
+### 4.2 SSD水位检查
+
+```
+isSsdHighWatermark(segment_name)
+│
+├── 查询SsdMetricsProvider
+│ ├── total = getSsdTotalCapacity(segment_name)
+│ └── used = getSsdUsedBytes(segment_name)
+│
+├── total <= 0?
+│ └── 返回false（无SSD信息，不阻塞）
+│
+└── used/total >= ssd_high_watermark_ratio?
+ ├── YES → 排除该segment
+ └── NO → 允许分配
+```
+
+### 4.3 DDR准入水位检查
+
+```
+isDdrHighWatermark(segment_name)
+│
+├── ddr_admission_watermark_ <= 0.0?
+│ └── 返回false（未启用DDR准入）
+│
+├── ddr_admission_watermark_ >= 1.0?
+│ └── 返回false（显式禁用）
+│
+├── 查询SsdMetricsProvider
+│ └── ratio = getDdrUsedRatio(segment_name)
+│ └── MasterMetricManager.get_segment_mem_used_ratio()
+│
+└── ratio >= ddr_admission_watermark_?
+ ├── YES → 排除该segment
+ └── NO → 允许分配
+```
+
+## 5. SSD使用量追踪
+
+### 5.1 数据结构
+
+`LocalDiskSegment` 新增字段：
+
+```cpp
+std::atomic ssd_used_bytes{0};
+```
+
+### 5.2 更新时机
+
+| 事件 | 操作 | 触发位置 |
+|------|------|----------|
+| offload成功 | `ssd_used_bytes += data_size` | `NotifyOffloadSuccess` |
+| 磁盘replica被驱逐 | `ssd_used_bytes -= object_size` | `EvictDiskReplica` |
+
+### 5.3 暴露接口
+
+通过 `SsdMetricsProvider` 接口：
+
+```cpp
+class SsdMetricsProvider {
+ virtual int64_t getSsdTotalCapacity(const std::string& segment_name) const = 0;
+ virtual int64_t getSsdUsedBytes(const std::string& segment_name) const = 0;
+ virtual double getDdrUsedRatio(const std::string& segment_name) const {
+ return 0.0; // 默认不检查DDR
+ }
+};
+```
+
+`ScopedLocalDiskSegmentAccess` 实现该接口，通过 segment_name → client_id → LocalDiskSegment 查找。
+
+## 6. 配置参数
+
+### 6.1 Master 启动参数
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `--allocation_strategy` | `random` | 设为 `ssd_balance` 启用本策略 |
+| `--ssd_high_watermark_ratio` | `0.90` | SSD使用率上限，超过则禁止向该节点写入 |
+| `--ddr_admission_watermark_ratio` | `0.0` | DDR准入水位（0.0 = 禁用），低于此值则禁止向该segment分配 |
+
+### 6.2 环境变量（存储后端驱逐保护）
+
+| 变量 | 默认值 | 说明 |
+|------|--------|------|
+| `MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION` | `false` | 强制禁止SSD驱逐，即使 eviction_policy 非 NONE 也不驱逐 |
+
+### 6.3 错误码
+
+| 错误码 | 值 | 触发条件 |
+|--------|-----|----------|
+| `NO_AVAILABLE_HANDLE` | -200 | 分配失败（段满或其他原因），触发 eviction |
+| `DDR_ADMISSION_REJECTED` | -201 | DDR准入水位拒绝分配，**不触发** eviction |
+
+启用方式：
+
+```bash
+./mooncake_master --allocation_strategy=ssd_balance \
+ --ssd_high_watermark_ratio=0.90 \
+ --ddr_admission_watermark_ratio=0.90
+```
+
+## 7. 代码结构
+
+### 7.1 新增/修改文件
+
+| 文件 | 变更类型 | 说明 |
+|------|----------|------|
+| `include/allocation_strategy.h` | 修改 | 新增 `SsdMetricsProvider` 接口（含 `getDdrUsedRatio`）、`SsdBalanceAllocationStrategy` 类（含 `isDdrHighWatermark`）、更新工厂函数 |
+| `include/types.h` | 修改 | `AllocationStrategyType` 枚举新增 `SSD_BALANCE`；新增 `DDR_ADMISSION_REJECTED` 错误码 |
+| `include/segment.h` | 修改 | `LocalDiskSegment` 新增 `ssd_used_bytes`；`ScopedLocalDiskSegmentAccess` 实现 `SsdMetricsProvider`（含 `getDdrUsedRatio`） |
+| `src/segment.cpp` | 修改 | 实现 `getSsdTotalCapacity`、`getSsdUsedBytes`、`getDdrUsedRatio` |
+| `include/master_config.h` | 修改 | 新增 `ssd_high_watermark_ratio`、`ddr_admission_watermark_ratio` 配置字段 |
+| `src/master.cpp` | 修改 | 新增 `--ssd_high_watermark_ratio`、`--ddr_admission_watermark_ratio` gflag |
+| `include/master_service.h` | 修改 | 新增 `ssd_high_watermark_ratio_` 成员 |
+| `src/master_service.cpp` | 修改 | 分配策略传 SSD/DDR provider、SSD使用量追踪、分发 DDR_ADMISSION_REJECTED（不触发 eviction） |
+| `src/client_service.cpp` | 修改 | 处理 `DDR_ADMISSION_REJECTED` 错误码（日志 + 重试） |
+| `include/storage_backend.h` | 修改 | `BucketBackendConfig` 新增 `disable_ssd_eviction` 字段 |
+| `src/storage_backend.cpp` | 修改 | `PrepareEviction` 检查 `disable_ssd_eviction`；`IsEnableOffloading` 跳过eviction分支 |
+
+### 7.2 类继承关系
+
+```
+AllocationStrategy (抽象基类)
+├── RandomAllocationStrategy
+│ └── FreeRatioFirstAllocationStrategy
+│ └── SsdBalanceAllocationStrategy ← 新增
+└── CxlAllocationStrategy
+
+SsdMetricsProvider (抽象接口)
+└── ScopedLocalDiskSegmentAccess ← 新增实现
+```
+
+## 8. 验证方案
+
+详见 `mooncake-wheel/tests/verify_ssd_balance.py` 和 `tests/ssd_balance_test_guide.md`。
+
+| 测试 | 验证内容 |
+|------|----------|
+| `load_balancing` | 2个Client不对称SSD，验证数据按SSD空闲比例分布 |
+| `ssd_high_watermark_blocking` | SSD达到90%高水位后offload完成，验证新分配被拒绝 + 初始数据可读 |
+| `ssd_eviction_protection` | 启用FIFO驱逐+`MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION=true`，验证已有SSD数据不被驱逐 |
+| `ddr_admission` | 设置 `--ddr_admission_watermark_ratio=0.90`，DDR满时拒绝写入，不触发eviction |
+| `all_ssd_full` | 所有节点SSD满后全局拒绝，释放后恢复 |
diff --git a/docs/spdiag_integration_guide.md b/docs/spdiag_integration_guide.md
new file mode 100644
index 0000000000..0c37fda692
--- /dev/null
+++ b/docs/spdiag_integration_guide.md
@@ -0,0 +1,205 @@
+# Mooncake SpDiag 两层集成使用指南
+
+Mooncake 在配置阶段提供两个互斥的 SpDiag 编译层。构建完成后不能在运行时切换层级；如需切换，应重新配置并编译。
+
+| 层 | 配置 | SpDiag 来源 | 生成结果 |
+|---|---|---|---|
+| Layer 0：Mock | `MOONCAKE_ENABLE_SPDIAG=OFF` | Mooncake 拉取固定源码头文件 | PerfPoint 为空实现，无 SpDiag 运行时依赖 |
+| Layer 1：System | `MOONCAKE_ENABLE_SPDIAG=ON` | 用户预装的 SpDiag RPM | Mooncake 链接系统 `.so`，并可将同版本 CLI/`.so` 打入单一 RPM |
+
+## 1. Layer 0：默认 Mock
+
+### 1.1 配置与编译
+
+```bash
+cmake -S . -B build \
+ -DMOONCAKE_ENABLE_SPDIAG=OFF
+cmake --build build --parallel
+```
+
+`MOONCAKE_ENABLE_SPDIAG` 默认值为 `OFF`，因此该参数可以省略。
+
+Mooncake 使用 FetchContent 拉取固定修订的 SpDiag 源码，只消费 `include/spdiag` 公共头文件，并通过统一目标 `SpDiag::spdiag_lib` 传播 `SPDIAG_DISABLE`。
+
+SpDiag 头文件中的 `SPDIAG_DISABLE` 分支将 PerfPoint 构造、`Start()`、`End()` 和 `Abandon()` 编译为空实现。因此：
+
+- Mooncake 打点源码保持不变；
+- Mooncake ELF 不依赖 `libspdiag.so`；
+- 不生成或打包 SpDiag CLI；
+- 不创建 SpDiag SHM；
+- 系统中已安装的 SpDiag 不参与该构建。
+
+离线环境可以指定已准备好的 SpDiag 源码目录：
+
+```bash
+cmake -S . -B build \
+ -DMOONCAKE_ENABLE_SPDIAG=OFF \
+ -DMOONCAKE_SPDIAG_SOURCE_DIR=/opt/src/spdiag
+```
+
+## 2. Layer 1：系统 SpDiag
+
+### 2.1 前置条件
+
+Layer 1 不下载或编译真实 SpDiag。配置 Mooncake 前，用户必须先安装完整的 SpDiag 运行时 RPM 和开发 RPM。安装结果必须同时提供：
+
+- `SpDiagConfig.cmake`；
+- 导入目标 `SpDiag::spdiag_lib`；
+- 共享库 `libspdiag.so`；
+- 可被系统找到的 `spdiag` CLI；
+- `SPDIAG_ENABLE_PERCENTILE`；
+- `SPDIAG_ENABLE_PERFLOG`。
+
+SpDiag RPM 构建时应至少启用：
+
+```text
+SPDIAG_BUILD_SHARED=ON
+ENABLE_PERCENTILE=ON
+ENABLE_PERFLOG=ON
+```
+
+Mooncake 不固定 Layer 1 的 SpDiag 版本或源码 SHA。它通过 RPM 数据库分别查询 `libspdiag.so` 与 CLI 的 `VERSION-RELEASE.ARCH`，并要求二者完全一致。SpDiag RPM 中其他功能的启用情况和功能正确性由 SpDiag 发布包负责。
+
+### 2.2 配置与编译
+
+标准系统路径：
+
+```bash
+cmake -S . -B build-spdiag \
+ -DMOONCAKE_ENABLE_SPDIAG=ON
+cmake --build build-spdiag --parallel
+```
+
+自定义 RPM 安装前缀：
+
+```bash
+cmake -S . -B build-spdiag \
+ -DMOONCAKE_ENABLE_SPDIAG=ON \
+ -DCMAKE_PREFIX_PATH=/opt/spdiag \
+ -DCMAKE_PROGRAM_PATH=/opt/spdiag/bin
+```
+
+也可以直接指定 CMake package 和 CLI：
+
+```bash
+cmake -S . -B build-spdiag \
+ -DMOONCAKE_ENABLE_SPDIAG=ON \
+ -DSpDiag_DIR=/opt/spdiag/lib64/cmake/SpDiag \
+ -DMOONCAKE_SPDIAG_SYSTEM_CLI=/opt/spdiag/bin/spdiag
+```
+
+### 2.3 配置检查
+
+ON 模式依次确认：
+
+1. 找到 SpDiag CMake package 和 `SpDiag::spdiag_lib`；
+2. package 导出 `SHARED_LIBRARY` 目标；
+3. target 传播 P99 与 PerfLog 能力宏；
+4. 找到 `spdiag` CLI；
+5. `.so` 与 CLI 均由已安装的 RPM 提供；
+6. 两者的 `VERSION-RELEASE.ARCH` 完全一致。
+
+任一条件不满足都会停止配置。Mooncake 不会回退到源码构建；应先修复或重新安装 SpDiag RPM，再重新配置 Mooncake。
+
+## 3. 构建 Mooncake RPM
+
+### 3.1 Mock RPM
+
+```bash
+bash scripts/build_rpm.sh build rpm-output "$(uname -m)"
+rpm -qlp rpm-output/mooncake-*.rpm
+```
+
+Mock RPM 包含 Mooncake 二进制，但不包含：
+
+```text
+/usr/bin/spdiag
+/usr/lib64/libspdiag.so*
+/etc/spdiag/spdiag.conf
+```
+
+### 3.2 System RPM
+
+```bash
+bash scripts/build_rpm.sh build-spdiag rpm-output "$(uname -m)"
+rpm -qlp rpm-output/mooncake-*.rpm
+```
+
+System RPM 包含：
+
+```text
+/usr/bin/mooncake_master
+/usr/bin/mooncake_client
+/usr/bin/spdiag
+/usr/lib64/libspdiag.so*
+/etc/spdiag/spdiag.conf # 系统安装提供该配置时
+```
+
+打包脚本读取 `build-spdiag/mooncake_spdiag.env`，并复制 CMake 配置阶段已经选中的 CLI、共享库及其符号链接。CLI 与共享库的版本一致性在 CMake 配置阶段完成检查；如果构建机上的 SpDiag RPM 随后发生变化，应重新配置后再打包。
+
+## 4. 安装与运行
+
+Mooncake RPM 使用标准系统路径安装 Mooncake 和 SpDiag 运行时：
+
+```text
+/usr/bin/mooncake_master
+/usr/bin/mooncake_client
+/usr/bin/spdiag
+/usr/lib64/libspdiag.so*
+/etc/spdiag/spdiag.conf
+```
+
+安装和基础运行命令：
+
+```bash
+sudo rpm -Uvh rpm-output/mooncake-*.rpm
+sudo ldconfig
+
+spdiag --version
+spdiag start
+spdiag status
+
+# 启动 Mooncake master/client 并执行实际读写负载
+
+spdiag show
+spdiag show --detail
+```
+
+PerfLog、P99、历史数据和 CSV 参数以所安装 SpDiag CLI 的帮助为准：
+
+```bash
+spdiag --help
+spdiag show --help
+```
+
+## 5. 常见错误
+
+### `.so` 或 CLI 不受 RPM 管理
+
+Layer 1 面向系统 SpDiag RPM，不接受手工复制的散装文件。请安装完整的 SpDiag 运行时 RPM 和开发 RPM 后重新配置 Mooncake。
+
+### `.so` 与 CLI 的 RPM 版本不同
+
+卸载冲突版本，并安装同一发布批次的 SpDiag RPM。Mooncake 比较 `VERSION-RELEASE.ARCH`，不通过文件 SHA 判断。
+
+### 找到静态库
+
+重新构建 SpDiag RPM，并设置 `SPDIAG_BUILD_SHARED=ON`。
+
+### 缺少 P99 或 PerfLog
+
+重新构建 SpDiag RPM，并设置：
+
+```text
+ENABLE_PERCENTILE=ON
+ENABLE_PERFLOG=ON
+```
+
+### 从 Mock 切换到 System
+
+推荐使用新的构建目录：
+
+```bash
+cmake -S . -B build-spdiag \
+ -DMOONCAKE_ENABLE_SPDIAG=ON
+```
diff --git a/docs/yh/log-reference.md b/docs/yh/log-reference.md
new file mode 100644
index 0000000000..2bcc96b6d3
--- /dev/null
+++ b/docs/yh/log-reference.md
@@ -0,0 +1,1153 @@
+# Mooncake Store 日志参考手册
+
+本文档描述 `get` / `get_batch` / `get_into` / `batch_get_into` / `put` / `put_batch` 六个操作的全链路日志输出，以及传输任务层、URMA 建连层、Client 创建等新增日志。
+
+---
+
+## 0. 日志系统概览
+
+### 0.1 两套日志系统
+
+当前代码中存在两套日志系统：
+
+| 系统 | 使用文件 | 宏 | trace_id 前缀 | 输出方式 |
+|------|---------|-----|--------------|---------|
+| **MC_LOG 系统** | `real_client.cpp`、`client_service.cpp`、`transfer_task.cpp` | `MC_LOG` / `MC_VLOG` | 是 | 异步队列 |
+| **原生 glog** | `store_py.cpp`（batch 操作）、`urma_endpoint.cpp` | `LOG` / `VLOG` | 否 | 同步直接输出 |
+
+MC_LOG 系统的每条日志自动带有 `trace_id[xxx] ` 前缀（无 trace 上下文时为 `trace_id[none] `）。原生 glog 日志无此前缀。
+
+**注意**：当两层日志混合输出时（如 `get_batch`），原生 glog 日志与 MC_LOG 日志可能因异步队列导致顺序不完全一致。
+
+### 0.2 MC_LOG 异步队列
+
+MC_LOG 通过 `AsyncLogMessage` 临时对象在析构时将日志条目入队，后台单线程消费并调用 glog 输出。队列容量 8192 条，满时阻塞写入线程。FATAL 级别绕过队列直接同步输出。进程退出时通过 `atexit` 处理器刷出剩余日志。
+
+实现文件：`mooncake-common/include/mooncake_logging.h`、`mooncake-common/src/mooncake_logging.cpp`。
+
+### 0.3 TraceId 系统
+
+- **生成**：`NewTraceId()` 使用 `(PID << 48) ^ (steady_clock_ns & 0x0000FFFFFFFF0000) ^ atomic_counter++` 生成全局唯一 ID
+- **线程传递**：`ScopedTraceId` 通过 `thread_local` 保存/恢复当前线程的 trace_id
+- **同步调用链传递**：`RealClient` 入口创建 `ScopedTraceId(NewTraceId())` 后，同线程下游函数通过 `CurrentTraceId()` 读取同一个 trace_id
+- **异步任务传播**：提交线程用 `CurrentTraceId()` 捕获当前 trace_id，并写入 `MemcpyTask`、`FilereadTask` 或线程池 lambda；工作线程执行时通过 `ScopedTraceId` 恢复，确保异步路径日志可追踪
+- **上下文恢复**：`ScopedTraceId` 析构时恢复旧值，避免线程池复用、嵌套调用或提前返回后把上一个请求的 trace_id 带到后续日志
+
+---
+
+## 1. `get` 日志链路
+
+> Python 绑定层（`store_py.cpp`）的单 key `get` 生命周期日志（`get start/complete/slow`）已移除，改为由 `real_client.cpp` 输出慢操作告警。
+
+正常路径日志按调用顺序：
+
+```
+real_client::get_buffer
+ ├ real_client::get_buffer_internal
+ │ ├ query_success
+ │ ├ replica_selected
+ │ ├ [SSD 路径] ssd_read_detail
+ │ ├ get_breakdown
+ │ └ [慢操作] get_buffer_slow
+ ├ client_service::Get
+ │ └ transfer_read_completed
+ ├ client_service::TransferData
+ │ └ transfer_data op[READ]
+ └ [慢操作] get_buffer_slow（在 real_client::get_buffer 层）
+```
+
+### 1.1 核心逻辑层 — `real_client.cpp::get_buffer_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `query_success` | INFO | `query_success key[{key}] replicas[{n}]` | Master 查询成功，返回 n 个副本 |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[{type}] endpoint[{ip:port}] size[{bytes}]` | Memory/LocalDisk 副本选中，含 endpoint |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[{type}] file_path[{path}] size[{bytes}]` | Disk 副本选中，含文件路径 |
+| `get_breakdown` | INFO | `get_breakdown key[{key}] query_us[{t1}] select_us[{t2}] alloc_us[{t3}] read_us[{t4}] total_us[{total}] type[{type}] status[{status}]` | 分阶段耗时汇总 |
+
+**`get_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | Master 查询耗时（微秒） |
+| `select_us` | 副本选择耗时 |
+| `alloc_us` | 缓冲区分配耗时 |
+| `read_us` | 数据读取耗时（RDMA/文件IO/SSD RPC） |
+| `total_us` | 总耗时 |
+| `type` | 副本类型：`memory_local` / `memory_remote` / `local_disk_local` / `local_disk_remote` / `disk` |
+| `status` | 结果：`read_ok` / `read_fail` / `ssd_ok` / `ssd_fail` |
+
+### 1.2 传输服务层 — `client_service.cpp::Get`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_read_completed` | INFO | `transfer_read_completed key[{key}] elapsed_us[{us}] data_size[{bytes}] cache_hit[{0/1}]` | RDMA/文件传输完成 |
+| `transfer_read_failed` | ERROR | `transfer_read_failed key={key}` | 传输失败 |
+| `lease_expired_before_data_transfer_completed` | WARNING | `lease_expired_before_data_transfer_completed key={key}` | 租约过期 |
+
+**`transfer_read_completed` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `elapsed_us` | 传输总耗时（微秒） |
+| `data_size` | 传输数据大小（字节） |
+| `cache_hit` | 是否命中热缓存：`1` 命中，`0` 未命中 |
+
+### 1.3 传输引擎层 — `client_service.cpp::TransferData`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_data` | INFO | `transfer_data first_transfer_data[{0/1}] op[{READ/WRITE}] strategy[{int}] submit_us[{t1}] wait_us[{t2}] result[{code}]` | 传输耗时拆分 |
+
+**字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `first_transfer_data` | 进程首次传输数据输出 `1`，后续输出 `0`（用于区分首次建连开销） |
+| `op` | 操作类型：`READ` 或 `WRITE` |
+| `strategy` | `TransferFuture` 传输策略整数值（对应传输引擎内部策略枚举） |
+| `submit_us` | 提交传输请求耗时（微秒） |
+| `wait_us` | 等待传输完成耗时（微秒） |
+| `result` | 传输结果，`OK` 表示成功 |
+
+### 1.4 SSD Offload 路径 — `real_client.cpp::batch_get_into_offload_object_internal`
+
+仅当副本类型为 `local_disk`（远端 SSD）时触发。
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `ssd_read_detail` | INFO | `ssd_read_detail endpoint[{ip:port}] num_keys[{n}] total_size[{bytes}] elapsed_ms[{ms}] batch_id[{id}]` | SSD RPC 读取详情 |
+
+**`ssd_read_detail` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `endpoint` | SSD offload RPC 服务端地址 |
+| `num_keys` | 本批次读取的 key 数量 |
+| `total_size` | 本批次读取的总字节数 |
+| `elapsed_ms` | 整批 RPC 耗时（毫秒） |
+| `batch_id` | 批次 ID |
+
+---
+
+## 2. `get_batch` 日志链路
+
+> **注意**：Python 绑定层（`store_py.cpp`）的 batch 操作仍使用原生 `LOG()`，无 `trace_id` 前缀。下层（real_client/client_service）使用 `MC_LOG`，带 `trace_id` 前缀。两层日志混合输出时可能因异步队列导致顺序不完全一致。
+
+```
+store_py::get_batch
+ ├ get_batch start
+ ├ real_client::batch_get_buffer_internal
+ │ ├ batch_query_result
+ │ ├ [逐 key] replica_selected (无此日志，batch 不逐 key 输出)
+ │ ├ [SSD 路径] ssd_read_detail
+ │ ├ batch_get_breakdown
+ │ └ [慢操作] batch_get_buffer_slow
+ ├ client_service::BatchGet
+ │ └ batch_get_transfer_complete
+ ├ client_service::TransferData (多次)
+ │ └ transfer_data op[READ]
+ └ get_batch complete
+```
+
+### 2.1 Python 绑定层 — `store_py.cpp::get_batch`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `get_batch start` | INFO | `get_batch start num_keys[{n}]` | 操作开始 |
+| `get_batch complete` | INFO | `get_batch complete num_keys[{n}] success[{s}] rc[0] elapsed_us[{us}]` | 操作成功完成 |
+| `get_batch complete` | INFO | `get_batch complete num_keys[{n}] rc[-1] elapsed_us[{us}]` | 操作失败 |
+| `get_batch_slow` | WARNING | `get_batch_slow num_keys[{n}] elapsed_us[{us}]` | 耗时超过 3ms 触发慢操作告警 |
+
+### 2.2 核心逻辑层 — `real_client.cpp::batch_get_buffer_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_query_result` | INFO | `batch_query_result num_keys[{n}] num_found[{f}]` | 批量查询结果，f 为找到的 key 数 |
+| `batch_get_breakdown` | INFO | `batch_get_breakdown num_keys[{n}] query_us[{t1}] prep_us[{t2}] read_us[{t3}] total_us[{total}] batch_get_ops[{m}] ssd_offload_ops[{s}] success[{ok}]` | 分阶段耗时汇总 |
+
+**`batch_get_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | 批量 Master 查询耗时 |
+| `prep_us` | 准备阶段耗时（副本选择 + 缓冲区分配，逐 key 循环） |
+| `read_us` | 数据读取耗时（BatchGet + SSD RPC） |
+| `total_us` | 总耗时 |
+| `batch_get_ops` | 走 BatchGet 的 key 数（MEMORY + DISK 副本） |
+| `ssd_offload_ops` | 走 SSD RPC 的 key 数（LOCAL_DISK 副本） |
+| `success` | 成功读取的 key 数 |
+
+### 2.3 传输服务层 — `client_service.cpp::BatchGet`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_get_transfer_complete` | INFO | `batch_get_transfer_complete num_keys[{n}] success[{s}] elapsed_us[{us}] pending_count[{c}]` | 批量传输完成 |
+
+**字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `num_keys` | 批量传输的 key 总数 |
+| `success` | 成功传输的 key 数 |
+| `elapsed_us` | 传输总耗时（微秒） |
+| `pending_count` | 总传输任务数（提交的 TransferFuture 数量） |
+
+### 2.4 传输引擎层 — 同第 1 节的 `transfer_data`
+
+### 2.5 SSD Offload 路径 — 同第 1 节的 `ssd_read_detail`
+
+---
+
+## 3. `get_into` 日志链路
+
+`get_into` 将对象直接读入调用方提供的 buffer，核心日志来自 `real_client.cpp`。
+
+```
+real_client::get_into
+ ├ real_client::resolve_ranged_read_metadata
+ │ ├ query_success
+ │ └ replica_selected
+ ├ real_client::execute_ranged_read
+ │ ├ [MEMORY/DISK] client_service::Get
+ │ ├ [LOCAL_DISK] ssd_read_detail
+ │ └ [失败] SSD/DISK/scatter/Get error
+ ├ get_into_breakdown
+ └ [慢操作] get_into_slow
+```
+
+### 3.1 核心逻辑层 — `real_client.cpp::get_into_range_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `query_success` | INFO | `query_success key[{key}] replicas[{n}]` | Master 查询成功，返回 n 个副本 |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[{type}] endpoint[{ip:port}] size[{bytes}]` | Memory/LocalDisk 副本选中，含 endpoint |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[{type}] file_path[{path}] size[{bytes}]` | Disk 副本选中，含文件路径 |
+| `get_into_breakdown` | INFO | `get_into_breakdown key[{key}] query_us[{t1}] select_us[{t2}] read_us[{t3}] total_us[{total}] type[{type}] mode[{mode}] status[{status}]` | 分阶段耗时汇总 |
+| `get_into_breakdown` | INFO | `get_into_breakdown key[{key}] query_us[0] select_us[0] read_us[0] total_us[{total}] type[unknown] mode[unknown] status[{error}]` | metadata 查询/选副本失败时的汇总 |
+
+**`get_into_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | Master 查询耗时（微秒） |
+| `select_us` | 副本选择耗时 |
+| `read_us` | 数据读取耗时（RDMA/文件IO/SSD RPC/scatter） |
+| `total_us` | 总耗时 |
+| `type` | 副本类型：`memory_local` / `memory_remote` / `local_disk_local` / `local_disk_remote` / `disk` / `unknown` |
+| `mode` | 读取模式：`full` 完整对象读取，`range` 部分读取，`unknown` 表示 metadata 阶段失败 |
+| `status` | 结果：`read_ok` / `mem_fail` / `disk_fail` / `ssd_fail` / 错误码字符串 |
+
+### 3.2 读取失败日志 — `real_client.cpp::execute_ranged_read`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `SSD read failed` | ERROR | `SSD read failed for key '{key}': {error}` | LOCAL_DISK 完整读取失败 |
+| `Ranged SSD read failed` | ERROR | `Ranged SSD read failed for key '{key}': {error}` | LOCAL_DISK 范围读取失败 |
+| `DISK Get failed` | ERROR | `DISK Get failed for key: {key} with error: {error}` | DISK 文件读取失败 |
+| `DISK full read scatter failed` | ERROR | `DISK full read scatter failed for key '{key}': {error}` | DISK 完整读取后 scatter 到用户 buffer 失败 |
+| `Ranged disk read scatter failed` | ERROR | `Ranged disk read scatter failed for key '{key}': {error}` | DISK/LOCAL_DISK 范围读取后 scatter 失败 |
+| `Ranged Get failed` | ERROR | `Ranged Get failed for key: {key} with error: {error}` | MEMORY 范围读取失败 |
+
+### 3.3 下层日志
+
+- MEMORY / DISK 通过 `Client::Get` 时，会继续产生 `transfer_read_completed` / `transfer_data op[READ]`。
+- LOCAL_DISK 通过 SSD offload 时，会继续产生 `ssd_read_detail`。
+
+---
+
+## 4. `batch_get_into` 日志链路
+
+`batch_get_into` 将多个对象分别读入调用方提供的 buffers，批量成功项不逐 key 打 INFO，失败项仍逐 key 打 ERROR。
+
+```
+real_client::batch_get_into
+ ├ batch_get_into_query_result
+ ├ [逐 key 失败] Query/Select/Buffer/DISK error
+ ├ [MEMORY] client_service::BatchGet
+ ├ [DISK] client_service::BatchGet + scatter
+ ├ [LOCAL_DISK] ssd_read_detail
+ ├ batch_get_into_breakdown
+ └ [慢操作] batch_get_into_slow
+```
+
+### 4.1 核心逻辑层 — `real_client.cpp::batch_get_into_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_get_into_query_result` | INFO | `batch_get_into_query_result num_keys[{n}] num_found[{f}]` | 批量查询结果，f 为找到的 key 数 |
+| `batch_get_into_breakdown` | INFO | `batch_get_into_breakdown num_keys[{n}] query_us[{t1}] prep_us[{t2}] read_us[{t3}] total_us[{total}] mem_ops[{m}] disk_ops[{d}] ssd_offload_ops[{s}] success[{ok}]` | 分阶段耗时汇总 |
+
+**`batch_get_into_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | 批量 Master 查询耗时 |
+| `prep_us` | 准备阶段耗时（逐 key 副本选择、容量校验、分类、slice 准备） |
+| `read_us` | 数据读取耗时（MEMORY BatchGet + DISK BatchGet/scatter + SSD RPC） |
+| `total_us` | 总耗时 |
+| `mem_ops` | 走 MEMORY `BatchGet` 的 key 数 |
+| `disk_ops` | 走 DISK 临时 buffer + `BatchGet` + scatter 的 key 数 |
+| `ssd_offload_ops` | 实际提交到 SSD offload 的 key 数（LOCAL_DISK） |
+| `success` | 成功读取且返回正数字节数的 key 数 |
+
+### 4.2 逐 key / endpoint 失败日志
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `Query failed` | ERROR | `Query failed for key '{key}': {error}` | 单个 key 查询失败（非 NOT_FOUND/NOT_READY） |
+| `Empty replica list` | ERROR | `Empty replica list for key: {key}` | 查询结果没有 replica |
+| `No usable replica` | ERROR | `No usable replica for key: {key}` | 无可用 COMPLETE 副本 |
+| `Buffer too small` | ERROR | `Buffer too small for key '{key}': required={r}, available={a}` | 用户 buffer 容量不足 |
+| `BatchGet failed` | ERROR | `BatchGet failed for key '{key}': {error}` | MEMORY BatchGet 失败 |
+| `DISK BatchGet failed` | ERROR | `DISK BatchGet failed for key '{key}': {error}` | DISK BatchGet 失败 |
+| `DISK scatter failed` | ERROR | `DISK scatter failed for key '{key}': {error}` | DISK 临时 buffer scatter 到用户 buffer 失败 |
+| `Batch get store object failed` | ERROR | `Batch get store object failed endpoint[{endpoint}] objects[{n}] error[{error}]` | LOCAL_DISK endpoint 级 offload 失败 |
+
+### 4.3 下层日志
+
+- MEMORY / DISK 批量读取会继续产生 `batch_get_transfer_complete` / `transfer_data op[READ]`。
+- LOCAL_DISK offload 会继续产生 `ssd_read_detail`。
+
+---
+
+## 5. `put` 日志链路
+
+> Python 绑定层（`store_py.cpp`）的单 key `put` 生命周期日志（`put start/complete/slow`）已移除，改为由 `real_client.cpp` 输出慢操作告警。
+
+```
+real_client::put_internal
+ ├ put_result
+ └ [慢操作] put_slow
+client_service::Put
+ ├ put_start_success (或 OBJECT_ALREADY_EXISTS)
+ └ put_end_success
+client_service::TransferData
+ └ transfer_data op[WRITE]
+```
+
+### 5.1 核心逻辑层 — `real_client.cpp::put_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_result` | INFO | `put_result key[{key}] rc[0] size[{bytes}]` | Put 成功 |
+| `put_result` | INFO | `put_result key[{key}] rc[{code}] size[{bytes}]` | Put 失败，code 为错误码 |
+
+### 5.2 传输服务层 — `client_service.cpp::Put`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_start` | INFO | `put_start key[{key}] rc[OBJECT_ALREADY_EXISTS]` | 对象已存在，直接返回成功 |
+| `put_start_success` | INFO | `put_start_success key[{key}] replicas[{n}]` | Master 分配 replica 成功 |
+| `put_end_success` | INFO | `put_end_success key[{key}] transfer_us[{us}] data_size[{bytes}]` | Put 完成，数据写入成功 |
+
+**`put_end_success` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `transfer_us` | 传输阶段总耗时（含磁盘写入 + RDMA 传输） |
+| `data_size` | 写入数据大小 |
+
+### 5.3 传输引擎层 — `client_service.cpp::TransferData`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_data` | INFO | `transfer_data first_transfer_data[{0/1}] op[WRITE] strategy[{int}] submit_us[{t1}] wait_us[{t2}] result[{code}]` | 传输耗时拆分 |
+
+字段含义同第 1.3 节 `transfer_data`。
+
+---
+
+## 6. `put_batch` 日志链路
+
+> **注意**：Python 绑定层（`store_py.cpp`）的 batch 操作仍使用原生 `LOG()`，无 `trace_id` 前缀。下层使用 `MC_LOG`，带 `trace_id` 前缀。
+
+```
+store_py::put_batch
+ ├ put_batch start
+ ├ real_client::put_batch_internal
+ │ └ batch_put_result
+ ├ client_service::BatchPut
+ │ ├ batch_put start
+ │ └ batch_put complete
+ ├ client_service::TransferData (多次)
+ │ └ transfer_data op[WRITE]
+ └ put_batch complete
+```
+
+### 6.1 Python 绑定层 — `store_py.cpp::put_batch`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_batch start` | INFO | `put_batch start num_keys[{n}] total_size[{bytes}]` | 操作开始 |
+| `put_batch complete` | INFO | `put_batch complete num_keys[{n}] rc[{ret}] elapsed_us[{us}]` | 操作完成，rc=0 成功 |
+| `put_batch_slow` | WARNING | `put_batch_slow num_keys[{n}] elapsed_us[{us}]` | 耗时超过 10ms 触发慢操作告警 |
+
+### 6.2 核心逻辑层 — `real_client.cpp::put_batch_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_put_result` | INFO | `batch_put_result num_keys[{n}] num_failed[{f}]` | 批量 Put 结果 |
+
+### 6.3 传输服务层 — `client_service.cpp::BatchPut`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_put start` | INFO | `batch_put start num_keys[{n}]` | 批量 Put 传输开始 |
+| `batch_put complete` | INFO | `batch_put complete num_keys[{n}] num_failed[{f}] transfer_us[{us}] total_size[{bytes}]` | 批量 Put 完成（正常路径） |
+| `batch_put complete` | INFO | `batch_put complete num_keys[{n}] num_failed[{f}] total_size[{bytes}]` | 批量 Put 完成（prefer_same_node 路径，无 transfer_us） |
+
+**`batch_put complete` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `num_keys` | 批量写入的 key 数量 |
+| `num_failed` | 失败的 key 数量 |
+| `transfer_us` | 传输阶段总耗时（微秒，prefer_same_node 路径无此字段） |
+| `total_size` | 写入的总数据大小（字节） |
+
+### 6.4 传输引擎层 — 同第 5 节的 `transfer_data`
+
+---
+
+## 7. 附录：PerfPoint 打点与日志对照表
+
+PerfPoint 定义在 `mooncake-integration/store/mooncake_perf_points.def`。
+使用 `spdiag show` 可查看实时性能数据，配合日志进行交叉分析。
+
+### 7.0 `get_into` / `batch_get_into` 与 `get_buffer` / `batch_get_buffer` 是否一致
+
+结论：**单 key 的 `get_into` 与 `get_buffer` 基本一致，批量的 `batch_get_into` 与 `batch_get_buffer` 还不完全一致。**
+
+单 key 路径中，`get_into_breakdown.type` 已经和 `get_breakdown.type` 一样细分为 `memory_local` / `memory_remote` / `local_disk_local` / `local_disk_remote` / `disk`。如果只看到 `memory` / `local_disk` / `disk` 三类，那是旧文档口径，不是当前源码口径。
+
+| 接口 | 入口日志 | 入口 PerfPoint | 汇总日志 | 子步骤对齐情况 |
+|------|----------|----------------|----------|----------------|
+| `get_buffer` | `get_buffer_start` | `GET_BUFFER_INTERNAL_FULL` | `get_breakdown` | `type` 细分 local/remote；有 `alloc_us` |
+| `get_into` | `get_into_start` | `GET_INTO_INTERNAL` | `get_into_breakdown` | `type` 细分 local/remote；没有独立 `alloc_us`，DISK/范围读临时 buffer 分配计入 `read_us` |
+| `batch_get_buffer` | `batch_get_buffer_start` | `GET_BATCH_BUFFER_INTERNAL_FULL` | `batch_get_breakdown` | `batch_get_ops` 合并 MEMORY + DISK，`ssd_offload_ops` 表示 LOCAL_DISK |
+| `batch_get_into` | `batch_get_into_start` | `GET_BATCH_INTO_INTERNAL` | `batch_get_into_breakdown` | `mem_ops` / `disk_ops` / `ssd_offload_ops` 拆开统计，比 `batch_get_buffer` 更细 |
+
+差异点：
+
+- `get_into` 的 replica type 现在与 `get_buffer` 一致，都是 local/remote 细分；文档已同步修正。
+- `get_into_breakdown` 没有 `alloc_us` 字段，因为它通常直接写入调用方 buffer；只有 DISK 或 range 读需要临时 CPU buffer，这部分耗时归入 `read_us`。
+- `batch_get_into_breakdown` 比 `batch_get_breakdown` 多拆了 `mem_ops` 和 `disk_ops`；而 `batch_get_breakdown` 用 `batch_get_ops` 合并 MEMORY + DISK。所以这两者目前不是完全一致口径。
+
+### GET 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_STORE_PY_GET` | store_py.cpp::get | Get | —（已移除） |
+| `GET_BUFFER_INTERNAL` | store_py.cpp::get | GetBuffer | `get_breakdown` |
+| `GET_BUFFER_INTERNAL_FULL` | real_client.cpp::get_buffer | GetBufferInternal | `get_breakdown` |
+| `GET_INTERNAL_QUERY` | real_client.cpp::get_buffer_internal | Query | `query_success` |
+| `GET_INTERNAL_SELECT_REPLICA` | real_client.cpp::get_buffer_internal | SelectReplica | `replica_selected` |
+| `GET_INTERNAL_ALLOC_BUFFER` | real_client.cpp::get_buffer_internal | AllocBuffer | `get_breakdown` alloc_us |
+| `GET_INTERNAL_SSD_READ` | real_client.cpp::get_buffer_internal | SSDRead | `ssd_read_detail` |
+| `GET_INTERNAL_MEM_READ` | real_client.cpp::get_buffer_internal | MemRead | `transfer_read_completed` |
+| `GET_INTERNAL_DISK_READ` | real_client.cpp::get_buffer_internal | DiskRead | `transfer_read_completed` |
+| `GET_SSD_OFFLOAD_RPC` | real_client.cpp::batch_get_into_offload_object_internal | OffloadRpc | `ssd_read_detail` |
+| `GET_SSD_TRANSFER_DATA` | real_client.cpp::batch_get_into_offload_object_internal | TransferData | `ssd_read_detail` |
+| `GET_SSD_RELEASE_BUFFER` | real_client.cpp::batch_get_into_offload_object_internal | ReleaseBuffer | — |
+| `GET_SINGLE_FIND_REPLICA` | client_service.cpp::Get | FindReplica | `transfer_read_completed` |
+| `GET_SINGLE_HOT_CACHE` | client_service.cpp::Get | HotCache | `transfer_read_completed` cache_hit |
+| `GET_SINGLE_TRANSFER_READ` | client_service.cpp::Get | TransferRead | `transfer_read_completed` |
+| `GET_SINGLE_RELEASE_CACHE` | client_service.cpp::Get | ReleaseCache | — |
+| `GET_SINGLE_ASYNC_CACHE` | client_service.cpp::Get | AsyncCache | — |
+| `GET_SINGLE_TRANSFER_FULL` | client_service.cpp::TransferData | TransferData | `transfer_data op[READ]` |
+| `GET_SINGLE_TRANSFER_SUBMIT` | client_service.cpp::TransferData | Submit | `transfer_data` submit_us |
+| `GET_SINGLE_TRANSFER_WAIT` | client_service.cpp::TransferData | Wait | `transfer_data` wait_us |
+
+### GET INTO 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_INTO_INTERNAL` | real_client.cpp::get_into | GetIntoInternal | `get_into_breakdown` |
+| `GET_INTO_INTERNAL_QUERY` | real_client.cpp::get_into_internal | Query | `query_success` |
+| `GET_INTO_INTERNAL_SELECT_REPLICA` | real_client.cpp::get_into_internal | SelectReplica | `replica_selected` |
+| `GET_INTO_INTERNAL_ALLOC_BUFFER` | real_client.cpp::get_into_internal | AllocBuffer | `get_into_breakdown` read_us |
+| `GET_INTO_INTERNAL_SSD_READ` | real_client.cpp::get_into_internal | SSDRead | `ssd_read_detail` / `SSD read failed` |
+| `GET_INTO_INTERNAL_MEM_READ` | real_client.cpp::get_into_internal | MemRead | `transfer_read_completed` / `get_into_breakdown` |
+| `GET_INTO_INTERNAL_DISK_READ` | real_client.cpp::get_into_internal | DiskRead | `transfer_read_completed` / `get_into_breakdown` |
+
+### GET BATCH 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_STORE_PY_GET_BATCH` | store_py.cpp::get_batch | GetBatch | `get_batch start` / `get_batch complete` |
+| `GET_BATCH_BUFFER_INTERNAL` | store_py.cpp::get_batch | BatchGetBuffer | `batch_get_breakdown` |
+| `GET_BATCH_BUFFER_INTERNAL_FULL` | real_client.cpp::batch_get_buffer | BatchGetBufferInternal | `batch_get_breakdown` |
+| `GET_BATCH_INTERNAL_QUERY` | real_client.cpp::batch_get_buffer_internal | BatchQuery | `batch_query_result` |
+| `GET_BATCH_INTERNAL_PREPARATION` | real_client.cpp::batch_get_buffer_internal | Preparation | —（仅定义，当前未在源码中使用） |
+| `GET_BATCH_INTERNAL_SELECT_REPLICA` | real_client.cpp::batch_get_buffer_internal | SelectReplica | `batch_get_breakdown` prep_us（逐 key 汇总） |
+| `GET_BATCH_INTERNAL_ALLOC_BUFFER` | real_client.cpp::batch_get_buffer_internal | AllocBuffer | `batch_get_breakdown` prep_us（逐 key 汇总） |
+| `GET_BATCH_INTERNAL_SSD_READ` | real_client.cpp::batch_get_buffer_internal | SSDRead | `ssd_read_detail` |
+| `GET_BATCH_INTERNAL_MEMDISH_READ` | real_client.cpp::batch_get_buffer_internal | MemDiskRead | `batch_get_transfer_complete` |
+| `GET_BATCH_FULL` | client_service.cpp::BatchGet | TransferBatchGet | `batch_get_transfer_complete` |
+| `GET_BATCH_FIND_REPLICA` | client_service.cpp::BatchGet | FindReplica | — |
+| `GET_BATCH_HOT_CACHE` | client_service.cpp::BatchGet | HotCache | — |
+| `GET_BATCH_SUBMIT` | client_service.cpp::BatchGet | Submit | — |
+| `GET_BATCH_WAIT` | client_service.cpp::BatchGet | Wait | — |
+| `GET_BATCH_RELEASE_CACHE` | client_service.cpp::BatchGet | ReleaseCache | — |
+| `GET_BATCH_ASYNC_CACHE` | client_service.cpp::BatchGet | AsyncCache | — |
+
+### BATCH GET INTO 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_BATCH_INTO_INTERNAL` | real_client.cpp::batch_get_into | BatchGetIntoInternal | `batch_get_into_breakdown` |
+| `GET_BATCH_INTO_INTERNAL_QUERY` | real_client.cpp::batch_get_into_internal | BatchQuery | `batch_get_into_query_result` |
+| `GET_BATCH_INTO_INTERNAL_SELECT_REPLICA` | real_client.cpp::batch_get_into_internal | SelectReplica | `batch_get_into_breakdown` prep_us（逐 key 汇总） |
+| `GET_BATCH_INTO_INTERNAL_ALLOC_BUFFER` | real_client.cpp::batch_get_into_internal | AllocBuffer | `batch_get_into_breakdown` read_us（仅 DISK 临时 buffer） |
+| `GET_BATCH_INTO_INTERNAL_MEM_READ` | real_client.cpp::batch_get_into_internal | MemRead | `batch_get_transfer_complete` / `batch_get_into_breakdown` |
+| `GET_BATCH_INTO_INTERNAL_DISK_READ` | real_client.cpp::batch_get_into_internal | DiskRead | `DISK BatchGet failed` / `DISK scatter failed` |
+| `GET_BATCH_INTO_INTERNAL_SSD_READ` | real_client.cpp::batch_get_into_internal | SSDRead | `ssd_read_detail` / `Batch get store object failed` |
+
+### PUT 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `PUT_STORE_PY_PUT` | store_py.cpp::put | Put | —（已移除） |
+| `PUT_INTERNAL_FULL` | store_py.cpp::put | PutBuffer | `put_result` |
+| `PUT_INTERNAL_ALLOC_BUFFER` | real_client.cpp::put_internal | AllocBuffer | — |
+| `PUT_INTERNAL_MEM_COPY` | real_client.cpp::put_internal | MemCopy | — |
+| `PUT_INTERNAL_SPLIT_SLICES` | real_client.cpp::put_internal | SplitSlices | — |
+| `PUT_SINGLE_FULL` | client_service.cpp::Put | TransferPut | `put_end_success` |
+| `PUT_SINGLE_PUT_START` | client_service.cpp::Put | PutStart | `put_start_success` |
+| `PUT_SINGLE_DISK_WRITE` | client_service.cpp::Put | DiskWrite | `put_end_success` |
+| `PUT_SINGLE_TRANSFER_WRITE` | client_service.cpp::Put | TransferWrite | `put_end_success` |
+| `PUT_SINGLE_PUT_END` | client_service.cpp::Put | PutEnd | `put_end_success` |
+| `PUT_SINGLE_PUT_REVOKE` | client_service.cpp::Put | PutRevoke | — |
+| `PUT_SINGLE_TRANSFER_FULL` | client_service.cpp::TransferData | TransferData | `transfer_data op[WRITE]` |
+| `PUT_SINGLE_TRANSFER_SUBMIT` | client_service.cpp::TransferData | Submit | `transfer_data` submit_us |
+| `PUT_SINGLE_TRANSFER_WAIT` | client_service.cpp::TransferData | Wait | `transfer_data` wait_us |
+
+### PUT BATCH 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `PUT_STORE_PY_PUT_BATCH` | store_py.cpp::put_batch | PutBatch | `put_batch start` / `put_batch complete` |
+| `PUT_BATCH_INTERNAL_FULL` | store_py.cpp::put_batch | BatchPutBuffer | `batch_put_result` |
+| `PUT_BATCH_INTERNAL_ALLOC_BUFFER` | real_client.cpp::put_batch_internal | AllocBuffer | — |
+| `PUT_BATCH_INTERNAL_MEM_COPY` | real_client.cpp::put_batch_internal | MemCopy | — |
+| `PUT_BATCH_INTERNAL_SPLIT_SLICES` | real_client.cpp::put_batch_internal | SplitSlices | — |
+| `PUT_BATCH_FULL` | client_service.cpp::BatchPut | TransferBatchPut | `batch_put complete` |
+| `PUT_BATCH_CREATE_OPS` | client_service.cpp::BatchPut | CreateOps | — |
+| `PUT_BATCH_PUT_START` | client_service.cpp::StartBatchPut | PutStart | — |
+| `PUT_BATCH_SUBMIT` | client_service.cpp::SubmitTransfers | Submit | — |
+| `PUT_BATCH_DISK_WRITE` | client_service.cpp::SubmitTransfers | DiskWrite | — |
+| `PUT_BATCH_WAIT` | client_service.cpp::WaitForTransfers | Wait | — |
+| `PUT_BATCH_PUT_END` | client_service.cpp::FinalizeBatchPut | PutEnd | — |
+| `PUT_BATCH_PUT_REVOKE` | client_service.cpp::FinalizeBatchPut | PutRevoke | — |
+| `PUT_BATCH_COLLECT_RESULTS` | client_service.cpp::BatchPut | CollectResults | — |
+
+### UB/URMA 建连侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `UB_HANDSHAKE_ENCODE` | transfer_metadata.cpp::encode | Encode | — |
+| `UB_HANDSHAKE_DECODE` | transfer_metadata.cpp::decode | Decode | — |
+| `UB_ENDPOINT_CONSTRUCT` | urma_endpoint.cpp::construct | Construct | `urma_endpoint_construct_breakdown` |
+| `UB_ENDPOINT_CREATE_JETTY` | urma_endpoint.cpp::construct | CreateJetty | `urma_create_jetty_breakdown` |
+| `UB_ENDPOINT_ACTIVE_SETUP` | urma_endpoint.cpp::setupConnectionsByActive | ActiveSetup | `urma_active_setup_breakdown` |
+| `UB_ENDPOINT_ACTIVE_HANDSHAKE` | urma_endpoint.cpp::setupConnectionsByActive | SendHandshake | — |
+| `UB_ENDPOINT_PASSIVE_SETUP` | urma_endpoint.cpp::setupConnectionsByPassive | PassiveSetup | `urma_passive_setup_breakdown` |
+| `UB_ENDPOINT_DO_SETUP_ALL` | urma_endpoint.cpp::doSetupConnection | DoSetupAll | `urma_do_setup_all_breakdown` |
+| `UB_ENDPOINT_IMPORT_JETTY` | urma_endpoint.cpp::doSetupConnection | ImportJetty | `urma_import_jetty_breakdown` |
+| `UB_ENDPOINT_BIND_JETTY` | urma_endpoint.cpp::doSetupConnection | BindJetty | `urma_bind_jetty_breakdown` |
+
+---
+
+## 8. 日志配置
+
+Mooncake 使用 glog 作为日志库，通过环境变量控制日志级别、输出位置和开关。
+
+### 8.1 环境变量
+
+| 变量 | 默认值 | 说明 |
+|------|-------|------|
+| `MC_LOG_LEVEL` | `INFO` | 日志输出级别 |
+| `MC_LOG_DIR` | 空（stderr） | 日志文件输出目录 |
+| `MC_LOG_ENABLE` | 禁用 | 日志总开关 |
+| `MC_HIFREQ_LOG_SAMPLE_RATE` | `0.1` | 高频 breakdown 日志采样率，详见 §8.6 |
+
+代码来源：`mooncake-transfer-engine/src/config.cpp`（MC_LOG_LEVEL）、`mooncake-common/src/mooncake_logging.cpp`（MC_LOG_ENABLE、MC_HIFREQ_LOG_SAMPLE_RATE）
+
+### 8.2 设置日志级别 — `MC_LOG_LEVEL`
+
+可选值：
+
+| 值 | 效果 |
+|----|------|
+| `TRACE` | 最详细，输出 INFO/WARNING/ERROR，额外启用 trace 标志 |
+| `INFO` | 默认，输出 INFO/WARNING/ERROR |
+| `WARNING` | 只输出 WARNING/ERROR |
+| `ERROR` | 只输出 ERROR |
+
+**操作方法：**
+
+```bash
+# 在启动 Mooncake 服务前设置
+export MC_LOG_LEVEL=WARNING
+./mooncake_master
+
+# 或在 Python 端（import mooncake 前）
+import os
+os.environ['MC_LOG_LEVEL'] = 'WARNING'
+import mooncake
+```
+
+**注意：** 设置日志级别只影响日志输出，不影响时间记录代码（`steady_clock::now()` 调用仍会执行）。
+
+### 8.3 设置日志输出位置 — `MC_LOG_DIR`
+
+| 情况 | 行为 |
+|------|------|
+| 未设置或为空 | 日志输出到 stderr（终端） |
+| 目录不存在 | 输出 WARNING，回退到 stderr |
+| 目录不可写 | 输出 WARNING，回退到 stderr |
+| 目录存在且可写 | 日志写入该目录下的文件 |
+
+**操作方法：**
+
+```bash
+# 输出到指定目录
+export MC_LOG_DIR=/var/log/mooncake
+./mooncake_master
+
+# 需要先确保目录存在且可写
+mkdir -p /var/log/mooncake
+chmod 755 /var/log/mooncake
+```
+
+### 8.4 设置日志总开关 — `MC_LOG_ENABLE`
+
+控制 Mooncake 日志输出开关。未显式设置时默认关闭；显式启用后，`MC_LOG`/`MC_VLOG` 按此开关输出，Transfer Engine 初始化时也会按此开关设置 glog 的最低输出级别。
+
+| 值 | 效果 |
+|----|------|
+| 未设置 | 默认禁用 |
+| `off` / `0` / `false` / `no` | 关闭日志（不区分大小写） |
+| 其他值 | 启用 |
+
+**注意：**
+- FATAL 级别日志不受此开关影响，始终输出
+- 此开关与 `MC_LOG_LEVEL` 独立：两者都允许时日志才输出
+- 仅使用原生 `LOG()` 且未经过 Transfer Engine 配置初始化的进程，仍可能受 glog 自身 flag 控制
+
+**操作方法：**
+
+```bash
+# 启用日志
+export MC_LOG_ENABLE=on
+./mooncake_master
+```
+
+### 8.5 常用配置场景
+
+| 场景 | 配置 |
+|------|------|
+| 生产环境 | `export MC_LOG_ENABLE=on && export MC_LOG_LEVEL=WARNING && export MC_LOG_DIR=/var/log/mooncake` |
+| 调试排查 | `export MC_LOG_ENABLE=on && export MC_LOG_LEVEL=INFO`（输出到 stderr，除非设置 `MC_LOG_DIR`） |
+| 性能测试（减少日志） | `export MC_LOG_ENABLE=on && export MC_LOG_LEVEL=ERROR` |
+| 启用日志输出 | `export MC_LOG_ENABLE=on` |
+
+### 8.6 高频 breakdown 日志采样 — `MC_HIFREQ_LOG_SAMPLE_RATE`
+
+`get_buffer` / `batch_get_buffer` / `get_into` / `batch_get_into` 四条链路精简后，每个请求只剩一条
+`*_breakdown` 汇总日志（原生 `LOG(INFO)`）。由于 get 类操作调用极频繁，这条"每请求一条"的日志本身即为高频
+日志，可用本变量按概率采样。
+
+| 取值 | 效果 |
+|------|------|
+| `1.0` | 每个请求都输出其 breakdown（100%） |
+| `0.1`（默认） | 每个请求 10% 概率输出 breakdown |
+| `0` | 完全不输出 breakdown（全部静默） |
+| 非法/越界 | 非数值回退 `0.1`；`<0` 截断为 `0`；`>1` 截断为 `1` |
+
+**实现要点：**
+- 每个请求只掷一次骰子（`mooncake::logging::ShouldSampleHiFreqLog()`，线程本地无锁 RNG）；命中才**既输出
+ 日志又记录其 steady/system clock 计时**，未命中则跳过计时与输出，开销接近零。
+- 仅作用于 breakdown 这一条文本日志。**SpDiag `PerfPoint` 打点始终记录，不受采样影响**；`ERROR`/`WARNING`
+ 也照常每次输出。
+- breakdown 是原生 `LOG(INFO)`，可见性由本变量控制，**与 `MC_LOG_ENABLE` 无关**。
+- 解析与缓存：`mooncake-common/src/mooncake_logging.cpp::ParseHiFreqLogSampleRate`（进程内只解析一次）。
+
+```bash
+# 全量输出 breakdown（排查/对账时用）
+export MC_HIFREQ_LOG_SAMPLE_RATE=1.0
+# 默认 10% 采样，无需设置；完全关闭：
+export MC_HIFREQ_LOG_SAMPLE_RATE=0
+```
+
+#### Correlated latency diagnostics
+
+High-frequency diagnostics now use deterministic `trace_id` sampling. A
+sampled request emits correlated `*_breakdown`, `transfer_diag`,
+`storage_read_breakdown`, `BatchGetReplicaListItem`, and
+`urma_queue_depth` records across worker threads and processes. SSD offload
+RPCs propagate the trace id to the storage owner. Calls without a trace id
+retain random local sampling.
+
+- `transfer_diag.submit_us`: validation, segment lookup, request construction,
+ and submission until a future is returned.
+- `transfer_diag.wait_us`: time blocked in `future.get()`.
+- `local_endpoints_us`: time spent locking and copying the locally mounted
+ Transfer Engine endpoint set.
+- `select_replica_us`: time spent scanning the returned replicas and choosing
+ the preferred complete replica.
+- Transfer-layer records intentionally do not carry object keys; correlate
+ them with Store records through the existing trace id.
+- `storage_read_breakdown.alloc_us`: owner ClientBuffer batch allocation.
+- `storage_read_breakdown.plan_us`: metadata lookup and read-plan construction.
+- `storage_read_breakdown.file_open_us`: path resolution and file open time.
+- `storage_read_breakdown.disk_read_us`: cumulative time inside actual
+ `read_aligned`/`vector_read` calls.
+- `storage_read_breakdown.total_us`: wall time of the owner `BatchGet`.
+
+Batch endpoint records are emitted as `batch_transfer_item`, one record per
+key. Transfer failure ERROR records are never sampled and include the key.
+
+---
+
+## 9. 异步日志与 TraceId
+
+### 9.1 异步日志架构
+
+实现文件：`mooncake-common/include/mooncake_logging.h`、`mooncake-common/src/mooncake_logging.cpp`
+
+**流程：**
+
+```
+调用 MC_LOG(severity) << "message"
+ → 创建 AsyncLogMessage 临时对象
+ → 捕获当前 trace_id（CurrentTraceId()）
+ → 构造日志消息到 ostringstream
+ → 析构时判断：
+ - FATAL：直接同步调用 glog 输出（带 trace_id 前缀）
+ - 其他：构造 LogEntry{file, line, severity, trace_id, message} 入队
+ → AsyncLogQueue 后台线程消费
+ → WriteSync() 调用 google::LogMessage 输出（自动带 trace_id 前缀）
+```
+
+**队列参数：**
+
+| 参数 | 值 |
+|------|-----|
+| 最大队列长度 | 8192 条 |
+| 队列满策略 | 阻塞写入线程（condition_variable wait） |
+| 后台线程数 | 1 |
+| 退出处理 | `atexit` 注册 `Stop()`，刷出剩余日志 |
+
+### 9.2 TraceId 系统
+
+**ID 生成算法：**
+
+```
+process_seed = (PID << 48) ^ (steady_clock_ns & 0x0000FFFFFFFF0000)
+trace_id = process_seed ^ atomic_counter++
+```
+
+PID 保证跨进程唯一，steady_clock_ns 保证同进程每次启动不同，atomic_counter 保证同进程内递增唯一。
+
+**线程传递机制：**
+
+- `thread_local uint64_t current_trace_id` 存储当前线程的 trace_id
+- `ScopedTraceId` 在构造时保存旧值、设置新值，析构时恢复旧值
+- `AsyncLogMessage` 构造时读取 `CurrentTraceId()`，并把该值固化到日志条目中；日志之后即使由后台线程异步落盘，也不会丢失原始 trace_id
+- RAII 模式，支持嵌套、提前返回和异常退出
+
+恢复旧值的原因是 `current_trace_id` 绑定在线程上，而 Mooncake 中大量使用线程池和后台 worker。线程处理完一个请求或任务后会继续处理其他工作；如果不恢复，后续不属于该请求的日志可能仍然带着旧 trace_id，导致排查时误以为它们属于同一条调用链。
+
+**函数间传递链路：**
+
+1. 入口函数生成 trace：`real_client.cpp` 中 `get_buffer`、`get_into`、`batch_get_into`、`put`、`put_batch` 等公开入口创建 `ScopedTraceId trace(NewTraceId())`，从这里开始一次用户操作拥有独立 trace_id。
+2. 同步函数调用自动继承：入口函数继续调用 `*_internal`、`client_service`、`TransferSubmitter` 等下游函数时，只要仍在同一线程执行，下游 `MC_LOG` 会通过 `CurrentTraceId()` 读到同一个 thread-local trace_id，不需要把 trace_id 作为函数参数层层传递。
+3. MC_LOG 捕获当前 trace：每条 `MC_LOG` / `MC_VLOG` 在构造 `AsyncLogMessage` 时立即捕获当前 trace_id，并随 `LogEntry` 放入异步日志队列。后台日志线程只负责输出已经捕获好的 trace_id。
+4. 跨线程提交前显式捕获：当执行流要进入线程池或 worker 队列时，提交线程先调用 `CurrentTraceId()` 取出当前 trace_id，并把它放进任务对象或 lambda 捕获列表。
+5. 工作线程恢复上下文：worker 取出任务后创建 `ScopedTraceId trace(task.trace_id)` 或 `ScopedTraceId trace(trace_id)`，让该任务执行期间的所有 `MC_LOG` 都继续带原始请求的 trace_id。
+6. 任务结束自动恢复：worker 任务作用域结束后 `ScopedTraceId` 析构，恢复该线程之前的 trace_id，通常恢复为 `0`，后续空闲、清理或下一任务日志不会串到刚完成的请求上。
+
+典型同步链路：
+
+```
+RealClient::get_buffer
+ -> ScopedTraceId(NewTraceId())
+ -> RealClient::get_buffer_internal
+ -> Client::Get / TransferSubmitter
+ -> MC_LOG 捕获 CurrentTraceId()
+```
+
+典型异步链路：
+
+```
+RealClient::put
+ -> ScopedTraceId(NewTraceId())
+ -> client_service 提交异步任务前 CurrentTraceId()
+ -> write_thread_pool_.enqueue(..., trace_id)
+ -> worker lambda 内 ScopedTraceId(trace_id)
+ -> StoreObject / PutEnd 相关 MC_LOG 继续使用同一 trace_id
+```
+
+典型传输任务链路：
+
+```
+TransferSubmitter::submitTransfer
+ -> MemcpyTask(..., CurrentTraceId())
+ -> MemcpyWorkerPool::workerThread
+ -> ScopedTraceId(task.trace_id)
+ -> memcpy / GPU copy 相关 MC_LOG 使用原始 trace_id
+
+TransferSubmitter::submitFileReadOperation
+ -> FilereadTask(..., CurrentTraceId())
+ -> FilereadWorkerPool::workerThread
+ -> ScopedTraceId(task.trace_id)
+ -> LoadObject 相关 MC_LOG 使用原始 trace_id
+```
+
+**异步任务传播：**
+
+- `MemcpyTask` 和 `FilereadTask` 携带 `trace_id` 字段
+- `client_service.cpp` 中异步 `StoreObject + PutEnd` 的线程池 lambda 通过捕获列表携带 `trace_id`
+- 工作线程取出任务后通过 `ScopedTraceId trace(task.trace_id)` 或 `ScopedTraceId trace(trace_id)` 恢复上下文
+- 确保异步路径的日志可关联到原始操作
+
+**日志格式：**
+
+```
+I0527 14:30:00.123456 12345 real_client.cpp:2682] trace_id[123456789abcdef0] get_breakdown key[k1] ...
+I0527 14:30:00.123789 12345 real_client.cpp:3600] trace_id[none] Cleaning up ...
+```
+
+### 9.3 FlushAsyncLogs
+
+调用 `mooncake::logging::FlushAsyncLogs()` 可手动刷出异步队列中所有待输出日志，内部会等待队列为空且无活跃写入后调用 `google::FlushLogFiles(google::INFO)`。
+
+---
+
+## 10. `client_create_breakdown` 日志
+
+来源：`client_service.cpp::Client::Create`
+
+记录 Client 对象创建过程各阶段耗时。有两种变体：
+
+### 10.1 RPC-only 模式（`protocol == "rpc_only"`）
+
+```
+client_create_breakdown protocol[rpc_only] connect_master_us[{t1}] storage_config_us[{t2}] init_transfer_engine_us[0] init_transfer_submitter_us[0] total_us[{t5}]
+```
+
+### 10.2 完整模式
+
+```
+client_create_breakdown protocol[{p}] connect_master_us[{t1}] storage_config_us[{t2}] init_transfer_engine_us[{t3}] init_transfer_submitter_us[{t4}] total_us[{t5}]
+```
+
+### 字段说明
+
+| 字段 | 含义 |
+|------|------|
+| `protocol` | 传输协议：`rdma` / `tcp` / `ub` / `ascend` / `cxl` / `rpc_only` 等 |
+| `connect_master_us` | 连接 Master 服务耗时（微秒），含 HA 视图读取（如启用） |
+| `storage_config_us` | 获取存储配置耗时（微秒），含 GetStorageConfig 或回退 GetFsdir |
+| `init_transfer_engine_us` | 初始化传输引擎耗时（微秒），rpc_only 模式为 0 |
+| `init_transfer_submitter_us` | 初始化传输提交器耗时（微秒），rpc_only 模式为 0 |
+| `total_us` | `Client::Create` 总耗时（微秒），从函数入口到返回 |
+
+---
+
+## 11. 传输任务层日志
+
+来源：`mooncake-store/src/transfer_task.cpp`
+
+记录传输任务执行过程中的关键步骤耗时。
+
+### 11.1 `transfer_future_wait`
+
+```
+transfer_future_wait first_wait[{0/1}] ready_before_wait[{0/1}] strategy[{int}] wait_us[{us}] result[{code}]
+```
+
+记录等待传输 Future 完成的过程。
+
+| 字段 | 含义 |
+|------|------|
+| `first_wait` | 是否为进程首次调用 wait：`1` 首次，`0` 后续（用于区分首次建连开销） |
+| `ready_before_wait` | wait 前传输是否已完成：`1` 已完成（无需等待），`0` 需要等待 |
+| `strategy` | `TransferFuture` 的传输策略整数值 |
+| `wait_us` | 等待耗时（微秒） |
+| `result` | 传输结果：`OK` 表示成功，其他为错误码 |
+
+### 11.2 `open_segment_breakdown`
+
+```
+open_segment_breakdown endpoint[{addr}] open_segment_us[{us}] status[{0/-1}]
+```
+
+记录打开远端 Segment（建立传输连接）的耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `endpoint` | 传输引擎端点地址字符串 |
+| `open_segment_us` | `openSegment()` 调用耗时（微秒） |
+| `status` | 结果：`0` 成功，`-1` 失败（`ERR_INVALID_ARGUMENT`） |
+
+### 11.3 `submit_transfer_breakdown`
+
+```
+submit_transfer_breakdown first_transfer[{0/1}] batch_id[{id}] request_count[{n}] alloc_batch_id_us[{us}] submit_transfer_us[{us}] status[{0/err}]
+```
+
+记录提交批量传输请求的耗时拆分。
+
+| 字段 | 含义 |
+|------|------|
+| `first_transfer` | 是否为首次提交传输：`1` 首次，`0` 后续 |
+| `batch_id` | 批量传输 ID |
+| `request_count` | 本次提交的传输请求数量 |
+| `alloc_batch_id_us` | 分配 batch ID 耗时（微秒） |
+| `submit_transfer_us` | 提交传输请求耗时（微秒） |
+| `status` | 提交结果：`0` 成功，其他为 gRPC 状态码 |
+
+---
+
+## 12. URMA 端建连日志（第一次建立连接）
+
+来源：`mooncake-transfer-engine/src/transport/kunpeng_transport/urma/urma_endpoint.cpp`
+
+> **注意**：本节的 `*_breakdown` 日志用的是 `MC_LOG(INFO)`，**会带 `trace_id` 前缀**——主动端在握手时通过 `local_desc.trace_id = CurrentTraceId()` 把 trace_id 传给对端，所以一次建连的主动端和被动端日志可以用同一个 `trace_id` 串起来。少量纯状态日志（如 `"Connection has been established"`）仍是原生 `LOG()`，无 trace_id。
+
+记录 UB/URMA 传输端点的构建和建连过程各步骤耗时。
+
+### 12.0 先搞清楚：连接是什么时候建的、日志按什么顺序打
+
+**何时触发**：URMA 连接是**懒建立**的——`openSegment()` 只拿元数据，并不建连；直到第一次真正往某个 `peer_nic_path`（远端节点+网卡）提交传输时，worker 线程发现这个 endpoint 还没 connected，才触发建连。这也是为什么进程第一次传输某个对端时会有一段额外开销（对应 `first_transfer` / `first_wait` 字段的 `1`）。
+
+**建连分主动端（发起方）和被动端（响应方）两侧**，各自打不同的 breakdown 日志。把它们按发生顺序串起来，就是下面这张图——读真实日志时照着对号入座即可：
+
+```
+主动端 Node A 被动端 Node B
+（worker 发现 endpoint 未连）
+setupConnectionsByActive() ← 计时起点 t0
+ │
+ ├─[情况①: peer_nic == 本机自己] 同节点直连，跳过握手，直接 doSetupConnection
+ │ → urma_active_setup_breakdown ... local_peer[1] handshake_us[0] ... (§12.3 变体1)
+ │
+ └─[情况②: 跨节点] 需要握手
+ sendHandshake(local_desc{nic,jetty_num,trace_id}) ──RPC──▶ onSetupConnections()
+ setupConnectionsByPassive()
+ doSetupConnection() ← 被动端也建
+ 每个 jetty: import + bind
+ → urma_import_jetty_breakdown (§12.6)
+ → urma_bind_jetty_breakdown (§12.7)
+ → urma_do_setup_all_breakdown (§12.5)
+ ◀──返回 peer_desc{eid,jetty_num}──
+ → urma_passive_setup_breakdown (§12.4)
+ ├─ 握手失败 → urma_active_setup_breakdown ... handshake_us[..] do_setup_us[0] status[err] (§12.3 变体2)
+ └─ 握手成功 → doSetupConnection(peer_eid, peer_jetty_num) ← 主动端建
+ 每个 jetty: import + bind
+ → urma_import_jetty_breakdown (§12.6)
+ → urma_bind_jetty_breakdown (§12.7)
+ → urma_do_setup_all_breakdown (§12.5)
+ → urma_active_setup_breakdown ... local_peer[0] handshake_us[..] do_setup_us[..] status[0] (§12.3 变体3)
+```
+
+**各日志在排查里的分工**（拿到一条慢建连日志时这样定位）：
+
+- `urma_active_setup_breakdown` 是**主动端的总账**：先看 `total_us` 判断这次建连慢不慢，再看是 `handshake_us`（握手 RPC 慢，多半是网络/对端响应慢）还是 `do_setup_us`（本地 import+bind 慢）占大头。
+- `urma_passive_setup_breakdown` 是**被动端的总账**：和主动端用同一 `trace_id` 配对，看对端响应那一侧花了多久。
+- `urma_do_setup_all_breakdown` 把一次 `doSetupConnection` 里**所有 jetty 的 import+bind 循环**汇总；想进一步拆到单个 jetty，再看 `urma_import_jetty_breakdown` / `urma_bind_jetty_breakdown`。
+- `urma_endpoint_construct_breakdown` / `urma_create_jetty_breakdown` 属于**更早的端点构建期**（创建 jetty 资源，发生在 init/首次用到该 context 时），不在每次建连路径上，但首次开销分析要一并看。
+
+> 想从代码角度理解这套握手/Jetty 绑定流程（`setupConnectionsByActive` → `sendHandshake` → `doSetupConnection` → `import/bind jetty`），见 `transfer-engine-deep-dive.md` 第 8 站与 `urma-transfer-engine-flow.md` 第 5 站。对应的 PerfPoint 打点见本手册 §545「UB/URMA 建连侧」（`UB_ENDPOINT_ACTIVE_SETUP` / `ACTIVE_HANDSHAKE` / `PASSIVE_SETUP` 等）。
+
+下面是每条日志的逐字段参考。
+
+### 12.1 `urma_endpoint_construct_breakdown`
+
+```
+urma_endpoint_construct_breakdown local_nic[{nic}] jetty_count[{n}] construct_us[{us}] status[0]
+```
+
+记录 UrmaEndpoint 构建过程总耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `local_nic` | 本端 NIC 路径标识 |
+| `jetty_count` | 创建的 jetty 数量（即 `num_jetty_per_ep` 配置值） |
+| `construct_us` | 整个 construct 过程耗时（微秒），含所有 jetty 创建 |
+| `status` | 结果：`0` 成功 |
+
+### 12.2 `urma_create_jetty_breakdown`
+
+**成功时：**
+```
+urma_create_jetty_breakdown index[{i}] jetty_id[{id}] jfc_id[{id}] create_us[{us}] status[0]
+```
+
+**失败时：**
+```
+urma_create_jetty_breakdown index[{i}] create_us[{us}] status[-1]
+```
+
+记录单个 jetty 创建耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `index` | jetty 在 jetty_list 中的索引 |
+| `jetty_id` | 创建成功后的 jetty ID |
+| `jfc_id` | jetty 关联的 JFC（Jetty Flow Control）ID |
+| `create_us` | `urma_create_jetty()` 调用耗时（微秒） |
+| `status` | 结果：`0` 成功，`-1` 失败 |
+
+### 12.3 `urma_active_setup_breakdown`
+
+记录主动建连（active side）过程耗时。有三种变体：
+
+**本端对端（local_peer=1，同节点通信）：**
+```
+urma_active_setup_breakdown local_nic[{nic}] peer_nic[{nic}] local_peer[1] handshake_us[0] do_setup_us[{us}] total_us[{us}] status[{rc}]
+```
+
+**握手失败：**
+```
+urma_active_setup_breakdown local_nic[{nic}] peer_nic[{nic}] local_peer[0] handshake_us[{us}] do_setup_us[0] total_us[{us}] status[{rc}]
+```
+
+**成功路径：**
+```
+urma_active_setup_breakdown local_nic[{nic}] peer_nic[{nic}] local_peer[0] handshake_us[{us}] do_setup_us[{us}] total_us[{us}] status[{rc}]
+```
+
+| 字段 | 含义 |
+|------|------|
+| `local_nic` | 本端 NIC 路径标识 |
+| `peer_nic` | 对端 NIC 路径标识 |
+| `local_peer` | 是否为本节点内通信：`1` 是（无需握手，直接 doSetup），`0` 否（需跨节点握手） |
+| `handshake_us` | 握手耗时（微秒），本端对端时为 `0` |
+| `do_setup_us` | `doSetupConnection()` 耗时（微秒），握手失败时为 `0` |
+| `total_us` | 主动建连总耗时（微秒） |
+| `status` | 结果：`0` 成功，其他为错误码（如 `ERR_DEVICE_NOT_FOUND`、`ERR_REJECT_HANDSHAKE`） |
+
+### 12.4 `urma_passive_setup_breakdown`
+
+```
+urma_passive_setup_breakdown local_nic[{nic}] peer_nic[{nic}] total_us[{us}] status[{rc}]
+```
+
+记录被动建连（passive side，响应握手请求）过程耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `local_nic` | 本端 NIC 路径标识 |
+| `peer_nic` | 对端 NIC 路径标识 |
+| `total_us` | 被动建连总耗时（微秒），含 `doSetupConnection()` |
+| `status` | 结果：`0` 成功，其他为错误码 |
+
+### 12.5 `urma_do_setup_all_breakdown`
+
+```
+urma_do_setup_all_breakdown local_nic[{nic}] peer_nic[{nic}] jetty_count[{n}] total_us[{us}] status[0]
+```
+
+记录 `doSetupConnection()` 整体耗时（包含所有 jetty 的 import + bind 循环）。
+
+| 字段 | 含义 |
+|------|------|
+| `local_nic` | 本端 NIC 路径标识 |
+| `peer_nic` | 对端 NIC 路径标识 |
+| `jetty_count` | 设置的 jetty 数量（本端与对端匹配） |
+| `total_us` | 全部 jetty 设置总耗时（微秒） |
+| `status` | 结果：`0` 成功 |
+
+### 12.6 `urma_import_jetty_breakdown`
+
+```
+urma_import_jetty_breakdown index[{i}] peer_jetty_id[{id}] import_us[{us}] status[{0/-1}]
+```
+
+记录单个 jetty 的 `urma_import_jetty()` 调用耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `index` | jetty 在 jetty_list 中的索引 |
+| `peer_jetty_id` | 对端 jetty ID |
+| `import_us` | `urma_import_jetty()` 调用耗时（微秒） |
+| `status` | 结果：`0` 成功，`-1` 失败 |
+
+### 12.7 `urma_bind_jetty_breakdown`
+
+```
+urma_bind_jetty_breakdown index[{i}] local_jetty_id[{id}] peer_jetty_id[{id}] bind_us[{us}] status[0]
+```
+
+记录单个 jetty 的 `urma_bind_jetty()` 调用耗时。
+
+| 字段 | 含义 |
+|------|------|
+| `index` | jetty 在 jetty_list 中的索引 |
+| `local_jetty_id` | 本端 jetty ID |
+| `peer_jetty_id` | 对端 jetty ID（即 imported jetty） |
+| `bind_us` | `urma_bind_jetty()` 调用耗时（微秒） |
+| `status` | 结果：`0` 成功 |
+
+---
+
+## 13. 慢操作告警汇总
+
+来源：`mooncake-store/src/real_client.cpp`
+
+慢操作告警统一由 `real_client.cpp` 输出，阈值为 **3000us（3ms）**。仅当操作耗时超过阈值时才输出 WARNING 级别日志。
+
+> **注意**：这些告警日志由 MC_LOG 输出，自动带 `trace_id` 前缀。
+
+### 13.1 单 key 操作
+
+```
+trace_id[xxx] {op}_slow key[{key}] size[{size}] elapsed_us[{us}] rc[{rc}]
+```
+
+| 字段 | 含义 |
+|------|------|
+| `{op}` | 操作名：`get_buffer` / `get_into` / `put` / `put_parts` / `put_from` |
+| `key` | 对象 key |
+| `size` | 数据大小（字节） |
+| `elapsed_us` | 操作总耗时（微秒） |
+| `rc` | 返回码：`0` 成功（但慢），`-1` 失败 |
+
+### 13.2 批量操作（带 success 计数）
+
+```
+trace_id[xxx] {op}_slow num_keys[{n}] size[{size}] elapsed_us[{us}] success[{s}]
+```
+
+| 字段 | 含义 |
+|------|------|
+| `{op}` | 操作名：`batch_get_buffer` / `batch_get_into` / `batch_put_from` / `batch_put_from_multi_buffers` |
+| `num_keys` | 批量操作的 key 数量 |
+| `size` | 批量操作的总数据大小（字节） |
+| `elapsed_us` | 操作总耗时（微秒） |
+| `success` | 成功的 key 数量 |
+
+### 13.3 批量操作（带 rc 返回码）
+
+```
+trace_id[xxx] {op}_slow num_keys[{n}] size[{size}] elapsed_us[{us}] rc[{rc}]
+```
+
+| 字段 | 含义 |
+|------|------|
+| `{op}` | 操作名：`put_batch` |
+| `num_keys` | 批量操作的 key 数量 |
+| `size` | 批量操作的总数据大小（字节） |
+| `elapsed_us` | 操作总耗时（微秒） |
+| `rc` | 返回码：`0` 成功（但慢），其他为错误码 |
+
+### 13.4 与旧版 Python 层慢日志的关系
+
+旧版 `store_py.cpp` 的 `get_slow` / `put_slow` 已移除。旧版 `get_batch_slow` / `put_batch_slow` 仍在 `store_py.cpp` 中保留（使用原生 `LOG()`，无 trace_id）。
+
+当操作超过 3ms 时：
+- 单 key `get`/`put`：仅 real_client.cpp 输出慢日志
+- 批量 `get_batch`/`put_batch`：可能同时出现 `store_py.cpp` 的慢日志（无 trace_id）和 `real_client.cpp` 的慢日志（有 trace_id）
diff --git a/docs/yh/pipline.md b/docs/yh/pipline.md
new file mode 100644
index 0000000000..f0d5a0d0f5
--- /dev/null
+++ b/docs/yh/pipline.md
@@ -0,0 +1,317 @@
+## 打点流程图
+
+### `get` 流程
+
+```mermaid
+flowchart TB
+ Start["store_py::get(key)
+#include
+#include
+#include
#include
+#include
+#include
+#include
+#include
#include
+#include
#include
#include
@@ -148,36 +156,63 @@ class DefaultConfig {
std::unordered_map data_;
};
-// usage: export MC_YLT_LOG_LEVEL=info or export MC_YLT_LOG_LEVEL=debug etc.
+// Configure yalantinglibs logging with MC_YLT_LOG_LEVEL, MC_YLT_LOG_PATH,
+// MC_YLT_LOG_MAX_FILE_SIZE (bytes), and MC_YLT_LOG_MAX_FILES.
inline void init_ylt_log_level() {
- const char* env_level = std::getenv("MC_YLT_LOG_LEVEL");
- if (!env_level || !*env_level) {
- // default is WARN
- easylog::set_min_severity(easylog::Severity::WARN);
- return;
- }
- std::string level_str(env_level);
- std::transform(level_str.begin(), level_str.end(), level_str.begin(),
- [](unsigned char c) { return std::tolower(c); });
- easylog::Severity severity;
- if (level_str == "trace") {
- severity = easylog::Severity::TRACE;
- } else if (level_str == "debug") {
- severity = easylog::Severity::DEBUG;
- } else if (level_str == "info") {
- severity = easylog::Severity::INFO;
- } else if (level_str == "warn" || level_str == "warning") {
- severity = easylog::Severity::WARN;
- } else if (level_str == "error") {
- severity = easylog::Severity::ERROR;
- } else if (level_str == "critical") {
- severity = easylog::Severity::CRITICAL;
- } else {
- // rollback to WARN
- severity = easylog::Severity::WARN;
- }
+ static std::once_flag once;
+ std::call_once(once, [] {
+ easylog::Severity severity = easylog::Severity::WARN;
+ const char* env_level = std::getenv("MC_YLT_LOG_LEVEL");
+ if (env_level && *env_level) {
+ std::string level_str(env_level);
+ std::transform(level_str.begin(), level_str.end(),
+ level_str.begin(),
+ [](unsigned char c) { return std::tolower(c); });
+ if (level_str == "trace") {
+ severity = easylog::Severity::TRACE;
+ } else if (level_str == "debug") {
+ severity = easylog::Severity::DEBUG;
+ } else if (level_str == "info") {
+ severity = easylog::Severity::INFO;
+ } else if (level_str == "warn" || level_str == "warning") {
+ severity = easylog::Severity::WARN;
+ } else if (level_str == "error") {
+ severity = easylog::Severity::ERROR;
+ } else if (level_str == "critical") {
+ severity = easylog::Severity::CRITICAL;
+ }
+ }
+
+ std::string log_path = "logs/rpc.log";
+ const char* env_log_path = std::getenv("MC_YLT_LOG_PATH");
+ if (env_log_path && *env_log_path) {
+ log_path = env_log_path;
+ }
- easylog::set_min_severity(severity);
+ auto get_size_env = [](const char* name, size_t default_value) {
+ const char* value = std::getenv(name);
+ if (!value || !*value) return default_value;
+
+ uint64_t parsed = 0;
+ const char* end = value + std::strlen(value);
+ const auto result = std::from_chars(value, end, parsed);
+ if (result.ec != std::errc() || result.ptr != end ||
+ parsed > std::numeric_limits::max()) {
+ return default_value;
+ }
+ return static_cast(parsed);
+ };
+
+ constexpr size_t kDefaultMaxFileSize = 1000ULL * 1024 * 1024;
+ constexpr size_t kDefaultMaxFiles = 3;
+ const size_t max_file_size = get_size_env(
+ "MC_YLT_LOG_MAX_FILE_SIZE", kDefaultMaxFileSize);
+ const size_t max_files =
+ get_size_env("MC_YLT_LOG_MAX_FILES", kDefaultMaxFiles);
+
+ easylog::init_log(severity, log_path, true, false, max_file_size,
+ max_files, false);
+ });
}
} // namespace mooncake
diff --git a/mooncake-common/include/environ.h b/mooncake-common/include/environ.h
index 3ff16e4723..197551161c 100644
--- a/mooncake-common/include/environ.h
+++ b/mooncake-common/include/environ.h
@@ -61,6 +61,13 @@ class Environ {
bool GetPathRoundrobin() const { return path_roundrobin_; }
bool GetWithNvidiaPeermem() const { return with_nvidia_peermem_; }
int GetEfaCqThreads() const { return efa_cq_threads_; }
+ size_t GetOffloadRpcThreadNum(size_t default_value = 8) const;
+ uint32_t GetYltRpcPoolMaxConnection(uint32_t default_value = 100) const;
+ size_t GetYltRpcPoolIdleTimeoutMs(size_t default_value) const;
+ size_t GetYltRpcPoolShortIdleTimeoutMs(size_t default_value) const;
+ bool GetYltRpcPoolWarmupEnabled(bool default_value = true) const;
+ size_t GetYltRpcPoolWarmupConnections(size_t default_value) const;
+ bool GetStoreWarmupEnabled(bool default_value = false) const;
bool GetStoreChecksumEnabled() const { return store_checksum_enabled_; }
// AWS / S3 client configuration
diff --git a/mooncake-common/include/mooncake_logging.h b/mooncake-common/include/mooncake_logging.h
new file mode 100644
index 0000000000..df7ef39564
--- /dev/null
+++ b/mooncake-common/include/mooncake_logging.h
@@ -0,0 +1,72 @@
+#pragma once
+
+#include
+#include
+#include
+
+#include
+
+namespace mooncake::logging {
+
+uint64_t NewTraceId();
+uint64_t CurrentTraceId();
+bool ShouldLog(google::LogSeverity severity);
+bool ShouldVLog(int level);
+void ApplyMooncakeLogEnableToGlog();
+
+// High-frequency log sampling. Parses MC_HIFREQ_LOG_SAMPLE_RATE once (default
+// 0.1, clamped to [0,1]) and caches it process-wide. ShouldSampleHiFreqLog()
+// without an id rolls a thread-local RNG. The trace-id overload hashes the id
+// so all layers handling the same request make the same sampling decision.
+double HiFreqLogSampleRate();
+bool ShouldSampleHiFreqLog();
+// Deterministic variant used to correlate high-frequency logs across
+// processes and worker threads. A zero trace id falls back to random sampling.
+bool ShouldSampleHiFreqLog(uint64_t trace_id);
+
+class ScopedTraceId {
+ public:
+ explicit ScopedTraceId(uint64_t trace_id);
+ ~ScopedTraceId();
+
+ ScopedTraceId(const ScopedTraceId&) = delete;
+ ScopedTraceId& operator=(const ScopedTraceId&) = delete;
+
+ private:
+ uint64_t previous_trace_id_;
+};
+
+class AsyncLogMessage {
+ public:
+ AsyncLogMessage(const char* file, int line, google::LogSeverity severity,
+ bool enabled);
+ ~AsyncLogMessage();
+
+ AsyncLogMessage(const AsyncLogMessage&) = delete;
+ AsyncLogMessage& operator=(const AsyncLogMessage&) = delete;
+
+ std::ostream& stream();
+
+ private:
+ const char* file_;
+ int line_;
+ google::LogSeverity severity_;
+ bool enabled_;
+ uint64_t trace_id_;
+ std::ostringstream stream_;
+};
+
+void FlushAsyncLogs();
+
+} // namespace mooncake::logging
+
+#define MC_LOG(severity) \
+ mooncake::logging::AsyncLogMessage( \
+ __FILE__, __LINE__, google::severity, \
+ mooncake::logging::ShouldLog(google::severity)) \
+ .stream()
+
+#define MC_VLOG(level) \
+ mooncake::logging::AsyncLogMessage(__FILE__, __LINE__, google::INFO, \
+ mooncake::logging::ShouldVLog(level)) \
+ .stream()
diff --git a/mooncake-common/src/CMakeLists.txt b/mooncake-common/src/CMakeLists.txt
index 2730485cf1..fcfb77ab2a 100644
--- a/mooncake-common/src/CMakeLists.txt
+++ b/mooncake-common/src/CMakeLists.txt
@@ -1,7 +1,8 @@
find_package(yaml-cpp REQUIRED)
set(MOONCAKE_COMMON_SOURCES crc_checksum.cpp default_config.cpp environ.cpp
- rpc_client_io_context.cpp)
+ rpc_client_io_context.cpp mooncake_logging.cpp
+)
add_library(asio_shared SHARED asio_impl.cpp)
@@ -40,7 +41,9 @@ target_include_directories(
$)
target_link_libraries(mooncake_common PUBLIC asio_shared yaml-cpp jsoncpp
- yalantinglibs::yalantinglibs)
+ glog::glog
+ gflags::gflags
+ yalantinglibs::yalantinglibs)
if(BUILD_SHARED_LIBS)
install(TARGETS mooncake_common DESTINATION lib)
diff --git a/mooncake-common/src/environ.cpp b/mooncake-common/src/environ.cpp
index de6faec6d2..b6b5b4e9cf 100644
--- a/mooncake-common/src/environ.cpp
+++ b/mooncake-common/src/environ.cpp
@@ -111,6 +111,44 @@ uint32_t ResolveRpcClientIoThreads(const EnvironSource& source,
} // namespace
+size_t Environ::GetOffloadRpcThreadNum(size_t default_value) const {
+ const size_t value = GetSizeT("MC_OFFLOAD_RPC_THREAD_NUM", default_value);
+ return value == 0 ? default_value : value;
+}
+
+uint32_t Environ::GetYltRpcPoolMaxConnection(uint32_t default_value) const {
+ const size_t value =
+ GetSizeT("MC_YLT_RPC_POOL_MAX_CONNECTION", default_value);
+ if (value == 0 || value > UINT32_MAX) {
+ std::cerr << "[Mooncake] Warning: invalid value for env "
+ << "MC_YLT_RPC_POOL_MAX_CONNECTION, using default "
+ << default_value << std::endl;
+ return default_value;
+ }
+ return static_cast(value);
+}
+
+size_t Environ::GetYltRpcPoolIdleTimeoutMs(size_t default_value) const {
+ return GetSizeT("MC_YLT_RPC_POOL_IDLE_TIMEOUT_MS", default_value);
+}
+
+size_t Environ::GetYltRpcPoolShortIdleTimeoutMs(
+ size_t default_value) const {
+ return GetSizeT("MC_YLT_RPC_POOL_SHORT_IDLE_TIMEOUT_MS", default_value);
+}
+
+bool Environ::GetYltRpcPoolWarmupEnabled(bool default_value) const {
+ return GetBool("MC_YLT_RPC_POOL_WARMUP", default_value);
+}
+
+size_t Environ::GetYltRpcPoolWarmupConnections(size_t default_value) const {
+ return GetSizeT("MC_YLT_RPC_POOL_WARMUP_CONNECTIONS", default_value);
+}
+
+bool Environ::GetStoreWarmupEnabled(bool default_value) const {
+ return GetBool("MC_STORE_WARMUP", default_value);
+}
+
Environ& Environ::Get() {
static Environ instance(GetOsEnvironSource());
return instance;
diff --git a/mooncake-common/src/mooncake_logging.cpp b/mooncake-common/src/mooncake_logging.cpp
new file mode 100644
index 0000000000..e79b6cb27c
--- /dev/null
+++ b/mooncake-common/src/mooncake_logging.cpp
@@ -0,0 +1,392 @@
+#include "mooncake_logging.h"
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#ifdef _WIN32
+#include
+#else
+#include
+#endif
+
+namespace mooncake::logging {
+namespace {
+
+thread_local uint64_t current_trace_id = 0;
+
+uint64_t GetPidForTrace() {
+#ifdef _WIN32
+ return static_cast(_getpid());
+#else
+ return static_cast(getpid());
+#endif
+}
+
+uint64_t SteadyClockNs() {
+ return static_cast(
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now().time_since_epoch())
+ .count());
+}
+
+double ParseHiFreqLogSampleRate() {
+ const char* value = std::getenv("MC_HIFREQ_LOG_SAMPLE_RATE");
+ if (value == nullptr || *value == '\0')
+ return 1.0; // default: full sampling
+ errno = 0;
+ char* end = nullptr;
+ double rate = std::strtod(value, &end);
+ if (end == value || errno != 0) return 1.0; // non-numeric / overflow
+ if (rate < 0.0) return 0.0;
+ if (rate > 1.0) return 1.0;
+ return rate;
+}
+
+// Interval for the background periodic flush performed by the async worker
+// thread. Default 1s; tunable via MC_LOG_FLUSH_SECS (accepts fractional
+// seconds, e.g. "0.5"). Clamped to a 50ms floor to avoid pathological busy
+// flushing.
+std::chrono::milliseconds ParseFlushIntervalMs() {
+ double secs = 1.0;
+ const char* value = std::getenv("MC_LOG_FLUSH_SECS");
+ if (value != nullptr && *value != '\0') {
+ errno = 0;
+ char* end = nullptr;
+ double parsed = std::strtod(value, &end);
+ if (end != value && errno == 0 && parsed > 0.0) secs = parsed;
+ }
+ if (secs < 0.05) secs = 0.05;
+ return std::chrono::milliseconds(static_cast(secs * 1000.0));
+}
+
+// Ring buffer capacity (messages). Default 65536; tunable via
+// MC_LOG_QUEUE_SIZE. Floored at 64 so the buffer stays usable.
+size_t ParseQueueSize() {
+ const char* value = std::getenv("MC_LOG_QUEUE_SIZE");
+ if (value == nullptr || *value == '\0') return 65536;
+ long parsed = std::strtol(value, nullptr, 10);
+ if (parsed < 64) return 64;
+ return static_cast(parsed);
+}
+
+// Number of background writer threads. Default 2; tunable via MC_LOG_WORKERS,
+// clamped to [1, 16]. Note glog serializes the actual file write internally,
+// so extra workers mostly overlap message formatting with IO and keep the ring
+// drained rather than parallelizing disk writes.
+size_t ParseWorkerCount() {
+ const char* value = std::getenv("MC_LOG_WORKERS");
+ if (value == nullptr || *value == '\0') return 2;
+ long parsed = std::strtol(value, nullptr, 10);
+ if (parsed < 1) return 1;
+ if (parsed > 16) return 16;
+ return static_cast(parsed);
+}
+
+struct LogEntry {
+ const char* file;
+ int line;
+ google::LogSeverity severity;
+ uint64_t trace_id;
+ std::string message;
+};
+
+// Async log pipeline backing MC_LOG().
+//
+// Hot path (Enqueue): take the mutex, move the entry into a PRE-ALLOCATED ring
+// buffer, release. No per-message heap allocation for queue nodes and no
+// blocking: when the ring is full the OLDEST entry is overwritten (overrun) and
+// a counter is bumped. This bounds producer latency -- a hot business thread
+// is never stalled waiting on slow log IO -- at the cost of dropping the oldest
+// pending lines under sustained overload. The dropped count is reported
+// periodically as a WARNING so loss is never silent.
+//
+// Background: ParseWorkerCount() writer threads drain the ring via glog. One
+// of them also performs the periodic FlushLogFiles (coordinated by an atomic so
+// it runs once per interval regardless of worker count) and the overrun report
+// -- both off the hot path.
+class AsyncLogQueue {
+ public:
+ static AsyncLogQueue& Instance() {
+ static AsyncLogQueue* queue = new AsyncLogQueue();
+ return *queue;
+ }
+
+ void Enqueue(LogEntry entry) {
+ EnsureStarted();
+ {
+ std::lock_guard lock(mutex_);
+ if (stopped_) {
+ WriteSync(entry);
+ return;
+ }
+ RingPush(std::move(entry)); // overruns oldest if full
+ }
+ queue_not_empty_.notify_one();
+ }
+
+ void Flush() {
+ EnsureStarted();
+ {
+ std::unique_lock lock(mutex_);
+ queue_empty_.wait(
+ lock, [this] { return RingEmpty() && active_writes_ == 0; });
+ }
+ google::FlushLogFiles(google::INFO);
+ }
+
+ private:
+ AsyncLogQueue()
+ : capacity_(ParseQueueSize() + 1), // +1 slot reserved as full marker
+ worker_count_(ParseWorkerCount()),
+ flush_interval_(ParseFlushIntervalMs()),
+ ring_(capacity_) {}
+
+ void EnsureStarted() {
+ std::call_once(start_once_, [this] {
+ for (size_t i = 0; i < worker_count_; ++i) {
+ workers_.emplace_back([this] { WorkerLoop(); });
+ }
+ std::atexit([] { AsyncLogQueue::Instance().Stop(); });
+ });
+ }
+
+ void Stop() {
+ {
+ std::lock_guard lock(mutex_);
+ stopped_ = true;
+ }
+ queue_not_empty_.notify_all();
+ for (auto& w : workers_) {
+ if (w.joinable()) w.join();
+ }
+ ReportOverruns();
+ google::FlushLogFiles(google::INFO);
+ }
+
+ void WorkerLoop() {
+ // The periodic flush runs here, off the hot path: log producers never
+ // pay a flush cost. google::FlushLogFiles drains glog's file buffer,
+ // covering BOTH the async MC_LOG output written below AND synchronous
+ // LOG()/MC_LOG emitted on business threads (they share the same glog
+ // file). This keeps the tail of the log (e.g. get_into_breakdown /
+ // put_result) from being lost when the host process is torn down
+ // without running atexit/static destructors (Go's os.Exit, SIGKILL
+ // under k8s).
+ while (true) {
+ LogEntry entry;
+ bool have_entry = false;
+ {
+ std::unique_lock lock(mutex_);
+ queue_not_empty_.wait_for(lock, flush_interval_, [this] {
+ return stopped_ || !RingEmpty();
+ });
+ if (!RingEmpty()) {
+ entry = std::move(ring_[head_]);
+ head_ = (head_ + 1) % capacity_;
+ ++active_writes_;
+ have_entry = true;
+ } else {
+ queue_empty_.notify_all();
+ if (stopped_) return;
+ }
+ }
+ if (have_entry) {
+ WriteSync(entry);
+ std::lock_guard lock(mutex_);
+ --active_writes_;
+ if (RingEmpty() && active_writes_ == 0) {
+ queue_empty_.notify_all();
+ }
+ }
+ MaybePeriodicFlush();
+ }
+ }
+
+ // Flush + overrun report, gated by an atomic so it runs at most once per
+ // interval across all workers. Bounds the worst-case loss window on a hard
+ // kill to one interval, with no per-message flush overhead.
+ void MaybePeriodicFlush() {
+ const int64_t now_ns =
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now().time_since_epoch())
+ .count();
+ const int64_t interval_ns =
+ std::chrono::duration_cast(
+ flush_interval_)
+ .count();
+ int64_t last = last_flush_ns_.load(std::memory_order_relaxed);
+ if (now_ns - last < interval_ns) return;
+ if (!last_flush_ns_.compare_exchange_strong(
+ last, now_ns, std::memory_order_relaxed)) {
+ return; // another worker won this interval
+ }
+ google::FlushLogFiles(google::INFO);
+ ReportOverruns();
+ }
+
+ // Emit one synchronous WARNING if entries were overrun since the last
+ // report, so dropped logs are never silent. The counter is read+reset
+ // under the queue mutex; the WriteSync itself does not touch the ring.
+ void ReportOverruns() {
+ uint64_t dropped = 0;
+ {
+ std::lock_guard lock(mutex_);
+ dropped = overrun_count_;
+ overrun_count_ = 0;
+ }
+ if (dropped > 0) {
+ google::LogMessage(__FILE__, __LINE__, google::WARNING).stream()
+ << "trace_id[none] async log ring overran, dropped " << dropped
+ << " message(s); raise MC_LOG_QUEUE_SIZE or MC_LOG_WORKERS";
+ }
+ }
+
+ // --- ring buffer, all access guarded by mutex_ ---
+ bool RingEmpty() const { return head_ == tail_; }
+
+ void RingPush(LogEntry&& entry) {
+ ring_[tail_] = std::move(entry);
+ tail_ = (tail_ + 1) % capacity_;
+ if (tail_ == head_) { // full: overwrite oldest
+ head_ = (head_ + 1) % capacity_;
+ ++overrun_count_;
+ }
+ }
+
+ static void WriteSync(const LogEntry& entry) {
+ google::LogMessage log_message(entry.file, entry.line, entry.severity);
+ auto& stream = log_message.stream();
+ if (entry.trace_id != 0) {
+ stream << "trace_id[" << entry.trace_id << "] ";
+ } else {
+ stream << "trace_id[none] ";
+ }
+ stream << entry.message;
+ if (entry.severity == google::FATAL)
+ google::FlushLogFiles(google::INFO);
+ }
+
+ const size_t capacity_;
+ const size_t worker_count_;
+ const std::chrono::milliseconds flush_interval_;
+
+ std::once_flag start_once_;
+ std::vector workers_;
+
+ std::mutex mutex_;
+ std::condition_variable queue_not_empty_;
+ std::condition_variable queue_empty_;
+
+ std::vector ring_;
+ size_t head_ = 0;
+ size_t tail_ = 0;
+ uint64_t overrun_count_ = 0;
+ size_t active_writes_ = 0;
+ bool stopped_ = false;
+
+ std::atomic last_flush_ns_{0};
+};
+
+} // namespace
+
+uint64_t NewTraceId() {
+ static const uint64_t process_seed =
+ (GetPidForTrace() << 48) ^ (SteadyClockNs() & 0x0000FFFFFFFF0000ULL);
+ static std::atomic counter{1};
+ return process_seed ^ counter.fetch_add(1, std::memory_order_relaxed);
+}
+
+uint64_t CurrentTraceId() { return current_trace_id; }
+
+double HiFreqLogSampleRate() {
+ static const double rate = ParseHiFreqLogSampleRate();
+ return rate;
+}
+
+bool ShouldSampleHiFreqLog() {
+ const double rate = HiFreqLogSampleRate();
+ if (rate >= 1.0) return true;
+ if (rate <= 0.0) return false;
+ thread_local std::mt19937 rng(static_cast(
+ SteadyClockNs() ^ reinterpret_cast(&rng)));
+ thread_local std::uniform_real_distribution dist(0.0, 1.0);
+ return dist(rng) < rate;
+}
+
+bool ShouldSampleHiFreqLog(uint64_t trace_id) {
+ if (trace_id == 0) return ShouldSampleHiFreqLog();
+ const double rate = HiFreqLogSampleRate();
+ if (rate >= 1.0) return true;
+ if (rate <= 0.0) return false;
+
+ // SplitMix64 finalizer: stable, cheap, and sufficiently uniform for
+ // deterministic sampling. Use the top 53 bits to match double precision.
+ uint64_t value = trace_id + 0x9e3779b97f4a7c15ULL;
+ value = (value ^ (value >> 30)) * 0xbf58476d1ce4e5b9ULL;
+ value = (value ^ (value >> 27)) * 0x94d049bb133111ebULL;
+ value ^= value >> 31;
+ constexpr double kScale = 1.0 / static_cast(1ULL << 53);
+ return static_cast(value >> 11) * kScale < rate;
+}
+
+bool ShouldLog(google::LogSeverity severity) {
+ // MC_LOG_ENABLE was removed: MC_LOG now behaves like plain glog LOG and is
+ // gated only by glog's own severity threshold (still async + trace_id).
+ if (severity == google::FATAL) return true;
+ return severity >= FLAGS_minloglevel;
+}
+
+bool ShouldVLog(int level) { return VLOG_IS_ON(level); }
+
+void ApplyMooncakeLogEnableToGlog() {
+ // No-op retained for call-site compatibility (master/real_client main).
+ // MC_LOG_ENABLE was removed; nothing to apply.
+}
+
+ScopedTraceId::ScopedTraceId(uint64_t trace_id)
+ : previous_trace_id_(current_trace_id) {
+ current_trace_id = trace_id;
+}
+
+ScopedTraceId::~ScopedTraceId() { current_trace_id = previous_trace_id_; }
+
+AsyncLogMessage::AsyncLogMessage(const char* file, int line,
+ google::LogSeverity severity, bool enabled)
+ : file_(file),
+ line_(line),
+ severity_(severity),
+ enabled_(enabled),
+ trace_id_(CurrentTraceId()) {}
+
+AsyncLogMessage::~AsyncLogMessage() {
+ if (!enabled_) return;
+ if (severity_ == google::FATAL) {
+ google::LogMessage log_message(file_, line_, severity_);
+ auto& output = log_message.stream();
+ if (trace_id_ != 0) {
+ output << "trace_id[" << trace_id_ << "] ";
+ } else {
+ output << "trace_id[none] ";
+ }
+ output << stream_.str();
+ return;
+ }
+ AsyncLogQueue::Instance().Enqueue(
+ LogEntry{file_, line_, severity_, trace_id_, stream_.str()});
+}
+
+std::ostream& AsyncLogMessage::stream() { return stream_; }
+
+void FlushAsyncLogs() { AsyncLogQueue::Instance().Flush(); }
+
+} // namespace mooncake::logging
diff --git a/mooncake-integration/CMakeLists.txt b/mooncake-integration/CMakeLists.txt
index c7fe22818b..ba9767de27 100644
--- a/mooncake-integration/CMakeLists.txt
+++ b/mooncake-integration/CMakeLists.txt
@@ -113,6 +113,11 @@ if(WITH_STORE)
store/engram_store_py.cpp
integration_utils.h)
set_target_properties(store PROPERTIES INSTALL_RPATH "$ORIGIN")
+
+ include(${CMAKE_SOURCE_DIR}/mooncake-common/FindSpDiag.cmake)
+ target_include_directories(store PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/store)
+ target_link_libraries(store PRIVATE SpDiag::spdiag_lib)
+
if(USE_ASCEND_DIRECT)
target_link_libraries(
store PUBLIC ascendcl transfer_engine glog::glog gflags::gflags
diff --git a/mooncake-integration/store/mooncake_perf_points.def b/mooncake-integration/store/mooncake_perf_points.def
new file mode 100644
index 0000000000..eaf6dd992d
--- /dev/null
+++ b/mooncake-integration/store/mooncake_perf_points.def
@@ -0,0 +1,216 @@
+// === Python绑定层 ===
+PERF_KEY_DEF(GET_STORE_PY_GET, "store_py.cpp::get", "Get")
+PERF_KEY_DEF(GET_STORE_PY_GET_BATCH, "store_py.cpp::get_batch", "GetBatch")
+
+// === RealClient核心逻辑层 ===
+PERF_KEY_DEF(GET_BUFFER_INTERNAL, "store_py.cpp::get", "GetBuffer")
+PERF_KEY_DEF(GET_BATCH_BUFFER_INTERNAL, "store_py.cpp::get_batch", "BatchGetBuffer")
+PERF_KEY_DEF(GET_BUFFER_INTERNAL_FULL, "real_client.cpp::get_buffer", "GetBufferInternal")
+PERF_KEY_DEF(GET_BATCH_BUFFER_INTERNAL_FULL, "real_client.cpp::batch_get_buffer", "BatchGetBufferInternal")
+PERF_KEY_DEF(GET_INTO_INTERNAL, "real_client.cpp::get_into", "GetIntoInternal")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL, "real_client.cpp::batch_get_into", "BatchGetIntoInternal")
+
+// === get_buffer_internal 子步骤 ===
+PERF_KEY_DEF(GET_INTERNAL_QUERY, "real_client.cpp::get_buffer_internal", "Query")
+PERF_KEY_DEF(GET_INTERNAL_SELECT_REPLICA, "real_client.cpp::get_buffer_internal", "SelectReplica")
+PERF_KEY_DEF(GET_INTERNAL_ALLOC_BUFFER, "real_client.cpp::get_buffer_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_INTERNAL_SSD_READ, "real_client.cpp::get_buffer_internal", "SSDRead")
+PERF_KEY_DEF(GET_INTERNAL_MEM_READ, "real_client.cpp::get_buffer_internal", "MemRead")
+PERF_KEY_DEF(GET_INTERNAL_DISK_READ, "real_client.cpp::get_buffer_internal", "DiskRead")
+
+// === batch_get_buffer_internal 子步骤 ===
+PERF_KEY_DEF(GET_BATCH_INTERNAL_QUERY, "real_client.cpp::batch_get_buffer_internal", "BatchQuery")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_PREPARATION, "real_client.cpp::batch_get_buffer_internal", "Preparation")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_SELECT_REPLICA, "real_client.cpp::batch_get_buffer_internal", "SelectReplica")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_ALLOC_BUFFER, "real_client.cpp::batch_get_buffer_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_SSD_READ, "real_client.cpp::batch_get_buffer_internal", "SSDRead")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_MEMDISH_READ,"real_client.cpp::batch_get_buffer_internal", "MemDiskRead")
+
+// === get_into_internal steps ===
+PERF_KEY_DEF(GET_INTO_INTERNAL_QUERY, "real_client.cpp::get_into_internal", "Query")
+PERF_KEY_DEF(GET_INTO_INTERNAL_SELECT_REPLICA, "real_client.cpp::get_into_internal", "SelectReplica")
+PERF_KEY_DEF(GET_INTO_INTERNAL_ALLOC_BUFFER, "real_client.cpp::get_into_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_INTO_INTERNAL_SSD_READ, "real_client.cpp::get_into_internal", "SSDRead")
+PERF_KEY_DEF(GET_INTO_INTERNAL_MEM_READ, "real_client.cpp::get_into_internal", "MemRead")
+PERF_KEY_DEF(GET_INTO_INTERNAL_DISK_READ, "real_client.cpp::get_into_internal", "DiskRead")
+
+// === batch_get_into_internal steps ===
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_QUERY, "real_client.cpp::batch_get_into_internal", "BatchQuery")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_SELECT_REPLICA, "real_client.cpp::batch_get_into_internal", "SelectReplica")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_ALLOC_BUFFER, "real_client.cpp::batch_get_into_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_MEM_READ, "real_client.cpp::batch_get_into_internal", "MemRead")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_DISK_READ, "real_client.cpp::batch_get_into_internal", "DiskRead")
+PERF_KEY_DEF(GET_BATCH_INTO_INTERNAL_SSD_READ, "real_client.cpp::batch_get_into_internal", "SSDRead")
+
+// === Client::Get (单key) ===
+PERF_KEY_DEF(GET_SINGLE_FULL, "client_service.cpp::Get", "TransferGet")
+PERF_KEY_DEF(GET_SINGLE_FIND_REPLICA, "client_service.cpp::Get", "FindReplica")
+PERF_KEY_DEF(GET_SINGLE_HOT_CACHE, "client_service.cpp::Get", "HotCache")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_READ, "client_service.cpp::Get", "TransferRead")
+PERF_KEY_DEF(GET_SINGLE_RELEASE_CACHE, "client_service.cpp::Get", "ReleaseCache")
+PERF_KEY_DEF(GET_SINGLE_ASYNC_CACHE, "client_service.cpp::Get", "AsyncCache")
+
+// === Client::Get 内 TransferData ===
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_FULL, "client_service.cpp::TransferData", "TransferData[G]")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_SUBMIT, "client_service.cpp::TransferData", "Submit[G]")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_WAIT, "client_service.cpp::TransferData", "Wait[G]")
+
+// === Client::BatchGet (批量) ===
+PERF_KEY_DEF(GET_BATCH_FULL, "client_service.cpp::BatchGet", "TransferBatchGet")
+PERF_KEY_DEF(GET_BATCH_FIND_REPLICA, "client_service.cpp::BatchGet", "FindReplica")
+PERF_KEY_DEF(GET_BATCH_HOT_CACHE, "client_service.cpp::BatchGet", "HotCache")
+PERF_KEY_DEF(GET_BATCH_SUBMIT, "client_service.cpp::BatchGet", "Submit")
+PERF_KEY_DEF(GET_BATCH_WAIT, "client_service.cpp::BatchGet", "Wait")
+PERF_KEY_DEF(GET_BATCH_RELEASE_CACHE, "client_service.cpp::BatchGet", "ReleaseCache")
+PERF_KEY_DEF(GET_BATCH_ASYNC_CACHE, "client_service.cpp::BatchGet", "AsyncCache")
+
+// === Python绑定层 ===
+PERF_KEY_DEF(PUT_STORE_PY_PUT, "store_py.cpp::put", "Put")
+PERF_KEY_DEF(PUT_STORE_PY_PUT_BATCH, "store_py.cpp::put_batch", "PutBatch")
+
+// === RealClient核心逻辑层 ===
+PERF_KEY_DEF(PUT_INTERNAL_FULL, "store_py.cpp::put", "PutBuffer")
+PERF_KEY_DEF(PUT_INTERNAL_ALLOC_BUFFER, "real_client.cpp::put_internal", "AllocBuffer")
+PERF_KEY_DEF(PUT_INTERNAL_MEM_COPY, "real_client.cpp::put_internal", "MemCopy")
+PERF_KEY_DEF(PUT_INTERNAL_SPLIT_SLICES, "real_client.cpp::put_internal", "SplitSlices")
+
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_FULL, "store_py.cpp::put_batch", "BatchPutBuffer")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_ALLOC_BUFFER,"real_client.cpp::put_batch_internal", "AllocBuffer")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_MEM_COPY, "real_client.cpp::put_batch_internal", "MemCopy")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_SPLIT_SLICES,"real_client.cpp::put_batch_internal", "SplitSlices")
+
+// === Client::Put (单key) ===
+PERF_KEY_DEF(PUT_SINGLE_FULL, "client_service.cpp::Put", "TransferPut")
+PERF_KEY_DEF(PUT_SINGLE_PUT_START, "client_service.cpp::Put", "PutStart")
+PERF_KEY_DEF(PUT_SINGLE_DISK_WRITE, "client_service.cpp::Put", "DiskWrite")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_WRITE, "client_service.cpp::Put", "TransferWrite")
+PERF_KEY_DEF(PUT_SINGLE_PUT_END, "client_service.cpp::Put", "PutEnd")
+PERF_KEY_DEF(PUT_SINGLE_PUT_REVOKE, "client_service.cpp::Put", "PutRevoke")
+
+// === Client::Put 内 TransferData ===
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_FULL, "client_service.cpp::TransferData", "TransferData[W]")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_SUBMIT, "client_service.cpp::TransferData", "Submit[W]")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_WAIT, "client_service.cpp::TransferData", "Wait[W]")
+
+// === Client::BatchPut (批量) ===
+PERF_KEY_DEF(PUT_BATCH_FULL, "client_service.cpp::BatchPut", "TransferBatchPut")
+PERF_KEY_DEF(PUT_BATCH_CREATE_OPS, "client_service.cpp::BatchPut", "CreateOps")
+PERF_KEY_DEF(PUT_BATCH_PUT_START, "client_service.cpp::StartBatchPut", "PutStart")
+PERF_KEY_DEF(PUT_BATCH_SUBMIT, "client_service.cpp::SubmitTransfers", "Submit")
+PERF_KEY_DEF(PUT_BATCH_DISK_WRITE, "client_service.cpp::SubmitTransfers", "DiskWrite")
+PERF_KEY_DEF(PUT_BATCH_WAIT, "client_service.cpp::WaitForTransfers", "Wait")
+PERF_KEY_DEF(PUT_BATCH_PUT_END, "client_service.cpp::FinalizeBatchPut", "PutEnd")
+PERF_KEY_DEF(PUT_BATCH_PUT_REVOKE, "client_service.cpp::FinalizeBatchPut", "PutRevoke")
+PERF_KEY_DEF(PUT_BATCH_COLLECT_RESULTS, "client_service.cpp::BatchPut", "CollectResults")
+
+// === batch_get_into_offload_object_internal 子步骤 (SSD read) ===
+PERF_KEY_DEF(GET_SSD_OFFLOAD_RPC, "real_client.cpp::batch_get_into_offload_object_internal", "OffloadRpc")
+PERF_KEY_DEF(GET_SSD_OFFLOAD_RPC_POOL, "real_client.cpp::ClientRequester::invoke_rpc", "RpcPoolLookup")
+PERF_KEY_DEF(GET_SSD_OFFLOAD_RPC_CALL, "real_client.cpp::ClientRequester::invoke_rpc", "RpcCall")
+PERF_KEY_DEF(GET_SSD_OFFLOAD_RPC_RESULT_GET, "real_client.cpp::ClientRequester::invoke_rpc", "RpcResultGet")
+PERF_KEY_DEF(GET_SSD_TRANSFER_DATA, "real_client.cpp::batch_get_into_offload_object_internal", "TransferData")
+PERF_KEY_DEF(GET_SSD_RELEASE_BUFFER, "real_client.cpp::batch_get_into_offload_object_internal", "ReleaseBuffer")
+
+// === Owner-side offload sub-steps (pull & push) ===
+PERF_KEY_DEF(GET_SSD_OWNER_READ, "file_storage.cpp::BatchGet", "OwnerSsdRead")
+PERF_KEY_DEF(GET_SSD_OWNER_ALLOC, "file_storage.cpp::BatchGet", "OwnerAllocBuffer")
+PERF_KEY_DEF(GET_SSD_OWNER_LOAD, "file_storage.cpp::BatchGet", "OwnerDiskLoad")
+PERF_KEY_DEF(GET_SSD_OWNER_LOAD_PLAN, "storage_backend.cpp::BatchLoad", "OwnerLoadPlan")
+PERF_KEY_DEF(GET_SSD_OWNER_LOAD_URING, "storage_backend.cpp::BatchLoad", "OwnerLoadUring")
+PERF_KEY_DEF(GET_SSD_OWNER_LOAD_POSIX, "storage_backend.cpp::BatchLoad", "OwnerLoadPosix")
+PERF_KEY_DEF(GET_SSD_OWNER_PUSH_WRITE, "real_client.cpp::batch_get_offload_object_push", "OwnerPushWrite")
+PERF_KEY_DEF(GET_SSD_OWNER_RPC_QUEUE, "real_client.cpp::batch_get_offload_object", "OwnerRpcQueue")
+PERF_KEY_DEF(GET_SSD_OWNER_RPC_BATCH_GET, "real_client.cpp::batch_get_offload_object", "OwnerRpcBatchGet")
+PERF_KEY_DEF(GET_SSD_OWNER_RPC_RESUME, "real_client.cpp::batch_get_offload_object", "OwnerRpcResume")
+PERF_KEY_DEF(GET_SSD_OWNER_RELEASE, "file_storage.cpp::ReleaseBuffer", "OwnerReleaseBuffer")
+
+// === UB / URMA endpoint first connection path ===
+PERF_KEY_DEF(UB_HANDSHAKE_ENCODE, "transfer_metadata.cpp::TransferHandshakeUtil::encode", "Encode")
+PERF_KEY_DEF(UB_HANDSHAKE_DECODE, "transfer_metadata.cpp::TransferHandshakeUtil::decode", "Decode")
+PERF_KEY_DEF(UB_ENDPOINT_CONSTRUCT, "urma_endpoint.cpp::construct", "Construct")
+PERF_KEY_DEF(UB_ENDPOINT_CREATE_JETTY, "urma_endpoint.cpp::construct", "CreateJetty")
+PERF_KEY_DEF(UB_ENDPOINT_ACTIVE_SETUP, "urma_endpoint.cpp::setupConnectionsByActive", "ActiveSetup")
+PERF_KEY_DEF(UB_ENDPOINT_ACTIVE_HANDSHAKE, "urma_endpoint.cpp::setupConnectionsByActive", "SendHandshake")
+PERF_KEY_DEF(UB_ENDPOINT_PASSIVE_SETUP, "urma_endpoint.cpp::setupConnectionsByPassive", "PassiveSetup")
+PERF_KEY_DEF(UB_ENDPOINT_DO_SETUP_ALL, "urma_endpoint.cpp::doSetupConnection", "DoSetupAll")
+PERF_KEY_DEF(UB_ENDPOINT_IMPORT_JETTY, "urma_endpoint.cpp::doSetupConnection", "ImportJetty")
+PERF_KEY_DEF(UB_ENDPOINT_BIND_JETTY, "urma_endpoint.cpp::doSetupConnection", "BindJetty")
+
+// ============================================================
+// === Master 服务端打点（程序名 mooncake_master） ===
+// ============================================================
+
+// --- P0: RPC 热路径（单 key，走 execute_rpc）---
+PERF_KEY_DEF(MASTER_RPC_PUT_START, "rpc_service.cpp::PutStart", "PutStart")
+PERF_KEY_DEF(MASTER_RPC_PUT_END, "rpc_service.cpp::PutEnd", "PutEnd")
+PERF_KEY_DEF(MASTER_RPC_PUT_REVOKE, "rpc_service.cpp::PutRevoke", "PutRevoke")
+PERF_KEY_DEF(MASTER_RPC_GET_REPLICA_LIST, "rpc_service.cpp::GetReplicaList", "GetReplicaList")
+PERF_KEY_DEF(MASTER_RPC_EXIST_KEY, "rpc_service.cpp::ExistKey", "ExistKey")
+
+// --- P0: Batch get/put（手动插点，不走 execute_rpc）---
+PERF_KEY_DEF(MASTER_RPC_BATCH_PUT_START, "rpc_service.cpp::BatchPutStart", "BatchPutStart")
+PERF_KEY_DEF(MASTER_RPC_BATCH_PUT_END, "rpc_service.cpp::BatchPutEnd", "BatchPutEnd")
+PERF_KEY_DEF(MASTER_RPC_BATCH_PUT_REVOKE, "rpc_service.cpp::BatchPutRevoke", "BatchPutRevoke")
+PERF_KEY_DEF(MASTER_RPC_BATCH_GET_REPLICA,"rpc_service.cpp::BatchGetReplicaList","BatchGetReplicaList")
+
+// --- P1: 次热路径（走 execute_rpc）---
+PERF_KEY_DEF(MASTER_RPC_UPSERT_START, "rpc_service.cpp::UpsertStart", "UpsertStart")
+PERF_KEY_DEF(MASTER_RPC_UPSERT_END, "rpc_service.cpp::UpsertEnd", "UpsertEnd")
+PERF_KEY_DEF(MASTER_RPC_UPSERT_REVOKE, "rpc_service.cpp::UpsertRevoke", "UpsertRevoke")
+PERF_KEY_DEF(MASTER_RPC_REMOVE, "rpc_service.cpp::Remove", "Remove")
+PERF_KEY_DEF(MASTER_RPC_REMOVE_BY_REGEX, "rpc_service.cpp::RemoveByRegex", "RemoveByRegex")
+PERF_KEY_DEF(MASTER_RPC_MOUNT_SEGMENT, "rpc_service.cpp::MountSegment", "MountSegment")
+PERF_KEY_DEF(MASTER_RPC_UNMOUNT_SEGMENT, "rpc_service.cpp::UnmountSegment", "UnmountSegment")
+PERF_KEY_DEF(MASTER_RPC_COPY_START, "rpc_service.cpp::CopyStart", "CopyStart")
+PERF_KEY_DEF(MASTER_RPC_COPY_END, "rpc_service.cpp::CopyEnd", "CopyEnd")
+PERF_KEY_DEF(MASTER_RPC_COPY_REVOKE, "rpc_service.cpp::CopyRevoke", "CopyRevoke")
+PERF_KEY_DEF(MASTER_RPC_PING, "rpc_service.cpp::Ping", "Ping")
+
+// --- RPC 内部子步骤 ---
+PERF_KEY_DEF(MASTER_PUT_ALLOCATE_MEM, "master_service.cpp::AllocateAndInsertMetadata", "AllocateMemory")
+PERF_KEY_DEF(MASTER_PUT_ALLOCATE_NOF, "master_service.cpp::AllocateAndInsertMetadata", "AllocateNoF")
+PERF_KEY_DEF(MASTER_PUT_SHARD_LOCK, "master_service.cpp::PutStart", "AcquireShardLock")
+PERF_KEY_DEF(MASTER_SNAPSHOT_LOCK, "master_service.cpp::SnapshotThreadFunc", "AcquireSnapshotLock")
+
+// --- 后台线程 ---
+PERF_KEY_DEF(MASTER_BG_BATCH_EVICT, "master_service.cpp::BatchEvict", "BatchEvict")
+PERF_KEY_DEF(MASTER_BG_NOF_BATCH_EVICT, "master_service.cpp::NoFBatchEvict", "NoFBatchEvict")
+PERF_KEY_DEF(MASTER_BG_DISCARD_EXPIRED, "master_service.cpp::EvictionThreadFunc", "DiscardExpired")
+PERF_KEY_DEF(MASTER_BG_SNAPSHOT_PERSIST, "master_service.cpp::SnapshotThreadFunc", "SnapshotPersist")
+PERF_KEY_DEF(MASTER_BG_CLIENT_MONITOR, "master_service.cpp::ClientMonitorFunc", "ClientMonitorScan")
+PERF_KEY_DEF(MASTER_BG_CLIENT_UNMOUNT, "master_service.cpp::ClientMonitorFunc", "ExpiredClientUnmount")
+
+// ============================================================
+// === vLLM MooncakeStoreConnector 路径打点（vllm 0.26.1rc0）===
+// === 覆盖 S2-S9 入口 + T1-T8 下沉 + Client 服务层 ===
+// ============================================================
+
+// === store_py.cpp Python 绑定层（vllm 直接调用入口，S2-S9）===
+PERF_KEY_DEF(STORE_PY_SETUP, "store_py.cpp::setup", "Setup")
+PERF_KEY_DEF(STORE_PY_REGISTER_BUFFER, "store_py.cpp::register_buffer", "RegisterBuffer")
+PERF_KEY_DEF(STORE_PY_BATCH_PUT_MULTI, "store_py.cpp::batch_put_from_multi_buffers","BatchPutMultiBuf")
+PERF_KEY_DEF(STORE_PY_BATCH_GET_INTO_MULTI, "store_py.cpp::batch_get_into_multi_buffers","BatchGetIntoMultiBuf")
+PERF_KEY_DEF(STORE_PY_BATCH_IS_EXIST, "store_py.cpp::batch_is_exist", "BatchIsExist")
+PERF_KEY_DEF(STORE_PY_BATCH_GET_REPLICA_DESC, "store_py.cpp::batch_get_replica_desc", "BatchGetReplicaDesc")
+PERF_KEY_DEF(STORE_PY_REMOVE_ALL, "store_py.cpp::remove_all", "RemoveAll")
+PERF_KEY_DEF(STORE_PY_CLOSE, "store_py.cpp::close", "Close")
+
+// === RealClient 核心逻辑层（下沉路径，T1-T8）===
+PERF_KEY_DEF(RC_SETUP_REAL, "real_client.cpp::setup_real", "SetupReal")
+PERF_KEY_DEF(RC_SETUP_INTERNAL, "real_client.cpp::setup_internal", "SetupInternal")
+PERF_KEY_DEF(RC_TEARDOWN_ALL, "real_client.cpp::tearDownAll", "TeardownAll")
+PERF_KEY_DEF(RC_TEARDOWN_ALL_INTERNAL, "real_client.cpp::tearDownAll_internal", "TeardownAllInternal")
+PERF_KEY_DEF(RC_REMOVE_ALL, "real_client.cpp::removeAll", "RemoveAll")
+PERF_KEY_DEF(RC_REMOVE_ALL_INTERNAL, "real_client.cpp::removeAll_internal", "RemoveAllInternal")
+PERF_KEY_DEF(RC_BATCH_IS_EXIST, "real_client.cpp::batchIsExist", "BatchIsExist")
+PERF_KEY_DEF(RC_BATCH_IS_EXIST_INTERNAL, "real_client.cpp::batchIsExist_internal", "BatchIsExistInternal")
+PERF_KEY_DEF(RC_REGISTER_BUFFER, "real_client.cpp::register_buffer", "RegisterBuffer")
+PERF_KEY_DEF(RC_REGISTER_BUFFER_INTERNAL, "real_client.cpp::register_buffer_internal","RegisterBufferInternal")
+PERF_KEY_DEF(RC_BATCH_PUT_MULTI, "real_client.cpp::batch_put_from_multi_buffers", "BatchPutMultiBuf")
+PERF_KEY_DEF(RC_BATCH_PUT_MULTI_INTERNAL, "real_client.cpp::batch_put_from_multi_buffers_internal","BatchPutMultiBufInternal")
+PERF_KEY_DEF(RC_BATCH_GET_INTO_MULTI, "real_client.cpp::batch_get_into_multi_buffers", "BatchGetIntoMultiBuf")
+PERF_KEY_DEF(RC_BATCH_GET_INTO_MULTI_INTERNAL, "real_client.cpp::batch_get_into_multi_buffers_internal","BatchGetIntoMultiBufInternal")
+PERF_KEY_DEF(RC_BATCH_GET_REPLICA_DESC, "real_client.cpp::batch_get_replica_desc", "BatchGetReplicaDesc")
+
+// === Client 服务层（vllm 路径下沉，部分已有打点）===
+PERF_KEY_DEF(CLIENT_BATCH_QUERY, "client_service.cpp::BatchQuery", "BatchQuery")
diff --git a/mooncake-integration/store/store_py.cpp b/mooncake-integration/store/store_py.cpp
index 0ee3a30546..82f8e7f5d2 100644
--- a/mooncake-integration/store/store_py.cpp
+++ b/mooncake-integration/store/store_py.cpp
@@ -2,6 +2,7 @@
#include
#include
+#include
#include
#include
#include
@@ -15,6 +16,7 @@
#include "memory_alloc.h"
#include "ssd_register_client.h"
#include "device/accelerator_registry.h"
+#include "mooncake_logging.h" // MC_LOG
#include // for atexit
#include
@@ -22,6 +24,10 @@
#include "integration_utils.h"
#include "buffer_pool.h"
+#define SPDIAG_PERF_DEF_FILE "mooncake_perf_points.def"
+#define SPDIAG_PROGRAM_NAME "mooncake_store"
+#include "spdiag/auto_perf.h"
+
// Forward declaration for EngramStore bindings
namespace mooncake {
namespace engram {
@@ -470,8 +476,12 @@ class MooncakeStorePyWrapper {
}
pybind11::bytes get(const std::string &key) {
+ SpDiag::PerfPoint pt(PerfKey::GET_STORE_PY_GET,
+ SpDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
if (!is_client_initialized()) {
LOG(ERROR) << "Client is not initialized";
+ pt.End(-1);
return pybind11::bytes("\\0", 0);
}
@@ -479,13 +489,19 @@ class MooncakeStorePyWrapper {
{
py::gil_scoped_release release_gil;
+ SpDiag::PerfPoint pt_full(PerfKey::GET_BUFFER_INTERNAL,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
auto buffer_handle = store_->get_buffer(key);
+ pt_full.End(buffer_handle ? 0 : -1);
if (!buffer_handle) {
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
return kNullString;
}
py::gil_scoped_acquire acquire_gil;
+ pt.End(0);
auto runtime_accelerator =
mooncake::device::GetAcceleratorRegistry()
.RuntimeAccelerators();
@@ -507,18 +523,46 @@ class MooncakeStorePyWrapper {
std::vector get_batch(
const std::vector &keys) {
+ auto start = std::chrono::steady_clock::now();
+ LOG(INFO) << "get_batch start num_keys[" << keys.size() << "]";
+
+ SpDiag::PerfPoint pt(PerfKey::GET_STORE_PY_GET_BATCH,
+ SpDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
const auto kNullString = pybind11::bytes("\\0", 0);
if (!is_client_initialized()) {
LOG(ERROR) << "Client is not initialized";
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
+ auto elapsed_us =
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start)
+ .count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size()
+ << "] rc[-1] elapsed_us[" << elapsed_us << "]";
return {kNullString};
}
{
py::gil_scoped_release release_gil;
+ SpDiag::PerfPoint pt_full(PerfKey::GET_BATCH_BUFFER_INTERNAL,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
auto batch_data = store_->batch_get_buffer(keys);
+ pt_full.End(0);
if (batch_data.empty()) {
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
+ auto elapsed_us =
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start)
+ .count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size()
+ << "] rc[-1] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "get_batch_slow num_keys[" << keys.size()
+ << "] elapsed_us[" << elapsed_us << "]";
+ }
return {kNullString};
}
@@ -526,11 +570,13 @@ class MooncakeStorePyWrapper {
std::vector results;
results.reserve(batch_data.size());
+ size_t success_count = 0;
auto runtime_accelerator =
mooncake::device::GetAcceleratorRegistry()
.RuntimeAccelerators();
for (const auto &data : batch_data) {
+ if (data) success_count++;
if (!data) {
results.emplace_back(kNullString);
continue;
@@ -549,6 +595,18 @@ class MooncakeStorePyWrapper {
pybind11::bytes((char *)data->ptr(), data->size()));
}
}
+ pt.End(0);
+ auto elapsed_us =
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start)
+ .count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size()
+ << "] success[" << success_count << "] rc[0] elapsed_us["
+ << elapsed_us << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "get_batch_slow num_keys[" << keys.size()
+ << "] elapsed_us[" << elapsed_us << "]";
+ }
return results;
}
}
@@ -2027,6 +2085,9 @@ PYBIND11_MODULE(store, m) {
const std::string &tenant_id = "default",
bool enable_client_http_server = false,
int client_http_port = DEFAULT_CLIENT_HTTP_PORT) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_SETUP,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
auto real_client = self.init_real_client();
std::shared_ptr transfer_engine =
nullptr;
@@ -2034,12 +2095,15 @@ PYBIND11_MODULE(store, m) {
transfer_engine =
engine.cast>();
}
- return real_client->setup_real(
+ auto ret = real_client->setup_real(
local_hostname, metadata_server, global_segment_size,
local_buffer_size, protocol, rdma_devices,
master_server_addr, transfer_engine, "", enable_ssd_offload,
ssd_offload_path, tenant_id, enable_client_http_server,
client_http_port);
+ pt.End(ret == 0 ? 0 : -1);
+ // MC_LOG 在下沉层 setup_real 输出（Q1b）
+ return ret;
},
py::arg("local_hostname"), py::arg("metadata_server"),
py::arg("global_segment_size"), py::arg("local_buffer_size"),
@@ -2052,6 +2116,9 @@ PYBIND11_MODULE(store, m) {
.def(
"setup",
[](MooncakeStorePyWrapper &self, const py::dict &config_dict) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_SETUP,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
auto real_client = self.init_real_client();
// Convert py::dict to ConfigDict (all values as strings)
@@ -2063,8 +2130,11 @@ PYBIND11_MODULE(store, m) {
}
auto result = real_client->setup_internal(config);
- return result.has_value() ? 0
- : static_cast(result.error());
+ int ret = result.has_value() ? 0
+ : static_cast(result.error());
+ pt.End(ret == 0 ? 0 : -1);
+ // MC_LOG 在下沉层 setup_real 输出（Q1b）
+ return ret;
},
py::arg("config"),
"Setup the store with a configuration dictionary.\n"
@@ -2173,8 +2243,16 @@ PYBIND11_MODULE(store, m) {
.def(
"remove_all",
[](MooncakeStorePyWrapper &self, bool force) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_REMOVE_ALL,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->removeAll(force);
+ auto ret = self.store_->removeAll(force);
+ pt.End(ret == 0 ? 0 : -1);
+ // 下沉 removeAll 不加 MC_LOG，入口层输出汇总（Q1b）
+ MC_LOG(INFO) << "[remove_all] elapsed_us=" << pt.ElapsedMicros()
+ << " success=" << (ret == 0 ? 1 : 0);
+ return ret;
},
py::arg("force") = false,
"Remove all objects from the store. If force=True, skip lease "
@@ -2198,17 +2276,36 @@ PYBIND11_MODULE(store, m) {
"batch_is_exist",
[](MooncakeStorePyWrapper &self,
const std::vector &keys) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_BATCH_IS_EXIST,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->batchIsExist(keys);
+ auto ret = self.store_->batchIsExist(keys);
+ pt.End(0);
+ // MC_LOG 在下沉层 batchIsExist 输出 per-key（Q1b）
+ return ret;
},
py::arg("keys"),
"Check if multiple objects exist. Returns list of results: 1 if "
"exists, 0 if not exists, -1 if error")
.def("close",
[](MooncakeStorePyWrapper &self) {
- if (!self.store_) return 0;
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_CLOSE,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
+ if (!self.store_) {
+ pt.End(0);
+ // 无下沉 MC_LOG，入口层输出汇总（Q1b）
+ MC_LOG(INFO) << "[close] elapsed_us=" << pt.ElapsedMicros()
+ << " success=1";
+ return 0;
+ }
int rc = self.store_->tearDownAll();
self.store_.reset();
+ pt.End(rc == 0 ? 0 : -1);
+ // 下沉 tearDownAll 不加 MC_LOG，入口层输出汇总（Q1b）
+ MC_LOG(INFO) << "[close] elapsed_us=" << pt.ElapsedMicros()
+ << " success=" << (rc == 0 ? 1 : 0);
return rc;
})
.def("health_check", &MooncakeStorePyWrapper::health_check,
@@ -2592,10 +2689,16 @@ PYBIND11_MODULE(store, m) {
"register_buffer",
[](MooncakeStorePyWrapper &self, uintptr_t buffer_ptr,
size_t size) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_REGISTER_BUFFER,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
// Register memory buffer for RDMA operations
void *buffer = reinterpret_cast(buffer_ptr);
py::gil_scoped_release release;
- return self.store_->register_buffer(buffer, size);
+ auto ret = self.store_->register_buffer(buffer, size);
+ pt.End(ret == 0 ? 0 : -1);
+ // MC_LOG 在下沉层 register_buffer 输出汇总（Q1b）
+ return ret;
},
py::arg("buffer_ptr"), py::arg("size"),
"Register a memory buffer for direct access operations")
@@ -2729,12 +2832,23 @@ PYBIND11_MODULE(store, m) {
py::buffer buf,
const ReplicateConfig &config = ReplicateConfig{}) {
py::buffer_info info = buf.request(/*writable=*/false);
+
+ SpDiag::PerfPoint pt(PerfKey::PUT_STORE_PY_PUT,
+ SpDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->put(
+ SpDiag::PerfPoint pt_full(PerfKey::PUT_INTERNAL_FULL,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
+ auto ret = self.store_->put(
key,
std::span(static_cast(info.ptr),
static_cast(info.size)),
config);
+ pt_full.End(ret == 0 ? 0 : -1);
+ pt.End(ret == 0 ? 0 : -1);
+
+ return ret;
},
py::arg("key"), py::arg("value"),
py::arg("config") = ReplicateConfig{})
@@ -2772,21 +2886,49 @@ PYBIND11_MODULE(store, m) {
const std::vector &keys,
const std::vector &buffers,
const ReplicateConfig &config = ReplicateConfig{}) {
+ auto start = std::chrono::steady_clock::now();
+
+ SpDiag::PerfPoint pt(PerfKey::PUT_STORE_PY_PUT_BATCH,
+ SpDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
// Convert pybuffers to spans without copying
std::vector infos;
std::vector> spans;
infos.reserve(buffers.size());
spans.reserve(buffers.size());
+ size_t total_size = 0;
for (const auto &buf : buffers) {
infos.emplace_back(buf.request(/*writable=*/false));
const auto &info = infos.back();
+ total_size += static_cast(info.size);
spans.emplace_back(static_cast(info.ptr),
static_cast(info.size));
}
+ LOG(INFO) << "put_batch start num_keys[" << keys.size()
+ << "] total_size[" << total_size << "]";
+
py::gil_scoped_release release;
- return self.store_->put_batch(keys, spans, config);
+ SpDiag::PerfPoint pt_full(PerfKey::PUT_BATCH_INTERNAL_FULL,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
+ auto ret = self.store_->put_batch(keys, spans, config);
+ pt_full.End(ret == 0 ? 0 : -1);
+ pt.End(ret == 0 ? 0 : -1);
+
+ auto elapsed_us =
+ std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start)
+ .count();
+ LOG(INFO) << "put_batch complete num_keys[" << keys.size()
+ << "] rc[" << ret << "] elapsed_us[" << elapsed_us
+ << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "put_batch_slow num_keys[" << keys.size()
+ << "] elapsed_us[" << elapsed_us << "]";
+ }
+ return ret;
},
py::arg("keys"), py::arg("values"),
py::arg("config") = ReplicateConfig{})
@@ -2801,13 +2943,20 @@ PYBIND11_MODULE(store, m) {
const std::vector> &all_buffer_ptrs,
const std::vector> &all_sizes,
const ReplicateConfig &config = ReplicateConfig{}) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_BATCH_PUT_MULTI,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
if (!self.is_client_initialized()) {
LOG(ERROR) << "Client is not initialized";
+ pt.End(-1);
return std::vector{};
}
py::gil_scoped_release release;
- return self.store_->batch_put_from_multi_buffers(
+ auto ret = self.store_->batch_put_from_multi_buffers(
keys, CastAddrs2Ptrs(all_buffer_ptrs), all_sizes, config);
+ pt.End(ret.empty() ? -1 : 0);
+ // MC_LOG 在下沉层 *_internal 输出 汇总+per-key（Q1b）
+ return ret;
},
py::arg("keys"), py::arg("all_buffer_ptrs"), py::arg("all_sizes"),
py::arg("config") = ReplicateConfig{},
@@ -2821,10 +2970,16 @@ PYBIND11_MODULE(store, m) {
const std::vector> &all_buffer_ptrs,
const std::vector> &all_sizes,
bool prefer_alloc_in_same_node = false) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_BATCH_GET_INTO_MULTI,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->batch_get_into_multi_buffers(
+ auto ret = self.store_->batch_get_into_multi_buffers(
keys, CastAddrs2Ptrs(all_buffer_ptrs), all_sizes,
prefer_alloc_in_same_node);
+ pt.End(ret.empty() ? -1 : 0);
+ // MC_LOG 在下沉层 *_internal 输出 汇总+per-key（Q1b）
+ return ret;
},
py::arg("keys"), py::arg("all_buffer_ptrs"), py::arg("all_sizes"),
py::arg("prefer_alloc_in_same_node") = false,
@@ -2842,8 +2997,14 @@ PYBIND11_MODULE(store, m) {
"batch_get_replica_desc",
[](MooncakeStorePyWrapper &self,
const std::vector &keys) {
+ SpDiag::PerfPoint pt(PerfKey::STORE_PY_BATCH_GET_REPLICA_DESC,
+ SpDiag::PerfLevel::KEY_MODULE);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->batch_get_replica_desc(keys);
+ auto ret = self.store_->batch_get_replica_desc(keys);
+ pt.End(0);
+ // MC_LOG 在下沉层 batch_get_replica_desc 输出 per-key（Q1b）
+ return ret;
},
py::arg("keys"))
.def(
diff --git a/mooncake-p2p-store/CMakeLists.txt b/mooncake-p2p-store/CMakeLists.txt
index 5b9980de90..013e69514b 100644
--- a/mooncake-p2p-store/CMakeLists.txt
+++ b/mooncake-p2p-store/CMakeLists.txt
@@ -1,8 +1,11 @@
# Currently you have to manually execute makefile in the src subdirectory.
add_custom_target(build_p2p_store DEPENDS transfer_engine)
add_custom_command(
- TARGET build_p2p_store
- COMMAND bash build.sh ${CMAKE_CURRENT_BINARY_DIR} ${USE_ETCD} ${USE_REDIS} ${USE_HTTP} ${USE_ETCD_LEGACY} ${CMAKE_BINARY_DIR}
- WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
-)
+ TARGET build_p2p_store
+ COMMAND
+ bash build.sh ${CMAKE_CURRENT_BINARY_DIR} ${USE_ETCD} ${USE_REDIS}
+ ${USE_HTTP} ${USE_ETCD_LEGACY} ${CMAKE_BINARY_DIR}
+ "${MOONCAKE_SPDIAG_ACTIVE_LAYER}" "${MOONCAKE_SPDIAG_LIBRARY_DIR}"
+ WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+ VERBATIM)
set_property(TARGET build_p2p_store PROPERTY EXCLUDE_FROM_ALL FALSE)
diff --git a/mooncake-p2p-store/build.sh b/mooncake-p2p-store/build.sh
index 774e6ba688..f631b8d4fc 100644
--- a/mooncake-p2p-store/build.sh
+++ b/mooncake-p2p-store/build.sh
@@ -13,8 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-if [ "$#" -ne 6 ]; then
- echo "Usage: $0 TARGET_PATH USE_ETCD USE_REDIS USE_HTTP USE_ETCD_LEGACY BUILD_DIR"
+if [ "$#" -ne 8 ]; then
+ echo "Usage: $0 TARGET_PATH USE_ETCD USE_REDIS USE_HTTP USE_ETCD_LEGACY BUILD_DIR SPDIAG_LAYER SPDIAG_LIB_DIR"
exit 1
fi
@@ -24,6 +24,8 @@ USE_REDIS=$3
USE_HTTP=$4
USE_ETCD_LEGACY=$5
BUILD_DIR=$6
+SPDIAG_LAYER=$7
+SPDIAG_LIB_DIR=$8
cd "src/p2pstore"
if [ $? -ne 0 ]; then
@@ -37,6 +39,10 @@ EXT_LDFLAGS+=" -L$BUILD_DIR/mooncake-common"
EXT_LDFLAGS+=" -L$BUILD_DIR/mooncake-common/src"
EXT_LDFLAGS+=" -ltransfer_engine -lbase -lasio -lstdc++ -lnuma -lglog -libverbs -lmlx5 -ljsoncpp -lmooncake_common -lm"
+if [ "$SPDIAG_LAYER" = "system" ]; then
+ EXT_LDFLAGS+=" -L$SPDIAG_LIB_DIR -lspdiag"
+fi
+
if [ -d "/usr/local/cuda/lib64/stubs" ]; then
EXT_LDFLAGS+=" -L/usr/local/cuda/lib64/stubs"
fi
diff --git a/mooncake-store/benchmarks/CMakeLists.txt b/mooncake-store/benchmarks/CMakeLists.txt
index 63af87d0f3..003f0b8724 100644
--- a/mooncake-store/benchmarks/CMakeLists.txt
+++ b/mooncake-store/benchmarks/CMakeLists.txt
@@ -33,6 +33,19 @@ target_link_libraries(
allocation_strategy_bench PRIVATE mooncake_store cachelib_memory_allocator
gflags::gflags glog::glog pthread)
+add_executable(stress_cluster_bench stress_cluster_bench.cpp)
+target_link_libraries(
+ stress_cluster_bench PRIVATE mooncake_store transfer_engine asio_shared
+ gflags::gflags glog::glog pthread)
+
+# Benchmark for RealClient::get_into_ranges with configurable value size,
+# fragments per key and keys per query.
+add_executable(stress_cluster_ranges_bench stress_cluster_ranges_bench.cpp)
+target_link_libraries(
+ stress_cluster_ranges_bench PRIVATE mooncake_store transfer_engine
+ asio_shared gflags::gflags glog::glog
+ pthread)
+
# Add NoF worker pool benchmark executable
if(USE_NOF)
add_executable(nof_worker_pool_bench nof_worker_pool_bench.cpp)
diff --git a/mooncake-store/benchmarks/cluster_mooncake_diag.py b/mooncake-store/benchmarks/cluster_mooncake_diag.py
new file mode 100644
index 0000000000..86ba16c0d6
--- /dev/null
+++ b/mooncake-store/benchmarks/cluster_mooncake_diag.py
@@ -0,0 +1,2403 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""Mooncake 集群诊断工具（单文件、单 HTML、无配置文件）
+
+================================================================================
+【使用方法】
+================================================================================
+
+1. 必改：编辑本文件的 CONFIG 区域（约第 106 行），只需填：
+ - k8s：K8S_NAMESPACE + K8S_LOG_PATH（自动发现该 namespace 下所有 pod）
+ - client 超节点：CLIENT_HOSTS 列表 + CLIENT_LOG_PATH
+ - master 超节点：MASTER_HOSTS 列表 + MASTER_LOG_PATH
+ - SSH 密码：SSH_PASSWORD（如有必要，使用 sshpass；免密则留空）
+ - SSH 用户/端口：SSH_USER / SSH_PORT（默认 root:22）
+
+ 示例：
+ K8S_NAMESPACE = "default"
+ K8S_LOG_PATH = "/var/log/mooncake"
+ CLIENT_HOSTS = ["10.0.1.5", "10.0.1.6"]
+ CLIENT_LOG_PATH = "/var/log/mooncake"
+ MASTER_HOSTS = ["10.0.1.1"]
+ MASTER_LOG_PATH = "/var/log/mooncake"
+ SSH_USER = "root"
+ SSH_PASSWORD = "" # 免密则留空
+ SPDIAG_BIN = "spdiag"
+
+2. 常用命令：
+
+ # 默认模式：拉日志 + spdiag show + 分析 → 单 HTML
+ python cluster_mooncake_diag.py --since 30min -o report.html
+
+ # 仅在所有目标上执行命令（如 spdiag start / clear），不分析
+ python cluster_mooncake_diag.py --exec "spdiag start"
+ python cluster_mooncake_diag.py --exec "spdiag clear"
+
+ # 只重新分析已收集的日志（不连远程）
+ python cluster_mooncake_diag.py --analyze-only --since 30min
+
+ # 生成示例 HTML（用伪造数据，不连任何目标，用于查看报告样式）
+ python cluster_mooncake_diag.py --sample -o sample.html
+
+3. 时间窗口（--since / --until 在解析阶段按行时间戳真实过滤）：
+ --since 30min # 最近 30 分钟
+ --since 2h # 最近 2 小时
+ --since "2026-08-05 10:00:00" # 绝对时间
+ --until "2026-08-05 11:00:00" # 绝对时间（--until 仅支持绝对）
+
+4. 典型工作流：
+ a. python cluster_mooncake_diag.py --exec "spdiag start" # 启动 spdiag 共享内存
+ b. 重启 Mooncake 进程（Windows 必须重启；Linux 上 spdiag 会自动激活）
+ c. 跑你的 benchmark / 业务负载
+ d. python cluster_mooncake_diag.py --since 30min -o report.html
+ e. python cluster_mooncake_diag.py --exec "spdiag stop" # 可选，停止 spdiag
+
+5. 输出 HTML 包含的区块：
+ - 概览卡片（targets / spdiag ok / log files / get requests / p50/p99/max / 带宽）
+ - [仅 --exec 模式] Command Execution Results
+ - spdiag show 概览 + top 30 慢点位表
+ - QPS by slot 折线图 + 每秒 QPS 表格
+ - Bandwidth by slot 折线图 + 每秒带宽表格（MB/s）
+ - get_into 耗时拆解（按 slot 分面板，6 阶段折线，鼠标框选缩放）
+ - 8 类操作 Summary Statistics 表（count/avg/p50/p95/p99/p999/p9999/min/max）
+ - Per-pod file stats（get 多少文件，#c# chunk vs 其他，字节数）
+ - 最慢 100 个请求跨角色关联表（trace_id 关联 real_client/RPC/storage/master）
+ - Per-target 详情（spdiag 原始输出 + 日志文件列表）
+
+6. 可调常量（文件顶部）：
+ CHART_MAX_POINTS = 6000 # get_into 图每 slot 最大均匀抽样点数
+ CHART_SLOW_POINTS = 200 # get_into 图强制保留的最慢点数
+ SLOW_TRACE_COUNT = 100 # 最慢请求表行数
+ DEFAULT_CHUNK_SIZE = 4194304 # chunk 字节数，默认 4MB
+
+7. 前置依赖：
+ - Python 3.10+
+ - kubectl（仅当 TARGETS 中有 k8s_pod 时）
+ - ssh / scp / rsync（仅当 TARGETS 中有 supernode 时）
+ - 目标主机上 spdiag 可执行文件已就位
+
+================================================================================
+"""
+
+from __future__ import annotations
+
+import argparse
+import html
+import json
+import math
+import re
+import shlex
+import subprocess
+import sys
+from collections import Counter, defaultdict
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from pathlib import Path
+from typing import Iterable
+
+
+# ===========================================================================
+# CONFIG - just set these, then run. No other configuration needed.
+# ===========================================================================
+
+# --- SSH settings ---
+SSH_USER = "root" # SSH login user for all supernode targets
+SSH_PORT = 22 # SSH port
+SSH_PASSWORD = "" # Leave empty to use SSH keys.
+ # If set, sshpass is used (install: apt-get install sshpass).
+
+# --- k8s: auto-discover all pods in this namespace ---
+K8S_NAMESPACE = "e2b" # e.g. "default". Empty = skip k8s.
+K8S_LOG_PATH = "/var/log/mooncake"
+
+# --- client supernodes (store/client reader nodes) ---
+CLIENT_HOSTS: list[str] = [
+ # "10.0.1.5",
+ # "10.0.1.6",
+]
+CLIENT_LOG_PATH = "/var/log/mooncake"
+
+# --- master supernodes (master service nodes) ---
+MASTER_HOSTS: list[str] = [
+ # "10.0.1.1",
+]
+MASTER_LOG_PATH = "/var/log/mooncake"
+
+# --- spdiag binary path on remote targets ---
+SPDIAG_BIN = "spdiag"
+
+# ===========================================================================
+
+
+# ---------------------------------------------------------------------------
+# Defaults tunable via CLI
+# ---------------------------------------------------------------------------
+DEFAULT_CHUNK_SIZE = 4 * 1024 * 1024
+DEFAULT_CHUNK_MARKER = "#c#"
+DEFAULT_PATH_SEPARATOR = "/chunk/"
+DEFAULT_CHUNK_STYLE = "auto"
+CHART_MAX_POINTS = 6000
+CHART_SLOW_POINTS = 200
+SLOW_TRACE_COUNT = 100
+TOP_TEMPLATE_TABLE_N = 30
+
+# Operations recognized in breakdown logs
+OPS = (
+ "get_into_breakdown",
+ "batch_get_into_breakdown",
+ "put_into_breakdown",
+ "batch_put_into_breakdown",
+ "get_breakdown",
+ "batch_get_breakdown",
+ "put_breakdown",
+ "batch_put_breakdown",
+)
+OP_LABEL = {
+ "get_into_breakdown": "get_into",
+ "batch_get_into_breakdown": "batch_get_into",
+ "put_into_breakdown": "put_into",
+ "batch_put_into_breakdown": "batch_put_into",
+ "get_breakdown": "get",
+ "batch_get_breakdown": "batch_get",
+ "put_breakdown": "put",
+ "batch_put_breakdown": "batch_put",
+}
+
+# get_into stages for stacked area chart (from bench script)
+GET_INTO_STAGES = (
+ "query_us",
+ "select_us",
+ "offload_rpc_us",
+ "transfer_data_us",
+ "release_buffer_us",
+ "read_overhead_us",
+)
+GET_INTO_STAGE_LABEL = {
+ "query_us": "query",
+ "select_us": "select",
+ "offload_rpc_us": "offload RPC",
+ "transfer_data_us": "transfer",
+ "release_buffer_us": "release",
+ "read_overhead_us": "overhead",
+}
+
+# bench-style cross-role events
+BENCH_EVENTS = {
+ "get_into_breakdown",
+ "offload_rpc_client_breakdown",
+ "offload_rpc_server_breakdown",
+ "storage_read_breakdown",
+ "storage_release_breakdown",
+ "master_rpc_client_breakdown",
+}
+BENCH_STAGES = (
+ "query_us", "select_us", "read_us", "offload_rpc_us",
+ "transfer_data_us", "release_buffer_us", "read_overhead_us", "total_us",
+)
+RPC_STAGES = ("pool_lookup_us", "rpc_call_us", "result_get_us", "result_parse_us")
+
+# regexes
+GLOG_TS_RE = re.compile(
+ r"^[IWEF](\d{4})(\d{2})(\d{2}) (\d{2}:\d{2}:\d{2})\.(\d{6})"
+)
+ISO_TS_RE = re.compile(
+ r"(\d{4}-\d{2}-\d{2})[ T](\d{2}:\d{2}:\d{2})(?:\.(\d+))?"
+)
+BRACKET_RE = re.compile(r"([A-Za-z_][\w]*)\[([^\]]*)\]")
+FIELD_RE = re.compile(r"([A-Za-z_][\w]*)=([^\s]+)")
+TRACE_RE = re.compile(r"trace_id\[(\d+)\]")
+BENCH_LOG_RE = re.compile(
+ r"^[IWEF](?P\d{8}) (?P\d\d:\d\d:\d\d\.\d+).*?"
+ r"trace_id\[(?P\d+)\] (?P[a-z_]+) (?P.*)$"
+)
+MASTER_SERVER_RE = re.compile(
+ r"^[IWEF](?P\d{8}) (?P\d\d:\d\d:\d\d\.\d+).*?"
+ r"trace_id\[(?P\d+)\] GetReplicaList (?P.*)$"
+)
+OP_LINE_RE = re.compile(r"\b([a-z_]+breakdown)\b")
+KV_FIELD_RE = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]*)=([^\s]+)")
+SLOT_RE = re.compile(r"(?:^|[\\/])slot(\d+)_")
+
+# spdiag show row parser
+SPDIAG_ROW_RE = re.compile(
+ r"^\s*(?P\d+)\s+(?P\S+)\s+(?P\S+)\s+"
+ r"(?P\S+)\s+(?P\d+)\s+(?P\d+)\s+(?P\d+)\s+"
+ r"(?P\d+)\s+(?P\d+)\s+(?P\d+)\s+(?P\d+)\s+"
+ r"(?P\d+)\s+(?P\d+)"
+ r"(?:\s+(?P\d+|N/A))?"
+ r"(?:\s+(?P\d+|N/A))?"
+ r"(?:\s+(?P\d+|N/A))?"
+)
+
+
+# ---------------------------------------------------------------------------
+# Target abstraction
+# ---------------------------------------------------------------------------
+@dataclass
+class Target:
+ name: str
+ kind: str # "k8s_pod" or "supernode"
+ namespace: str = ""
+ pod: str = ""
+ container: str = ""
+ host: str = ""
+ log_path: str = ""
+ spdiag_bin: str = "spdiag"
+
+ # ----- ssh helpers (as private class vars, set once) -----
+ _ssh_port: int = 22
+ _ssh_user: str = "root"
+ _ssh_password: str = ""
+
+ def _ssh_base(self) -> list[str]:
+ """Build the ssh/sshpass command prefix."""
+ if self._ssh_password:
+ args = ["sshpass", "-p", self._ssh_password, "ssh"]
+ else:
+ args = ["ssh"]
+ args += ["-p", str(self._ssh_port)]
+ args += ["-o", "StrictHostKeyChecking=no",
+ "-o", "UserKnownHostsFile=/dev/null",
+ "-o", "ConnectTimeout=10",
+ "-o", "BatchMode=yes"]
+ return args
+
+ def _ssh_target(self) -> str:
+ return f"{self._ssh_user}@{self.host}" if self._ssh_user else self.host
+
+ def exec_command(self, command: str, timeout: int = 60) -> tuple[int, str]:
+ if self.kind == "k8s_pod":
+ args = ["kubectl", "-n", self.namespace, "exec", self.pod]
+ if self.container:
+ args += ["-c", self.container]
+ args += ["--", "sh", "-c", command]
+ else:
+ args = self._ssh_base() + [self._ssh_target(), command]
+ try:
+ proc = subprocess.run(
+ args, capture_output=True, text=True,
+ timeout=timeout, check=False,
+ )
+ return proc.returncode, proc.stdout + proc.stderr
+ except FileNotFoundError as exc:
+ return 127, f"command not found: {exc}"
+ except subprocess.TimeoutExpired:
+ return 124, f"timeout after {timeout}s"
+ except Exception as exc:
+ return 1, f"exec error: {exc}"
+
+ def pull_logs(self, dest_dir: Path) -> list[Path]:
+ dest_dir.mkdir(parents=True, exist_ok=True)
+ target_dir = dest_dir / self.name
+ target_dir.mkdir(parents=True, exist_ok=True)
+ if not self.log_path:
+ return []
+ if self.kind == "k8s_pod":
+ return self._pull_k8s(target_dir)
+ return self._pull_ssh(target_dir)
+
+ def _pull_k8s(self, target_dir: Path) -> list[Path]:
+ cp_args = ["kubectl", "-n", self.namespace, "cp"]
+ if self.container:
+ cp_args += ["-c", self.container]
+ cp_args += [f"{self.pod}:{self.log_path}", str(target_dir / "logs")]
+ subprocess.run(cp_args, capture_output=True, text=True, check=False)
+ files = list(_walk_files(target_dir / "logs"))
+ if not files:
+ # cat fallback for single file
+ cat_args = ["kubectl", "-n", self.namespace, "exec", self.pod]
+ if self.container:
+ cat_args += ["-c", self.container]
+ cat_args += ["--", "cat", self.log_path]
+ rc = subprocess.run(cat_args, capture_output=True, text=True, check=False)
+ if rc.returncode == 0:
+ f = target_dir / "pod.log"
+ f.write_text(rc.stdout, encoding="utf-8", errors="replace")
+ files = [f]
+ return files
+
+ def _pull_ssh(self, target_dir: Path) -> list[Path]:
+ target = self._ssh_target()
+ ssh_args = self._ssh_base() + [target]
+ detect = ssh_args.copy()
+ if self._ssh_password:
+ # sshpass takes the command as a single arg after the ssh flags
+ detect = self._ssh_base() + [target, f"test -d {shlex.quote(self.log_path)} && echo DIR || echo FILE"]
+ else:
+ detect = self._ssh_base() + [target, f"test -d {shlex.quote(self.log_path)} && echo DIR || echo FILE"]
+ rc = subprocess.run(detect, capture_output=True, text=True, check=False)
+ is_dir = "DIR" in rc.stdout
+ # Build rsync or scp command
+ if is_dir:
+ ssh_cmd_str = " ".join(self._ssh_base())
+ rsync_args = ["rsync", "-az", "-e", ssh_cmd_str,
+ f"{target}:{self.log_path}/", str(target_dir / "logs")]
+ rc2 = subprocess.run(rsync_args, capture_output=True, text=True, check=False)
+ if rc2.returncode != 0:
+ scp_args = ["scp", "-r"] + self._ssh_base()[1:] + \
+ [f"{target}:{self.log_path}", str(target_dir / "logs")]
+ subprocess.run(scp_args, capture_output=True, text=True, check=False)
+ else:
+ scp_args = ["scp"] + self._ssh_base()[1:] + \
+ [f"{target}:{self.log_path}", str(target_dir / "remote.log")]
+ subprocess.run(scp_args, capture_output=True, text=True, check=False)
+ return list(_walk_files(target_dir))
+
+
+def _walk_files(root: Path) -> Iterable[Path]:
+ if not root.exists():
+ return []
+ if root.is_file():
+ return [root]
+ return [p for p in root.rglob("*") if p.is_file()]
+
+
+def discover_k8s_pods(namespace: str) -> list[str]:
+ """List pod names in the given k8s namespace via kubectl."""
+ try:
+ proc = subprocess.run(
+ ["kubectl", "-n", namespace, "get", "pods", "-o", "name"],
+ capture_output=True, text=True, timeout=30, check=False,
+ )
+ except FileNotFoundError:
+ print("error: kubectl not found", file=sys.stderr)
+ return []
+ if proc.returncode != 0:
+ print(f"kubectl get pods failed: {proc.stderr.strip()}", file=sys.stderr)
+ return []
+ pods = [line.strip().removeprefix("pod/") for line in proc.stdout.splitlines() if line.strip()]
+ return pods
+
+
+def build_targets() -> list[Target]:
+ """Build Target list from CONFIG globals. Auto-discovers k8s pods."""
+ targets: list[Target] = []
+
+ # --- k8s pods ---
+ if K8S_NAMESPACE:
+ pods = discover_k8s_pods(K8S_NAMESPACE)
+ print(f"[k8s] {K8S_NAMESPACE}: found {len(pods)} pods")
+ for pod_name in pods:
+ targets.append(Target(
+ name=pod_name,
+ kind="k8s_pod",
+ namespace=K8S_NAMESPACE,
+ pod=pod_name,
+ log_path=K8S_LOG_PATH,
+ spdiag_bin=SPDIAG_BIN,
+ ))
+ else:
+ print("[k8s] namespace not set, skipping")
+
+ # --- client supernodes ---
+ for host in CLIENT_HOSTS:
+ targets.append(Target(
+ name=f"client-{host}",
+ kind="supernode",
+ host=host,
+ log_path=CLIENT_LOG_PATH,
+ spdiag_bin=SPDIAG_BIN,
+ ))
+
+ # --- master supernodes ---
+ for host in MASTER_HOSTS:
+ targets.append(Target(
+ name=f"master-{host}",
+ kind="supernode",
+ host=host,
+ log_path=MASTER_LOG_PATH,
+ spdiag_bin=SPDIAG_BIN,
+ ))
+
+ # Apply SSH settings to all supernode targets
+ for t in targets:
+ if t.kind == "supernode":
+ t._ssh_user = SSH_USER
+ t._ssh_port = SSH_PORT
+ t._ssh_password = SSH_PASSWORD
+
+ return targets
+
+
+# ---------------------------------------------------------------------------
+# Time parsing
+# ---------------------------------------------------------------------------
+def parse_since(value: str) -> datetime:
+ m = re.fullmatch(r"(\d+)\s*(s|min|h)", value)
+ if m:
+ n = int(m.group(1))
+ unit = m.group(2)
+ delta = (timedelta(seconds=n) if unit == "s"
+ else timedelta(minutes=n) if unit == "min"
+ else timedelta(hours=n))
+ return datetime.now() - delta
+ for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M"):
+ try:
+ return datetime.strptime(value, fmt)
+ except ValueError:
+ continue
+ raise SystemExit(f"unrecognized --since format: {value!r}")
+
+
+def parse_until(value: str) -> datetime:
+ for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M"):
+ try:
+ return datetime.strptime(value, fmt)
+ except ValueError:
+ continue
+ raise SystemExit(f"--until requires absolute time, got: {value!r}")
+
+
+def line_timestamp(line: str) -> datetime | None:
+ """Extract timestamp from a log line (glog prefix or ISO inside fields)."""
+ m = GLOG_TS_RE.match(line)
+ if m:
+ return datetime.strptime(
+ f"{m.group(1)}-{m.group(2)}-{m.group(3)} {m.group(4)}.{m.group(5)}",
+ "%Y-%m-%d %H:%M:%S.%f",
+ )
+ # Fallback: ISO timestamp inside the line
+ m = ISO_TS_RE.search(line)
+ if m:
+ micros = m.group(3) or "0"
+ micros = (micros + "000000")[:6]
+ return datetime.strptime(
+ f"{m.group(1)} {m.group(2)}.{micros}", "%Y-%m-%d %H:%M:%S.%f"
+ )
+ return None
+
+
+# ---------------------------------------------------------------------------
+# Field / key parsing
+# ---------------------------------------------------------------------------
+def parse_fields(body: str) -> dict[str, str]:
+ result = {k: v for k, v in BRACKET_RE.findall(body)}
+ result.update({k: v for k, v in FIELD_RE.findall(body)})
+ return result
+
+
+def split_chunk_key(
+ key: str, *, marker: str, path_separator: str, chunk_size: int, style: str
+) -> tuple[str, bool, int | None]:
+ if style in {"auto", "offset-marker"} and marker in key:
+ base, offset_text = key.rsplit(marker, 1)
+ try:
+ offset = int(offset_text)
+ return base, True, offset // chunk_size
+ except ValueError:
+ pass
+ if style in {"auto", "path-index"} and path_separator in key:
+ base, index_text = key.rsplit(path_separator, 1)
+ try:
+ idx = int(index_text)
+ if idx >= 0:
+ return base, True, idx
+ except ValueError:
+ pass
+ return key, False, None
+
+
+def number(value: str | None) -> float | None:
+ if value is None:
+ return None
+ try:
+ return float(value)
+ except ValueError:
+ return None
+
+
+def percentile(values: list[float], q: float) -> float:
+ if not values:
+ return 0.0
+ ordered = sorted(values)
+ pos = (len(ordered) - 1) * q
+ lo, hi = math.floor(pos), math.ceil(pos)
+ if lo == hi:
+ return ordered[lo]
+ return ordered[lo] + (ordered[hi] - ordered[lo]) * (pos - lo)
+
+
+def stats(values: list[float]) -> dict[str, float | int]:
+ data = [v for v in values if v is not None]
+ if not data:
+ return {"count": 0, "avg": 0, "p50": 0, "p95": 0, "p99": 0,
+ "p999": 0, "p9999": 0, "min": 0, "max": 0}
+ return {
+ "count": len(data),
+ "avg": sum(data) / len(data),
+ "p50": percentile(data, 0.50),
+ "p95": percentile(data, 0.95),
+ "p99": percentile(data, 0.99),
+ "p999": percentile(data, 0.999),
+ "p9999": percentile(data, 0.9999),
+ "min": min(data),
+ "max": max(data),
+ }
+
+
+def slot_from_source(source_file: str) -> str:
+ m = SLOT_RE.search(source_file)
+ if m:
+ return f"slot{m.group(1)}"
+ # fallback: use parent dir name
+ return Path(source_file).parent.name or Path(source_file).stem
+
+
+# ---------------------------------------------------------------------------
+# Data records
+# ---------------------------------------------------------------------------
+@dataclass
+class Sample:
+ """One breakdown line (any of the 8 ops)."""
+ op: str
+ ts: datetime
+ fields: dict[str, str]
+ timings: dict[str, float]
+ raw_line: str
+ source_file: str
+ line_no: int
+ slot: str
+ target: str
+
+
+@dataclass
+class BenchRequest:
+ """get_into_breakdown record for bench-style analysis."""
+ ts: datetime
+ slot: str
+ target: str
+ trace_id: str
+ key: str
+ fields: dict[str, str]
+ timings: dict[str, float]
+ raw_log: str
+
+
+@dataclass
+class PodFileStat:
+ """Per-pod file statistics for get ops (item #5)."""
+ target: str
+ slot: str
+ total_files: int = 0
+ chunk_files: int = 0 # key contains #c# or /chunk/
+ other_files: int = 0
+ chunk_bytes: int = 0 # chunk_files * chunk_size
+ other_bytes_known: int = 0 # sum of total_bytes for non-chunk keys
+ chunk_keys: set[str] = field(default_factory=set) # unique base keys
+ other_key_samples: list[str] = field(default_factory=list) # sample other keys
+
+
+@dataclass
+class SpDiagRow:
+ program: str
+ module: str
+ point: str
+ level: int
+ ticks: int
+ good: int
+ bad: int
+ not_done: int
+ total_ns: int
+ avg_ns: int
+ min_ns: int
+ max_ns: int
+ p99_ns: int | None = None
+ p999_ns: int | None = None
+ p9999_ns: int | None = None
+
+
+@dataclass
+class TargetSnapshot:
+ target: Target
+ spdiag_rows: list[SpDiagRow] = field(default_factory=list)
+ spdiag_raw: str = ""
+ spdiag_rc: int = 0
+ log_files: list[Path] = field(default_factory=list)
+ exec_rc: int | None = None
+ exec_out: str = ""
+ error: str = ""
+
+
+# ---------------------------------------------------------------------------
+# Spdiag parsing
+# ---------------------------------------------------------------------------
+def parse_spdiag_show(text: str) -> list[SpDiagRow]:
+ rows: list[SpDiagRow] = []
+ for line in text.splitlines():
+ m = SPDIAG_ROW_RE.match(line)
+ if not m:
+ continue
+ p99 = m.group("p99")
+ p999 = m.group("p999")
+ p9999 = m.group("p9999")
+ rows.append(SpDiagRow(
+ program=m.group("program"),
+ module=m.group("module"),
+ point=m.group("point"),
+ level=int(m.group("lvl")),
+ ticks=int(m.group("ticks")),
+ good=int(m.group("good")),
+ bad=int(m.group("bad")),
+ not_done=int(m.group("not")),
+ total_ns=int(m.group("total")),
+ avg_ns=int(m.group("avg")),
+ min_ns=int(m.group("min")),
+ max_ns=int(m.group("max")),
+ p99_ns=None if p99 in (None, "N/A") else int(p99),
+ p999_ns=None if p999 in (None, "N/A") else int(p999),
+ p9999_ns=None if p9999 in (None, "N/A") else int(p9999),
+ ))
+ return rows
+
+
+# ---------------------------------------------------------------------------
+# Log parsing
+# ---------------------------------------------------------------------------
+def discover_log_files(root: Path) -> list[Path]:
+ if not root.exists():
+ return []
+ patterns = ("*.log", "*.log.*", "*.txt", "*.out", "*.err", "*.INFO*", "*.WARNING*")
+ files: list[Path] = []
+ for pat in patterns:
+ files.extend(root.rglob(pat))
+ # Dedupe and sort
+ seen = set()
+ result = []
+ for f in sorted(files):
+ rp = f.resolve()
+ if rp in seen:
+ continue
+ seen.add(rp)
+ result.append(f)
+ return result
+
+
+def parse_breakdown_line(line: str) -> tuple[str, dict[str, str]] | None:
+ """If line contains an op breakdown, return (op, fields). Else None."""
+ m = OP_LINE_RE.search(line)
+ if not m:
+ return None
+ op = m.group(1)
+ if op not in OPS:
+ return None
+ fields = parse_fields(line)
+ fields.update(dict(KV_FIELD_RE.findall(line)))
+ return op, fields
+
+
+def parse_logs(
+ log_root: Path,
+ *,
+ since: datetime | None,
+ until: datetime | None,
+ chunk_size: int,
+ chunk_marker: str,
+ path_separator: str,
+ chunk_style: str,
+) -> tuple[list[Sample], list[BenchRequest], list[PodFileStat], dict[str, int]]:
+ """Parse all log files under log_root. Returns (samples, bench_requests,
+ pod_file_stats, counts)."""
+ samples: list[Sample] = []
+ bench_requests: list[BenchRequest] = []
+ counts: Counter[str] = Counter()
+ # pod_file_stats keyed by (target, slot)
+ pod_stats: dict[tuple[str, str], PodFileStat] = {}
+
+ for path in discover_log_files(log_root):
+ target_name = _infer_target_name(path, log_root)
+ slot = slot_from_source(str(path))
+ counts["files_seen"] += 1
+ try:
+ with path.open("r", encoding="utf-8", errors="replace") as fh:
+ for line_no, line in enumerate(fh, 1):
+ counts["lines_seen"] += 1
+ parsed = parse_breakdown_line(line)
+ if parsed is None:
+ continue
+ op, fields = parsed
+ ts = line_timestamp(line)
+ if ts is None:
+ continue
+ if since and ts < since:
+ continue
+ if until and ts > until:
+ continue
+ counts["target_lines"] += 1
+ timings = {
+ k: float(v) for k, v in fields.items()
+ if k.endswith("_us") and _is_number(v)
+ }
+ if not timings:
+ counts["malformed"] += 1
+ continue
+ samples.append(Sample(
+ op=op, ts=ts, fields=fields, timings=timings,
+ raw_line=line.rstrip(), source_file=str(path),
+ line_no=line_no, slot=slot, target=target_name,
+ ))
+ counts[op] += 1
+
+ # Bench-style: get_into_breakdown for trace correlation
+ if op == "get_into_breakdown":
+ trace_m = TRACE_RE.search(line)
+ trace_id = trace_m.group(1) if trace_m else ""
+ bench_requests.append(BenchRequest(
+ ts=ts, slot=slot, target=target_name,
+ trace_id=trace_id, key=fields.get("key", "-"),
+ fields=fields, timings=timings,
+ raw_log=line.rstrip(),
+ ))
+
+ # Pod file stats for get-class ops
+ if op.startswith("get") and "key" in fields:
+ _accumulate_pod_file_stat(
+ pod_stats, target_name, slot, fields["key"],
+ fields, chunk_size, chunk_marker,
+ path_separator, chunk_style,
+ )
+ except OSError:
+ counts["files_failed"] += 1
+
+ samples.sort(key=lambda s: s.ts)
+ bench_requests.sort(key=lambda r: r.ts)
+ pod_file_stats = list(pod_stats.values())
+ pod_file_stats.sort(key=lambda p: (p.target, p.slot))
+ return samples, bench_requests, pod_file_stats, dict(counts)
+
+
+def _infer_target_name(path: Path, log_root: Path) -> str:
+ """Infer target name from path relative to log_root (first dir component)."""
+ try:
+ rel = path.relative_to(log_root)
+ parts = rel.parts
+ if len(parts) > 1:
+ return parts[0]
+ except ValueError:
+ pass
+ return path.parent.name or "unknown"
+
+
+def _is_number(value: str) -> bool:
+ try:
+ float(value)
+ return True
+ except ValueError:
+ return False
+
+
+def _accumulate_pod_file_stat(
+ pod_stats: dict[tuple[str, str], PodFileStat],
+ target: str,
+ slot: str,
+ key: str,
+ fields: dict[str, str],
+ chunk_size: int,
+ chunk_marker: str,
+ path_separator: str,
+ chunk_style: str,
+) -> None:
+ stat = pod_stats.setdefault((target, slot), PodFileStat(
+ target=target, slot=slot
+ ))
+ stat.total_files += 1
+ base_key, is_chunk, _ = split_chunk_key(
+ key, marker=chunk_marker, path_separator=path_separator,
+ chunk_size=chunk_size, style=chunk_style,
+ )
+ total_bytes = number(fields.get("total_bytes")) or 0
+ if is_chunk:
+ stat.chunk_files += 1
+ stat.chunk_bytes += chunk_size # standard 4MB chunk
+ stat.chunk_keys.add(base_key)
+ else:
+ stat.other_files += 1
+ stat.other_bytes_known += int(total_bytes)
+ if len(stat.other_key_samples) < 10:
+ stat.other_key_samples.append(key)
+
+
+# ---------------------------------------------------------------------------
+# Bench-style cross-role correlation (from analyze_mooncake_bench_web.py)
+# ---------------------------------------------------------------------------
+def parse_bench_aux_logs(
+ log_root: Path,
+ *,
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[
+ list[BenchRequest],
+ dict[str, dict[str, str]], # rpc (client)
+ dict[str, dict[str, str]], # storage_read
+ dict[str, dict[str, str]], # storage_rpc (server)
+ dict[str, dict[str, str]], # release
+ dict[str, dict[str, str]], # master_rpc
+ dict[str, int], # counts
+]:
+ requests: list[BenchRequest] = []
+ rpc: dict[str, dict[str, str]] = {}
+ storage_read: dict[str, dict[str, str]] = {}
+ storage_rpc: dict[str, dict[str, str]] = {}
+ release: dict[str, dict[str, str]] = {}
+ master_rpc: dict[str, dict[str, str]] = {}
+ counts: Counter[str] = Counter()
+
+ for path in discover_log_files(log_root):
+ slot = slot_from_source(str(path))
+ try:
+ fh = path.open("r", encoding="utf-8", errors="replace")
+ except OSError:
+ counts["files_failed"] += 1
+ continue
+ with fh:
+ for line in fh:
+ m = BENCH_LOG_RE.match(line)
+ if not m:
+ mm = MASTER_SERVER_RE.match(line)
+ if mm:
+ ts = datetime.strptime(
+ mm.group("date") + " " + mm.group("time"),
+ "%Y%m%d %H:%M:%S.%f",
+ )
+ if since and ts < since:
+ continue
+ if until and ts > until:
+ continue
+ counts["master_rpc_server_breakdown"] += 1
+ trace = mm.group("trace")
+ fields = parse_fields(mm.group("body"))
+ target = master_rpc.setdefault(trace, {})
+ target["server_latency_us"] = fields.get("latency_us", "")
+ target["server_status"] = fields.get("status", "")
+ continue
+ event = m.group("event")
+ if event not in BENCH_EVENTS:
+ continue
+ ts = datetime.strptime(
+ m.group("date") + " " + m.group("time"),
+ "%Y%m%d %H:%M:%S.%f",
+ )
+ if since and ts < since:
+ continue
+ if until and ts > until:
+ continue
+ counts[event] += 1
+ trace = m.group("trace")
+ fields = parse_fields(m.group("body"))
+ fields["_slot"] = slot
+ if event == "get_into_breakdown":
+ timings = {
+ k: float(v) for k, v in fields.items()
+ if k in BENCH_STAGES and _is_number(v)
+ }
+ requests.append(BenchRequest(
+ ts=ts, slot=slot, target=_infer_target_name(path, log_root),
+ trace_id=trace, key=fields.get("key", "-"),
+ fields=fields, timings=timings,
+ raw_log=line.rstrip(),
+ ))
+ elif event == "offload_rpc_client_breakdown":
+ rpc[trace] = fields
+ elif event == "storage_read_breakdown":
+ storage_read[trace] = fields
+ elif event == "offload_rpc_server_breakdown":
+ storage_rpc[trace] = fields
+ elif event == "storage_release_breakdown":
+ release[trace] = fields
+ else:
+ master_rpc.setdefault(trace, {}).update(fields)
+ return requests, rpc, storage_read, storage_rpc, release, master_rpc, dict(counts)
+
+
+# ---------------------------------------------------------------------------
+# Aggregation
+# ---------------------------------------------------------------------------
+def aggregate_op_summary(samples: list[Sample]) -> list[dict]:
+ """8-op summary: each op x each *_us segment -> stats row."""
+ # group by (op, segment)
+ buckets: dict[tuple[str, str], list[float]] = defaultdict(list)
+ for s in samples:
+ for seg, val in s.timings.items():
+ buckets[(s.op, seg)].append(val)
+ rows = []
+ for op in OPS:
+ # collect all segments observed for this op
+ segs = sorted({seg for (o, seg) in buckets if o == op})
+ for seg in segs:
+ values = buckets[(op, seg)]
+ st = stats(values)
+ rows.append({
+ "op": OP_LABEL.get(op, op),
+ "segment": seg,
+ **st,
+ })
+ return rows
+
+
+def aggregate_qps_by_slot(samples: list[Sample]) -> dict[str, dict[str, int]]:
+ """QPS by slot for get_into/batch_get_into ops (per-second buckets)."""
+ by_slot: dict[str, list[Sample]] = defaultdict(list)
+ for s in samples:
+ if s.op in ("get_into_breakdown", "batch_get_into_breakdown"):
+ by_slot[f"{s.target}/{s.slot}"].append(s)
+ qps: dict[str, dict[str, int]] = {}
+ for slot_key, items in sorted(by_slot.items()):
+ buckets = Counter(item.ts.strftime("%H:%M:%S") for item in items)
+ qps[slot_key] = dict(sorted(buckets.items()))
+ return qps
+
+
+def aggregate_bandwidth_by_slot(samples: list[Sample], *, chunk_size: int = DEFAULT_CHUNK_SIZE) -> dict[str, dict[str, float]]:
+ """Bandwidth by slot for get_into/batch_get_into ops (per-second bytes/s -> MB/s).
+ Falls back to chunk_size when total_bytes is missing/zero for get-class ops."""
+ by_slot: dict[str, dict[str, float]] = defaultdict(lambda: defaultdict(float))
+ for s in samples:
+ if s.op in ("get_into_breakdown", "batch_get_into_breakdown"):
+ key = f"{s.target}/{s.slot}"
+ ts_bucket = s.ts.strftime("%H:%M:%S")
+ val = float(s.fields.get("total_bytes", 0))
+ if val == 0 and s.op.startswith("get"):
+ val = float(chunk_size) # fallback: assume 4MB per get request
+ by_slot[key][ts_bucket] += val
+ bw: dict[str, dict[str, float]] = {}
+ for slot_key in sorted(by_slot):
+ bw[slot_key] = {t: v / (1024 * 1024) for t, v in sorted(by_slot[slot_key].items())}
+ return bw
+
+
+def aggregate_get_into_chart(
+ requests: list[BenchRequest],
+ *,
+ max_points: int = CHART_MAX_POINTS,
+ slow_points: int = CHART_SLOW_POINTS,
+) -> tuple[list[dict], int]:
+ """Downsample get_into requests for the stacked area chart.
+
+ Returns (chart_points, stride). Each chart point has time, slot, trace_id,
+ key, the 6 stage values, and total_us.
+ """
+ ordered = sorted(requests, key=lambda r: r.ts)
+ max_points = max(1, max_points)
+ slow_points = max(0, slow_points)
+ stride = max(1, math.ceil(len(ordered) / max_points))
+ uniform = ordered[::stride]
+ slow = sorted(
+ requests,
+ key=lambda r: r.timings.get("total_us", 0),
+ reverse=True,
+ )[:slow_points]
+ combined = sorted(
+ {id(r): r for r in uniform + slow}.values(),
+ key=lambda r: r.ts,
+ )
+ chart = [
+ {
+ "time": r.ts.strftime("%H:%M:%S.%f")[:-3],
+ "slot": f"{r.target}/{r.slot}",
+ "trace_id": r.trace_id,
+ "key": r.key,
+ **{seg: r.timings.get(seg, 0) for seg in GET_INTO_STAGES},
+ "total_us": r.timings.get("total_us", 0),
+ "log": r.raw_log,
+ }
+ for r in combined
+ ]
+ return chart, stride
+
+
+def aggregate_slowest_requests(
+ requests: list[BenchRequest],
+ *,
+ rpc: dict[str, dict[str, str]],
+ storage_read: dict[str, dict[str, str]],
+ storage_rpc: dict[str, dict[str, str]],
+ release: dict[str, dict[str, str]],
+ master_rpc: dict[str, dict[str, str]],
+ n: int = SLOW_TRACE_COUNT,
+) -> list[dict]:
+ slowest = sorted(
+ requests, key=lambda r: r.timings.get("total_us", 0), reverse=True
+ )[:n]
+ rows = []
+ for rank, r in enumerate(slowest, 1):
+ rpc_f = rpc.get(r.trace_id, {})
+ read_f = storage_read.get(r.trace_id, {})
+ server_f = storage_rpc.get(r.trace_id, {})
+ release_f = release.get(r.trace_id, {})
+ master_f = master_rpc.get(r.trace_id, {})
+ rows.append({
+ "rank": rank,
+ "time": r.ts.strftime("%H:%M:%S.%f")[:-3],
+ "slot": f"{r.target}/{r.slot}",
+ "trace_id": r.trace_id,
+ "key": r.key,
+ "remote": r.fields.get("remote_endpoint", "-"),
+ **{k: r.timings.get(k) for k in BENCH_STAGES},
+ "rpc_pool_lookup_us": number(rpc_f.get("pool_lookup_us")),
+ "rpc_rpc_call_us": number(rpc_f.get("rpc_call_us")),
+ "rpc_result_get_us": number(rpc_f.get("result_get_us")),
+ "storage_queue_us": number(server_f.get("queue_us")),
+ "storage_file_open_us": number(read_f.get("file_open_us")),
+ "storage_disk_read_us": number(read_f.get("disk_read_us")),
+ "storage_total_us": number(read_f.get("total_us")),
+ "storage_release_us": number(release_f.get("total_us")),
+ "master_pool_lookup_us": number(master_f.get("pool_lookup_us")),
+ "master_rpc_call_us": number(master_f.get("rpc_call_us")),
+ "master_result_get_us": number(master_f.get("result_get_us")),
+ "master_rpc_total_us": number(master_f.get("total_us")),
+ "master_server_us": number(master_f.get("server_latency_us")),
+ })
+ return rows
+
+
+def aggregate_latency_overview(requests: list[BenchRequest]) -> dict:
+ total_values = [r.timings["total_us"] for r in requests if "total_us" in r.timings]
+ bytes_total = sum(number(r.fields.get("total_bytes")) or 0 for r in requests)
+ successful = [r for r in requests if r.fields.get("status") == "read_ok"]
+ correlated = sum(
+ 1 for r in requests
+ if r.trace_id in rpc_dummy and r.trace_id in storage_read_dummy
+ ) if False else 0 # correlated computed by caller
+ return {
+ "request_count": len(requests),
+ "success_count": len(successful),
+ "failed_count": len(requests) - len(successful),
+ "bytes_total": bytes_total,
+ "latency": stats(total_values),
+ }
+
+# dummy refs (avoid unused warning in aggregate_latency_overview)
+rpc_dummy: dict = {}
+storage_read_dummy: dict = {}
+
+
+# ---------------------------------------------------------------------------
+# Collection
+# ---------------------------------------------------------------------------
+def collect_snapshots(
+ targets: list[Target],
+ work_dir: Path,
+ *,
+ run_spdiag: bool,
+ pull_logs: bool,
+ timeout: int,
+) -> list[TargetSnapshot]:
+ snapshots: list[TargetSnapshot] = []
+ logs_root = work_dir / "collected_logs"
+ spdiag_root = work_dir / "spdiag_show"
+ logs_root.mkdir(parents=True, exist_ok=True)
+ spdiag_root.mkdir(parents=True, exist_ok=True)
+
+ for idx, target in enumerate(targets, 1):
+ snap = TargetSnapshot(target=target)
+ print(f"[{idx}/{len(targets)}] {target.name}:", flush=True)
+
+ if pull_logs and target.log_path:
+ print(f" pulling logs from {target.log_path} ...", flush=True)
+ files = target.pull_logs(logs_root)
+ snap.log_files = files
+ print(f" got {len(files)} file(s)", flush=True)
+
+ if run_spdiag:
+ cmd = f"{target.spdiag_bin} show"
+ print(f" running: {cmd}", flush=True)
+ rc, out = target.exec_command(cmd, timeout=timeout)
+ snap.spdiag_rc = rc
+ snap.spdiag_raw = out
+ if rc == 0:
+ snap.spdiag_rows = parse_spdiag_show(out)
+ print(f" spdiag show: {len(snap.spdiag_rows)} rows", flush=True)
+ else:
+ print(f" spdiag show failed (rc={rc})", flush=True)
+ (spdiag_root / f"{target.name}.txt").write_text(
+ out, encoding="utf-8", errors="replace"
+ )
+
+ snapshots.append(snap)
+ return snapshots
+
+
+def exec_on_targets(
+ targets: list[Target], command: str, timeout: int
+) -> list[TargetSnapshot]:
+ snapshots: list[TargetSnapshot] = []
+ for idx, target in enumerate(targets, 1):
+ snap = TargetSnapshot(target=target)
+ print(f"[{idx}/{len(targets)}] {target.name}: exec {command!r}", flush=True)
+ rc, out = target.exec_command(command, timeout=timeout)
+ snap.exec_rc = rc
+ snap.exec_out = out
+ print(f" rc={rc}", flush=True)
+ snapshots.append(snap)
+ return snapshots
+
+
+# ===========================================================================
+# HTML rendering
+# ===========================================================================
+CSS = """
+:root {
+ --bg: #f4f7fb;
+ --panel: #ffffff;
+ --line: #dbe3ee;
+ --text: #172033;
+ --muted: #667085;
+ --blue: #1769aa;
+ --red: #c9362b;
+ --green: #1f9d55;
+ --amber: #b7791f;
+}
+* { box-sizing: border-box; }
+body {
+ margin: 0;
+ background: var(--bg);
+ color: var(--text);
+ font: 14px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC",
+ "Microsoft YaHei", sans-serif;
+}
+header {
+ padding: 26px 32px;
+ background: linear-gradient(135deg, #102a43 0%, #1f3a5f 100%);
+ color: #fff;
+}
+header h1 { margin: 0 0 6px; font-size: 24px; }
+header .meta { opacity: 0.85; font-size: 13px; }
+header .meta span {
+ display: inline-block;
+ padding: 3px 9px;
+ margin-right: 8px;
+ margin-top: 6px;
+ background: rgba(255,255,255,0.12);
+ border-radius: 6px;
+}
+main { padding: 22px 30px 60px; }
+h2 {
+ margin: 28px 0 13px;
+ font-size: 18px;
+ font-weight: 650;
+ border-left: 4px solid var(--blue);
+ padding-left: 10px;
+}
+h3 { margin: 18px 0 8px; font-size: 15px; }
+.cards {
+ display: grid;
+ grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
+ gap: 12px;
+}
+.card {
+ background: var(--panel);
+ border: 1px solid var(--line);
+ border-radius: 9px;
+ padding: 14px;
+}
+.card b { display: block; font-size: 22px; color: var(--blue); }
+.card span { color: var(--muted); font-size: 12px; }
+.panel {
+ background: var(--panel);
+ border: 1px solid var(--line);
+ border-radius: 9px;
+ margin-top: 14px;
+ overflow: hidden;
+}
+.panel-head {
+ padding: 12px 16px;
+ background: #edf3f8;
+ border-bottom: 1px solid var(--line);
+ font-weight: 650;
+ display: flex;
+ justify-content: space-between;
+ align-items: center;
+}
+.panel-head .count {
+ color: var(--muted);
+ font-size: 12px;
+ font-weight: 400;
+}
+.panel-body { padding: 14px 16px; overflow-x: auto; }
+.panel-body.no-pad { padding: 0; }
+table { border-collapse: collapse; width: 100%; font-variant-numeric: tabular-nums; }
+th, td {
+ padding: 7px 10px;
+ border-bottom: 1px solid var(--line);
+ text-align: right;
+ white-space: nowrap;
+}
+th:first-child, td:first-child, th.col-name, td.col-name { text-align: left; }
+th {
+ position: sticky; top: 0;
+ background: #edf3f8;
+ color: #344054;
+ font-weight: 650;
+}
+tr:hover td { background: #f8fafc; }
+.rc-ok { color: var(--green); font-weight: 600; }
+.rc-fail { color: var(--red); font-weight: 600; }
+pre {
+ margin: 0;
+ padding: 10px 12px;
+ max-height: 320px;
+ overflow: auto;
+ background: #0f1e2e;
+ color: #d6e4f0;
+ font: 12px/1.5 "SFMono-Regular", Consolas, "Liberation Mono", monospace;
+ white-space: pre-wrap;
+ word-break: break-all;
+}
+.muted { color: var(--muted); }
+.target-section { margin-top: 18px; }
+.target-section h3 {
+ display: flex;
+ align-items: center;
+ gap: 8px;
+}
+.badge {
+ display: inline-block;
+ padding: 2px 8px;
+ border-radius: 11px;
+ font-size: 11px;
+ font-weight: 600;
+ background: #e1eaf5;
+ color: var(--blue);
+}
+.badge.k8s { background: #326ce5; color: #fff; }
+.badge.ssh { background: #1f9d55; color: #fff; }
+.empty {
+ padding: 40px;
+ text-align: center;
+ color: var(--muted);
+ font-style: italic;
+}
+.legend {
+ display: flex;
+ flex-wrap: wrap;
+ gap: 12px;
+ margin: 8px 0;
+ color: var(--muted);
+ font-size: 13px;
+}
+svg {
+ width: 100%;
+ height: 330px;
+ background: #fbfdff;
+ border: 1px solid var(--line);
+ display: block;
+}
+.slot-chart { margin-top: 16px; padding-top: 12px; border-top: 1px solid var(--line); }
+.slot-chart h3 { margin: 0 0 5px; font-size: 15px; }
+.slot-chart svg { height: 300px; }
+.chart-wrap { position: relative; }
+.tooltip {
+ display: none;
+ position: absolute;
+ z-index: 5;
+ max-width: 720px;
+ padding: 10px 12px;
+ background: rgba(16,42,67,.96);
+ color: #fff;
+ border-radius: 7px;
+ box-shadow: 0 5px 20px #0004;
+ pointer-events: none;
+ font-size: 12px;
+ white-space: normal;
+}
+.tooltip code {
+ display: block;
+ margin-top: 6px;
+ color: #d9efff;
+ word-break: break-all;
+}
+"""
+
+
+def esc(value) -> str:
+ return html.escape(str(value) if value is not None else "")
+
+
+def fmt_us(value: float | int | None) -> str:
+ if value is None:
+ return "-"
+ return f"{float(value):,.1f}"
+
+
+def fmt_bytes(value: int | float) -> str:
+ v = float(value)
+ if v < 1024:
+ return f"{v:.0f} B"
+ if v < 1024 * 1024:
+ return f"{v/1024:.1f} KB"
+ if v < 1024 * 1024 * 1024:
+ return f"{v/(1024*1024):.1f} MB"
+ return f"{v/(1024*1024*1024):.2f} GB"
+
+
+def _render_qps_table(qps_by_slot: dict[str, dict[str, int]]) -> str:
+ """Render per-second QPS table with one row per second, columns per slot."""
+ if not qps_by_slot:
+ return ''
+ slots = sorted(qps_by_slot.keys())
+ all_times = sorted({t for s in slots for t in qps_by_slot[s]})
+ if not all_times:
+ return ''
+ th = "time " + "".join(f"{esc(s)} " for s in slots) + "{esc(t)} "
+ for s in slots:
+ v = qps_by_slot[s].get(t, 0)
+ cells += f"{v:,} "
+ rows.append(f"{cells} ")
+ return f''
+
+
+def _render_bandwidth_table(bw_by_slot: dict[str, dict[str, float]]) -> str:
+ """Render per-second bandwidth table with one row per second, columns per slot."""
+ if not bw_by_slot:
+ return ''
+ slots = sorted(bw_by_slot.keys())
+ all_times = sorted({t for s in slots for t in bw_by_slot[s]})
+ if not all_times:
+ return ''
+ th = "time " + "".join(f"{esc(s)} " for s in slots) + "{esc(t)} "
+ for s in slots:
+ v = bw_by_slot[s].get(t, 0)
+ cells += f"{v:,.1f} "
+ rows.append(f"{cells} ")
+ return f''
+
+
+HTML_TEMPLATE = """
+Mooncake Cluster Diagnostics
+
+
+
+__OVERVIEW_CARDS__
+__EXEC_SECTION__
+__SPDIAG_OVERVIEW__
+__SPDIAG_TOP_SLOW__
+__QPS_SECTION__
+__QPS_TABLE_SECTION__
+__BANDWIDTH_SECTION__
+__GET_INTO_CHART_SECTION__
+__OP_SUMMARY_SECTION__
+__POD_FILE_STATS_SECTION__
+__SLOW_REQUESTS_SECTION__
+__PER_TARGET_SECTION__
+
+
+
+"""
+
+
+def render_html(
+ *,
+ output: Path,
+ work_dir: Path,
+ since: datetime | None,
+ until: datetime | None,
+ snapshots: list[TargetSnapshot],
+ samples: list[Sample],
+ bench_requests: list[BenchRequest],
+ pod_file_stats: list[PodFileStat],
+ op_summary: list[dict],
+ qps_by_slot: dict[str, dict[str, int]],
+ bandwidth_by_slot: dict[str, dict[str, float]],
+ get_into_chart: list[dict],
+ chart_stride: int,
+ slow_requests: list[dict],
+ latency_overview: dict,
+ correlated_count: int,
+ parse_counts: dict[str, int],
+ chunk_size: int,
+) -> None:
+ """Render the single self-contained HTML report."""
+ time_desc = "all time"
+ if since and until:
+ time_desc = f"{since.isoformat(sep=' ')} -> {until.isoformat(sep=' ')}"
+ elif since:
+ time_desc = f"since {since.isoformat(sep=' ')}"
+ elif until:
+ time_desc = f"until {until.isoformat(sep=' ')}"
+
+ total_targets = len(snapshots)
+ ok_spdiag = sum(1 for s in snapshots if s.spdiag_rc == 0 and s.spdiag_rows)
+ fail_spdiag = sum(1 for s in snapshots if s.spdiag_rc != 0)
+ total_log_files = sum(len(s.log_files) for s in snapshots)
+ total_perf_points = sum(len(s.spdiag_rows) for s in snapshots)
+
+ # Overview cards
+ cards = ['']
+ cards.append(f'
{total_targets} targets
')
+ cards.append(f'
{ok_spdiag} spdiag ok
')
+ cards.append(f'
{fail_spdiag} spdiag failed
')
+ cards.append(f'
{total_log_files} log files
')
+ cards.append(f'
{total_perf_points} perf points
')
+ cards.append(f'
{latency_overview["request_count"]:,} get requests
')
+ cards.append(f'
{latency_overview["success_count"]:,} successful
')
+ cards.append(f'
{latency_overview["failed_count"]:,} failed
')
+ cards.append(f'
{correlated_count:,} full trace correlation
')
+ cards.append(f'
{fmt_us(latency_overview["latency"]["p50"])} p50 (us)
')
+ cards.append(f'
{fmt_us(latency_overview["latency"]["p99"])} p99 (us)
')
+ cards.append(f'
{fmt_us(latency_overview["latency"]["max"])} max (us)
')
+ cards.append(f'
{fmt_bytes(latency_overview["bytes_total"])} total bytes
')
+ # average bandwidth
+ total_bw = sum(sum(v for v in b.values()) for b in bandwidth_by_slot.values())
+ elapsed_s = latency_overview.get("elapsed_s", 1) or 1
+ avg_bw = total_bw / elapsed_s if elapsed_s > 0 else 0
+ cards.append(f'
{avg_bw:,.1f} avg bandwidth (MB/s)
')
+ cards.append('
{esc(s.target.name)} "
+ f"{esc(s.target.kind)} "
+ f"{esc(s.exec_rc)} "
+ f"{esc(preview)} Command Execution Results (--exec)
+--exec results
+
+target kind rc output {''.join(rows_html)}
+
{esc(s.target.name)} "
+ f"{esc(s.target.kind)} "
+ f"{esc(', '.join(programs))} "
+ f"{len(s.spdiag_rows)} "
+ f"{ticks:,} "
+ f"{fmt_us(max_avg)} "
+ f"{fmt_us(max_max)} spdiag show 概览
+per-target perf points
+{ok_spdiag}/{total_targets} targets
+
+target kind program points
+total ticks max avg (us) max max (us) {''.join(spdiag_overview_rows)}
+
spdiag show 概览
+No spdiag show data. Run `spdiag start` on targets first.
{esc(target_name)} "
+ f"{esc(r.program)} {esc(r.module)} "
+ f"{esc(r.point)} {r.level} "
+ f"{r.ticks:,} {r.good:,} {r.bad:,} "
+ f"{fmt_us(r.avg_ns)} {fmt_us(r.min_ns)} "
+ f"{fmt_us(r.max_ns)} {fmt_us(r.p99_ns)} "
+ f"{fmt_us(r.p999_ns)} {fmt_us(r.p9999_ns)} top 30 slowest perf points (by avg)
+{len(all_rows)} total
+
+target program module point
+lvl ticks good bad
+avg (us) min (us) max (us)
+p99 (us) p999 (us) p9999 (us) {''.join(top_rows_html)}
+
QPS by slot
+"""
+
+ # QPS per-second table
+ qps_table_html = _render_qps_table(qps_by_slot)
+ qps_table_section = f"""
+每秒 QPS 明细表
+"""
+
+ # Bandwidth section (chart + table)
+ bw_section = f"""
+Bandwidth by slot (MB/s)
+"""
+ bw_table_html = _render_bandwidth_table(bandwidth_by_slot)
+ bw_section += f"""
+每秒带宽明细表 (MB/s)
+
+{bw_table_html}
+
get_into 耗时拆解 (us)
+
+
+
+
+ 按 __CHART_STRIDE__ 个请求抽样（目标最多 {CHART_MAX_POINTS:,} 个基础点），
+ 并强制加入全局耗时最高的 {CHART_SLOW_POINTS} 个请求。
+ 按住鼠标左键拖拽框选放大，双击恢复。
+
+
{esc(row['op'])} "
+ f"{esc(row['segment'])} "
+ f"{row['count']:,} "
+ f"{row['avg']:,.1f} "
+ f"{row['p50']:,.1f} "
+ f"{row['p95']:,.1f} "
+ f"{row['p99']:,.1f} "
+ f"{row['p999']:,.1f} "
+ f"{row['p9999']:,.1f} "
+ f"{row['min']:,.1f} "
+ f"{row['max']:,.1f} 8 类操作 Summary Statistics (us)
+
+operation segment count avg
+p50 p95 p99 p999 p9999
+min max {''.join(op_rows_html) or "no breakdown logs found
+
{esc(p.target)} "
+ f"{esc(p.slot)} "
+ f"{p.total_files:,} "
+ f"{p.chunk_files:,} "
+ f"{fmt_bytes(p.chunk_bytes)} "
+ f"{len(p.chunk_keys):,} "
+ f"{p.other_files:,} "
+ f"{fmt_bytes(p.other_bytes_known)} "
+ f"{esc(chunk_keys_str)} "
+ f"{esc(other_samples_str)} Per-pod file stats (get class)
+每个 pod get 了多少文件，#c# chunk vs 其他
+chunk size = {fmt_bytes(chunk_size)}
+
+target slot total files
+chunk files (#c#) chunk bytes unique chunk keys
+other files other bytes (known)
+sample chunk keys sample other keys {''.join(pod_rows_html) or "no get-class logs found
+
{row['rank']} "
+ f"{esc(row['time'])} "
+ f"{esc(row['slot'])} "
+ f"{esc(row['trace_id'])} "
+ f"{esc(row['key'])} "
+ f"{esc(row['remote'])} "
+ + "".join(f"{fmt_us(row.get(k))} " for k in BENCH_STAGES)
+ + f"{fmt_us(row.get('rpc_pool_lookup_us'))} "
+ + f"{fmt_us(row.get('rpc_rpc_call_us'))} "
+ + f"{fmt_us(row.get('rpc_result_get_us'))} "
+ + f"{fmt_us(row.get('storage_queue_us'))} "
+ + f"{fmt_us(row.get('storage_file_open_us'))} "
+ + f"{fmt_us(row.get('storage_disk_read_us'))} "
+ + f"{fmt_us(row.get('storage_total_us'))} "
+ + f"{fmt_us(row.get('storage_release_us'))} "
+ + f"{fmt_us(row.get('master_pool_lookup_us'))} "
+ + f"{fmt_us(row.get('master_rpc_call_us'))} "
+ + f"{fmt_us(row.get('master_result_get_us'))} "
+ + f"{fmt_us(row.get('master_rpc_total_us'))} "
+ + f"{fmt_us(row.get('master_server_us'))} 最慢 {SLOW_TRACE_COUNT} 个请求：跨角色关联 (trace_id)
+
+rank time slot trace id key remote
+query select read offload RPC transfer
+release overhead total
+RPC pool RPC call RPC result
+ST queue ST file open ST disk read ST total ST release
+M pool M rpc M result M total M server
+{''.join(slow_rows_html) or "no get_into_breakdown records
+
Per-Target Details ']
+ for s in snapshots:
+ badge_cls = "k8s" if s.target.kind == "k8s_pod" else "ssh"
+ per_target_parts.append(f'')
+ per_target_parts.append(
+ f"
{esc(s.target.name)} "
+ f"{esc(s.target.kind)} "
+ )
+ if s.spdiag_raw:
+ rc_cls = "rc-ok" if s.spdiag_rc == 0 else "rc-fail"
+ per_target_parts.append(
+ f"
"
+ f"spdiag show "
+ f"rc={s.spdiag_rc} "
+ f" · {len(s.spdiag_rows)} rows
"
+ f"
"
+ )
+ if s.log_files:
+ file_rows = []
+ for f in s.log_files[:50]:
+ try:
+ size = f.stat().st_size
+ except OSError:
+ size = -1
+ file_rows.append(
+ f"
{esc(f)} {size:,} "
+ )
+ extra = ""
+ if len(s.log_files) > 50:
+ extra = (f"
"
+ f"... and {len(s.log_files) - 50} more ")
+ per_target_parts.append(
+ f"
"
+ f"collected log files "
+ f"{len(s.log_files)} files
"
+ f"
"
+ f"path size {''.join(file_rows)}{extra}
"
+ )
+ if s.error:
+ per_target_parts.append(
+ f"
"
+ f"error: {esc(s.error)} "
+ f"
"
+ )
+ per_target_parts.append('
no get_into data
"; return; }}
+ const times = [...new Set(slots.flatMap(s => Object.keys(qps[s])))].sort();
+ const tip = document.getElementById("qps-tip");
+ const W = 1200, H = 330, p = {{l:58, r:18, t:16, b:42}};
+ svg.setAttribute("viewBox", `0 0 ${{W}} ${{H}}`);
+ const ymax = Math.max(1, ...slots.flatMap(s => Object.values(qps[s])));
+ const x = i => p.l + i * (W - p.l - p.r) / Math.max(1, times.length - 1);
+ const y = v => H - p.b - v * (H - p.t - p.b) / ymax;
+ let out = `${{Math.round(v)}} `;
+ }}
+ slots.forEach((s, si) => {{
+ const pts = times.map((t, i) => `${{x(i)}},${{y(qps[s][t] || 0)}}`).join(" ");
+ out += `${{times[0]}} `;
+ out += `${{times[times.length-1]}} `;
+ }}
+ svg.innerHTML = out;
+ document.getElementById("qps-legend").innerHTML = slots.map((s, i) =>
+ `● ${{esc(s)}} `).join("");
+ attachHover(svg, tip, times.length, x, i =>
+ `${{times[i]}} ● ${{s}}: ${{qps[s][times[i]] || 0}} QPS `).join("no bandwidth data
"; return; }}
+ const times = [...new Set(slots.flatMap(s => Object.keys(bw[s])))].sort();
+ const tip = document.getElementById("bw-tip");
+ const W = 1200, H = 330, p = {{l:58, r:18, t:16, b:42}};
+ svg.setAttribute("viewBox", `0 0 ${{W}} ${{H}}`);
+ const ymax = Math.max(1, ...slots.flatMap(s => Object.values(bw[s])));
+ const x = i => p.l + i * (W - p.l - p.r) / Math.max(1, times.length - 1);
+ const y = v => H - p.b - v * (H - p.t - p.b) / ymax;
+ let out = `${{(v).toFixed(1)}} `;
+ }}
+ slots.forEach((s, si) => {{
+ const pts = times.map((t, i) => `${{x(i)}},${{y(bw[s][t] || 0)}}`).join(" ");
+ out += `${{times[0]}} `;
+ out += `${{times[times.length-1]}} `;
+ }}
+ svg.innerHTML = out;
+ document.getElementById("bw-legend").innerHTML = slots.map((s, i) =>
+ `● ${{esc(s)}} `).join("");
+ attachHover(svg, tip, times.length, x, i =>
+ `${{times[i]}} ● ${{s}}: ${{(bw[s][times[i]] || 0).toFixed(1)}} MB/s `).join("no get_into_breakdown records
"; return; }}
+ const labels = STAGE_LABELS;
+ document.getElementById("get-legend").innerHTML = STAGES.map((s, i) =>
+ `● ${{labels[s]}} `).join("");
+ const slots = [...new Set(all.map(v => v.slot))].sort((a, b) =>
+ Number(a.replace(/\\D/g, "")) - Number(b.replace(/\\D/g, "")));
+ const niceMax = v => {{
+ if (v <= 0) return 1;
+ const power = 10 ** Math.floor(Math.log10(v));
+ const n = v / power;
+ return (n <= 1 ? 1 : n <= 2 ? 2 : n <= 5 ? 5 : 10) * power;
+ }};
+ slots.forEach(slot => {{
+ const pts = all.filter(v => v.slot === slot);
+ const section = document.createElement("div");
+ section.className = "slot-chart";
+ section.innerHTML = `${{esc(slot)}} · ${{pts.length.toLocaleString()}} plotted points
+ `;
+ host.appendChild(section);
+ const svg = section.querySelector("svg");
+ const tip = section.querySelector(".tooltip");
+ const H = 300, p = {{l:68, r:18, t:16, b:44}};
+ let zoomStart = 0, zoomEnd = Math.max(0, pts.length - 1), dragX = null;
+ let W = 1200, currentVisible = [], currentX = () => 0, currentY = () => 0;
+ const updateWidth = () => {{
+ W = Math.max(700, Math.round(svg.getBoundingClientRect().width || 1200));
+ svg.setAttribute("viewBox", `0 0 ${{W}} ${{H}}`);
+ }};
+ const toViewX = e => {{
+ const r = svg.getBoundingClientRect();
+ return (e.clientX - r.left) * W / r.width;
+ }};
+ const toViewY = e => {{
+ const r = svg.getBoundingClientRect();
+ return (e.clientY - r.top) * H / r.height;
+ }};
+ function render() {{
+ updateWidth();
+ const visible = pts.slice(zoomStart, zoomEnd + 1);
+ const rawMax = Math.max(1, ...visible.map(v => v.total_us || 0));
+ const ymax = niceMax(rawMax);
+ const x = i => p.l + i * (W - p.l - p.r) / Math.max(1, visible.length - 1);
+ const y = v => H - p.b - v * (H - p.t - p.b) / ymax;
+ currentVisible = visible; currentX = x; currentY = y;
+ let out = `${{Math.round(v).toLocaleString()}} `;
+ }}
+ STAGES.forEach((s, si) => {{
+ out += `${{esc(visible[idx].time)}} `;
+ }}
+ }}
+ out += `${{esc(v.time)}} · ${{esc(v.slot)}} ● ${{labels[s]}}: ${{Number(v[s] || 0).toLocaleString()}} us `).join("${{Number(v.total_us || 0).toLocaleString()}} us ` +
+ `${{esc(v.log)}}`;
+ tip.style.display = "block";
+ const gap = 24, tw = tip.offsetWidth, th = tip.offsetHeight;
+ const useRB = mx < wr.width / 2;
+ tip.style.left = Math.max(4, Math.min(wr.width - tw - 4, useRB ? mx + gap : mx - gap - tw)) + "px";
+ tip.style.top = Math.max(4, Math.min(wr.height - th - 4, useRB ? my + gap : my - gap - th)) + "px";
+ }});
+ svg.addEventListener("mouseup", e => {{
+ if (dragX === null) return;
+ const endX = toViewX(e);
+ const a = Math.max(p.l, Math.min(dragX, endX));
+ const b = Math.min(W - p.r, Math.max(dragX, endX));
+ const count = zoomEnd - zoomStart + 1;
+ if (b - a > 8 && count > 2) {{
+ const oldStart = zoomStart;
+ zoomStart = oldStart + Math.floor((a - p.l) * Math.max(1, count - 1) / (W - p.l - p.r));
+ zoomEnd = Math.min(pts.length - 1, oldStart + Math.ceil((b - p.l) * Math.max(1, count - 1) / (W - p.l - p.r)));
+ zoomEnd = Math.max(zoomStart + 1, zoomEnd);
+ }}
+ dragX = null;
+ render();
+ }});
+ svg.addEventListener("mouseleave", () => {{
+ tip.style.display = "none";
+ if (dragX !== null) {{ dragX = null; render(); }}
+ }});
+ svg.addEventListener("dblclick", () => {{
+ zoomStart = 0; zoomEnd = Math.max(0, pts.length - 1);
+ tip.style.display = "none"; render();
+ }});
+ render();
+ }});
+}}
+
+drawQps();
+drawBandwidth();
+drawGetInto();
+"""
+
+
+# ===========================================================================
+# CLI
+# ===========================================================================
+def build_arg_parser() -> argparse.ArgumentParser:
+ p = argparse.ArgumentParser(
+ description="Cluster-wide Mooncake diagnostics: collect + analyze + single HTML.",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=(
+ "Edit the CONFIG section at the top of this file, then run:\n"
+ " python cluster_mooncake_diag.py\n\n"
+ "Time window:\n"
+ " --since 30min | --since '2026-08-05 10:00:00'\n"
+ " --until '2026-08-05 11:00:00'\n\n"
+ "Exec mode (only run a command, no analysis):\n"
+ " --exec 'spdiag start'\n"
+ " --exec 'spdiag clear'\n"
+ ),
+ )
+ p.add_argument("-w", "--work-dir", type=Path,
+ default=Path("./cluster_diag_work"),
+ help="Working dir for collected files and reports.")
+ p.add_argument("-o", "--output", type=Path,
+ default=Path("cluster_diag_report.html"),
+ help="Output HTML path.")
+ p.add_argument("--since", default=None,
+ help="Start of time window. Relative (30min, 2h, 90s) "
+ "or absolute (YYYY-MM-DD HH:MM:SS).")
+ p.add_argument("--until", default=None,
+ help="End of time window (absolute only).")
+ p.add_argument("--exec", dest="exec_cmd", default=None,
+ help="Only execute this command on every target, then exit. "
+ "Example: --exec 'spdiag start'")
+ p.add_argument("--analyze-only", action="store_true",
+ help="Skip collection, analyze already-collected logs under --work-dir.")
+ p.add_argument("--no-spdiag", action="store_true",
+ help="Skip running spdiag show.")
+ p.add_argument("--no-logs", action="store_true",
+ help="Skip pulling logs.")
+ p.add_argument("--exec-timeout", type=int, default=60,
+ help="Timeout (s) for remote commands. Default 60.")
+ p.add_argument("--chunk-size", type=int, default=DEFAULT_CHUNK_SIZE,
+ help=f"Chunk size in bytes. Default {DEFAULT_CHUNK_SIZE}.")
+ p.add_argument("--chunk-marker", default=DEFAULT_CHUNK_MARKER,
+ help=f"Chunk key marker. Default {DEFAULT_CHUNK_MARKER!r}.")
+ p.add_argument("--path-separator", default=DEFAULT_PATH_SEPARATOR,
+ help=f"Chunk path separator. Default {DEFAULT_PATH_SEPARATOR!r}.")
+ p.add_argument("--chunk-style", default=DEFAULT_CHUNK_STYLE,
+ choices=["auto", "offset-marker", "path-index"],
+ help=f"Chunk key style. Default {DEFAULT_CHUNK_STYLE!r}.")
+ p.add_argument("--sample", action="store_true",
+ help="Generate sample HTML using synthetic data.")
+ return p
+
+
+def cmd_sample(args) -> int:
+ """Generate a sample HTML report using synthetic data."""
+ output = args.output.resolve()
+ output.parent.mkdir(parents=True, exist_ok=True)
+
+ # Synthetic targets
+ targets = [
+ Target(name="slot1_master", kind="k8s_pod", namespace="default",
+ pod="mooncake-master-xxxxx", log_path="/var/log/mooncake"),
+ Target(name="slot2_store", kind="k8s_pod", namespace="default",
+ pod="mooncake-store-yyyyy", log_path="/var/log/mooncake"),
+ Target(name="supernode-10.0.1.5", kind="supernode", host="10.0.1.5",
+ log_path="/var/log/mooncake"),
+ ]
+
+ def make_rows(program: str, points: list[tuple[str, str, int, int, int]]) -> list[SpDiagRow]:
+ out = []
+ for module, point, ticks, avg_ns, max_ns in points:
+ out.append(SpDiagRow(
+ program=program, module=module, point=point,
+ level=3, ticks=ticks, good=ticks, bad=0, not_done=0,
+ total_ns=ticks * avg_ns, avg_ns=avg_ns,
+ min_ns=max(100, avg_ns // 3), max_ns=max_ns,
+ p99_ns=int(avg_ns * 1.5), p999_ns=int(avg_ns * 2.0),
+ p9999_ns=int(avg_ns * 3.0),
+ ))
+ return out
+
+ snapshots = [
+ TargetSnapshot(
+ target=targets[0],
+ spdiag_rows=make_rows("mooncake_master", [
+ ("rpc_service.cpp::PutStart", "PutStart", 12500, 8500, 42000),
+ ("rpc_service.cpp::PutEnd", "PutEnd", 12500, 12300, 58000),
+ ("rpc_service.cpp::GetReplicaList", "GetReplicaList", 24800, 4200, 18900),
+ ("master_service.cpp::AllocateAndInsertMetadata", "AllocateMemory", 12500, 2100, 8800),
+ ("master_service.cpp::PutStart", "AcquireShardLock", 12500, 180, 950),
+ ]),
+ spdiag_raw=(
+ "spdiag started (SHM: /spdiag_shm_default)\n"
+ "# Program Module Point Lvl Ticks Good Bad Not Total(ns) Avg(ns) Min(ns) Max(ns) P99 P999 P9999\n"
+ "1 mooncake_master rpc_service.cpp::PutStart PutStart 3 12500 12500 0 0 106250000 8500 1200 42000 12750 18000 31000\n"
+ "2 mooncake_master rpc_service.cpp::PutEnd PutEnd 3 12500 12500 0 0 153750000 12300 2100 58000 18450 26000 44000\n"
+ "3 mooncake_master rpc_service.cpp::GetReplicaList GetReplicaList 3 24800 24800 0 0 104160000 4200 800 18900 6300 8900 15000\n"
+ ),
+ spdiag_rc=0,
+ log_files=[
+ Path("/tmp/cluster_logs/slot1_master/mooncake_master.log.INFO.20260805-101530.12345"),
+ ],
+ ),
+ TargetSnapshot(
+ target=targets[1],
+ spdiag_rows=make_rows("mooncake_store", [
+ ("real_client.cpp::get_into_internal", "GetIntoInternal", 25000, 45000, 230000),
+ ("real_client.cpp::get_buffer_internal", "AllocBuffer", 25000, 3200, 18500),
+ ("real_client.cpp::get_buffer_internal", "SSDRead", 18000, 28000, 145000),
+ ("client_service.cpp::Get", "TransferGet", 25000, 52000, 260000),
+ ]),
+ spdiag_raw=(
+ "# Program Module Point Lvl Ticks Good Bad Not Total(ns) Avg(ns) Min(ns) Max(ns) P99 P999 P9999\n"
+ "1 mooncake_store real_client.cpp::get_into_internal GetIntoInternal 3 25000 25000 0 0 1125000000 45000 5200 230000 67500 95000 160000\n"
+ "2 mooncake_store real_client.cpp::get_buffer_internal AllocBuffer 3 25000 25000 0 0 80000000 3200 800 18500 4800 6700 12000\n"
+ ),
+ spdiag_rc=0,
+ log_files=[
+ Path("/tmp/cluster_logs/slot2_store/mooncake_client.log.INFO.20260805-101600.23456"),
+ ],
+ ),
+ TargetSnapshot(
+ target=targets[2],
+ spdiag_rows=[],
+ spdiag_raw="spdiag is not running (SHM: /spdiag_shm_default)\n",
+ spdiag_rc=1,
+ log_files=[],
+ error="spdiag not started",
+ ),
+ ]
+
+ # Synthetic samples for op summary
+ base_ts = datetime(2026, 8, 5, 10, 0, 0)
+ samples: list[Sample] = []
+ for i in range(500):
+ ts = base_ts + timedelta(seconds=i)
+ for op in OPS[:4]: # first 4 ops
+ samples.append(Sample(
+ op=op, ts=ts,
+ fields={"key": f"e2b-dev-fc-templates/abc/rootfs.ext4#c#{i*4194304}",
+ "total_bytes": str(4 * 1024 * 1024),
+ "query_us": str(500 + i % 100),
+ "select_us": str(200 + i % 50),
+ "read_us": str(8000 + i % 2000),
+ "total_us": str(45000 + i % 5000)},
+ timings={"query_us": 500 + i % 100, "select_us": 200 + i % 50,
+ "read_us": 8000 + i % 2000, "total_us": 45000 + i % 5000},
+ raw_line=f"I20260805 10:00:{i:02d}.000000 trace_id[{i}] {op} ...",
+ source_file="synthetic.log", line_no=i, slot="slot1",
+ target="slot2_store",
+ ))
+
+ # Synthetic bench requests
+ bench_requests: list[BenchRequest] = []
+ for i in range(2000):
+ ts = base_ts + timedelta(milliseconds=i * 30)
+ is_chunk = i % 5 != 0 # 80% chunk, 20% other
+ if is_chunk:
+ key = f"e2b-dev-fc-templates/tmpl-{i%5}/rootfs.ext4#c#{i * 4194304}"
+ else:
+ key = f"e2b-dev-fc-templates/tmpl-{i%5}/snapfile.bin"
+ bench_requests.append(BenchRequest(
+ ts=ts, slot="slot2_store", target="slot2_store",
+ trace_id=str(1000 + i), key=key,
+ fields={"status": "read_ok", "total_bytes": str(4 * 1024 * 1024),
+ "remote_endpoint": "10.0.1.5:50051"},
+ timings={
+ "query_us": 500 + i % 200,
+ "select_us": 200 + i % 80,
+ "read_us": 8000 + (i * 17) % 3000,
+ "offload_rpc_us": 3000 + (i * 13) % 1500,
+ "transfer_data_us": 15000 + (i * 23) % 8000,
+ "release_buffer_us": 800 + i % 200,
+ "read_overhead_us": 200 + i % 100,
+ "total_us": 45000 + (i * 37) % 30000,
+ },
+ raw_log=f"I20260805 ... trace_id[{1000+i}] get_into_breakdown ...",
+ ))
+
+ # Pod file stats
+ pod_file_stats = [
+ PodFileStat(
+ target="slot2_store", slot="slot2",
+ total_files=2000, chunk_files=1600, other_files=400,
+ chunk_bytes=1600 * 4 * 1024 * 1024,
+ other_bytes_known=400 * 1024 * 1024,
+ chunk_keys={"e2b-dev-fc-templates/tmpl-0/rootfs.ext4",
+ "e2b-dev-fc-templates/tmpl-1/rootfs.ext4",
+ "e2b-dev-fc-templates/tmpl-2/rootfs.ext4",
+ "e2b-dev-fc-templates/tmpl-3/rootfs.ext4",
+ "e2b-dev-fc-templates/tmpl-4/rootfs.ext4"},
+ other_key_samples=["e2b-dev-fc-templates/tmpl-0/snapfile.bin",
+ "e2b-dev-fc-templates/tmpl-1/memfile.bin"],
+ ),
+ ]
+
+ op_summary = aggregate_op_summary(samples)
+ qps_by_slot = aggregate_qps_by_slot(samples)
+ bandwidth_by_slot = aggregate_bandwidth_by_slot(samples, chunk_size=args.chunk_size)
+ get_into_chart, chart_stride = aggregate_get_into_chart(bench_requests)
+ slow_requests = aggregate_slowest_requests(
+ bench_requests,
+ rpc={}, storage_read={}, storage_rpc={}, release={}, master_rpc={},
+ )
+ latency_overview = {
+ "request_count": len(bench_requests),
+ "success_count": len(bench_requests),
+ "failed_count": 0,
+ "bytes_total": sum(4 * 1024 * 1024 for _ in bench_requests),
+ "latency": stats([r.timings["total_us"] for r in bench_requests]),
+ "elapsed_s": (bench_requests[-1].ts - bench_requests[0].ts).total_seconds() if bench_requests else 1,
+ }
+
+ render_html(
+ output=output,
+ work_dir=Path("/tmp/cluster_diag_work"),
+ since=base_ts,
+ until=base_ts + timedelta(hours=1),
+ snapshots=snapshots,
+ samples=samples,
+ bench_requests=bench_requests,
+ pod_file_stats=pod_file_stats,
+ op_summary=op_summary,
+ qps_by_slot=qps_by_slot,
+ bandwidth_by_slot=bandwidth_by_slot,
+ get_into_chart=get_into_chart,
+ chart_stride=chart_stride,
+ slow_requests=slow_requests,
+ latency_overview=latency_overview,
+ correlated_count=0,
+ parse_counts={"files_seen": 5, "lines_seen": 50000, "target_lines": 12000},
+ chunk_size=args.chunk_size,
+ )
+ print(f"sample report: {output}")
+ return 0
+
+
+def main(argv: list[str]) -> int:
+ args = build_arg_parser().parse_args(argv)
+
+ if args.sample:
+ return cmd_sample(args)
+
+ since_dt = parse_since(args.since) if args.since else None
+ until_dt = parse_until(args.until) if args.until else None
+ work_dir = args.work_dir.resolve()
+ work_dir.mkdir(parents=True, exist_ok=True)
+
+ targets = build_targets()
+
+ # --exec mode: only run command, no analysis
+ if args.exec_cmd:
+ snapshots = exec_on_targets(targets, args.exec_cmd, args.exec_timeout)
+ # Render a minimal exec-only HTML
+ render_html(
+ output=args.output.resolve(),
+ work_dir=work_dir,
+ since=since_dt,
+ until=until_dt,
+ snapshots=snapshots,
+ samples=[], bench_requests=[], pod_file_stats=[],
+ op_summary=[], qps_by_slot={}, bandwidth_by_slot={}, get_into_chart=[], chart_stride=1,
+ slow_requests=[], latency_overview={
+ "request_count": 0, "success_count": 0, "failed_count": 0,
+ "bytes_total": 0, "latency": stats([]), "elapsed_s": 1,
+ },
+ correlated_count=0, parse_counts={}, chunk_size=args.chunk_size,
+ )
+ print(f"\nexec report: {args.output.resolve()}")
+ return 0
+
+ # Collection phase
+ if not args.analyze_only:
+ snapshots = collect_snapshots(
+ targets, work_dir,
+ run_spdiag=not args.no_spdiag,
+ pull_logs=not args.no_logs,
+ timeout=args.exec_timeout,
+ )
+ else:
+ # Reconstruct snapshots from existing files
+ spdiag_dir = work_dir / "spdiag_show"
+ snapshots = []
+ for target in targets:
+ spdiag_file = spdiag_dir / f"{target.name}.txt"
+ snap = TargetSnapshot(target=target)
+ if spdiag_file.exists():
+ snap.spdiag_raw = spdiag_file.read_text(encoding="utf-8", errors="replace")
+ snap.spdiag_rows = parse_spdiag_show(snap.spdiag_raw)
+ snap.spdiag_rc = 0
+ log_root = work_dir / "collected_logs" / target.name
+ if log_root.exists():
+ snap.log_files = list(_walk_files(log_root))
+ snapshots.append(snap)
+
+ # Analysis phase
+ collected_logs = work_dir / "collected_logs"
+ print(f"\nparsing logs under {collected_logs} ...", flush=True)
+ samples, bench_requests, pod_file_stats, parse_counts = parse_logs(
+ collected_logs,
+ since=since_dt, until=until_dt,
+ chunk_size=args.chunk_size, chunk_marker=args.chunk_marker,
+ path_separator=args.path_separator, chunk_style=args.chunk_style,
+ )
+ print(f" samples: {len(samples)}, bench_requests: {len(bench_requests)}, "
+ f"pod_file_stats: {len(pod_file_stats)}", flush=True)
+
+ # Bench-style aux parsing for trace correlation
+ bench_req2, rpc, storage_read, storage_rpc, release, master_rpc, bench_counts = (
+ parse_bench_aux_logs(
+ collected_logs, since=since_dt, until=until_dt,
+ )
+ )
+ # Use bench_requests from main parse if bench parse found nothing
+ if bench_req2:
+ bench_requests_final = bench_req2
+ else:
+ bench_requests_final = bench_requests
+ rpc, storage_read, storage_rpc, release, master_rpc = {}, {}, {}, {}, {}
+
+ correlated_count = sum(
+ 1 for r in bench_requests_final
+ if r.trace_id in rpc and r.trace_id in storage_read and r.trace_id in storage_rpc
+ )
+
+ # Aggregations
+ op_summary = aggregate_op_summary(samples)
+ qps_by_slot = aggregate_qps_by_slot(samples)
+ bandwidth_by_slot = aggregate_bandwidth_by_slot(samples, chunk_size=args.chunk_size)
+ get_into_chart, chart_stride = aggregate_get_into_chart(bench_requests_final)
+ slow_requests = aggregate_slowest_requests(
+ bench_requests_final,
+ rpc=rpc, storage_read=storage_read, storage_rpc=storage_rpc,
+ release=release, master_rpc=master_rpc,
+ )
+ elapsed_s = (
+ (bench_requests_final[-1].ts - bench_requests_final[0].ts).total_seconds()
+ if bench_requests_final else 1
+ )
+ latency_overview = {
+ "request_count": len(bench_requests_final),
+ "success_count": sum(1 for r in bench_requests_final
+ if r.fields.get("status") == "read_ok"),
+ "failed_count": sum(1 for r in bench_requests_final
+ if r.fields.get("status") != "read_ok"),
+ "bytes_total": sum(number(r.fields.get("total_bytes")) or 0
+ for r in bench_requests_final),
+ "latency": stats([r.timings.get("total_us", 0) for r in bench_requests_final
+ if "total_us" in r.timings]),
+ "elapsed_s": elapsed_s,
+ }
+
+ # Render
+ render_html(
+ output=args.output.resolve(),
+ work_dir=work_dir,
+ since=since_dt,
+ until=until_dt,
+ snapshots=snapshots,
+ samples=samples,
+ bench_requests=bench_requests_final,
+ pod_file_stats=pod_file_stats,
+ op_summary=op_summary,
+ qps_by_slot=qps_by_slot,
+ bandwidth_by_slot=bandwidth_by_slot,
+ get_into_chart=get_into_chart,
+ chart_stride=chart_stride,
+ slow_requests=slow_requests,
+ latency_overview=latency_overview,
+ correlated_count=correlated_count,
+ parse_counts=parse_counts,
+ chunk_size=args.chunk_size,
+ )
+ print(f"\ncluster report: {args.output.resolve()}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/mooncake-store/benchmarks/stress_cluster_bench.cpp b/mooncake-store/benchmarks/stress_cluster_bench.cpp
new file mode 100644
index 0000000000..5da134b429
--- /dev/null
+++ b/mooncake-store/benchmarks/stress_cluster_bench.cpp
@@ -0,0 +1,1618 @@
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "gflags/gflags.h"
+#include "glog/logging.h"
+#include "mooncake_logging.h"
+#include "real_client.h"
+
+#include
+#include
+#include
+#include
+
+namespace {
+constexpr size_t KB = 1024;
+constexpr size_t MB = 1024 * KB;
+constexpr size_t GB = 1024 * MB;
+
+const static int NR_SOCKETS =
+ numa_available() == 0 ? numa_num_configured_nodes() : 1;
+
+static void bindToSocket(int socket_id) {
+ if (numa_available() < 0) return;
+ cpu_set_t cpu_set;
+ CPU_ZERO(&cpu_set);
+ if (socket_id < 0 || socket_id >= numa_num_configured_nodes())
+ socket_id = 0;
+ struct bitmask* cpu_list = numa_allocate_cpumask();
+ numa_node_to_cpus(socket_id, cpu_list);
+ int nr_possible_cpus = numa_num_possible_cpus();
+ int nr_cpus = 0;
+ for (int cpu = 0; cpu < nr_possible_cpus; ++cpu) {
+ if (numa_bitmask_isbitset(cpu_list, cpu) &&
+ numa_bitmask_isbitset(numa_all_cpus_ptr, cpu)) {
+ CPU_SET(cpu, &cpu_set);
+ ++nr_cpus;
+ }
+ }
+ numa_free_cpumask(cpu_list);
+ if (nr_cpus > 0) {
+ if (sched_setaffinity(0, sizeof(cpu_set), &cpu_set) != 0) {
+ PLOG(WARNING) << "Failed to set CPU affinity for NUMA socket "
+ << socket_id;
+ }
+ }
+}
+
+static std::string FormatBytes(size_t bytes) {
+ if (bytes == 0) return "0 B";
+ const char* units[] = {"B", "KB", "MB", "GB", "TB"};
+ int i = static_cast(std::floor(std::log2(bytes) / 10));
+ if (i > 4) i = 4;
+ double val = static_cast(bytes) / std::pow(1024, i);
+ std::ostringstream oss;
+ oss << std::fixed << std::setprecision(2) << val << " " << units[i];
+ return oss.str();
+}
+
+static std::vector DiscoverSegmentsFromMaster(
+ const std::string& master_host, int master_admin_port) {
+ std::vector segments;
+
+ int sockfd = socket(AF_INET, SOCK_STREAM, 0);
+ if (sockfd < 0) {
+ LOG(ERROR) << "Failed to create socket for discovering segments";
+ return segments;
+ }
+
+ struct timeval timeout;
+ timeout.tv_sec = 5;
+ timeout.tv_usec = 0;
+ setsockopt(sockfd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout));
+ setsockopt(sockfd, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout));
+
+ struct sockaddr_in addr;
+ memset(&addr, 0, sizeof(addr));
+ addr.sin_family = AF_INET;
+ addr.sin_port = htons(master_admin_port);
+
+ std::string host = master_host;
+ size_t colon_pos = host.find(':');
+ if (colon_pos != std::string::npos) {
+ host = host.substr(0, colon_pos);
+ }
+
+ if (inet_pton(AF_INET, host.c_str(), &addr.sin_addr) <= 0) {
+ LOG(ERROR) << "Invalid master host: " << host;
+ close(sockfd);
+ return segments;
+ }
+
+ if (connect(sockfd, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
+ LOG(ERROR) << "Failed to connect to master admin at " << host << ":"
+ << master_admin_port;
+ close(sockfd);
+ return segments;
+ }
+
+ std::string request = "GET /get_all_segments HTTP/1.0\r\nHost: " + host +
+ "\r\nConnection: close\r\n\r\n";
+ if (send(sockfd, request.c_str(), request.size(), 0) < 0) {
+ LOG(ERROR) << "Failed to send HTTP request to master";
+ close(sockfd);
+ return segments;
+ }
+
+ std::string response;
+ char buf[4096];
+ ssize_t n;
+ while ((n = recv(sockfd, buf, sizeof(buf), 0)) > 0) {
+ response.append(buf, n);
+ }
+ close(sockfd);
+
+ size_t header_end = response.find("\r\n\r\n");
+ if (header_end == std::string::npos) {
+ LOG(ERROR) << "Invalid HTTP response from master";
+ return segments;
+ }
+
+ std::string header = response.substr(0, header_end);
+ size_t status_pos = header.find(' ');
+ if (status_pos == std::string::npos) {
+ LOG(ERROR) << "Invalid HTTP response header from master";
+ return segments;
+ }
+ size_t status_code_start = status_pos + 1;
+ size_t status_code_end = header.find(' ', status_code_start);
+ if (status_code_end == std::string::npos) {
+ LOG(ERROR) << "Invalid HTTP status line from master";
+ return segments;
+ }
+ std::string status_code =
+ header.substr(status_code_start, status_code_end - status_code_start);
+ if (status_code != "200") {
+ LOG(ERROR) << "HTTP request failed with status " << status_code
+ << " from master at " << master_host << ":"
+ << master_admin_port;
+ return segments;
+ }
+
+ std::string body = response.substr(header_end + 4);
+ std::istringstream iss(body);
+ std::string line;
+ // Use an unordered_set to track already-seen segments and avoid duplicates
+ std::unordered_set seen;
+ while (std::getline(iss, line)) {
+ while (!line.empty() && (line.back() == '\r' || line.back() == '\n' ||
+ line.back() == ' ' || line.back() == '\t')) {
+ line.pop_back();
+ }
+ if (!line.empty() && seen.insert(line).second) {
+ segments.push_back(line);
+ }
+ }
+
+ return segments;
+}
+} // namespace
+
+DEFINE_string(local_hostname, "localhost",
+ "Local hostname (with optional port, e.g. node1:12345)");
+DEFINE_string(metadata_server, "http://127.0.0.1:8080/metadata",
+ "Metadata server URL");
+DEFINE_string(master_server, "127.0.0.1:50051", "Master server address");
+DEFINE_string(protocol, "tcp", "Transport protocol: tcp, rdma, ub");
+DEFINE_string(device_name, "", "RDMA/UB device name (comma-separated)");
+DEFINE_uint64(global_segment_size, 4 * GB, "Global segment size in bytes");
+DEFINE_uint64(local_buffer_size, 512 * MB, "Local buffer size in bytes");
+DEFINE_bool(enable_ssd_offload, false, "Enable SSD offload on this client");
+DEFINE_string(ssd_offload_path, "", "SSD offload directory path");
+
+DEFINE_string(scenario, "local_memory",
+ "Benchmark scenario: local_memory, remote_memory, local_disk, "
+ "remote_disk, segment_write, segment_read");
+DEFINE_string(role, "writer",
+ "Node role: writer (prefill data) or reader (benchmark reads)");
+DEFINE_uint64(value_size, 4 * MB, "Size of each value in bytes");
+DEFINE_uint64(num_keys, 100, "Number of keys to write/read");
+DEFINE_uint64(batch_size, 32, "Batch size for put/get operations");
+DEFINE_uint64(num_threads, 1, "Number of concurrent reader threads");
+DEFINE_uint64(warmup_keys, 5, "Number of warmup keys (not counted in stats)");
+DEFINE_uint64(wait_seconds, 5,
+ "Seconds to wait before reading (for remote scenarios)");
+DEFINE_uint64(client_init_wait_seconds, 0,
+ "Seconds to wait after RealClient setup succeeds and before "
+ "benchmark run starts. This can give asynchronous RPC/transfer "
+ "warmup and background initialization time to settle.");
+DEFINE_bool(verify, true, "Verify data integrity after read");
+DEFINE_uint64(replica_num, 1, "Number of replicas for each object");
+DEFINE_bool(hard_pin, false,
+ "Pin objects to prevent eviction during benchmark");
+
+DEFINE_string(segments, "",
+ "Comma-separated segment names for segment_write/segment_read "
+ "scenarios. Use segment 'name' (typically hostname), NOT "
+ "IP:port. Leave empty to auto-discover from master.");
+DEFINE_uint64(master_admin_port, 9003,
+ "Master admin HTTP port for auto-discovering segments");
+DEFINE_uint64(read_segment_nums, 0,
+ "Number of segments to read from in segment_read scenario (0 = "
+ "read from all segments)");
+DEFINE_uint64(duration, 0,
+ "Duration in seconds for continuous reading in segment_read "
+ "scenario (0 = read num_keys once)");
+DEFINE_uint64(statis_interval, 5,
+ "Statistics print interval in seconds for segment_read scenario");
+DEFINE_uint64(shuffle_seed, 0,
+ "Seed for deterministic shuffle of segment_read keys "
+ "(0 = disabled)");
+
+using Clock = std::chrono::steady_clock;
+using Nanos = std::chrono::nanoseconds;
+
+inline int64_t ElapsedNanos(Clock::time_point t0, Clock::time_point t1) {
+ return std::chrono::duration_cast(t1 - t0).count();
+}
+
+inline double NanosToUs(int64_t ns) { return static_cast(ns) / 1000.0; }
+inline double NanosToMs(int64_t ns) {
+ return static_cast(ns) / 1000000.0;
+}
+inline double NanosToSec(int64_t ns) { return static_cast(ns) / 1e9; }
+
+struct ThreadResult {
+ std::vector latencies_ns; // per-query latency
+ size_t total_bytes = 0;
+ size_t total_keys = 0; // number of keys processed
+ size_t total_queries =
+ 0; // number of API calls (get_into / batch_get_into)
+ size_t failed_ops = 0;
+};
+
+class BenchmarkStats {
+ public:
+ void InitThreads(size_t n, size_t expected_per_thread) {
+ thread_results_.resize(n);
+ expected_per_thread_ = expected_per_thread;
+ }
+
+ ThreadResult& GetThreadResult(size_t tid) { return thread_results_[tid]; }
+
+ void StartTimer() { start_ = Clock::now(); }
+ void StopTimer() { end_ = Clock::now(); }
+
+ double WallSeconds() const {
+ return NanosToSec(ElapsedNanos(start_, end_));
+ }
+
+ void Finalize() {
+ merged_latencies_ns_.clear();
+ total_bytes_ = 0;
+ total_keys_ = 0;
+ total_queries_ = 0;
+ total_failed_ = 0;
+
+ for (auto& tr : thread_results_) {
+ merged_latencies_ns_.insert(merged_latencies_ns_.end(),
+ tr.latencies_ns.begin(),
+ tr.latencies_ns.end());
+ total_bytes_ += tr.total_bytes;
+ total_keys_ += tr.total_keys;
+ total_queries_ += tr.total_queries;
+ total_failed_ += tr.failed_ops;
+ }
+ std::sort(merged_latencies_ns_.begin(), merged_latencies_ns_.end());
+ }
+
+ double PercentileUs(double p) const {
+ if (merged_latencies_ns_.empty()) return 0.0;
+ double rank = (p / 100.0) * (merged_latencies_ns_.size() - 1);
+ size_t lo = static_cast(rank);
+ size_t hi = std::min(lo + 1, merged_latencies_ns_.size() - 1);
+ double frac = rank - lo;
+ int64_t ns_val =
+ static_cast(merged_latencies_ns_[lo] * (1.0 - frac) +
+ merged_latencies_ns_[hi] * frac);
+ return NanosToUs(ns_val);
+ }
+
+ double MeanLatencyUs() const {
+ if (merged_latencies_ns_.empty()) return 0.0;
+ double sum = static_cast(
+ std::accumulate(merged_latencies_ns_.begin(),
+ merged_latencies_ns_.end(), int64_t(0)));
+ return NanosToUs(sum /
+ static_cast(merged_latencies_ns_.size()));
+ }
+
+ double ThroughputMBps() const {
+ double wall = WallSeconds();
+ return (wall > 0) ? (static_cast(total_bytes_) / MB) / wall : 0;
+ }
+
+ double KeysPerSec() const {
+ double wall = WallSeconds();
+ return (wall > 0) ? static_cast(total_keys_) / wall : 0;
+ }
+
+ double QueriesPerSec() const {
+ double wall = WallSeconds();
+ return (wall > 0) ? static_cast(total_queries_) / wall : 0;
+ }
+
+ void Print(const std::string& title) const {
+ std::cout << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << " " << title << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << std::fixed << std::setprecision(2);
+
+ double wall = WallSeconds();
+ std::cout << " Wall time: " << wall << " s\n";
+ std::cout << " Total queries: " << total_queries_
+ << " (failed: " << total_failed_ << ")\n";
+ std::cout << " Total keys: " << total_keys_ << "\n";
+ std::cout << " Total data: " << FormatBytes(total_bytes_)
+ << "\n";
+ std::cout << " Throughput: " << ThroughputMBps() << " MB/s";
+ if (ThroughputMBps() > 1024) {
+ std::cout << " (" << ThroughputMBps() / 1024 << " GB/s)";
+ }
+ std::cout << "\n";
+ std::cout << " Keys/sec: " << KeysPerSec() << "\n";
+ std::cout << " Queries/sec: " << QueriesPerSec() << "\n";
+
+ if (!merged_latencies_ns_.empty()) {
+ size_t n = merged_latencies_ns_.size();
+ std::cout << "\n Latency (us) [n=" << n << ", per-query]\n";
+ std::cout << " Min: " << std::setw(12)
+ << NanosToUs(merged_latencies_ns_.front()) << "\n";
+ std::cout << " Avg: " << std::setw(12) << MeanLatencyUs()
+ << "\n";
+ std::cout << " P50: " << std::setw(12) << PercentileUs(50)
+ << "\n";
+ std::cout << " P90: " << std::setw(12) << PercentileUs(90)
+ << "\n";
+ std::cout << " P99: " << std::setw(12) << PercentileUs(99);
+ if (n < 100) std::cout << " (n<100)";
+ std::cout << "\n";
+ std::cout << " P999: " << std::setw(12) << PercentileUs(99.9);
+ if (n < 1000) std::cout << " (n<1000)";
+ std::cout << "\n";
+ std::cout << " Max: " << std::setw(12)
+ << NanosToUs(merged_latencies_ns_.back()) << "\n";
+ }
+ std::cout << "========================================"
+ << "========================================\n\n";
+ }
+
+ size_t total_bytes() const { return total_bytes_; }
+ size_t total_keys() const { return total_keys_; }
+ size_t total_queries() const { return total_queries_; }
+ size_t total_failed() const { return total_failed_; }
+
+ private:
+ std::vector thread_results_;
+ std::vector merged_latencies_ns_;
+ size_t total_bytes_ = 0;
+ size_t total_keys_ = 0;
+ size_t total_queries_ = 0;
+ size_t total_failed_ = 0;
+ size_t expected_per_thread_ = 0;
+ Clock::time_point start_;
+ Clock::time_point end_;
+};
+
+class StressBenchmark {
+ public:
+ StressBenchmark()
+ : client_(mooncake::RealClient::create()),
+ buffer_(nullptr),
+ buffer_size_(0) {}
+
+ ~StressBenchmark() {
+ // Early return if already cleaned up
+ if (!client_) {
+ return;
+ }
+
+ // Unregister and free thread buffers (these are allocated by this
+ // class)
+ for (auto& tb : thread_buffers_) {
+ if (tb.ptr) {
+ try {
+ client_->unregister_buffer(tb.ptr);
+ } catch (...) {
+ LOG(WARNING)
+ << "Failed to unregister thread buffer, ignoring";
+ }
+ numa_free(tb.ptr, tb.size);
+ tb.ptr = nullptr;
+ }
+ }
+ thread_buffers_.clear();
+
+ // Unregister and free main buffer (allocated by this class)
+ if (buffer_) {
+ try {
+ client_->unregister_buffer(buffer_);
+ } catch (...) {
+ LOG(WARNING) << "Failed to unregister main buffer, ignoring";
+ }
+ numa_free(buffer_, buffer_size_);
+ buffer_ = nullptr;
+ }
+ client_ = nullptr;
+ }
+
+ int Setup() {
+ int ret = client_->setup_real(
+ FLAGS_local_hostname, FLAGS_metadata_server,
+ FLAGS_global_segment_size, FLAGS_local_buffer_size, FLAGS_protocol,
+ FLAGS_device_name, FLAGS_master_server, nullptr, "",
+ FLAGS_enable_ssd_offload, FLAGS_ssd_offload_path);
+ if (ret != 0) {
+ LOG(ERROR) << "RealClient setup_real failed, ret=" << ret;
+ return ret;
+ }
+ LOG(INFO) << "RealClient setup succeeded"
+ << (FLAGS_enable_ssd_offload ? " (SSD offload enabled)" : "");
+
+ buffer_size_ = FLAGS_batch_size * FLAGS_value_size;
+ buffer_ = reinterpret_cast(numa_alloc_local(buffer_size_));
+ if (!buffer_) {
+ LOG(ERROR) << "Failed to allocate buffer of " << buffer_size_
+ << " bytes";
+ return -1;
+ }
+ std::memset(buffer_, 0, buffer_size_);
+
+ ret = client_->register_buffer(buffer_, buffer_size_);
+ if (ret != 0) {
+ LOG(ERROR) << "register_buffer failed, ret=" << ret;
+ return ret;
+ }
+ LOG(INFO) << "Registered buffer of " << buffer_size_ / MB << " MB";
+ return 0;
+ }
+
+ int RunWriter() {
+ LOG(INFO) << "=== WRITER MODE ===";
+ LOG(INFO) << "Writing " << FLAGS_num_keys << " keys, each "
+ << FLAGS_value_size / MB << " MB";
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ size_t written = 0;
+ size_t failed = 0;
+
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+
+ auto t0 = Clock::now();
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ auto t1 = Clock::now();
+
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key
+ << " ret=" << ret;
+ ++failed;
+ continue;
+ }
+ ++written;
+
+ if ((i + 1) % 10 == 0 || i == FLAGS_num_keys - 1) {
+ double elapsed_us = NanosToUs(ElapsedNanos(t0, t1));
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys
+ << " last_latency=" << elapsed_us << " us";
+ }
+ }
+
+ LOG(INFO) << "Write complete: " << written << " succeeded, " << failed
+ << " failed";
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for reader to connect...";
+ std::this_thread::sleep_for(std::chrono::seconds(FLAGS_wait_seconds));
+
+ return (failed > 0) ? -1 : 0;
+ }
+
+ int RunReader() {
+ LOG(INFO) << "=== READER MODE ===";
+ LOG(INFO) << "Scenario: " << FLAGS_scenario;
+ LOG(INFO) << "Reading " << FLAGS_num_keys << " keys with "
+ << FLAGS_num_threads
+ << " threads, batch_size=" << FLAGS_batch_size;
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ if (FLAGS_scenario == "remote_memory" ||
+ FLAGS_scenario == "remote_disk") {
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for writer to finish prefill...";
+ std::this_thread::sleep_for(
+ std::chrono::seconds(FLAGS_wait_seconds));
+ }
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+ system("spdiag clear");
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ auto threads = LaunchReadWorkers(
+ FLAGS_num_threads, FLAGS_num_keys, stats, start_latch, done_latch,
+ [](size_t idx) { return MakeKey(idx); });
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+
+ std::string title = "READ BENCHMARK [" + FLAGS_scenario + "]";
+ stats.Print(title);
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ if (v != 0) {
+ LOG(ERROR) << "Data verification FAILED";
+ } else {
+ LOG(INFO) << "Data verification PASSED";
+ }
+ }
+
+ return 0;
+ }
+
+ int RunLocalMemory() {
+ LOG(INFO) << "=== LOCAL MEMORY BENCHMARK ===";
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ LOG(INFO) << "Phase 1: Writing " << FLAGS_num_keys << " keys...";
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key;
+ return ret;
+ }
+ if ((i + 1) % 50 == 0) {
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys;
+ }
+ }
+ LOG(INFO) << "Write phase complete";
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+
+ LOG(INFO) << "Phase 2: Concurrent reads with " << FLAGS_num_threads
+ << " threads";
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ auto threads = LaunchReadWorkers(
+ FLAGS_num_threads, FLAGS_num_keys, stats, start_latch, done_latch,
+ [](size_t idx) { return MakeKey(idx); });
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+ stats.Print("LOCAL MEMORY READ BENCHMARK");
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ LOG_IF(INFO, v == 0) << "Data verification PASSED";
+ LOG_IF(ERROR, v != 0) << "Data verification FAILED";
+ }
+
+ return 0;
+ }
+
+ int RunLocalDisk() {
+ LOG(INFO) << "=== LOCAL DISK BENCHMARK ===";
+ LOG(INFO) << "NOTE: Disk reads require Master with enable_offload=true "
+ << "and client with enable_ssd_offload=true";
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ LOG(INFO) << "Phase 1: Writing " << FLAGS_num_keys
+ << " keys (data may be offloaded to SSD)...";
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key;
+ return ret;
+ }
+ if ((i + 1) % 50 == 0) {
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys;
+ }
+ }
+ LOG(INFO) << "Write phase complete";
+
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for offload/eviction to complete...";
+ std::this_thread::sleep_for(std::chrono::seconds(FLAGS_wait_seconds));
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+
+ LOG(INFO) << "Phase 2: Concurrent disk reads with " << FLAGS_num_threads
+ << " threads";
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ auto threads = LaunchReadWorkers(
+ FLAGS_num_threads, FLAGS_num_keys, stats, start_latch, done_latch,
+ [](size_t idx) { return MakeKey(idx); });
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+ stats.Print("LOCAL DISK READ BENCHMARK");
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ LOG_IF(INFO, v == 0) << "Data verification PASSED";
+ LOG_IF(ERROR, v != 0) << "Data verification FAILED";
+ }
+
+ return 0;
+ }
+
+ static std::vector ParseSegments() {
+ std::vector segments;
+ std::istringstream iss(FLAGS_segments);
+ std::string seg;
+ while (std::getline(iss, seg, ',')) {
+ size_t start = seg.find_first_not_of(" \t");
+ size_t end = seg.find_last_not_of(" \t");
+ if (start != std::string::npos && end != std::string::npos) {
+ segments.push_back(seg.substr(start, end - start + 1));
+ }
+ }
+ return segments;
+ }
+
+ static std::string MakeSegmentKey(const std::string& segment, size_t idx) {
+ static const char* kSpecialChars = ".:-/\\[]{}()@#$%^&*+=|<>,;!?`'\"~";
+ std::string sanitized = segment;
+ for (char& c : sanitized) {
+ if (std::strchr(kSpecialChars, c) != nullptr || std::isspace(c)) {
+ c = '_';
+ }
+ }
+ return "seg_" + sanitized + "_key_" + std::to_string(idx);
+ }
+
+ int RunSegmentWrite() {
+ auto segments = DiscoverSegmentsIfNeeded(
+ "--segments not specified, auto-discovering");
+ if (segments.empty()) {
+ return -1;
+ }
+ LOG(INFO) << "Discovered " << segments.size()
+ << " segments from master";
+
+ LOG(INFO) << "=== SEGMENT WRITE MODE ===";
+ LOG(INFO) << "Writing to " << segments.size() << " segments, "
+ << FLAGS_num_keys << " keys per segment (interleaved), each "
+ << FLAGS_value_size / MB << " MB";
+
+ std::vector seg_written(segments.size(), 0);
+ std::vector seg_failed(segments.size(), 0);
+ std::vector configs(segments.size());
+ for (size_t s = 0; s < segments.size(); ++s) {
+ configs[s].replica_num = FLAGS_replica_num;
+ configs[s].with_hard_pin = FLAGS_hard_pin;
+ configs[s].preferred_segments = {segments[s]};
+ }
+
+ size_t total_written = 0;
+ size_t total_failed = 0;
+
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ for (size_t s = 0; s < segments.size(); ++s) {
+ const auto& segment = segments[s];
+ std::string key = MakeSegmentKey(segment, i);
+ FillBuffer(i);
+
+ auto t0 = Clock::now();
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size,
+ configs[s]);
+ auto t1 = Clock::now();
+
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key
+ << " segment=" << segment << " ret=" << ret;
+ ++seg_failed[s];
+ continue;
+ }
+ ++seg_written[s];
+ }
+
+ if ((i + 1) % 10 == 0 || i == FLAGS_num_keys - 1) {
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys
+ << " keys to all " << segments.size() << " segments";
+ }
+ }
+
+ for (size_t s = 0; s < segments.size(); ++s) {
+ total_written += seg_written[s];
+ total_failed += seg_failed[s];
+ LOG(INFO) << "Segment [" << s << "] " << segments[s]
+ << " complete: " << seg_written[s] << " succeeded, "
+ << seg_failed[s] << " failed";
+ }
+
+ LOG(INFO) << "All segments write complete: " << total_written
+ << " succeeded, " << total_failed << " failed";
+
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for reader to connect...";
+ std::this_thread::sleep_for(std::chrono::seconds(FLAGS_wait_seconds));
+
+ return (total_failed > 0) ? -1 : 0;
+ }
+
+ int RunSegmentRead() {
+ auto segments = DiscoverSegmentsIfNeeded(
+ "--segments not specified, auto-discovering");
+ if (segments.empty()) {
+ return -1;
+ }
+ LOG(INFO) << "Discovered " << segments.size()
+ << " segments from master";
+
+ size_t read_segment_nums = FLAGS_read_segment_nums;
+ if (read_segment_nums == 0 || read_segment_nums > segments.size()) {
+ read_segment_nums = segments.size();
+ }
+
+ std::vector read_segments(
+ segments.begin(), segments.begin() + read_segment_nums);
+
+ LOG(INFO) << "=== SEGMENT READ MODE ===";
+ LOG(INFO) << "Reading from " << read_segment_nums << " segments ("
+ << read_segment_nums << " nodes)";
+ for (size_t s = 0; s < read_segments.size(); ++s) {
+ LOG(INFO) << " Segment [" << s << "]: " << read_segments[s];
+ }
+ LOG(INFO) << "Keys per segment: " << FLAGS_num_keys;
+ LOG(INFO) << "Duration: "
+ << (FLAGS_duration > 0 ? std::to_string(FLAGS_duration) + "s"
+ : "single pass");
+ LOG(INFO) << "Stats interval: " << FLAGS_statis_interval << "s";
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ std::vector all_keys;
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ for (size_t s = 0; s < read_segments.size(); ++s) {
+ all_keys.push_back(MakeSegmentKey(read_segments[s], i));
+ }
+ }
+ if (FLAGS_shuffle_seed != 0) {
+ std::mt19937_64 rng(FLAGS_shuffle_seed);
+ std::shuffle(all_keys.begin(), all_keys.end(), rng);
+ LOG(INFO) << "Shuffled segment_read keys with seed="
+ << FLAGS_shuffle_seed;
+ }
+ LOG(INFO) << "Total keys to read: " << all_keys.size();
+
+ size_t warmup_end =
+ std::min(static_cast(FLAGS_warmup_keys), all_keys.size());
+ if (warmup_end > 0) {
+ LOG(INFO) << "Warmup: reading " << warmup_end << " keys...";
+ for (size_t i = 0; i < warmup_end; ++i) {
+ int64_t ret =
+ client_->get_into(all_keys[i], buffer_, FLAGS_value_size);
+ if (ret < 0) {
+ LOG(WARNING)
+ << "Warmup get_into failed for key=" << all_keys[i]
+ << " ret=" << ret;
+ }
+ }
+ LOG(INFO) << "Warmup complete";
+ }
+
+ if (FLAGS_duration == 0) {
+ return RunSegmentReadSinglePass(read_segments, all_keys);
+ }
+ return RunSegmentReadDuration(read_segments, all_keys);
+ }
+
+ int RunSegmentReadSinglePass(const std::vector& read_segments,
+ const std::vector& all_keys) {
+ LOG(INFO) << "Single-pass read with " << FLAGS_num_threads
+ << " threads";
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ all_keys.size() / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ auto threads =
+ LaunchReadWorkers(FLAGS_num_threads, all_keys.size(), stats,
+ start_latch, done_latch, [&all_keys](size_t idx) {
+ return all_keys[idx % all_keys.size()];
+ });
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+
+ std::string title = "SEGMENT READ BENCHMARK [segments=" +
+ std::to_string(read_segments.size()) + "]";
+ stats.Print(title);
+ return 0;
+ }
+
+ struct IntervalLatencyStats {
+ std::vector latencies_ns;
+ int64_t min_latency_ns = std::numeric_limits::max();
+ int64_t max_latency_ns = 0;
+ double p50_latency_ns = 0;
+ double p90_latency_ns = 0;
+ double p99_latency_ns = 0;
+ double p999_latency_ns = 0;
+ double p9999_latency_ns = 0;
+ double avg_latency_ns = 0;
+ double throughput_mbps = 0;
+ double keys_per_sec = 0;
+ double queries_per_sec = 0;
+
+ void Finalize() {
+ if (latencies_ns.empty()) return;
+ std::sort(latencies_ns.begin(), latencies_ns.end());
+ min_latency_ns = latencies_ns.front();
+ max_latency_ns = latencies_ns.back();
+ size_t n = latencies_ns.size();
+ avg_latency_ns =
+ std::accumulate(latencies_ns.begin(), latencies_ns.end(), 0.0) /
+ n;
+ auto percentile = [&](double p) -> double {
+ if (n == 0) return 0;
+ size_t idx = static_cast(p / 100.0 * (n - 1));
+ return static_cast(latencies_ns[idx]);
+ };
+ p50_latency_ns = percentile(50);
+ p90_latency_ns = percentile(90);
+ if (n >= 100) {
+ p99_latency_ns = latencies_ns[static_cast(n * 0.99)];
+ }
+ if (n >= 1000) {
+ p999_latency_ns = latencies_ns[static_cast(n * 0.999)];
+ }
+ if (n >= 10000) {
+ p9999_latency_ns =
+ latencies_ns[static_cast(n * 0.9999)];
+ }
+ }
+
+ void Aggregate(const IntervalLatencyStats& other) {
+ min_latency_ns = std::min(min_latency_ns, other.min_latency_ns);
+ max_latency_ns = std::max(max_latency_ns, other.max_latency_ns);
+ p50_latency_ns = std::max(p50_latency_ns, other.p50_latency_ns);
+ p90_latency_ns = std::max(p90_latency_ns, other.p90_latency_ns);
+ p99_latency_ns = std::max(p99_latency_ns, other.p99_latency_ns);
+ p999_latency_ns = std::max(p999_latency_ns, other.p999_latency_ns);
+ p9999_latency_ns =
+ std::max(p9999_latency_ns, other.p9999_latency_ns);
+ throughput_mbps += other.throughput_mbps;
+ keys_per_sec += other.keys_per_sec;
+ queries_per_sec += other.queries_per_sec;
+ // Weighted average: accumulate sum and count
+ total_latency_sum_ns +=
+ other.avg_latency_ns *
+ static_cast(other.latencies_ns.size());
+ total_samples += other.latencies_ns.size();
+ if (total_samples > 0) {
+ avg_latency_ns =
+ total_latency_sum_ns / static_cast(total_samples);
+ }
+ }
+
+ size_t total_samples = 0;
+ double total_latency_sum_ns = 0;
+ };
+
+ int RunSegmentReadDuration(const std::vector& read_segments,
+ const std::vector& all_keys) {
+ LOG(INFO) << "Duration-based continuous read with " << FLAGS_num_threads
+ << " threads for " << FLAGS_duration << "s, stats every "
+ << FLAGS_statis_interval << "s";
+
+ std::atomic stop_flag{false};
+ std::atomic global_keys{0};
+ std::atomic global_queries{0};
+ std::atomic global_bytes{0};
+ std::atomic global_failed{0};
+
+ std::vector> thread_latencies(FLAGS_num_threads);
+ std::vector latency_mutexes(FLAGS_num_threads);
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::vector threads;
+
+ size_t total_keys = all_keys.size();
+ size_t keys_per_thread =
+ (total_keys + FLAGS_num_threads - 1) / FLAGS_num_threads;
+
+ for (size_t t = 0; t < FLAGS_num_threads; ++t) {
+ threads.emplace_back([&, t, keys_per_thread, total_keys]() {
+ bindToSocket(t % NR_SOCKETS);
+ char* my_buf = thread_buffers_[t].ptr;
+
+ start_latch.arrive_and_wait();
+
+ size_t key_offset = t * keys_per_thread;
+ size_t key_idx = key_offset;
+
+ if (FLAGS_batch_size <= 1) {
+ while (!stop_flag.load(std::memory_order_relaxed)) {
+ const std::string& key = all_keys[key_idx % total_keys];
+ auto t0 = Clock::now();
+ int64_t ret =
+ client_->get_into(key, my_buf, FLAGS_value_size);
+ auto t1 = Clock::now();
+ int64_t latency_ns = ElapsedNanos(t0, t1);
+
+ {
+ std::lock_guard lock(
+ latency_mutexes[t]);
+ thread_latencies[t].push_back(latency_ns);
+ }
+
+ if (ret < 0) {
+ global_failed.fetch_add(1,
+ std::memory_order_relaxed);
+ } else {
+ global_bytes.fetch_add(static_cast(ret),
+ std::memory_order_relaxed);
+ }
+ global_keys.fetch_add(1, std::memory_order_relaxed);
+ global_queries.fetch_add(1, std::memory_order_relaxed);
+ ++key_idx;
+ }
+ } else {
+ size_t per_key_buf = FLAGS_value_size;
+ while (!stop_flag.load(std::memory_order_relaxed)) {
+ std::vector keys;
+ std::vector bufs;
+ std::vector sizes;
+ keys.reserve(FLAGS_batch_size);
+ bufs.reserve(FLAGS_batch_size);
+ sizes.reserve(FLAGS_batch_size);
+
+ for (size_t b = 0; b < FLAGS_batch_size; ++b) {
+ const std::string& key =
+ all_keys[key_idx % total_keys];
+ keys.push_back(key);
+ bufs.push_back(my_buf + b * per_key_buf);
+ sizes.push_back(FLAGS_value_size);
+ ++key_idx;
+ }
+
+ auto t0 = Clock::now();
+ auto results =
+ client_->batch_get_into(keys, bufs, sizes);
+ auto t1 = Clock::now();
+ int64_t latency_ns = ElapsedNanos(t0, t1);
+
+ {
+ std::lock_guard lock(
+ latency_mutexes[t]);
+ thread_latencies[t].push_back(latency_ns);
+ }
+
+ for (size_t k = 0; k < results.size(); ++k) {
+ if (results[k] < 0) {
+ global_failed.fetch_add(
+ 1, std::memory_order_relaxed);
+ } else {
+ global_bytes.fetch_add(
+ static_cast(results[k]),
+ std::memory_order_relaxed);
+ }
+ global_keys.fetch_add(1, std::memory_order_relaxed);
+ }
+ global_queries.fetch_add(1, std::memory_order_relaxed);
+ }
+ }
+ });
+ }
+
+ auto bench_start = Clock::now();
+ auto bench_end = bench_start + std::chrono::seconds(FLAGS_duration);
+ auto next_statis =
+ bench_start + std::chrono::seconds(FLAGS_statis_interval);
+
+ size_t prev_keys = 0;
+ size_t prev_queries = 0;
+ size_t prev_bytes = 0;
+ size_t prev_failed = 0;
+ auto prev_time = bench_start;
+
+ std::vector interval_stats_list;
+
+ std::cout << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << " SEGMENT READ DURATION BENCHMARK [segments="
+ << read_segments.size() << "]\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << std::fixed << std::setprecision(2);
+
+ while (Clock::now() < bench_end) {
+ auto now = Clock::now();
+ if (now >= next_statis) {
+ size_t cur_keys = global_keys.load(std::memory_order_relaxed);
+ size_t cur_queries =
+ global_queries.load(std::memory_order_relaxed);
+ size_t cur_bytes = global_bytes.load(std::memory_order_relaxed);
+ size_t cur_failed =
+ global_failed.load(std::memory_order_relaxed);
+
+ double interval_sec = NanosToSec(ElapsedNanos(prev_time, now));
+ size_t interval_keys = cur_keys - prev_keys;
+ size_t interval_queries = cur_queries - prev_queries;
+ size_t interval_bytes = cur_bytes - prev_bytes;
+ size_t interval_failed = cur_failed - prev_failed;
+
+ double interval_throughput_mbps =
+ (interval_sec > 0)
+ ? (static_cast(interval_bytes) / MB) /
+ interval_sec
+ : 0;
+ double interval_keys_per_sec =
+ (interval_sec > 0)
+ ? static_cast(interval_keys) / interval_sec
+ : 0;
+ double interval_queries_per_sec =
+ (interval_sec > 0)
+ ? static_cast(interval_queries) / interval_sec
+ : 0;
+
+ IntervalLatencyStats interval_stats;
+ interval_stats.throughput_mbps = interval_throughput_mbps;
+ interval_stats.keys_per_sec = interval_keys_per_sec;
+ interval_stats.queries_per_sec = interval_queries_per_sec;
+ for (size_t t = 0; t < FLAGS_num_threads; ++t) {
+ std::lock_guard lock(latency_mutexes[t]);
+ interval_stats.latencies_ns.insert(
+ interval_stats.latencies_ns.end(),
+ thread_latencies[t].begin(), thread_latencies[t].end());
+ thread_latencies[t].clear();
+ }
+ interval_stats.Finalize();
+ interval_stats_list.push_back(interval_stats);
+
+ double total_sec = NanosToSec(ElapsedNanos(bench_start, now));
+ double total_throughput_mbps =
+ (total_sec > 0)
+ ? (static_cast(cur_bytes) / MB) / total_sec
+ : 0;
+ double total_keys_per_sec =
+ (total_sec > 0) ? static_cast(cur_keys) / total_sec
+ : 0;
+ double total_queries_per_sec =
+ (total_sec > 0)
+ ? static_cast(cur_queries) / total_sec
+ : 0;
+
+ std::cout << " [t=" << std::setw(6) << total_sec << "s]"
+ << " interval: " << interval_throughput_mbps
+ << " MB/s, " << interval_keys_per_sec << " keys/s, "
+ << interval_queries_per_sec << " qps"
+ << " (failed=" << interval_failed << ")"
+ << " lat[us]: avg="
+ << NanosToUs(interval_stats.avg_latency_ns)
+ << ", P50="
+ << NanosToUs(interval_stats.p50_latency_ns)
+ << ", P90="
+ << NanosToUs(interval_stats.p90_latency_ns)
+ << ", P99="
+ << NanosToUs(interval_stats.p99_latency_ns)
+ << " total: " << cur_queries << " queries, "
+ << cur_keys << " keys, " << total_throughput_mbps
+ << " MB/s, " << total_keys_per_sec << " keys/s, "
+ << total_queries_per_sec << " qps"
+ << " (failed=" << cur_failed << ")\n";
+
+ prev_keys = cur_keys;
+ prev_queries = cur_queries;
+ prev_bytes = cur_bytes;
+ prev_failed = cur_failed;
+ prev_time = now;
+ next_statis += std::chrono::seconds(FLAGS_statis_interval);
+ }
+ std::this_thread::sleep_for(std::chrono::milliseconds(100));
+ }
+
+ stop_flag.store(true, std::memory_order_relaxed);
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ auto final_time = Clock::now();
+ double total_sec = NanosToSec(ElapsedNanos(bench_start, final_time));
+ size_t final_keys = global_keys.load(std::memory_order_relaxed);
+ size_t final_queries = global_queries.load(std::memory_order_relaxed);
+ size_t final_bytes = global_bytes.load(std::memory_order_relaxed);
+ size_t final_failed = global_failed.load(std::memory_order_relaxed);
+
+ double final_throughput_mbps =
+ (total_sec > 0)
+ ? (static_cast(final_bytes) / MB) / total_sec
+ : 0;
+ double final_keys_per_sec =
+ (total_sec > 0) ? static_cast(final_keys) / total_sec : 0;
+ double final_queries_per_sec =
+ (total_sec > 0) ? static_cast(final_queries) / total_sec
+ : 0;
+
+ IntervalLatencyStats overall;
+ for (const auto& stats : interval_stats_list) {
+ overall.Aggregate(stats);
+ }
+ double avg_throughput_mbps =
+ !interval_stats_list.empty()
+ ? overall.throughput_mbps / interval_stats_list.size()
+ : 0;
+ size_t total_latency_samples = overall.total_samples;
+
+ std::cout << "\n FINAL SUMMARY\n";
+ std::cout << " Total time: " << total_sec << " s\n";
+ std::cout << " Total queries: " << final_queries
+ << " (failed: " << final_failed << ")\n";
+ std::cout << " Total keys: " << final_keys << "\n";
+ std::cout << " Total data: " << FormatBytes(final_bytes) << "\n";
+ std::cout << " Throughput: " << final_throughput_mbps
+ << " MB/s (avg: " << avg_throughput_mbps << " MB/s)";
+ if (final_throughput_mbps > 1024) {
+ std::cout << " (" << final_throughput_mbps / 1024 << " GB/s)";
+ }
+ std::cout << "\n";
+ std::cout << " Keys/sec: " << final_keys_per_sec << "\n";
+ std::cout << " Queries/sec: " << final_queries_per_sec << "\n";
+
+ if (total_latency_samples > 0) {
+ std::cout << "\n Latency (us) [n=" << total_latency_samples
+ << ", per-query]\n";
+ std::cout << " Min: " << std::setw(12)
+ << NanosToUs(overall.min_latency_ns) << "\n";
+ std::cout << " Avg: " << std::setw(12)
+ << NanosToUs(overall.avg_latency_ns) << "\n";
+ std::cout << " P50: " << std::setw(12)
+ << NanosToUs(overall.p50_latency_ns) << "\n";
+ std::cout << " P90: " << std::setw(12)
+ << NanosToUs(overall.p90_latency_ns) << "\n";
+ std::cout << " P99: " << std::setw(12)
+ << NanosToUs(overall.p99_latency_ns);
+ if (total_latency_samples < 100) std::cout << " (n<100)";
+ std::cout << "\n";
+ std::cout << " P999: " << std::setw(12)
+ << NanosToUs(overall.p999_latency_ns);
+ if (total_latency_samples < 1000) std::cout << " (n<1000)";
+ std::cout << "\n";
+ std::cout << " P9999: " << std::setw(12)
+ << NanosToUs(overall.p9999_latency_ns);
+ if (total_latency_samples < 10000) std::cout << " (n<10000)";
+ std::cout << "\n";
+ std::cout << " Max: " << std::setw(12)
+ << NanosToUs(overall.max_latency_ns) << "\n";
+ }
+
+ std::cout << "========================================"
+ << "========================================\n\n";
+
+ return 0;
+ }
+
+ int RunListSegments() {
+ LOG(INFO) << "Discovering segments from master at "
+ << FLAGS_master_server << ":" << FLAGS_master_admin_port;
+
+ auto segments = DiscoverSegmentsFromMaster(
+ FLAGS_master_server, static_cast(FLAGS_master_admin_port));
+
+ if (segments.empty()) {
+ LOG(ERROR) << "No segments discovered from master. "
+ << "Check master connectivity at " << FLAGS_master_server
+ << ":" << FLAGS_master_admin_port;
+ return -1;
+ }
+
+ std::cout << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << " DISCOVERED SEGMENTS [count=" << segments.size()
+ << "]\n";
+ std::cout << "========================================"
+ << "========================================\n";
+
+ for (size_t i = 0; i < segments.size(); ++i) {
+ std::cout << " [" << std::setw(4) << i << "] " << segments[i]
+ << "\n";
+ }
+
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << " Total segments: " << segments.size() << "\n";
+ std::cout << "========================================"
+ << "========================================\n\n";
+
+ return 0;
+ }
+
+ int Run() {
+ if (FLAGS_scenario == "local_memory") {
+ return RunLocalMemory();
+ } else if (FLAGS_scenario == "local_disk") {
+ return RunLocalDisk();
+ } else if (FLAGS_scenario == "segment_write") {
+ return RunSegmentWrite();
+ } else if (FLAGS_scenario == "segment_read") {
+ return RunSegmentRead();
+ } else if (FLAGS_scenario == "list_segments") {
+ return RunListSegments();
+ } else if (FLAGS_scenario == "remote_memory" ||
+ FLAGS_scenario == "remote_disk") {
+ if (FLAGS_role == "writer") {
+ return RunWriter();
+ } else {
+ return RunReader();
+ }
+ } else {
+ LOG(ERROR) << "Unknown scenario: " << FLAGS_scenario;
+ return -1;
+ }
+ }
+
+ private:
+ static std::string MakeKey(size_t idx) {
+ return "bench_key_" + std::to_string(idx);
+ }
+
+ void FillBuffer(size_t seed) {
+ uint64_t* ptr = reinterpret_cast(buffer_);
+ size_t num_words = FLAGS_value_size / sizeof(uint64_t);
+ uint64_t pattern = static_cast(seed) * 0x9E3779B97F4A7C15ULL;
+ for (size_t w = 0; w < num_words; ++w) {
+ pattern = (pattern ^ (pattern >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ pattern = (pattern ^ (pattern >> 27)) * 0x94D049BB133111EBULL;
+ ptr[w] = pattern ^ (pattern >> 31);
+ }
+ }
+
+ bool CheckBuffer(size_t seed, const void* data, size_t size) const {
+ const uint64_t* ptr = reinterpret_cast(data);
+ size_t num_words = size / sizeof(uint64_t);
+ uint64_t pattern = static_cast(seed) * 0x9E3779B97F4A7C15ULL;
+ for (size_t w = 0; w < num_words; ++w) {
+ pattern = (pattern ^ (pattern >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ pattern = (pattern ^ (pattern >> 27)) * 0x94D049BB133111EBULL;
+ uint64_t expected = pattern ^ (pattern >> 31);
+ if (ptr[w] != expected) {
+ LOG(ERROR) << "Checksum mismatch at word " << w
+ << " for seed=" << seed << " expected=" << std::hex
+ << expected << " got=" << ptr[w] << std::dec;
+ return false;
+ }
+ }
+ return true;
+ }
+
+ int DoWarmup() {
+ if (FLAGS_warmup_keys == 0) return 0;
+ LOG(INFO) << "Warmup: reading " << FLAGS_warmup_keys << " keys...";
+
+ size_t warmup_end = std::min(static_cast(FLAGS_warmup_keys),
+ static_cast(FLAGS_num_keys));
+ for (size_t i = 0; i < warmup_end; ++i) {
+ std::string key = MakeKey(i);
+ int64_t ret = client_->get_into(key, buffer_, FLAGS_value_size);
+ if (ret < 0) {
+ LOG(WARNING) << "Warmup get_into failed for key=" << key
+ << " ret=" << ret;
+ }
+ }
+ LOG(INFO) << "Warmup complete";
+ return 0;
+ }
+
+ void BatchReadWorker(size_t tid, size_t my_keys, size_t key_offset,
+ BenchmarkStats& stats, std::latch& start_latch,
+ std::latch& done_latch,
+ const std::function& key_func) {
+ bindToSocket(tid % NR_SOCKETS);
+
+ ThreadResult& result = stats.GetThreadResult(tid);
+ result.latencies_ns.reserve(my_keys);
+
+ char* my_buf = thread_buffers_[tid].ptr;
+
+ start_latch.arrive_and_wait();
+
+ size_t keys = 0;
+ size_t queries = 0;
+ size_t failed = 0;
+ size_t bytes = 0;
+
+ if (FLAGS_batch_size <= 1) {
+ for (size_t i = 0; i < my_keys; ++i) {
+ size_t key_idx = key_offset + i;
+ std::string key = key_func(key_idx);
+
+ auto t0 = Clock::now();
+ int64_t ret = client_->get_into(key, my_buf, FLAGS_value_size);
+ auto t1 = Clock::now();
+
+ int64_t lat_ns = ElapsedNanos(t0, t1);
+
+ if (ret < 0) {
+ ++failed;
+ LOG_EVERY_N(ERROR, 100)
+ << "get_into failed key=" << key << " ret=" << ret;
+ } else {
+ bytes += static_cast(ret);
+ }
+ result.latencies_ns.push_back(lat_ns);
+ ++keys;
+ ++queries;
+ }
+ } else {
+ size_t per_key_buf = FLAGS_value_size;
+ size_t i = 0;
+ while (i < my_keys) {
+ std::vector key_list;
+ std::vector bufs;
+ std::vector sizes;
+ size_t batch_end = std::min(i + FLAGS_batch_size, my_keys);
+ key_list.reserve(batch_end - i);
+ bufs.reserve(batch_end - i);
+ sizes.reserve(batch_end - i);
+
+ for (size_t j = i; j < batch_end; ++j) {
+ size_t key_idx = key_offset + j;
+ key_list.push_back(key_func(key_idx));
+ bufs.push_back(my_buf + (j - i) * per_key_buf);
+ sizes.push_back(FLAGS_value_size);
+ }
+
+ auto t0 = Clock::now();
+ auto results = client_->batch_get_into(key_list, bufs, sizes);
+ auto t1 = Clock::now();
+
+ int64_t lat_ns = ElapsedNanos(t0, t1);
+ result.latencies_ns.push_back(lat_ns);
+
+ for (size_t k = 0; k < results.size(); ++k) {
+ if (results[k] < 0) {
+ ++failed;
+ } else {
+ bytes += static_cast(results[k]);
+ }
+ ++keys;
+ }
+ ++queries;
+
+ i = batch_end;
+ }
+ }
+
+ result.total_bytes = bytes;
+ result.total_keys = keys;
+ result.total_queries = queries;
+ result.failed_ops = failed;
+
+ done_latch.arrive_and_wait();
+ }
+
+ std::vector LaunchReadWorkers(
+ size_t num_threads, size_t total_keys, BenchmarkStats& stats,
+ std::latch& start_latch, std::latch& done_latch,
+ const std::function& key_func) {
+ std::vector threads;
+ size_t keys_per_thread = total_keys / num_threads;
+ size_t remainder = total_keys % num_threads;
+
+ for (size_t t = 0; t < num_threads; ++t) {
+ size_t my_keys = keys_per_thread + (t < remainder ? 1 : 0);
+ size_t key_offset = t * keys_per_thread + std::min(t, remainder);
+
+ threads.emplace_back([&, t, my_keys, key_offset]() {
+ BatchReadWorker(t, my_keys, key_offset, stats, start_latch,
+ done_latch, key_func);
+ });
+ }
+ return threads;
+ }
+
+ std::vector DiscoverSegmentsIfNeeded(
+ const std::string& context) {
+ auto segments = ParseSegments();
+ if (!segments.empty()) {
+ return segments;
+ }
+
+ LOG(INFO) << context << ", auto-discovering from master at "
+ << FLAGS_master_server << ":" << FLAGS_master_admin_port;
+ segments = DiscoverSegmentsFromMaster(
+ FLAGS_master_server, static_cast(FLAGS_master_admin_port));
+ if (segments.empty()) {
+ LOG(ERROR) << "No segments discovered from master. "
+ << "Check master connectivity.";
+ }
+ return segments;
+ }
+
+ int VerifyData() {
+ LOG(INFO) << "Verifying data integrity for " << FLAGS_num_keys
+ << " keys...";
+ int errors = 0;
+
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ int64_t ret = client_->get_into(key, buffer_, FLAGS_value_size);
+ if (ret < 0) {
+ LOG(ERROR) << "Verify: get_into failed for key=" << key;
+ ++errors;
+ continue;
+ }
+ if (!CheckBuffer(i, buffer_, static_cast(ret))) {
+ LOG(ERROR) << "Verify: data mismatch for key=" << key;
+ ++errors;
+ }
+ }
+
+ LOG(INFO) << "Verification complete: " << errors << " errors out of "
+ << FLAGS_num_keys << " keys";
+ return errors > 0 ? -1 : 0;
+ }
+
+ std::shared_ptr client_;
+ char* buffer_;
+ size_t buffer_size_;
+
+ struct ThreadBuffer {
+ char* ptr = nullptr;
+ size_t size = 0;
+ int numa_node = -1;
+ };
+ std::vector