Feat spdiag logging - #36

Closed

ZhiningPan wants to merge 291 commits into

LinQuickDev:mainfrom

ZhiningPan:feat-spdiag-logging

This pull request is big! We're only showing the most recent 250 commits

Commits on May 26, 2026

Merge branch 'supercache_dev' into ubdiag
yuanhao-426
committed
3adc78c
Merge pull request #5 from yyyuanhao426-hash/ubdiag
zchuango
authored
b50bf0b
feat(allocation): 添加独立的 DDR 准入水位控制
liusiyu60
committed
60ddbcf
docs: 补充存储后端配置参数说明
liusiyu60
committed
41e12b1
Merge branch 'supercache_dev' into ssd_balance
liusiyu60
committed
16b946d
feat(allocation): 新增DDR水位线拒绝分配的错误码
liusiyu60
committed
858e81e
docs: 添加日志配置和异步改造方案文档
 yuanhao-426
committed
ce1c963
fix: 修复 DDR 高水位判断逻辑错误
liusiyu60
committed
0f418c8
Merge pull request #7 from syliudf/ssd_balance
zchuango
authored
8a9bfc9
feat(tests): 添加冷热数据交换机制验证脚本和指南
liusiyu60
committed
d6475cb
docs: update SSD balance allocation design with DDR admission control
liusiyu60
committed
c55d1db
docs: 更新 offload promotion 测试指南
liusiyu60
committed
e8d7f9a
docs(tests): 更新测试指南中的命令以指定master地址
liusiyu60
committed
16d9e4f
test: 确保SSD卸载路径目录存在
liusiyu60
committed
17a0759
test: 分批写入以支持 SSD 卸载和提升测试
liusiyu60
committed
9e71f87

Commits on May 27, 2026

test: 将默认段大小从32MB增加到64MB
liusiyu60
committed
019e35a
docs(tests): 更新 offload promotion 测试为 10 倍规模并调整参数
liusiyu60
committed
2d89f88
feat(logging): 为读写链路添加异步日志、TraceId 和性能监控
 yuanhao-426
committed
eaa76d1
fix(测试): 调整offload/promotion测试参数以应对流水线瓶颈
liusiyu60
committed
da89e61
docs(tests): 更新测试指南
liusiyu60
committed
40cb75b
perf: 集成性能监控与慢操作告警
 yuanhao-426
committed
c6084d2
feat(logging): 细化副本类型日志并更新文档
 yuanhao-426
committed
ecec9ca
refactor(测试): 引入安全清理函数避免段错误
liusiyu60
committed
7aff875

Commits on May 28, 2026

docs(tests): 更新测试指南和脚本以反映实际行为
liusiyu60
committed
dddb030
feat(transfer): 为异步传输任务添加 trace_id 支持
 yuanhao-426
committed
dfb496c
Merge pull request #9 from syliudf/ssd_balance
zchuango
authored
9a16629
Merge branch 'main' into supercache_dev
zchuango
committed
7cb6e76
format and clean code
zchuango
committed
ebe9064
build: 在链接库中添加 ubdiag 依赖
 yuanhao-426
committed
45e4ea8
feat: 添加 trace_id 支持以增强分布式追踪能力
 yuanhao-426
committed
fffd272
update log-reference.md
yuanhao-426
committed
2c79876

Commits on May 29, 2026

add the dummy client for go store
zchuango
committed
0bfae4f

Commits on May 30, 2026

Merge branch 'supercache_dev' into ubdiag
yuanhao-426
committed
6c7fa2e
feat(性能监控): 为传输数据性能点添加操作类型后缀
 yuanhao-426
committed
2ff72ad
fix: 防止重复初始化 Google Logging
yuanhao-426
committed
696bbac
fix: 防止重复初始化 Google Logging
yuanhao-426
committed
a7e3d27
fix: 仅在未设置MC_LOG_DIR环境变量时启用日志输出到标准错误
 yuanhao-426
committed
d430beb
add the segment read/write for stress cluster bench
zchuango
committed
f2197d9
add the autodiscovery segments
zchuango
committed
17334e5

Commits on May 31, 2026

remove some log
zchuango
committed
6b768a7
fix the bug
zchuango
committed
628b0e1
modify the code for coredump
zchuango
committed
f9b3060

Commits on Jun 1, 2026

refine the data for stress bench
zchuango
committed
d94b47f
optimize the code for urma endpoint
zchuango
committed
ff965c1
fix the bug for data
zchuango
committed
e98a784
docs: 补充 URMA 端建连日志的触发时机、顺序和排查指引
 yuanhao-426
committed
52a2890
feat(urma): 支持通过环境变量配置 URMA 传输模式
 yuanhao-426
committed
e3902f6
Merge pull request #10 from yyyuanhao426-hash/urma_mode
zchuango
authored
d4dd370

Commits on Jun 2, 2026

- 在 `UrmaContext` 中缓存传输模式，避免重复解析配置。
yuanhao-426
committed
4429cb6
Merge pull request #11 from yyyuanhao426-hash/urma_mode
zchuango
authored
98212f5
fix the numa affiniaty for segment
zchuango
committed
56b49c8
feat(master): 集成性能打点并优化客户端日志开销
Saturday
committed
d5cf2e9
add the rpm_build
zchuango
committed
d080c58
fix(logging): 修复日志总开关默认行为并统一配置
Saturday
committed
0f5edfe
Merge branch 'supercache_dev' into ubdiag
Saturday
committed
375e6e8
fix: 移除日志中多余的bind_us字段
Saturday
committed
1b079da
fix(urma): 在绑定jetty日志中添加bind_us时间信息
Saturday
committed
6d24ff1
fix the numa affiniaty for segment and add the rpm build script
zchuango
committed
ae4e64a
Merge branch 'supercache_dev' into ubdiag
Saturday
committed
34ff8bf
merge supercache_dev
Saturday
committed
3a33bd5
Merge pull request #12 from yyyuanhao426-hash/ubdiag
zchuango
authored
380bbaf

Commits on Jun 3, 2026

add the urma bonding multipath entable
zchuango
committed
7ac14b5
fix: 移除 MC_LOG_ENABLE 对 FLAGS_minloglevel 的全局影响
Saturday
committed
c7a0a3e
refactor: 统一日志宏从 MC_LOG 改为 LOG
Saturday
committed
ce7b686
Merge pull request #13 from yyyuanhao426-hash/ubdiag
zchuango
authored
a6a480f
add the rpm build script
zchuango
committed
ccf5499
perf(logging): 移除冗余日志以提升性能
Saturday
committed
79295d3
feat(logging): 添加高频 breakdown 日志采样支持
Saturday
committed
78a02e2
Merge pull request #18 from yyyuanhao426-hash/log
zchuango
authored
934e94c

Commits on Jun 4, 2026

test: 为 MemcpyTask 构造函数添加 trace_id 参数
Saturday
committed
79fd242
Merge pull request #19 from yyyuanhao426-hash/log
zchuango
authored
253beb2

Commits on Jun 6, 2026

rollback for store go api
zchuango
committed
3beb861

Commits on Jun 11, 2026

fix: remove stale hot cache on remove

Le1zyCatt
and
wangyuqi0429
committed
bfbbf6b
fix: refine hot cache invalidation on remove

Le1zyCatt
and
wangyuqi0429
committed
3f739f9

Commits on Jun 13, 2026

fix: 防止重复初始化glog并添加客户端主机信息到RPC日志
Saturday
committed
c380d18
optimize the stress cluster bench bug
zchuango
committed
03616e8
add the cuda dependency on store
zchuango
committed
b12de66
refactor(logging): 移除 MC_LOG_ENABLE 环境变量并简化日志控制
Saturday
committed
0481e75
Merge pull request #22 from yyyuanhao426-hash/supercache_log
zchuango
authored
c51bc4d
Merge branch 'main' into supercache_dev
zchuango
committed
ef840d2
format and fix some bug
zchuango
committed
f9371ca

Commits on Jun 15, 2026

Commits on Jun 16, 2026

Merge branch 'supercache_dev' into supercache_log
Saturday
committed
6ccb0a5
Merge branch 'fix-hot-cache-precise-invalidation' into supercache_dev
Le1zyCatt
committed
977fd58
perf(logging): 用环形缓冲区替换队列以提升异步日志性能
Saturday
committed
cc414b4
feat(日志与监控): 为关键操作添加详细性能日志和采样支持
Saturday
committed
3ff8ed9
perf: 添加性能监控点以跟踪关键操作耗时
Saturday
committed
c23d3fd
fix(master): 调整 glog 初始化顺序以正确处理 MC_LOG_DIR
Saturday
committed
01f9498
feat: 为关键操作添加跟踪ID以增强可观测性
Saturday
committed
db09972
fix(master_client): 将硬编码的trace_id替换为实际值
Saturday
committed
eb40c13

Commits on Jun 18, 2026

fix(urma): 为导入段操作添加互斥锁保护
Saturday
committed
950d09f

Commits on Jun 20, 2026

fix: 修正批量获取卸载对象时参数名错误
Saturday
committed
b20acb2

Commits on Jun 21, 2026

Merge pull request #25 from yyyuanhao426-hash/debug/malloc
zchuango
authored
0c7b5b6
Merge pull request #24 from liangxu2000/supercache_dev
zchuango
authored
ae1277a
optimize the stress cluster bench
zchuango
committed
b181c79
Merge pull request #26 from yyyuanhao426-hash/debug/storage_key
zchuango
authored
5b5ba1a
add the port env parameter
zchuango
committed
8c9e67f
modify the active port type to int8_t
zchuango
committed
c510dc2

Commits on Jun 23, 2026

feat(offload): 新增推送模式卸载以优化远程数据获取
Saturday
committed
66a8131
feat(perf): 为所有者端SSD卸载操作添加性能监控点
Saturday
committed
9cfdfba
perf(offload): 将性能点移至 FileStorage 以精确测量 SSD 读取阶段
Saturday
committed
aaa4456

Commits on Jun 24, 2026

修复urma rpc和urma transport冲突问题
 Gzure
committed
99983ae
feat(offload): 新增 SSD offload push 模式以优化读取性能
Saturday
committed
46c3cc4
支持urma rpc编译选项
 Gzure
committed
14bde4f
去掉ylt_urma选项
 Gzure
committed
662e105
Merge pull request #2 from Gzure/supercache_dev_zjw
JerryLX
authored
2554300

Commits on Jun 25, 2026

add doc for offload failure
lixu
committed
e5d16f9

Commits on Jun 27, 2026

add muiltiple threads handshake listener
zchuango
committed
af50016
add the warmup method
zchuango
committed
0dc18fd
fix the bug for warmup
zchuango
committed
d713e04
fix the get metadata bug
zchuango
committed
1de6449

Commits on Jul 2, 2026

Merge pull request #6 from LinQuickDev/supercache-warmup
zchuango
authored
279973d
Merge pull request #7 from yyyuanhao426-hash/feature/offloadrpc
zchuango
authored
c4d5c5a
feat: 为分布式存储系统添加跨进程和线程的高频日志采样与诊断
Saturday
committed
6323801
feat: 在文件存储模块中添加日志记录支持
Saturday
committed
7efe9df
feat(诊断): 在获取操作中增加本地端点与副本选择耗时细分
Saturday
committed
868ed55

Commits on Jul 3, 2026

perf(offload): 为SSD卸载路径添加详细的性能监控和日志
Saturday
committed
75dbd3d
perf: 优化日志初始化和RPC耗时统计
Saturday
committed
bdd1e8c
refactor(client): 重命名 RPC 时序字段以更清晰反映阶段
Saturday
committed
f5870a5

Commits on Jul 6, 2026

perf(master_client): 添加RPC调用的详细耗时日志记录
Saturday
committed
32d3aac
add the stress cluster rages benchmark
zchuango
committed
2300894
fix: 将RPC日志文件大小限制从10MB增加到100MB
Saturday
committed
e9cb815

Commits on Jul 7, 2026

fix: 将offload RPC服务器线程数从1增加到8
Saturday
committed
74ed2ee
feat: 为分段读取基准测试添加确定性键顺序支持
Saturday
committed
d80e169

Commits on Jul 8, 2026

feat(client): 增加预热卸载RPC连接池支持
Saturday
committed
2417a40
feat(offload): 增加RPC连接池预热和端点发现功能
Saturday
committed
dffa1db
feat(基准测试): 添加客户端初始化等待时间配置
Saturday
committed
f3a1b0c
perf: 将RPC日志文件大小限制从100MB增加到1000MB
Saturday
committed
6942716

Commits on Jul 9, 2026

Merge branch 'linqu_supercache' into supercache
Saturday
committed
e05ecf6

Commits on Jul 11, 2026

feat: 为 RPC 连接池添加空闲超时环境变量配置
 yyyuanhao426-hash
committed
336c53c

Commits on Jul 20, 2026

Commits on Jul 27, 2026

Merge pull request #9 from yyyuanhao426-hash/supercache
zchuango
authored
d74cea6

Commits on Jul 28, 2026

add design doc for distributed metadata plane
lixu
committed
749603c

Commits on Jul 30, 2026

feat: 基于 UBDIAG_DISABLE 重构 ubdiag 集成,FetchContent 替代 submodule
qinyufei63
committed
8baf433
fix: 修正 DISABLE 层源码拉取和编译开关传播
 qinyufei63
committed
411e9b8
feat: 切换 UbDiag GitHub 镜像并固定主仓提交
 qinyufei63
committed
14a15a7
fix: 固定 UbDiag v0.5.1 并修正全流程验证
 qinyufei63
committed
f240448
fix: 隔离UbDiag构建选项并裁剪CachePoint
qinyufei63
committed
e5b447b
docs: 补充UbDiag两层分发用户指南
 qinyufei63
committed
9dbac6b
fix: 锁定RPM内置UbDiag源码与构建产物
 qinyufei63
committed
8e8074c
fix: 修正UbDiag RPM来源校验与可选产物打包
 qinyufei63
committed
a52a405
fix: 更新UbDiag精确版本并包含聚合器修复
 qinyufei63
committed
0534957
fix: 清除RPM内UbDiag构建路径RPATH
qinyufei63
committed
0392794
docs: 完善UbDiag两层分发用户说明
 qinyufei63
committed
c09ee50
fix: 收紧UbDiag两层配置异常门禁
 qinyufei63
committed
05927d6
style: 清理RPM脚本行尾空格
 qinyufei63
committed
8a2304a
feat: 改为使用系统UbDiag完成两层分发
 qinyufei63
committed
ba2c643
docs: 更新UbDiag两层集成与单RPM说明
 qinyufei63
committed
480fe3d
refactor: 精简UbDiag两层集成保护逻辑
 qinyufei63
committed
362bbc8
docs: 精简并保留UbDiag用户指南
 qinyufei63
committed
9997920
fix: 修正系统UbDiag目标作用域与CLI选择
 qinyufei63
committed
64f45d2
Merge pull request #13 from qinyufei63/supercache_dev_ubdiag
zchuango
authored
e42c08c

Commits on Jul 31, 2026

feat(supercache): import supercache feature from LinQuickDev/Mooncake
Wang_YQi
committed
f8e2128
fix(environ): remove duplicate method defs and place them in mooncake namespace
Wang_YQi
committed
8d12958
fix(store): resolve merge conflicts in master_service and rpc_service
Wang_YQi
committed
8675c41
fix(transfer-engine): replace continue with return in handleConnection
Wang_YQi
committed
8f0d2f2
fix: add missing supercache headers and fix types.h enum comma
Wang_YQi
committed
9b4f4ee
fix: remove non-existent SsdFreeRatioFirstAllocationStrategy and fix allocateSingle args
Wang_YQi
committed
950f8c4
fix: bridge .hpp to .h to avoid redefinition errors
Wang_YQi
committed
8897477
fix: remove duplicate for-loop and redeclaration in storage_backend.cpp
Wang_YQi
committed
9247639
fix: fix header_iov→header_iovs typo and add missing metadata_store.h include
Wang_YQi
committed
e2a669e
fix(store): add missing headers - master_admin_service.h, ha_metric_manager.h for rpc_service; ubdiag for client_service
Wang_YQi
committed
8b48fb6
fix(store): convert client_buffer.hpp to bridge header to resolve redefinition
Wang_YQi
committed
628917d
fix(store): add missing mooncake_logging.h include for client_service
Wang_YQi
committed
d605ea7
fix(store): resolve header/impl mismatch and merge artifacts in master_admin/rpc/master_service
Wang_YQi
committed
7bd6187
fix(store): fix setup_internal param order, remove duplicate functions, add missing metadata ref
Wang_YQi
committed
df97067
fix(store): move HttpSegment types and shared helpers to master_admin_service.h
Wang_YQi
committed
60b0e96
fix(store): expose drain job types and helpers in master_admin_service.h
Wang_YQi
committed
36e4403
fix(store): fix decl order and remove anon-ns conflicts in master_admin_service
Wang_YQi
committed
e89d997
fix(store): remove duplicate default arg, expose HttpSegmentStatusResponse
Wang_YQi
committed
ce10f4d
fix(store): remove duplicate MasterAdminServer impls from rpc_service.cpp
Wang_YQi
committed
6641b3b
fix(store): add missing BuildHealthJson/BuildLeaderJson and Client method stubs
Wang_YQi
committed
475504b

Commits on Aug 1, 2026

Merge branch 'main' into supercache_ha
zchuango
committed
7a4348a
resolve confliction and remain
zchuango
committed
64292c2
resolve confliction for bug
zchuango
committed
8e677bf
conflictions for client
zchuango
committed
3ce536d
conflictions for storage_backend.h
zchuango
committed
d1d2274
Merge branch 'supercache_ha' into supercache
zchuango
committed
e615c88
fix the bug for date
zchuango
committed
a47a75c
[Store] Add strict_replica_allocation flag to enforce exact memory replica counts in PutStart (#16)

liangxu2000
and
zchuango
authored
7029995
[Store] Record one LOCAL_DISK replica per client in replicaList (#17)

liangxu2000
and
zchuango
authored
70afdf5
Urma mode (#18)

authored
d5c9e30
fix the code format error
zchuango
committed
e8a5b13

Commits on Aug 3, 2026

feat(ha): reduce DEFAULT_MASTER_VIEW_LEASE_TTL_SEC from 5s to 3s for faster warmup leadership during failover
Wang_YQi
committed
bfbaa0e
!28 merge sync-supercache into dev-master_ub_transport
Enigmo-x
committed
39b30f8

Commits on Aug 5, 2026

重构: 将 UbDiag 重命名为 SpDiag
yyyuanhao426-hash
committed
2bd73a1
etcd reporter, endpoint log
Wang_YQi
committed
8d0d7ce
fix: replace non-existent error_code.h with types.h in master_metrics_reporter.h
Wang_YQi
committed
bea906b
fix: resolve type mismatch and const-correctness errors in MasterMetricsReporter
Wang_YQi
committed
3e7e010
debug: add hex dump for transport_endpoint in NotifyOffloadSuccess
Wang_YQi
committed
3839141
debug: add hex dump after AddReplicas to trace transport_endpoint corruption
Wang_YQi
committed
3d30a4e

Commits on Aug 6, 2026

fix: replace insert+move_iterator with reserve+push_back in AddReplicas to prevent string corruption
Wang_YQi
committed
e81a19b
debug: add AfterConstruct and AfterEmplaceBack hex dumps to trace string loss point
Wang_YQi
committed
5c3222f
fix: resolve dangling references from get_descriptor() returning temporary, and fix silent no-op in update path
Wang_YQi
committed
c60e328
chore: remove debug hex dump logs, keep only the fix for dangling references and silent no-op
Wang_YQi
committed
58e5666
feat: change metrics etcd key to /masters/primary for single-primary tracking, format updated_at as ISO 8601 string
Wang_YQi
committed
7a61e6a
add cluster diag python
Wang_YQi
committed
a828e9e
fix: use localtime_r instead of gmtime_r for updated_at timestamp to match server timezone
Wang_YQi
committed
fd277cb
fix: filter stale LOCAL_DISK replica endpoints by client liveness
Wang_YQi
committed
f0d5f6d
fix: use CleanupStaleHandles for startup cleanup to handle all stale replica types
Wang_YQi
committed
3c88987
fix: clear standby_allocator_keepalive_ before startup cleanup to expire dummy allocator weak_ptrs
Wang_YQi
committed
d0ea077
diagnostic: add startup cleanup detail logs to identify surviving replica types
Wang_YQi
committed
1473755
fix: decrement allocated_mem_size metric after startup cleanup of stale replicas
Wang_YQi
committed
a7626f5
revert: remove overly aggressive startup cleanup to align with upstream
Wang_YQi
committed
5bf5e6c
pr 21 + ha
Wang_YQi
committed
e05ce63
post_promotion_cleanup_sec define
Wang_YQi
committed
95da514
add check client
Wang_YQi
committed
daf7bff
revert change
Wang_YQi
committed
2de0528
rm request
Wang_YQi
committed
fdf4581
fix bug for offload
Wang_YQi
committed
182ac3b
merge boostkit/dev-master_ub_transport into supercache_ha
Wang_YQi
committed
246bfb1
fix: remove undefined retry variable in persist_evict_oplog_or_skip
Wang_YQi
committed
b3349a4

Commits on Aug 7, 2026

fix: metrics reporter lease race + missing lease binding + remove unused iomanip
Wang_YQi
committed
7590edb
[Store] Protect all offload source replicas from eviction (#21)
liangxu2000
authored
bea2c4c
refactor: remove OpLog reservation retry loop in NoFBatchEvict
Wang_YQi
committed
f153498
fix: remove leftover retry reference in NoFBatchEvict log
Wang_YQi
committed
bfbb3c4
store: release offset recovery gaps

Yuchen Kou
authored and
Wang_YQi
committed
6128f2f
store: avoid false memory eviction triggers

Yuchen Kou
authored and
Wang_YQi
committed
fc6d0f1
store: grant lease after memory replica recovery

Yuchen Kou
authored and
Wang_YQi
committed
801fe07
store: stop batch eviction on oplog backpressure

Yuchen Kou
authored and
Wang_YQi
committed
1bbcf79
store: document recovery eviction invariants

Yuchen Kou
authored and
Wang_YQi
committed
a5b2301
store: clarify eviction oplog submission results

Yuchen Kou
authored and
Wang_YQi
committed
84ef383
store: protect restored replicas from eviction

Yuchen Kou
authored and
Wang_YQi
committed
8d1454d
[Bugfix][Store] Keep unreadable recovered replicas out of eviction

Yuchen Kou
authored and
Wang_YQi
committed
5f3c8d1
fix for oplog reserve
Wang_YQi
committed
6839d97
fix for local_disk remount
Wang_YQi
committed
233dc55
add for replica
Wang_YQi
committed
660f036
fix for local_disk replica
Wang_YQi
committed
3f9a19c
add for retry clearup for ha
Wang_YQi
committed
b9f4cee

Commits on Aug 8, 2026

fix whitebox
Wang_YQi
committed
a224d1d
chore(spdiag): 更新 SpDiag 仓库地址与分支
 yyyuanhao426-hash
committed
e7473de
fix for ub segment
Wang_YQi
committed
f101c64
fix for python version bug
Wang_YQi
committed
febacb7
Merge branch 'supercache' into supercache_ha
wangyuqi0429
authored
441ec78

Commits on Aug 10, 2026

Commits on Aug 14, 2026

fix(tracing): 添加缺失的 span 结束调用
 yyyuanhao426-hash
committed
5a10f35

Commits on Aug 17, 2026

Merge pull request #28 from yyyuanhao426-hash/supercache
zchuango
authored
ec7a438

Commits on Aug 27, 2026

feat(spdiag): 为 vLLM 调用路径添加 PerfPoint 打点与 MC_LOG 日志
 ZhiningPan
committed
b874ea5