Fix Mooncake replay cleanup with extended leases

[kiddyboots216]

Summary

• force-delete R3 replay payloads only after the distributed all-rank completion rendezvous
• preserve normal lease-aware deletion on failed or cancelled backend calls
• check Mooncake status codes, retry transient failures, and surface retained-key metrics
• roll back every unpublished partial-put chunk and block further publishing after incomplete cleanup

Validation

• 67 focused server tests passed
• Ruff, formatting, pre-commit, and diff checks passed
• real Mooncake v0.3.9 integration with a 300-second lease: normal delete returned -706 and retained the key; post-consumption forced cleanup removed it with zero failed or pending keys
• independent read-only review found no remaining blockers

The separate replay-payload packaging slowdown is intentionally not addressed here.

[broly-code-security-scanner]

Broly Security Scan

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.