feat: add klio.plugin.wal.restore_duration end-to-end metric - #206
Open
hh24k wants to merge 3 commits into
Open
Conversation
The CNPG-I plugin's RESTORE_WAL path measured its end-to-end duration but only logged it. Expose it as an OTel histogram (klio.plugin.wal.restore_duration, ns, per-file buckets) tagged with outcome, cache_hit, tier and cluster_name. This is the latency PostgreSQL actually experiences when it asks for a WAL segment. In a replica cluster whose designated primary replicates from an external Klio source, it is the speed of the replication path itself and so drives replica lag; during recovery it drives how fast a cluster catches up. Neither is visible today, and the server-side get_duration cannot stand in for it: that times only one WAL.Get call, while prefetch cache hits are served from the local spool and never reach the server at all. cache_hit and tier are threaded up from the prefetcher through restoreWAL via a small restoreOutcome; Restore records on every exit path via defer so failures are measured too. Assisted-by: Claude Signed-off-by: Hai He <hai.he@enterprisedb.com>
Add two panels to the Client / Plugin section for the new klio.plugin.wal.restore_duration metric: p95 end-to-end restore latency split by cache_hit and restore rate by outcome. A prefetch hit is a local rename while a miss waits on a download, so the two are orders of magnitude apart and are shown split rather than pooled; a falling hit ratio means prefetch is not keeping ahead of replay. Regenerated klio-dashboard.json via the builder. Assisted-by: Claude Signed-off-by: Hai He <hai.he@enterprisedb.com>
Keep WAL restore metrics aligned with the real restore path by retaining the last attempted tier on miss-all-tiers failures and expose a direct prefetch hit-ratio panel in the Grafana dashboard. Assisted-by: Claude Signed-off-by: Hai He <hai.he@enterprisedb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RESTORE_WALtiming only went to the log, so restore latency couldn't be graphedor alerted on. That latency sets the replica lag when a replica cluster
replicates from an external Klio source. See #139 for the background, and for why
klio.server.wal.get_durationdoesn't cover it.This adds
klio.plugin.wal.restore_duration, a histogram in nanoseconds. It usesthe same per-WAL-file buckets as the server's get and upload histograms, so the
three read alike. It is tagged with
outcome,cache_hit,tierandcluster_name.cache_hitandtierare only known deep in the restore path, so a smallrestoreOutcomecarries them back up throughrestoreWAL.Restorerecordsfrom a
defer, so failures are measured too, including the early ones thatreturn before any tier is tried.
A hit means the prefetch finished before PostgreSQL asked for the file. That
check is now named
walEntry.isReadyPrefetchand has a table test. If theprefetch is still running the caller waits for it, so it counts as a miss — which
is what makes the hit rate show whether prefetch is keeping up with replay.
Three dashboard panels in the Client / Plugin section: p95 latency split by
cache_hit, rate byoutcome, and the prefetch hit ratio. The hit ratio is theone to watch over time — it falling means prefetch is no longer keeping pace with
replay, which is fixable through the prefetch settings.
klio-dashboard.jsonis regenerated from the builder.Closes #139