Avoid deltas + simplify pebble metrics

[alrevuelta]

Describe your changes and provide context

This PR simplifies Pebble metrics. It drops the scrape-loop trick of storing the last Pebble totals in a wall of prev* fields and adding only the difference into OTel counters, and instead observes each db.Metrics() snapshot directly.

In other words, now we directly copy Pebble metrics from db.Metrics, instead of artificially keeping the prev values and adding them in each call. Less bug prone.

Testing performed to validate your change

• Tested in a live dashboard where I monitor most of Pebble metrics.

[cursor]

PR Summary

Medium Risk
Observability semantics change for many Pebble counters (cumulative observables vs. delta-scraped counters), which can break existing PromQL/dashboards until queries are updated; shutdown ordering is improved but metrics export is still non-critical to consensus.

Overview
Replaces the large scrape-loop PebbleMetrics implementation with a smaller observable-based exporter in pebble_metrics.go. Pebble-internal stats (compaction, flush, LSM levels, WAL, cache, etc.) are still exposed under the same pebble_* names with a db attribute, but collection no longer uses a wall of prev* fields and per-scrape Add(delta) on synchronous counters.

New behavior: NewPebbleMetrics drops the context argument and returns a shutdown func() that stops the refresh ticker, waits for the goroutine, and unregisters the OTel callback. A background ticker periodically stores db.Metrics() in an atomic pointer; OTel’s RegisterCallback reads that snapshot via Float64ObservableCounter / ObservableGauge helpers (declareDB, declareLevels). Cumulative Pebble totals are observed directly instead of being converted to increments each scrape.

Call sites (pebbledb/db.go, mvcc/db.go) store that shutdown function as metricsCancel and invoke it on Close. Open documents that the passed ctx does not control metrics lifecycle.

Dashboard note: counter series now reflect cumulative values from Pebble (suitable for rate() on the exported observable counters), not manually accumulated deltas between scrapes. Operation latencies (pebble_get_latency, etc.) were already defined in MVCC metrics and are not part of this refactor.

^{Reviewed by Cursor Bugbot for commit 304ff9e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 8, 2026, 9:35 AM

[codecov]

Codecov Report

❌ Patch coverage is 51.47059% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.00%. Comparing base (abdcc45) to head (304ff9e).

Files with missing lines	Patch %	Lines
sei-db/db_engine/pebbledb/pebble_metrics.go	50.89%	25 Missing and 140 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4100      +/-   ##
==========================================
- Coverage   61.21%   60.00%   -1.22%     
==========================================
  Files        2196     2079     -117     
  Lines      193395   178256   -15139     
==========================================
- Hits       118386   106958   -11428     
+ Misses      63793    61076    -2717     
+ Partials    11216    10222     -994     

Flag	Coverage Δ
sei-chain-pr	64.41% <51.47%> (?)
sei-db	69.80% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/pebbledb/db.go	74.33% <100.00%> (ø)
sei-db/db_engine/pebbledb/mvcc/db.go	67.89% <100.00%> (-0.09%)	⬇️
sei-db/db_engine/pebbledb/pebble_metrics.go	51.03% <50.89%> (-48.65%)	⬇️

... and 173 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

Solid simplification: the ~1700-line delta-tracking scrape loop is replaced by OTel observable instruments over an atomically-refreshed pebble.Metrics snapshot, and the removed latency/batch instruments were dead duplicates of the live ones in mvcc/metrics.go, so no metric is actually lost. Field mappings and the Close-before-db.Close() ordering check out; only non-blocking notes on the now-dead ctx parameter, two changed instrument kinds, and absent lifecycle tests.

Findings: 0 blocking | 3 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] No test covers the new lifecycle, which is the part with real failure modes: that the returned stop func is idempotent (sync.OnceFunc), that it waits for the refresher goroutine before returning so db.Close() is safe immediately after, and that a RegisterCallback failure degrades to a no-op stop func rather than a nil call. A small test in sei-db/db_engine/pebbledb opening a DB with EnableMetrics: true, calling the stop func twice, then closing, would pin all three cheaply.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] ctx is now entirely unused in Open — it was only ever wired to the metrics goroutine's cancellation. Documenting a parameter as ignored leaves a trap rather than removing it: every one of the ~16 call sites (s.ctx in flatkv/store.go:761, t.Context() in several tests, context.Background() elsewhere) still reads as if cancelling that context releases the DB's background work, and it no longer does. Since Close is now the single choke point for stopping collection, dropping the parameter from the signature would make that invariant unmissable instead of a comment callers have to find. If you prefer to keep the signature stable for now, consider renaming it _ context.Context so the compiler-visible intent matches the comment.

Relatedly, metricsCancel is still typed context.CancelFunc (line 24) though it no longer comes from a context; plain func() would match what it now holds.

[seidroid]

[suggestion] pebble_compaction_duration (and pebble_flush_duration on line 171) change instrument kind from Float64Histogram to Float64ObservableCounter. Recording a cumulative value into a histogram was meaningless, so this is the right fix — but it renames the exported Prometheus series (pebble_compaction_duration_seconds_bucket/_sum/_count → pebble_compaction_duration_seconds_total), so any existing dashboard panel or alert on these two will silently go empty rather than error. Every other metric keeps a compatible type, so it's worth calling out these two specifically in the PR description or a dashboard follow-up.

[alrevuelta]

yes. this specific metric breaks, but it was wrong before as you point out.