Skip to content

feat(sipflow): export raw-file flush and general sqlite pipeline metrics - #270

Open
ftong2010 wants to merge 3 commits into
restsend:mainfrom
ftong2010:sipflow-io-metrics
Open

feat(sipflow): export raw-file flush and general sqlite pipeline metrics#270
ftong2010 wants to merge 3 commits into
restsend:mainfrom
ftong2010:sipflow-io-metrics

Conversation

@ftong2010

Copy link
Copy Markdown

Motivation

Investigating sustained disk IO on the collector (measured ~7,200 write
IOPS / ~100 MB/s at 50 cps ingest) required ad-hoc probing of sqlite and
raw-file behavior. This PR makes the write/read pipeline permanently
observable so IO attribution is a metrics query, not an investigation.
Existing domain metrics (sipflow_flush_*, sipflow_wal_checkpoint_*,
sipflow_db_file_bytes, sipflow_raw_write_seconds) are unchanged; the
new metrics sit alongside them with a general taxonomy.

What this PR adds

1. Raw capture file (data.raw)

Metric Type Description
sipflow_raw_flush_total Counter Buffered writes to data.raw
sipflow_raw_flush_bytes Histogram Per-flush batch size (bytes)
sipflow_raw_write_seconds Histogram (existing) flush latency

2. General sqlite taxonomy (sqlite_*)

Every metric carries a database label (value sipflow) so the same
names apply to other sqlite usage later.

Metric Type Notes
sqlite_connections_opened_total{role=read|write} Counter read = per-query-per-bucket fan-out
sqlite_connections_open{role} Gauge drop-guard on read conns; explicit release on bucket rotate
sqlite_connection_open_seconds{role} Histogram connect (+ PRAGMA setup on read conns)
sqlite_connection_errors_total{phase} Counter
sqlite_statements_total{kind=select|insert|pragma|ddl} Counter
sqlite_statement_rows_total{kind,direction=read|written} Counter rows returned / rows affected
sqlite_statement_seconds{kind} Histogram statement-level only (batched ops, never per-row)
sqlite_errors_total{kind,error=busy|locked|constraint|other} Counter SQLITE 5/6/19/2067 classification
sqlite_transactions_total{outcome=commit|rollback} + sqlite_transaction_seconds Counter/Histogram wraps the flusher batch transaction
sqlite_wal_checkpoint_total{kind=passive|truncate} + _seconds + _busy_total Counter/Histogram PASSIVE cadence + rotate-time TRUNCATE
sqlite_db_bytes, sqlite_wal_bytes, sqlite_page_count, sqlite_freelist_pages Gauge stat() at flush cadence; page PRAGMAs piggybacked on the throttled checkpoint

Verification

50 cps × 300 s load against a rebuilt collector, all new metrics live:

  • sqlite_statement_rows_total{written} 10,730,964 vs
    sipflow_flush_rows_total 10,700,976 — Δ 0.28 % (same rows, two
    independent counters)
  • sqlite_transactions_total{commit} 5,965 → 1,800 rows/tx,
    matching the observed ~1,795-row flush batches
  • sqlite_wal_checkpoint_total{passive} 33 in 345 s — exactly the
    throttled 1 per 10 s
  • sqlite_connections_opened_total{read} 194,530 × 4 PRAGMAs each =
    778,120, matching sqlite_statements_total{pragma} 778,126 (the +6
    are the single write connection's setup)
  • sipflow_raw_flush_total 30,811 (~89/s), sipflow_raw_flush_bytes
    averaging 62.8 KB — the 64 KB batch threshold working as designed

Correlated against cgroup io.stat during the run: ~7,200 write IOPS /
~100 MB/s device writes sustained, dominated by WAL page appends +
checkpoint copying of ~33 k media_msgs rows/s.

Notes

  • One pre-existing test failure
    (backend::remote::tests::test_report_endpoint_returns_receive_counters)
    fails identically on unmodified main; unrelated.
  • flush_slice is split into a thin wrapper + flush_slice_inner to
    scope the transaction timer around the existing ? error paths; the
    silent .ok() on bucket-creation DDL now also increments
    sqlite_errors_total (still non-fatal).

tongfengyuan added 3 commits September 2, 2026 08:50
The raw capture file already recorded write latency
(sipflow_raw_write_seconds); add the flush counter and a per-flush
byte-size histogram so the data.raw write path can be correlated with
sqlite flush metrics (sipflow_flush_rows_total / flush_batch_size /
flush_db_seconds) against cgroup IO counters.
…w_* ones

Instrument the sqlite usage with a general taxonomy — connections
(opened/open/errors/latency, role=read|write), statements
(statements_total, rows read/written, latency by kind
select/insert/pragma/ddl), transactions (commit/rollback + latency),
WAL checkpoints (passive/truncate + busy) and file gauges
(db/wal bytes, page_count, freelist). All carry a database label
(value: sipflow) so the same taxonomy generalizes to other sqlite
usage later.

Existing domain metrics (sipflow_flush_*, sipflow_wal_checkpoint_*,
sipflow_db_file_bytes) are unchanged.
…sted

The bucket-rotate path released sqlite_connections_open{role=write}
unconditionally; on the first rotate with no open connection (startup
layout pass) that drove the gauge negative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant