Search before reporting
Motivation
brk_ml_cursor_persistLedgerErrors / brk_ml_cursor_persistLedgerSucceed and their ZooKeeper counterparts are the only metrics that reveal a cursor failing to persist its mark-delete position. They are behind exposeManagedCursorMetricsInPrometheus, which defaults to false (ServiceConfiguration.java:3883 on master 9ba61bd95de).
That default makes a whole class of failure undetectable in practice:
- a cursor whose ledger cannot be created or written keeps serving reads and produces no metric movement;
- most of the corresponding broker-side signals are DEBUG-level, so the logs do not fill the gap either;
pulsar_ml_AddEntryErrors is frequently assumed to cover it, but recordAddEntryError() has exactly two callers, both in OpAddEntry — that is the topic data write path, not the cursor persist path. Cursor persists are counted only by the cursor metrics above.
The result is that "the cursor is not advancing" has no first-class signal on a default deployment, and operators reach for metrics that cannot answer the question.
Solution
Any of, in decreasing order of preference:
- Export the cursor persist-error counters unconditionally, independent of
exposeManagedCursorMetricsInPrometheus — they are per-cursor counters with low cardinality relative to the full managed-cursor metric set that the flag was introduced to gate.
- Change the default of
exposeManagedCursorMetricsInPrometheus to true.
- Keep the flag as-is but emit a WARN when a cursor persist fails, so the condition is at least visible in logs at default levels.
Clarifying the documentation of pulsar_ml_AddEntryErrors to state that it does not cover cursor persists would also help.
Alternatives
Leaving it as-is and documenting that operators must enable the flag proactively. This does not help during an incident, since the flag is a broker restart away and the historical data does not exist.
Anything else?
Related: #25528 made acknowledgements with receipts reflect persistence, which improves client-side visibility. This request is about broker-side visibility for cursors with no client waiting on them — the compaction cursor in particular.
Are you willing to submit a PR?
Search before reporting
Motivation
brk_ml_cursor_persistLedgerErrors/brk_ml_cursor_persistLedgerSucceedand their ZooKeeper counterparts are the only metrics that reveal a cursor failing to persist its mark-delete position. They are behindexposeManagedCursorMetricsInPrometheus, which defaults tofalse(ServiceConfiguration.java:3883onmaster9ba61bd95de).That default makes a whole class of failure undetectable in practice:
pulsar_ml_AddEntryErrorsis frequently assumed to cover it, butrecordAddEntryError()has exactly two callers, both inOpAddEntry— that is the topic data write path, not the cursor persist path. Cursor persists are counted only by the cursor metrics above.The result is that "the cursor is not advancing" has no first-class signal on a default deployment, and operators reach for metrics that cannot answer the question.
Solution
Any of, in decreasing order of preference:
exposeManagedCursorMetricsInPrometheus— they are per-cursor counters with low cardinality relative to the full managed-cursor metric set that the flag was introduced to gate.exposeManagedCursorMetricsInPrometheustotrue.Clarifying the documentation of
pulsar_ml_AddEntryErrorsto state that it does not cover cursor persists would also help.Alternatives
Leaving it as-is and documenting that operators must enable the flag proactively. This does not help during an incident, since the flag is a broker restart away and the historical data does not exist.
Anything else?
Related: #25528 made acknowledgements with receipts reflect persistence, which improves client-side visibility. This request is about broker-side visibility for cursors with no client waiting on them — the compaction cursor in particular.
Are you willing to submit a PR?