feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue (+ call_ringing fix) - #267
Open
ftong2010 wants to merge 9 commits into
Open
Conversation
added 9 commits
September 2, 2026 07:12
Add a dedicated RWI webhook tokio runtime (configurable worker count via [proxy] rwi_webhook_worker_threads, default 2) so the webhook's sequential blocking-ish HTTP POSTs to the router (and any backpressure from a slow router) run off the SIP runtime shared by signalling, the HTTP route path and the CDR saver. - utils: set_rwi_webhook_runtime / rwi_webhook_spawn / rwi_webhook_runtime_handle helpers. - bin/rustpbx: build the rwi-webhook runtime alongside sip/media. - rwi/webhook: dispatch the handler via rwi_webhook_spawn; raise the webhook broadcast channel 512 -> 100000.
Add Prometheus metrics covering the RWI event pipeline: - rwi_events_sent_total: events pushed into the webhook broadcast channel by the gateway dispatch (fanout_webhook_tap). - rwi_events_pushed_total: events successfully delivered (2xx) by the webhook handler's HTTP push. - rwi_events_push_failed_total: webhook HTTP pushes that failed or returned a non-2xx status. - rwi_events_dropped_total: events lost to broadcast lag (the handler consumer fell behind the 100k channel and skipped events). - rwi_event_queue_size: webhook broadcast channel capacity. - rwi_event_queue_current: events currently queued in the channel (sampled every 5 s on the RWI webhook runtime; slow-router backpressure shows up as current climbing toward size). Queue gauges are sampled on the dedicated RWI webhook runtime so queue observation never contends with the SIP runtime.
Adds [proxy.locator_webhook] retries (default 0 = single attempt, hard cap 5). After a failed attempt the push retries with exponential backoff (200 ms base, doubling) for retryable outcomes: transport errors, 5xx and 429. Other 4xx are permanent and return immediately. Each attempt is bounded by the client's request timeout ([proxy.locator_webhook] timeout_ms, default 5 s), which already applies per request through build_keepalive_client. Also counts retries as rwi_events_push_retries_total, and documents the per-request timeout in send_payload.
[proxy] rwi_webhook_channel_size (default 100000) sets the capacity of the broadcast channel between the gateway and the webhook handler. The queue-size gauge reports the configured value. Also exports WEBHOOK_CHANNEL_SIZE so tests can pass an explicit size.
…dingly The opt-in latency histogram now measures the QUEUEING wait — from the gateway enqueuing the event to the webhook handler dequeuing it — instead of the end-to-end push duration. The HTTP push time is excluded on purpose: a slow router inflates push time, not queue wait. - rename rwi_event_push_latency_seconds -> rwi_event_queue_latency_seconds - sample at dequeue (rx.recv Ok), before dedup/push - exclude the HTTP push itself; slow-router backpressure does not show up here - update [rwi_webhook] track_latency doc
Matches the metric semantics after the queue-latency histogram rename: the flag gates rwi_event_queue_latency_seconds (queueing wait), not the HTTP push.
- rwi_events_reference (en/zh): [rwi_webhook] retries and track_queue_latency fields, [proxy] rwi_webhook_worker_threads / rwi_webhook_channel_size keys, and the webhook metrics table (enqueued/pushed/failed/retries/dropped/queue size/queue current/ queue latency). - observability.md: add the RWI Events section to the metrics reference.
The pipeline metrics commit shipped the gateway dispatch counter under its old name (rwi_events_sent_total) and without the event_type label. Rename to rwi_event_enqueued_total and tag with event_type, matching the other event pipeline metrics.
CallRinging was only emitted in the no-SDP (180) branch of the callee provisional handling; the 183-with-SDP (early media) branch — the majority of trunk calls — emitted nothing, so most calls never reported ringing. Emit CallRinging at the top of the provisional handling so BOTH a 183 with SDP and a 180 without fire it, guarded by a media.ringing_event_sent flag so it fires exactly once per call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The RWI webhook handler runs sequential, blocking-ish HTTP POSTs to the
router on the shared SIP runtime. A slow webhook consumer therefore
competes with SIP signalling, the HTTP route path and the CDR saver — and a
backpressured router can stall unrelated work. There was also no visibility
into the event pipeline (sent vs delivered vs dropped, queue depth), no retry
on transient push failures, a hardcoded queue capacity, and a gap in the
event stream:
call_ringingwas never emitted for calls whose trunkreplied with a 183 (early media), which is the majority of trunk calls.
What this PR adds
own tokio runtime (
rwi-webhookthreads), so webhook egress and anyslow-router backpressure never contend with the SIP runtime.
webhook path end-to-end (see table below).
429) are retried with exponential backoff (200 ms base, doubling, hard
cap 5); permanent 4xx are not retried.
capacity is now a config knob instead of a hardcoded constant.
in the queue (enqueued → handler dequeued). The HTTP push time is
excluded on purpose: a slow router inflates push time, not queue wait.
call_ringingfix — the event is now emitted on either provisional(183 with SDP or 180 without), exactly once per call.
Configuration
[proxy] rwi_webhook_worker_threads[proxy] rwi_webhook_channel_size[rwi_webhook] retries[rwi_webhook] track_queue_latencyrwi_event_queue_latency_seconds[rwi_webhook] timeout_msMetrics
rwi_event_enqueued_totalevent_typerwi_events_pushed_totalevent_typerwi_events_push_failed_totalevent_typerwi_events_push_retries_totalevent_typerwi_events_dropped_totalrwi_event_queue_sizerwi_event_queue_currentrwi_event_queue_latency_secondsevent_typetrack_queue_latencyVerification
SIP workers=8 Media workers=12 RWI webhook workers=2, with 2rwi-webhookthreads visible in theprocess.
rwi_event_enqueued_total209 ==rwi_events_pushed_total209,rwi_event_queue_current0 — no drops,no backlog.
20,065 pushed, queue stayed drained,
rwi_events_dropped_total0.call_ringingnow fires exactly once per call (600 calls → 600call_ringingenqueued/pushed), including 183-early-media calls thatpreviously emitted nothing.
rwi_webhook_channel_size = 1234→rwi_event_queue_size 1234.cargo check --workspace --binsclean; release image builds.Docs
docs/rwi_events_reference.md(zh) anddocs/rwi_events_reference_en.mdupdated with the new config keys and the metrics table;
docs/observability.mdgained an "RWI Events" section in the metricsreference.