Skip to content

feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue (+ call_ringing fix) - #267

Open
ftong2010 wants to merge 9 commits into
restsend:mainfrom
ftong2010:rwi-dedicated-worker
Open

feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue (+ call_ringing fix)#267
ftong2010 wants to merge 9 commits into
restsend:mainfrom
ftong2010:rwi-dedicated-worker

Conversation

@ftong2010

Copy link
Copy Markdown

Motivation

The RWI webhook handler runs sequential, blocking-ish HTTP POSTs to the
router on the shared SIP runtime. A slow webhook consumer therefore
competes with SIP signalling, the HTTP route path and the CDR saver — and a
backpressured router can stall unrelated work. There was also no visibility
into the event pipeline (sent vs delivered vs dropped, queue depth), no retry
on transient push failures, a hardcoded queue capacity, and a gap in the
event stream: call_ringing was never emitted for calls whose trunk
replied with a 183 (early media), which is the majority of trunk calls.

What this PR adds

  1. Dedicated RWI webhook runtime — the webhook handler now runs on its
    own tokio runtime (rwi-webhook threads), so webhook egress and any
    slow-router backpressure never contend with the SIP runtime.
  2. Event pipeline metrics — Prometheus counters/gauges covering the
    webhook path end-to-end (see table below).
  3. Push retry with backoff — transient failures (transport error, 5xx,
    429) are retried with exponential backoff (200 ms base, doubling, hard
    cap 5); permanent 4xx are not retried.
  4. Configurable queue length — the gateway→handler broadcast channel
    capacity is now a config knob instead of a hardcoded constant.
  5. Opt-in queueing-latency histogram — measures how long an event waits
    in the queue (enqueued → handler dequeued). The HTTP push time is
    excluded on purpose: a slow router inflates push time, not queue wait.
  6. call_ringing fix — the event is now emitted on either provisional
    (183 with SDP or 180 without), exactly once per call.

Configuration

Key Default Description
[proxy] rwi_webhook_worker_threads 2 Dedicated tokio workers for the webhook push consumer
[proxy] rwi_webhook_channel_size 512 Queue length (broadcast channel capacity)
[rwi_webhook] retries 0 Max retries after a failed push (hard cap 5, 200 ms base backoff doubling)
[rwi_webhook] track_queue_latency false Record the queueing-wait histogram rwi_event_queue_latency_seconds
[rwi_webhook] timeout_ms 5000 Per-request HTTP timeout (each retry attempt is bounded by it)

Metrics

Metric Type Labels Description
rwi_event_enqueued_total Counter event_type Events pushed into the queue by gateway dispatch
rwi_events_pushed_total Counter event_type Events delivered with a 2xx response
rwi_events_push_failed_total Counter event_type Pushes that errored or returned non-2xx
rwi_events_push_retries_total Counter event_type Retry attempts after a failed push
rwi_events_dropped_total Counter - Events lost to queue lag (consumer fell behind)
rwi_event_queue_size Gauge - Configured queue capacity
rwi_event_queue_current Gauge - Events currently queued (sampled every 5 s on the dedicated runtime)
rwi_event_queue_latency_seconds Histogram event_type Queueing wait (enqueued → handler dequeued); opt-in via track_queue_latency

Verification

  • Startup builds the dedicated runtime: SIP workers=8 Media workers=12 RWI webhook workers=2, with 2 rwi-webhook threads visible in the
    process.
  • Smoke test (5 cps, ~100 calls): rwi_event_enqueued_total 209 ==
    rwi_events_pushed_total 209, rwi_event_queue_current 0 — no drops,
    no backlog.
  • Load test (50 cps, recording + CDR + RWI enabled): 20,065 enqueued ==
    20,065 pushed, queue stayed drained, rwi_events_dropped_total 0.
  • call_ringing now fires exactly once per call (600 calls → 600
    call_ringing enqueued/pushed), including 183-early-media calls that
    previously emitted nothing.
  • Queue config honored end-to-end: rwi_webhook_channel_size = 1234
    rwi_event_queue_size 1234.
  • cargo check --workspace --bins clean; release image builds.

Docs

docs/rwi_events_reference.md (zh) and docs/rwi_events_reference_en.md
updated with the new config keys and the metrics table;
docs/observability.md gained an "RWI Events" section in the metrics
reference.

tongfengyuan added 9 commits September 2, 2026 07:12
Add a dedicated RWI webhook tokio runtime (configurable worker count via
[proxy] rwi_webhook_worker_threads, default 2) so the webhook's
sequential blocking-ish HTTP POSTs to the router (and any backpressure
from a slow router) run off the SIP runtime shared by signalling, the
HTTP route path and the CDR saver.

- utils: set_rwi_webhook_runtime / rwi_webhook_spawn /
  rwi_webhook_runtime_handle helpers.
- bin/rustpbx: build the rwi-webhook runtime alongside sip/media.
- rwi/webhook: dispatch the handler via rwi_webhook_spawn; raise the
  webhook broadcast channel 512 -> 100000.
Add Prometheus metrics covering the RWI event pipeline:

- rwi_events_sent_total: events pushed into the webhook broadcast
  channel by the gateway dispatch (fanout_webhook_tap).
- rwi_events_pushed_total: events successfully delivered (2xx) by the
  webhook handler's HTTP push.
- rwi_events_push_failed_total: webhook HTTP pushes that failed or
  returned a non-2xx status.
- rwi_events_dropped_total: events lost to broadcast lag (the handler
  consumer fell behind the 100k channel and skipped events).
- rwi_event_queue_size: webhook broadcast channel capacity.
- rwi_event_queue_current: events currently queued in the channel
  (sampled every 5 s on the RWI webhook runtime; slow-router
  backpressure shows up as current climbing toward size).

Queue gauges are sampled on the dedicated RWI webhook runtime so queue
observation never contends with the SIP runtime.
Adds [proxy.locator_webhook] retries (default 0 = single attempt, hard
cap 5). After a failed attempt the push retries with exponential backoff
(200 ms base, doubling) for retryable outcomes: transport errors, 5xx
and 429. Other 4xx are permanent and return immediately.

Each attempt is bounded by the client's request timeout
([proxy.locator_webhook] timeout_ms, default 5 s), which already applies
per request through build_keepalive_client.

Also counts retries as rwi_events_push_retries_total, and documents the
per-request timeout in send_payload.
[proxy] rwi_webhook_channel_size (default 100000) sets the capacity of
the broadcast channel between the gateway and the webhook handler. The
queue-size gauge reports the configured value.

Also exports WEBHOOK_CHANNEL_SIZE so tests can pass an explicit size.
…dingly

The opt-in latency histogram now measures the QUEUEING wait — from the
gateway enqueuing the event to the webhook handler dequeuing it — instead
of the end-to-end push duration. The HTTP push time is excluded on
purpose: a slow router inflates push time, not queue wait.

- rename rwi_event_push_latency_seconds ->
  rwi_event_queue_latency_seconds
- sample at dequeue (rx.recv Ok), before dedup/push
- exclude the HTTP push itself; slow-router backpressure does not show
  up here
- update [rwi_webhook] track_latency doc
Matches the metric semantics after the queue-latency histogram rename:
the flag gates rwi_event_queue_latency_seconds (queueing wait), not the
HTTP push.
- rwi_events_reference (en/zh): [rwi_webhook] retries and
  track_queue_latency fields, [proxy] rwi_webhook_worker_threads /
  rwi_webhook_channel_size keys, and the webhook metrics table
  (enqueued/pushed/failed/retries/dropped/queue size/queue current/
  queue latency).
- observability.md: add the RWI Events section to the metrics
  reference.
The pipeline metrics commit shipped the gateway dispatch counter under
its old name (rwi_events_sent_total) and without the event_type label.
Rename to rwi_event_enqueued_total and tag with event_type, matching
the other event pipeline metrics.
CallRinging was only emitted in the no-SDP (180) branch of the callee
provisional handling; the 183-with-SDP (early media) branch — the
 majority of trunk calls — emitted nothing, so most calls never
reported ringing.

Emit CallRinging at the top of the provisional handling so BOTH a 183
with SDP and a 180 without fire it, guarded by a
media.ringing_event_sent flag so it fires exactly once per call.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant