Skip to content

feat(webrtc): DataChannel support (SDP + ICE-lite + DTLS + SCTP over DTLS) - #310

Merged
gg582 merged 8 commits into
devfrom
feat/webrtc-datachannel
Oct 5, 2026
Merged

gg582 merged 8 commits into
devfrom
feat/webrtc-datachannel

Conversation

@gg582

@gg582 gg582 commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

DataChannel-only WebRTC (v3.9 scope, #308): SDP offer/answer, ICE-lite, DTLS, SCTP. No media/RTP.

Commits

  1. feat(webrtc) / test(webrtc) / fix(alloc) / fix(webrtc): the original stack.
  2. docs(webrtc): Doxygen comments for src/net/webrtc/ (comments only).
  3. feat(reactor): one-shot timers on the cwist reactor. This is new public API in reactor.h. The poll wait follows the earliest deadline and is still capped at 100 ms. cwist_reactor_stop() now writes the run flag atomically.
  4. perf(webrtc): the WebRTC event loop now runs on that reactor, with the browser interop fixes below.

Design

  • A ctx is one UDP socket served by one cwist reactor.
    • cwist_webrtc_ctx_new(port) creates its own reactor and thread.
    • cwist_webrtc_ctx_new_on(reactor, port) attaches to a reactor you already run.
  • All ctx/conn state is touched only on the reactor thread. send, close, handle_offer and ctx_free post to it when called from another thread.
  • Conns and ctxs are reference counted. A conn pointer is valid during a callback. To keep it longer, use cwist_webrtc_conn_retain()/release(). The new close handler tells you when a conn is gone.
  • Nothing runs on a fixed tick:
    • Each conn has one timer: STUN retransmit, DTLS retransmit, handshake deadline, 30 s liveness, or 30 s expiry for an answered offer that never connects.
    • usrsctp's clock runs only while associations exist: every 10 ms after recent traffic, every 250 ms when quiet.
  • I/O:
    • UDP is read with recvmmsg and written with sendmmsg.
    • DTLS runs on a datagram BIO.
    • Conns are looked up in a hash table.
    • Sends from other threads go through a lock-free per-conn inbox.
    • Sends are capped at 4 MiB queued per conn; see cwist_webrtc_conn_buffered_amount().

Bugs fixed along the way

  • Wrong PPIDs. The code had DCEP 53, string 50, binary 51; RFC 8831 says 50/51/53. Our loopback test passed because both ends were ours. Against Chrome, the channel open was echoed back as text and binary messages were dropped. Empty messages (PPID 56/57) are now supported.
  • Message callback API change. It now reports text vs binary (cwist_webrtc_data_type argument).
  • STUN USERNAME read from the wrong side. It is recipient:sender. As a result the conn parked for an answered offer was never adopted and leaked.
  • DTLS records could be split across UDP packets. The memory BIO was a byte stream read in 2 KiB slices.
  • SCTP partial delivery. Messages delivered in pieces were passed up as separate messages; they are now reassembled using MSG_EOR.
  • Data races. The pending queue and conn list were shared between threads without a lock. usrsctp's malloc'd buffers were freed with cwist_free. A static 1 MiB receive buffer was shared across ctxs. With several ctxs, the usrsctp clock ran fast.
  • Example crashed its workers. It created the ctx before cwist_app_listen() forked, and every worker freed the parent's copy at shutdown and segfaulted. A ctx inherited across fork() is now detected, and the example creates one ctx per worker.

Testing

  • make test passes, including test_webrtc and the new test_reactor_timer (io_uring and epoll).
  • test_webrtc covers:
    • ping/pong, a 200 000-byte message both ways, typed and empty messages
    • the max-message-size and 4 MiB queue limits
    • close from another thread reaching the peer's close handler
    • parked-conn adoption
    • a ctx on a caller-run reactor
  • test_webrtc is clean under ASan/UBSan/LSan and TSan (3 runs each). bench_webrtc under TSan is clean.
  • Real browser: make test_webrtc_browser drives headless Chromium over CDP against example/webrtc. Results:
    • The channel opens in about 8 ms.
    • Text comes back as a string.
    • A 200 000-byte binary message comes back intact.
    • An empty string round-trips.
    • Average RTT is about 0.08 ms.
    • It needs chromium and Node 22+, so it is not part of make test. Firefox was not tested.
  • make wasip2-smoke passes (WASI stubs for the timer API).

Numbers (loopback, ./bench_webrtc 200000 2000, 3 runs)

before after
idle ctx, context switches per second 101 11
64 B messages 195k msg/s 970k msg/s
1 KiB messages 26 MB/s 162 MB/s
64 KiB messages 37.6 MB/s 184 MB/s
RTT p50 / p99 25 / 40 µs 29 / 40–50 µs

The RTT p50 is about 4 µs worse. That is the reactor's one-shot re-arm plus the eventfd wake on each hop, and it is the same on io_uring and epoll.

Known limitations

  • ICE-lite only: no STUN/TURN servers, no trickle ICE, IPv4 host candidates only.
  • The peer certificate fingerprint is not yet checked against the SDP.
  • Ordered, reliable channels only; channels cannot be closed individually (no stream reset).
  • A ctx belongs to the process that created it. With cwist_app_listen() workers, create one per worker, as the example does.

gg582 added 4 commits October 4, 2026 22:16
…DTLS)

Adds a DataChannel-only WebRTC stack behind CWIST_WEBRTC=1 (default on),
per the v3.9 scope tracked in #308:

- SDP offer/answer for m=application datachannel (RFC 4566 subset):
  ice-ufrag/ice-pwd, fingerprint SHA-256, setup roles, mid, host candidate,
  sctp-port, ice-lite.
- ICE-lite agent (RFC 5389/8445): STUN binding request validation with
  MESSAGE-INTEGRITY (ice-pwd keyed HMAC-SHA1), XOR-MAPPED-ADDRESS responses
  with FINGERPRINT, USE-CANDIDATE nomination. The browser stays controlling.
- DTLS over the nominated UDP path using the vendored BoringSSL
  (DTLS_server_method / DTLS_client_method, memory BIOs, plain DTLS, no
  SRTP). Ephemeral self-signed P-256 certificate per ctx.
- SCTP DataChannels tunneled over DTLS (RFC 8831) via the new lib/usrsctp
  submodule in AF_CONN raw mode, plus DCEP open/ack (RFC 8832) and
  ordered binary/string message delivery.
- Public API in include/cwist/net/webrtc.h: ctx bound to a UDP port,
  cwist_webrtc_handle_offer(), message/channel callbacks, send, close.

tests/test_webrtc.c drives the full stack over UDP loopback between two
in-process ctxs (answering ICE-lite endpoint + offering peer) and
exchanges DataChannel messages in both directions; it joins make test.
example/webrtc contains a browser-interop demo (RTCPeerConnection +
POST /offer signaling).

Known limitations (MVP): ICE-lite only (no STUN/TURN server, no trickle
candidate parsing from offers), no peer fingerprint verification, ordered
delivery only, IPv4 host candidates, one bundled m-line.
check_test_wiring.py only scans the TEST_TARGETS = ... assignment block,
so the conditional TEST_TARGETS += from the feature commit was invisible
to the gate and CI flagged tests/test_webrtc.c as unwired. List
test_webrtc in the main block instead, keep the link rule next to the
other test rules, and add a CWIST_WEBRTC=0 skip stub so the target
resolves when the module is compiled out.
@gg582

gg582 commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

Well, well...things are just mixed. This should be two pull request: one for adding comments, one for webrtc only. +10000 is weird.

@gg582

gg582 commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

KIMI SLOP

@gg582 gg582 closed this Oct 5, 2026
@gg582 gg582 reopened this Oct 5, 2026
@gg582
gg582 force-pushed the feat/webrtc-datachannel branch from 9bbac1a to 6f681d7 Compare October 5, 2026 09:53
@gg582
gg582 marked this pull request as draft October 5, 2026 09:59
gg582 added 2 commits October 5, 2026 19:34
Add cwist_reactor_timer_{init,arm,cancel,armed}: caller-owned timers kept
in a min-heap per reactor and fired on the run thread.  Each backend's
poll wait (io_uring, epoll, kqueue) is shortened to the earliest
deadline, still capped at the existing 100 ms idle wait, so a reactor
with no armed timer wakes no more often than before.  Arm, cancel and the
callbacks run on the run thread only; foreign threads go through
cwist_reactor_post().

Until now anything that needed a deadline ran its own thread
(websocket_async's sweep watchdog) or polled on a fixed tick.  The WebRTC
event loop is the first user.

Also make the run flag atomic: cwist_reactor_stop() is called from
foreign threads and was writing a plain bool the run loop reads (found by
TSan in the new WebRTC tests).

WASI stubs cover the new functions.  tests/test_reactor_timer checks
deadline ordering under out-of-order arming, cancel, re-arm, self re-arm
from the callback, and never-early firing, on the default backend and on
forced epoll.
Event loop
- A ctx is now driven by a cwist reactor instead of a private poll()
  thread that woke every 10 ms.  cwist_webrtc_ctx_new() still creates its
  own reactor and thread; the new cwist_webrtc_ctx_new_on() attaches to a
  reactor the caller runs.
- Nothing runs on a fixed tick.  Each conn has one reactor timer (STUN
  retransmit with backoff, DTLS retransmit, 30 s handshake deadline,
  30 s liveness, 30 s expiry for answered offers that never connect), and
  usrsctp's clock is driven only while associations exist: every 10 ms
  after recent traffic, every 250 ms when quiet.
- UDP is read with recvmmsg() and written with sendmmsg(): datagrams
  queued during a callback leave in one call.  SCTP draining and queued
  sends are handled once per callback for every conn it touched.
- Connections are looked up in a hash table instead of a linked list.
- DTLS uses a datagram BIO.  The memory BIO was a byte stream: records
  written back to back were concatenated and read back in 2 KiB slices,
  so a record could be split across two UDP packets.
- Foreign-thread sends go through a per-conn lock-free inbox and wake the
  reactor once per burst.  The queue is capped at 4 MiB
  (cwist_webrtc_conn_send() returns -1 beyond it); queued bytes are
  reported by cwist_webrtc_conn_buffered_amount().
- usrsctp time advances by real elapsed time under one lock, whichever
  ctx ticks it (two ctxs used to each add their own elapsed time).  A
  packet usrsctp emits on another ctx's thread is marshalled to the owner.

Thread safety and lifetimes
- All ctx/conn state is touched on the owner thread only; send, close,
  handle_offer and ctx_free post when called elsewhere.  The old code read
  and wrote the pending queue and the conn list from two threads without
  a lock.
- Conns and ctxs are reference counted, with cwist_webrtc_conn_retain()/
  release() and a close handler (cwist_webrtc_ctx_set_close_handler), so
  closing no longer frees a conn under a pointer the application holds.
- usrsctp's malloc'd receive buffer is no longer passed to cwist_free():
  receive callbacks are gone, all data is drained with usrsctp_recvv().
- The shared static 1 MiB receive buffer is now per ctx.
- A ctx inherited across fork() is detected: handle_offer() returns -1 and
  ctx_free() does nothing in the child.  The example created its ctx
  before cwist_app_listen() forked workers; each worker freed the parent's
  copy at shutdown and segfaulted.  It now creates one ctx per worker.

Protocol fixes found by testing against Chromium
- PPIDs were wrong: DCEP is 50, string 51, binary 53 (RFC 8831 s8); the
  code used 53/50/51.  Our two ends agreed with each other, so the
  loopback test passed while a browser's channel open was echoed back as
  text and its binary messages were dropped as DCEP.  Empty messages now
  use PPID 56/57.
- The message callback now reports the message type so text can be
  echoed as text (API change; the callback gains a
  cwist_webrtc_data_type argument).
- Messages delivered in pieces (SCTP partial delivery) are reassembled
  using MSG_EOR instead of being passed up as separate messages.
- STUN USERNAME is "<recipient>:<sender>" (RFC 8445 s7.2.2); the remote
  ufrag was read from the wrong side, so the conn parked for an answered
  offer was never adopted and leaked.
- SCTP: 1024 streams, fixed 1160-byte path MTU, 10 s heartbeats, abort on
  close, association-loss notifications close the conn.

Tests
- test_webrtc: typed and empty messages, a 200 000-byte message both ways,
  the max-message-size and 4 MiB queue limits, close from a foreign thread
  reaching the peer's close handler, parked-conn adoption, and a ctx on a
  caller-run reactor freed from another thread.  Clean under ASan/UBSan/
  LSan and TSan.
- bench_webrtc (make bench_webrtc): idle cost, throughput, RTT.
- test_webrtc_browser (make target, not in `make test`): headless
  Chromium over CDP against example/webrtc.

Loopback, bench_webrtc 200000 2000, before -> after:
  idle ctx, 1 s           101 -> 11 context switches
  64 B messages           195k -> 970k msg/s
  1 KiB messages          26 -> 162 MB/s
  64 KiB messages         37.6 -> 184 MB/s
  RTT p50                 25 -> 29 us (reactor re-arm + eventfd wake)
@gg582
gg582 marked this pull request as ready for review October 5, 2026 10:38
@gg582
gg582 merged commit c1f24db into dev Oct 5, 2026
15 of 16 checks passed
gg582 added a commit that referenced this pull request Oct 5, 2026
…etection

The gates from #309 failed every run (7/7). The absolute keep-alive and
1 MiB backstops came from a Ryzen 5600X and an EPYC 9V45, and healthy code
on the EPYC 7763 runner lands under them (75k vs 110k keep-alive/s, 1.9 vs
2.5 GB/s). History is only recorded after a pass, so it never filled and
the same-CPU check never switched on.

- Gate on https/http ratios measured in the same run: churn, keep-alive,
  1 MiB. They hold within 0.11-0.13 / 0.55-0.66 / 0.34-0.52 across four
  CPU models while the absolute numbers move ~1.8x.
- Absolute backstops drop to ~60% of the slowest healthy runner: they now
  catch only breakage that is wrong on any hardware.
- Seed benchmarks/https_gates.json with the 7 CI runs and one local run,
  all on healthy trees, so the EPYC 7763 same-CPU check engages on its
  next run.
- https_perf_gates.sh: the plaintext 1 MiB number had no MB/s fallback.

Regression check (Ryzen 5600X, #307's TCP_QUICKACK re-arm disabled):
tls_rtt_p50_ms 0.083 -> 43.0, churn ratio 0.118 -> 0.052; the gate fails
RTT, churn ratio and absolute churn. All 7 recorded CI runs pass.
test_https_gates_eval.py replays both, the recorded history, and the
same-CPU path.

ROADMAP v3.9: (a) status updated; (c) corrected, the #294 premise
correction itself was wrong (cwist_app_listen does use the shepherds;
measured), #294 closed as completed; (d) done (#310).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant