diff --git a/_docs/21-tcp-sockets.md b/_docs/21-tcp-sockets.md new file mode 100644 index 0000000..a6e23ca --- /dev/null +++ b/_docs/21-tcp-sockets.md @@ -0,0 +1,487 @@ +--- +title: "TCP sockets from first principles" +order: 21 +part: "Networking" +description: "chatterd v0 opens Part 6 with a thread-per-connection TCP chat room: socket/bind/listen/accept4 and sockaddr_in by hand, SO_REUSEADDR and why a fresh serve needs it, TCP as a boundaryless byte stream that forces the length-prefix frame reused for the rest of the book, and the RAII Socket that owns every fd — cross-checked with ss, strace, and the wire bytes themselves." +duration: "50 minutes" +--- + +Part 5 spent nine chapters moving bytes between processes that already knew +each other — pipes an ancestor set up, UNIX sockets a stranger found by a +path on the local machine. `chatterd` opens Part 6 by dropping the last +assumption: the peer is on another machine, reachable only by an address and +a port, over a transport that can lose, reorder, and re-split everything you +send. This first cut is deliberately small — a thread-per-connection TCP +**chat room**, one binary that is both the daemon (`serve`) and the client +(`chatctl`) — so that every primitive underneath a network server is visible +with nothing hidden: `socket`, `bind`, `listen`, `accept4`, a `sockaddr_in` +filled in by hand, and the one idea that TCP forces on every protocol built +on it — that a byte stream has no message boundaries, so *you* must add them. + +The code is in `examples/21-tcp-sockets/`. `./demo.sh` there builds all three +implementations; its `README.md` fixes the CLI, the exit codes, and the wire +frame that chapters 22–24 and 27 grow but never replace. + +{% include excalidraw.html + file="21-server-lifecycle" + alt="A top band labeled serve — one listener thread holds five boxes left to right: socket() AF_INET SOCK_STREAM|CLOEXEC, setsockopt SO_REUSEADDR = 1 rebind past TIME_WAIT, bind() sockaddr_in 127.0.0.1:9000, listen(128) passive socket fd 3, and an amber accept4() loop SOCK_CLOEXEC producing new fd 4, 5. Two amber spawn-thread arrows fall from the accept loop into a lower band, one std::jthread or goroutine or std::thread per accepted connection, holding serve_client fd 4 (bob) and serve_client fd 5 (alice). An amber note reads SIGINT arrow g_stop = 1; poll(2) wakes arrow stop accepting; listener fd closes; threads join; exit 0." + caption="Figure 21.1 — chatterd serve's lifecycle: the four setup calls, the accept4 loop, and one thread per accepted fd; every fd number here reappears in the ss and strace captures later" %} + +> **Tools used** — `ss`, `strace`, `python3`, `mktemp`, `grep` (host); +> `tcpdump`/`perf` (host, where privileged), and bcc-tools +> (`tcplife`/`tcpconnect`/`offcputime`) on the lab VM, exercised in Part 8. +> Everything host-side is checked by `scripts/check-host.sh` or ships with +> Fedora. + +## Four calls to a listening socket + +A server socket is built by four syscalls in a fixed order, and `listen_tcp` +is the canonical spelling of all four, verbatim from +`examples/21-tcp-sockets/cpp/src/main.cpp`: + +```cpp +Result listen_tcp(const std::string& host, std::uint16_t port) { + Socket sock{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!sock.valid()) { + return std::unexpected(std::format("socket: {}", errno_text())); + } + int one = 1; + ::setsockopt(sock.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(std::format("{}: not an IPv4 address", host)); + } + if (::bind(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected(std::format("bind {}:{}: {}", host, port, errno_text())); + } + if (::listen(sock.get(), 128) < 0) { + return std::unexpected(std::format("listen: {}", errno_text())); + } + return sock; +} +``` + +`socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)` asks for an IPv4 +(`AF_INET`) reliable byte stream (`SOCK_STREAM` — TCP), and folds close-on-exec +into the creating call the same way chapter 7 insisted every fd source must: +an `accept4` a moment later has the same discipline, because a listener that +leaks into a forked child is a port you can never rebind. The returned fd is +not yet an address — `bind` gives it one. `sockaddr_in` is the IPv4 address +struct, and its two fields carry a trap worth stating once: `sin_port` and +`sin_addr` are **network byte order** (big-endian), which is why the port +goes through `htons` (host-to-network-short) and the dotted-quad string goes +through `inet_pton` (presentation-to-network) rather than being assigned +raw. Skip `htons` on a little-endian x86 box and you bind port 9000 as +0x2823 — 10275 — and spend an afternoon wondering why nobody can connect. +`inet_pton` here also does double duty as validation: it returns 1 only for a +real IPv4 literal, so a hostname or garbage gets the `not an IPv4 address` +error instead of a mysterious `bind` failure. Turning a *name* like +`chat.example.com` into a `sockaddr` is a different job — `getaddrinfo(3)`, +which also picks IPv4-vs-IPv6 for you; this example stays numeric on purpose +and leaves resolution to chapter 23. + +`bind` attaches the address; `listen(fd, 128)` flips the socket from active +to **passive** — it will never `connect` out, only accept in — and sizes the +*accept backlog*: the queue of connections the kernel completes the handshake +for while your code is between `accept` calls. 128 is the number of clients +that can be mid-connect before the kernel starts refusing; for a chat room it +is comfortably more than enough. + +### SO_REUSEADDR, and the TIME_WAIT you will hit in ten minutes + +The one line that looks optional and is not: `setsockopt(..., SO_REUSEADDR, +...)`. When a TCP connection closes, the end that closed *actively* parks the +socket in `TIME_WAIT` for a couple of minutes, holding the `(local addr, +local port, remote addr, remote port)` tuple so a late straggler packet from +the old connection cannot be mistaken for the new one. For a server that +means: kill `serve`, restart it within that window, and `bind` fails with +`EADDRINUSE` — "Address already in use" — because a lingering client +connection still pins the port. `SO_REUSEADDR` tells the kernel that a +`bind` to a port in `TIME_WAIT` is fine, and it is the reason you can Ctrl-C +this server and relaunch it immediately. Go sets it for you on every TCP +listener (which is why the Go `cmdServe` is a bare `net.Listen`); the Rust +build sets it explicitly through `nix` (`setsockopt(&fd, ReuseAddr, &true)`) +before `bind`, exactly as the C++ does. + +## TCP is a byte stream, so we frame it ourselves + +Here is the idea the rest of the book is built on. A UNIX pipe and a TCP +connection are both `SOCK_STREAM`: a reliable, ordered sequence of bytes with +**no message boundaries**. If a client sends `build_join("alice")` and then +`build_msg("hi")` back to back, the server's next `read` may return all of +the first frame and half the second, or two-and-a-half frames, or one byte. +TCP guarantees the *bytes* arrive in order and intact; it guarantees nothing +about how they are grouped into `read` calls. A protocol that assumed "one +`read` == one message" would work on loopback in testing and shatter under a +real network the first time a segment split. + +The fix is **length-prefix framing**, wrapped in a small fixed header, and it +is fixed for the whole book: chapters 22, 24, and 27 add message *types* to +this exact 6-byte header, never a new one. Every message, both directions, +opens with a 2-byte magic `0x43 0x48` (ASCII `"CH"`), a 1-byte version +(`0x01`), a 1-byte `type`, then a 2-byte **big-endian** payload length +(`0..65535`), and finally exactly that many payload bytes. + +{% include excalidraw.html + file="21-frame-format" + alt="A row of 21 byte cells for the server's DELIVER broadcast of alice's message to bob. The first six cells, amber, hold 43 48 01 03 00 0f labeled magic CH, version 0x01, type 0x03 DELIVER, length (u16BE) = 15. The next five cells hold 61 6c 69 63 65 labeled name = alice. One dark cell holds 00 labeled NUL. Nine cells hold 68 65 6c 6c 6f 20 62 6f 62 labeled text = hello bob. Below: 00 0f = 0x000f = 15 = len(alice) + 1 (NUL) + len(hello bob). Then the frames that got here: alice's own MSG frame (type 0x02) carried just hello bob as its payload, no name; JOIN (type 0x01) payload is name alone, sent once at connect. A footnote notes length is u16BE, 0..65535, counts payload bytes only, and the 6 header bytes are never counted; WELCOME (0x04) and PING (0x05) are declared and reserved but neither sent nor expected in this chapter." + caption="Figure 21.2 — the wire frame byte for byte: the DELIVER broadcast of alice's message to bob; 43 48 01 03 00 0f is the exact magic+version+type+length header captured off the socket in the cross-check below" %} + +The `type` byte is what makes the NUL separator mostly disappear: a **JOIN** +frame's payload is just the sender's `name`; a **MSG** frame's payload is the +chat `text` alone, nothing else; the server relays every MSG as a **DELIVER** +frame whose payload is `name` `0x00` `text` — the *only* frame that still +needs a NUL, because it is the only one carrying two fields to a reader who +never sent them itself. `WELCOME` and `PING` are declared and reserved for +chapters 22 and 24 but never sent or read here. The length prefix means the +reader never scans for a delimiter — it reads exactly the 6 header bytes, +learns the type and size from them, reads exactly that many payload bytes, +and is done. There is no separate "reject an oversized frame" check the way +an unbounded length would need: `length` is 16 bits, so 65535 bytes is the +largest payload the wire can even express, and that ceiling is a property of +the field, not a runtime guard the reader has to remember to add. + +## How the code works + +The reader is the heart of the program, and where the length-prefix +discipline lives. Here it is in all three languages, each verbatim from +`examples/21-tcp-sockets/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +std::optional read_frame(int fd) { + std::uint8_t hdr[kHeaderSize]; + if (!read_full(fd, hdr, kHeaderSize)) { + return std::nullopt; + } + if (hdr[0] != kMagic0 || hdr[1] != kMagic1 || hdr[2] != kWireVersion) { + return std::nullopt; // wrong magic/version: not a chatterd frame + } + std::uint8_t type = hdr[3]; + std::uint16_t len = (static_cast(hdr[4]) << 8) | hdr[5]; + std::string payload(len, '\0'); + if (len > 0 && !read_full(fd, payload.data(), len)) { + return std::nullopt; + } + return RawFrame{type, std::move(payload)}; +} +``` + +```go +// readFrame returns the frame's type and payload, or an error; io.EOF marks a +// clean close. +func readFrame(r *bufio.Reader) (byte, []byte, error) { + var hdr [headerSize]byte + if _, err := io.ReadFull(r, hdr[:]); err != nil { + return 0, nil, err + } + if hdr[0] != magic0 || hdr[1] != magic1 || hdr[2] != version { + return 0, nil, fmt.Errorf("not a chatterd frame (bad magic/version)") + } + typ := hdr[3] + n := binary.BigEndian.Uint16(hdr[4:6]) + payload := make([]byte, n) + if n > 0 { + if _, err := io.ReadFull(r, payload); err != nil { + return 0, nil, err + } + } + return typ, payload, nil +} +``` + +```rust +/// Read one frame; `Ok(None)` marks a clean EOF, `Err` a protocol fault. +fn read_frame(r: &mut R) -> Result)>> { + let mut hdr = [0u8; HEADER_SIZE]; + match r.read_exact(&mut hdr) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None), + Err(e) => return Err(e.into()), + } + if hdr[0..2] != MAGIC || hdr[2] != WIRE_VERSION { + bail!("not a chatterd frame (bad magic/version)"); + } + let frame_type = hdr[3]; + let len = u16::from_be_bytes([hdr[4], hdr[5]]); + let mut payload = vec![0u8; len as usize]; + if len > 0 { + match r.read_exact(&mut payload) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None), + Err(e) => return Err(e.into()), + } + } + Ok(Some((frame_type, payload))) +} +``` + +The three are the same state machine in three vocabularies. The load-bearing +call is still *read-exactly-n*: C++ hand-rolls it in `read_full`, a loop that +keeps calling `read` until `n` bytes have accumulated, retrying `EINTR` and +returning false on EOF; Go and Rust get it from the standard library +(`io.ReadFull`, `read_exact`). This loop is the whole point — a single +`read` returning fewer bytes than asked is not an error, it is Tuesday, and +the reader that treats a short read as a complete message is the bug the +length prefix exists to prevent. The 6-byte header is read whole, then +checked before anything in it is trusted: `hdr[0]`/`hdr[1]` must be `'C'`/`'H'` +and `hdr[2]` must be the version this build speaks (`0x01`), or the frame is +rejected outright — a stray non-chatterd byte stream, or a future version this +build doesn't understand, is refused before its length field is ever read as +a byte count. Only then are `type` (`hdr[3]`) and the big-endian 16-bit +`length` (`hdr[4..6]`, decoded with explicit shifts in C++, +`binary.BigEndian` in Go, `u16::from_be_bytes` in Rust) trusted, and the +payload allocated to exactly `length` bytes and read in full — capped at +65535 simply because the field is 16 bits, with no separate size check to +remember. None of the three distinguishes a bad magic/version from a short +read with a diagnostic in this chapter: a truncated header, a wrong magic, +and a truncated payload all end the connection the same quiet way a clean EOF +does — `read_frame` returns `std::nullopt`/an `error`/(mostly) `Ok(None)`, and +the caller simply stops reading that connection. Rust's return type is the +one place the distinction survives structurally (`Ok(None)` is EOF, +`Err(...)` is a bad magic/version via `bail!`), but `serve_client`'s +`_ => break` arm folds both back into the same outcome anyway. + +Around the reader sits the accept loop of Figure 21.1. `cmd_serve` installs a +SIGINT handler, then loops: `poll` the listener with a 200 ms timeout (so a +SIGINT that flips `g_stop` is noticed even between connections), +`accept4(..., SOCK_CLOEXEC)` a ready connection into a fresh `Socket`, bump an +id, and hand the connection to a `std::jthread` running `serve_client`. Each +connection thread reads frames forever; a JOIN logs `chatterd: joined` +and broadcasts a `server\0 joined` DELIVER notice to **all** members, +while a MSG is relayed as a DELIVER carrying `\0` to every client +**except** the sender — you already see what you typed. A `Hub` (a +`std::vector` under a `std::mutex`, a `map[int]*client` in Go, a +`Mutex` in Rust) is the shared roster every thread reads and writes. + +The RAII `Socket` is the quiet hero. It is a move-only handle that `close`s +its fd in the destructor and nowhere else — there is not a single naked +`close` in the C++ program. When `serve_client` returns, its `Socket` +parameter dies and the connection closes; when `cmd_serve` returns, the +listener `Socket` dies last and the port frees. Ownership *is* lifetime, so +"who closes this fd, and when" is answered by scope rather than by vigilance +— the same lesson chapter 7 drew for files, now load-bearing for a server +that opens and closes a socket per client. + +## Errors, three ways + +The contract is the book's usual triple: exit 0 clean, exit 1 for a runtime +failure with one diagnostic line, exit 2 for a usage error — and one +`verify.lua` asserts all three across the languages (19 behavioral checks +each). The instructive case is a client against a dead port. All three report +it and exit 1, and the errno underneath is identical (`ECONNREFUSED`) even as +each language's error type dresses it differently: + +```console +[host]$ ./cpp/build/release/app chatctl --port 9099 --name x < /dev/null +chatctl: error: connect 127.0.0.1:9099: Connection refused +[host]$ ./go/bin/app chatctl --port 9099 --name x < /dev/null +chatctl: error: connect 127.0.0.1:9099: dial tcp 127.0.0.1:9099: connect: connection refused +[host]$ ./rust/target/release/app chatctl --port 9099 --name x < /dev/null +chatctl: error: connect 127.0.0.1:9099: Connection refused (os error 111) +``` + +C++ carries the failure in `std::expected` and formats +the `std::error_code`; Go wraps with `%w` so the `dial tcp` context chains in; +Rust attaches `anyhow` context and prints the bare `os error 111`. A server-side +`bind` that loses (a privileged port, say) is the same shape from the other +side: `chatterd: error: bind 127.0.0.1:80: Permission denied`, exit 1. Usage +slips — no subcommand, `serve` without `--port`, `chatctl` without `--name` — +all short-circuit to the two-line usage text and exit 2 before any socket is +opened. Inside the connection threads, errors are handled by *ending that one +connection*, never the server: a peer that vanishes mid-write makes `write_all` +return quietly (there is nobody left to tell), and the accept loop takes the +next caller. + +## Concurrency lens + +`chatterd` v0 is **thread-per-connection**: one OS thread (or goroutine) blocked +in `read_frame` per client, plus the listener thread in `accept`. It is the +easiest server model to reason about — each connection is a straight-line +blocking program, and the only shared state is the `Hub` behind a mutex — and +it is exactly the model that stops scaling, which is why chapter 22 exists. +Two cracks are already visible. First, **cost per connection**: a thread is a +megabyte or two of stack plus a scheduler entry, so ten thousand idle chat +clients is gigabytes of RAM and a scheduler run queue that thrashes — the +classic C10K wall. Second, and more subtle, **head-of-line blocking in the +broadcast**: the C++ and Rust `broadcast` hold the `Hub` mutex while writing +to every client in turn, so one client whose kernel send buffer is full (a +slow reader) stalls `write` and freezes *every other* broadcast behind the +lock. Go sidesteps it structurally — each client owns a 64-deep buffered +channel and a writer goroutine, and `broadcast` does a non-blocking +`select { case c.out <- frame: case <-c.done: }`, so a wedged client is +skipped rather than allowed to block the room. That contrast — a mutex held +across I/O versus a bounded queue per consumer — is the seam chapter 22 opens +up when it moves the server onto one epoll loop and a thread pool. For now, the +three shutdown strategies all reach the same observable end: C++ interrupts +`poll` with SIGINT and lets each `std::jthread`'s `stop_callback` shut its +socket down; Go closes the listener out from under `Accept` via a cancelled +`context`; Rust flips an `AtomicBool` a non-blocking `accept` loop polls. All +three close the listener and exit 0. + +## Build, run, observe + +```bash +[host]$ cd examples/21-tcp-sockets && ./demo.sh build +``` + +`python3 scripts/test-all-examples.py --only 21-tcp-sockets` builds all three +and reports `PASS PASS PASS` (3 passed, 0 failed, 0 skipped; `verify.lua` 19/19 +per language). To drive it by hand the way this chapter's evidence was +produced, start a room and connect two clients — bob first, so he witnesses +alice arrive: + +```bash +[host]$ ./cpp/build/release/app serve --port 9001 --host 127.0.0.1 & +chatterd: listening on 127.0.0.1:9001 +[host]$ ./cpp/build/release/app chatctl --port 9001 --name bob # terminal 2 +[host]$ ./cpp/build/release/app chatctl --port 9001 --name alice # terminal 3 +``` + +With both connected, alice typing `hello bob` on stdin lands only in bob's +terminal, never echoed back to her. The observed transcripts and server log +from the scripted run were exactly: + +``` +# alice's terminal # bob's terminal # server stderr +server: alice joined server: bob joined chatterd: listening on 127.0.0.1:9001 + server: alice joined chatterd: bob joined + alice: hello bob chatterd: alice joined + server: alice left chatterd: alice left + chatterd: bob left + chatterd: shutting down +``` + +bob sees his own join notice (joins go to everyone), then alice's join, then +her message, then her leave when she Ctrl-Ds out; alice never sees `alice: +hello bob` because the sender is excluded from chat broadcasts. A SIGINT to +`serve` printed `chatterd: shutting down` and the process exited 0. + +## Cross-check: the listener, the connections, and the bytes on the wire + +Three independent tools confirm the mechanism rather than just the output. +**First, `ss -tlnp` on the running listener** shows the passive socket, its +backlog, and its owner: + +```console +[host]$ ss -tlnp | grep :9000 +LISTEN 0 128 127.0.0.1:9000 0.0.0.0:* users:(("app",pid=2816579,fd=3)) +``` + +`LISTEN`, the `128` backlog we passed to `listen`, the bound `127.0.0.1:9000`, +and `fd=3` in process `app` — Figure 21.1's listener box, live. **Second, +`ss -tnp` during a two-client session** shows four established sockets — two +in the server, one in each client — and the peer columns pair them by the +ephemeral ports the kernel assigned the clients: + +```console +[host]$ ss -tnp | grep :9001 +ESTAB 0 0 127.0.0.1:9001 127.0.0.1:47504 users:(("app",pid=2818525,fd=4)) +ESTAB 0 0 127.0.0.1:9001 127.0.0.1:47512 users:(("app",pid=2818525,fd=5)) +ESTAB 0 0 127.0.0.1:47504 127.0.0.1:9001 users:(("app",pid=2818649,fd=3)) +ESTAB 0 0 127.0.0.1:47512 127.0.0.1:9001 users:(("app",pid=2818655,fd=3)) +``` + +Server pid 2818525 holds `fd 4` and `fd 5` — the two `accept4` results from +Figure 21.1 — each paired with a client's ephemeral port (47504 for bob, +47512 for alice), while each client holds its `fd 3` connecting side. **Third, +the frame bytes themselves.** `tcpdump -i lo` reads them off the wire, but the +capture socket needs `CAP_NET_RAW`; on this unprivileged host the identical +bytes are visible one layer down, at the `read`/`write` syscalls, under +`strace` — the exact sequence the socket handed to (and took from) TCP during +alice's `hello bob`: + +```console +[host]$ strace -f -tt -e trace=accept4,read,write -s 80 ./cpp/build/release/app serve --port 9002 ... +accept4(3, NULL, NULL, SOCK_CLOEXEC) = 4 +accept4(3, NULL, NULL, SOCK_CLOEXEC) = 5 +read(5, "CH\1\1\0\5", 6) = 6 +read(5, "alice", 5) = 5 +read(5, "CH\1\2\0\t", 6) = 6 +read(5, "hello bob", 9) = 9 +write(4, "CH\1\3\0\17alice\0hello bob", 21) = 21 +``` + +Every claim in Figure 21.2 is on those lines (bob connected first onto `fd 4`; +his own JOIN read and the join-notice broadcasts are elided above for space — +they carry the same shape, just a different type and length). `accept4` +returns fds 4 and 5, each `SOCK_CLOEXEC` as the source asked. The reader +takes the fixed 6-byte header first — `read(5, "CH\1\1\0\5", 6)` is `43 48` +(`"CH"`), `01` (version), `01` (type = JOIN), `00 05` (length = 5, +big-endian) — then exactly that many payload bytes (`"alice"`), a strict +two-phase read that is the length-prefix discipline in the raw. alice's chat +line repeats the shape with a different type: `"CH\1\2\0\t"` is +`43 48 01 02 00 09` — type `02` = MSG, length 9 — followed by the 9-byte +payload `"hello bob"`, with no name in it at all; the connection already told +the server who she is. The server's relayed DELIVER, `write(4, +"CH\1\3\0\17alice\0hello bob", 21)`, is the whole broadcast in one write: +header `43 48 01 03 00 0f` (type `03` = DELIVER, length 15 = `len("alice")` +`+ 1 (NUL) + len("hello bob")`), then the 15-byte payload `alice\0hello bob` +— the NUL still separating name from text, because DELIVER is the only type +that carries both. That length prefix, octal `\17`, is `0f` in hex — 15 — +the same 6-byte magic/version/type/length header that would show up in a +`tcpdump` capture off `lo`, byte for byte the amber header of Figure 21.2. + +> **On the lab VM** unverified — +> the fleet view of these connections is bcc-tools, not `strace`: +> `tcpconnect`/`tcplife` trace every TCP session on the box with its bytes and +> lifetime, and `offcputime` shows the connection threads parked in +> `read`/`accept` (which is where a thread-per-connection server spends nearly +> all of its time). Those tools need root and a disposable kernel; the +> Debugging part (Part 8) runs them against `chatterd` on the +> `systems-target` VM. + +## What you learned + +- A server socket is four ordered calls — `socket` → `bind` → `listen` → + `accept4` — with `sockaddr_in` in network byte order (`htons`, `inet_pton`) + and `SO_REUSEADDR` set before `bind` so a restart can rebind a port still + held in `TIME_WAIT`; `ss -tlnp` showed the passive socket with backlog 128 + on `fd 3`, and `ss -tnp` the four established halves of a two-client + session with their paired ephemeral ports. +- TCP is a boundaryless byte stream: a `read` can return part of a frame or + several, so every message opens with a fixed 6-byte header — magic `CH`, + version, type, then a big-endian `u16` length — and a reader that loops + until it has exactly the header, then exactly the payload; `strace` caught + that two-phase read and the `43 48 01 03 00 0f` (DELIVER, length 15) header + on the wire. +- The RAII `Socket` owns every fd by scope, so there is no naked `close`; and + thread-per-connection is the simple model that chapter 22 must leave behind + — a thread and a stack per client, plus a broadcast that holds the roster + lock across I/O and stalls behind one slow reader. + +Next, **scaling the server**: chapter 22 takes the same `chatterd` frame off +threads and onto a single epoll loop with nonblocking sockets, so ten +thousand idle clients cost fds instead of stacks. + +--- + +

verified — every number and +output excerpt above was produced on the Fedora 44 reference host (kernel +7.1.3-200.fc44, strace 7.1) this session. The runner printed +21-tcp-sockets PASS PASS PASS (3 passed, 0 failed, 0 skipped; +verify.lua 19/19 per language). The hand-driven two-client +session produced the exact transcripts shown — bob received +alice: hello bob, alice saw no echo of her own line, both clients +saw a server-sender join notice, the server logged each join and +chatterd: shutting down, and SIGINT exited 0. ss -tlnp +showed LISTEN 0 128 127.0.0.1:9000 … fd=3; ss -tnp +showed the four established sockets with server pid 2818525 holding fd 4/5 and +clients on ephemeral ports 47504/47512. The connect-refused, bind-denied, and +usage error strings and exit codes (1/1/2) are real output. The +strace excerpt is real trimmed output re-captured against the +canonical CH frame: accept4 … = 4 and = 5, the +two-phase header-then-payload reads for alice's JOIN (CH\1\1\0\5 +then alice) and MSG (CH\1\2\0\t then +hello bob), and the server's relayed DELIVER, +write(4, "CH\1\3\0\17alice\0hello bob", 21) — the +\17 (0f in hex, 15) big-endian length half of the +6-byte magic/version/type/length header. tcpdump on +lo was not run (needs CAP_NET_RAW, unavailable in +this unprivileged session); the identical wire bytes were captured at the +syscall boundary instead. The bcc-tools "On the lab VM" callout is unverified +as marked and is exercised in Part 8.

diff --git a/_docs/22-scaling-the-server.md b/_docs/22-scaling-the-server.md new file mode 100644 index 0000000..fe856c5 --- /dev/null +++ b/_docs/22-scaling-the-server.md @@ -0,0 +1,439 @@ +--- +title: "Scaling the server" +order: 22 +part: "Networking" +description: "chatterd v1 holds many connections from one thread: nonblocking accept and per-connection read/write state machines, an outbound queue with EPOLLOUT backpressure reusing chapter 9's level-triggered epoll — then the chapter's real comparison, Go's goroutine-per-connection model with the runtime netpoller running epoll_pwait underneath, proven with per-process Threads counts, ss -tp, and strace." +duration: "55 minutes" +--- + +Chapter 21 gave `chatterd` a length-prefixed frame protocol and the obvious +server: one OS thread per connection, each blocked in `recv`. That design is +correct and it is easy to reason about — right up to the point a decade of +network programmers named the **C10K problem**: ten thousand idle clients means +ten thousand kernel threads, ten thousand stacks, and a scheduler run queue that +spends more time context-switching than working. This chapter keeps the threaded +engine (it still passes every test) and adds a second one that holds the same N +connections from a *single* thread. The one new idea is the **readiness model +from chapter 9, aimed at sockets**: put every fd in nonblocking mode, let one +`epoll_wait` tell you which are ready, and make each connection a little state +machine with an inbound buffer and an outbound queue so a partial read or a +blocked write is just *state*, never a stalled thread. Then the payoff — Go +writes the naive goroutine-per-connection code and gets the epoll engine for +free, because its runtime hides exactly this machinery underneath. + +The code is in `examples/22-scaling-the-server/`. `./demo.sh` there builds all +three implementations and runs an end-to-end demo; its `README.md` specifies the +CLI, the wire format (unchanged since chapter 21), and the exit codes all three +languages share. + +{% include excalidraw.html + file="22-connection-state-machine" + alt="Two bands. The top READ PATH band runs left to right: a recv() loop draining nonblocking until EAGAIN feeds an inbuf that reassembles partial frames across reads, into take_frame which asks whether one whole 6+LEN frame is ready, into MSG to broadcast which appends the frame to every connection's outbuf. The bottom WRITE PATH and BACKPRESSURE band: broadcast drops for each conn into queue() plus flush() which send()s what fits now; on send() EAGAIN the remainder stays in outbuf (the backpressure queue) because the kernel socket buffer is full; an amber arrow labelled full: arm EPOLLOUT runs down-left to an epoll_wait node that later reports EPOLLOUT for this conn, and an amber arrow labelled EPOLLOUT wakeup runs back up-right to flush(), closing the loop. A note reads: level-triggered, EPOLLIN always armed, EPOLLOUT armed only while outbuf is non-empty." + caption="Figure 22.1 — chatterd's epoll engine as one per-connection read/write state machine; the EPOLLOUT backpressure loop is the whole reason a slow client cannot stall the server" %} + +> **Tools used** — `strace`, `ss`, `python3` (host); `perf`, bcc-tools +> `offcputime`/`tcplife`/`tcpconnect` (lab VM, exercised in Part 8). Everything +> host-side is checked by `scripts/check-host.sh` or ships with Fedora. + +## Nonblocking sockets, EAGAIN, and the state machine + +A blocking server couples one thread to one connection because `recv` *waits*. +Break that coupling and a single thread can service everyone, but the price is +that every operation now returns "not yet" instead of blocking — and "not yet" +has a name: `EAGAIN` (equivalently `EWOULDBLOCK`). The listener is created +nonblocking (`SOCK_NONBLOCK` in the `socket` type), so `accept4` returns +`EAGAIN` when the backlog is drained; each client socket is nonblocking, so +`recv` returns `EAGAIN` when the kernel receive buffer is empty and `send` +returns `EAGAIN` when its *send* buffer is full. None of those are errors — they +are the readiness model telling you to go wait in `epoll_wait` and come back. + +Because a nonblocking `recv` can hand you *any* number of bytes — half a frame, +three frames, one frame plus a header — a connection cannot be a straight-line +function anymore. It has to be **state that survives across `epoll_wait` +iterations**. That is what `EConn` is: + +```cpp +struct EConn { + Fd fd; + std::string nick = "?"; + std::string inbuf; + std::string outbuf; // pending outbound bytes (the backpressure queue) + bool want_write = false; + bool dead = false; +}; +``` + +`inbuf` accumulates whatever `recv` returns until `take_frame` — the same +6-byte-header parser from chapter 21 — can lift a whole frame off the front; +leftover bytes stay for next time. `outbuf` is the mirror image and the heart of +the chapter: bytes we *want* to send but the kernel would not take. `want_write` +records whether we currently have `EPOLLOUT` armed for this connection (so we +only call `epoll_ctl` on a real transition), and `dead` marks a connection to +reap after the current event batch rather than deleting it mid-iteration. The +connections live in `std::unordered_map` keyed by an epoll +**token**, `next_tok_` starting at 2 because tokens 0 and 1 are reserved for the +listener and the `signalfd` — the same "let the kernel echo back a small integer +so dispatch is a `switch`, not an fd lookup" trick chapter 9 used. + +## How the code works + +Setup is four fallible steps, each guarded by `std::expected`: +`make_listener(port, nonblock=true)`, `make_signalfd` (block `SIGINT`/`SIGTERM` +process-wide, then read them as data), `epoll_create1(EPOLL_CLOEXEC)`, and two +`epoll_ctl` ADDs registering the listener and the signal fd with `EPOLLIN`. Then +`run()` is the entire server: a `switch` on the token the kernel returned. + +```cpp + for (int i = 0; i < n; ++i) { + const auto tok = events[i].data.u64; + if (tok == kListen) { + accept_ready(); + } else if (tok == kSignal) { + reap_closed(); + std::println(stderr, + "chatterd: stopped engine=epoll messages={} peak_conns={}", + messages_, peak_); + return 0; + } else { + if (!conns_.contains(tok)) { + continue; // reaped earlier in this batch + } + if ((events[i].events & (EPOLLHUP | EPOLLERR)) != 0) { + conns_.at(tok).dead = true; + } + if ((events[i].events & EPOLLIN) != 0) { + on_readable(tok); + } + if (conns_.contains(tok) && (events[i].events & EPOLLOUT) != 0) { + flush(tok); + } + } + } + reap_closed(); +``` + +Three details earn their place. The `conns_.contains(tok)` re-check before +`flush` matters because `on_readable` can mark a connection dead (or a broadcast +can), and acting on a fd we just decided to drop is a use-after-free waiting to +happen; deferring the actual erase to `reap_closed` at the end of the batch is +how a single-threaded loop safely mutates the container it is iterating events +from. `accept_ready` loops `accept4(…, SOCK_CLOEXEC | SOCK_NONBLOCK)` until +`EAGAIN` — one `EPOLLIN` on the listener can mean *many* pending connections, and +draining to `EAGAIN` is the level-triggered discipline from chapter 9 applied to +the backlog. Each new connection is registered `EPOLLIN`-only and immediately +handed a `WELCOME` frame via `queue`, which is the client's proof it is in the +fan-out set. `on_readable` drains `recv` into `inbuf` to `EAGAIN`, then pulls +every complete frame: a `JOIN` sets the nick, a `MSG` bumps the counter and +`broadcast`s a `DELIVER` frame — which is just `queue` called for every +connection. + +`queue` and `flush` are where backpressure lives, and where the three languages +are worth reading side by side. `queue` appends to `outbuf` and calls `flush`; +`flush` sends as much as the kernel will take and, crucially, *decides whether to +arm `EPOLLOUT`* based on what is left over: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp + void flush(std::uint64_t tok) { + auto& conn = conns_.at(tok); + while (!conn.outbuf.empty()) { + const ssize_t n = ::send(conn.fd.get(), conn.outbuf.data(), conn.outbuf.size(), + MSG_NOSIGNAL); + if (n > 0) { + conn.outbuf.erase(0, static_cast(n)); + } else if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) { + break; // socket buffer full: keep the rest, wait for EPOLLOUT + } else if (n < 0 && errno == EINTR) { + continue; + } else { + conn.dead = true; + return; + } + } + // Register/clear EPOLLOUT interest to match the backlog state. + const bool need = !conn.outbuf.empty(); + if (need != conn.want_write) { + conn.want_write = need; + epoll_event e{.events = EPOLLIN | (need ? EPOLLOUT : 0u), .data = {.u64 = tok}}; + ::epoll_ctl(epoll_.get(), EPOLL_CTL_MOD, conn.fd.get(), &e); + } + } +``` + +```go + case frame := <-h.broadcast: + h.messages.Add(1) + // Blocking send is the backpressure: a slow client slows the hub + // rather than dropping a frame. writePump keeps draining even on + // write errors, so this never deadlocks on a dead peer. + for c := range h.clients { + c.send <- frame + } +``` + +```rust +fn flush(reg: &mio::Registry, conns: &mut HashMap, tok: usize) { + let c = match conns.get_mut(&tok) { + Some(c) => c, + None => return, + }; + while !c.outbuf.is_empty() { + match c.stream.write(&c.outbuf) { + Ok(0) => { + c.dead = true; + return; + } + Ok(n) => { + c.outbuf.drain(0..n); + } + Err(e) if e.kind() == ErrorKind::WouldBlock => break, // buffer full: wait for WRITABLE + Err(e) if e.kind() == ErrorKind::Interrupted => continue, + Err(_) => { + c.dead = true; + return; + } + } + } + let need = !c.outbuf.is_empty(); + if need != c.want_write { + c.want_write = need; + let interest = if need { + Interest::READABLE | Interest::WRITABLE + } else { + Interest::READABLE + }; + let _ = reg.reregister(&mut c.stream, Token(tok), interest); + } +} +``` + +The C++ and Rust `flush` are the same state machine: drain until the socket says +`EAGAIN`/`WouldBlock`, then toggle write-interest to match. When a client stops +reading, its kernel send buffer fills, `send` returns `EAGAIN`, the remaining +bytes sit in `outbuf`, and `epoll_ctl`/`reregister` arms `EPOLLOUT` so the next +`epoll_wait` wakes us to try again — the amber loop in Figure 22.1. That is +backpressure with no thread blocked and no frame dropped: a slow reader grows its +own `outbuf` and nobody else's delivery is affected. The only asymmetry is the +library layer — mio's `reregister` takes an `Interest` set where raw epoll takes +a bitmask — but `EPOLLIN|EPOLLOUT` *is* `Interest::READABLE | Interest::WRITABLE` +underneath. Go's excerpt looks nothing alike, and that is the whole point of the +next section: its backpressure is a *blocking channel send*. `c.send` is a +buffered channel (capacity 64); when a client's `writePump` can't keep up the +channel fills, the send blocks, and the hub slows down — the same "slow reader +slows the writer, never drops" invariant, expressed as goroutines instead of +`EPOLLOUT`. + +## Go: goroutine-per-connection, epoll underneath + +{% include excalidraw.html + file="22-netpoller-vs-epoll" + alt="Two side-by-side bands. Left, titled C++/Rust — you write the epoll loop: one OS thread (Threads: 1) calling epoll_wait (4 calls for a 50-conn flood) connected by a bidirectional amber arrow labelled EPOLLIN/EPOLLOUT + token to a kernel eventpoll interest list holding tok 0 listener, tok 1 signalfd, tok 2..N every client socket, one process; a note reads backpressure = outbuf + EPOLLOUT (explicit). Right, titled Go — you write goroutines; the runtime hides the epoll: a goroutine per conn (read + write) plus hub with channels fanning out over ~7 OS threads total, a bidirectional amber arrow park/resume goroutine down to a dark runtime netpoller (hidden) box running epoll_pwait (165 calls, edge-triggered EPOLLET), then down to the same kernel sockets, one process, N conns (ss -tp); a note reads backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads." + caption="Figure 22.2 — the chapter's central comparison: the epoll loop you write by hand in C++/Rust is the epoll loop Go's runtime writes for you, driven by epoll_pwait one layer down" %} + +Go's server has no `epoll_create1`, no `epoll_ctl`, no `epoll_wait` — and it +still multiplexes thousands of sockets on a handful of threads. Each accepted +connection gets two goroutines (a `readPump` and a `writePump`) plus a shared +`hub` goroutine that owns the client set and fans `broadcast` out over channels. +The `readPump` calls `fr.next()`, which calls `conn.Read`, which *looks* +blocking; but `net.Conn` fds are registered with the runtime's **network +poller**, so a would-be-blocking `Read` parks the *goroutine* — not the OS +thread — until the netpoller's `epoll_pwait` reports the socket readable, then +resumes it. The `--engine threaded|epoll` flag on the Go binary changes only the +label it prints; both values run this exact code, because for Go there is no +second design to write. This is the payoff Figure 22.2 draws: the hand-rolled +epoll interest list on the left and the runtime netpoller on the right are the +same kernel object, and the `strace` cross-check below shows Go issuing the +`epoll_pwait` calls our C++ loop makes explicit — plus the edge-triggered +`EPOLLET` registration, because the runtime, unlike our level-triggered loop, +always drains to `EAGAIN` and parks on the transition. + +## Errors, three ways + +The contract is chapter 21's: exit 0 on success, 1 on a runtime failure with one +diagnostic line, 2 on a usage error — and `verify.lua` asserts all three per +language (24 checks each, 72 total). The interesting split is which side owns +which error. A client against a dead port fails `connect` and exits 1 with +`chatctl: cannot connect to 127.0.0.1:1: Connection refused` — `std::error_code` +text in C++, a `%w` chain in Go, `anyhow` context in Rust, the same errno +underneath. The *server* must survive its clients: a peer that hangs up mid-write +turns `send` into an error, which just marks that `EConn` dead and reaps it — +`MSG_NOSIGNAL` on every `send` is what stops a dead peer from killing the whole +process with `SIGPIPE`. A foreign speaker whose bytes don't start with the `CH` +magic desyncs one connection; `take_frame` clears its buffer and the stream is +dropped, never the server. And the two "not yet" returns are *successes*, not +failures: `EAGAIN` from `recv` means "drained, go wait", `EAGAIN` from `send` +means "buffered, arm `EPOLLOUT`". Confusing either for an error is how naive +nonblocking servers spin at 100% CPU or drop frames. + +## Concurrency lens + +Count the threads doing your work, and the three engines separate cleanly. The +epoll engine is **one** thread: `conns_` is touched by nobody else, so there are +no locks, no atomics, and the event loop serializes every mutation by +construction — the single-threaded design *is* the concurrency strategy. The +threaded engine is **1 + N**: a `Registry` behind a mutex, a per-socket write +mutex so two broadcasters can't interleave bytes on one wire, and `std::atomic` +counters; simple per connection, but every message wakes N reader threads. Go is +a bounded handful of runtime threads regardless of connection count, with +`pending`-style state confined to the hub goroutine and everything else +communicating by channel — chapter 6's "share memory by communicating" applied to +I/O. Measured on this host with 50 connections held open: the C++ epoll daemon +reports `Threads: 1`, the C++ threaded daemon `Threads: 51`, and the Go daemon +`Threads: 7` (near, but not pinned to, this box's 16 CPUs — the runtime grows +threads on demand up to `GOMAXPROCS` runnable). One process, one design decision, +three very different rows in `/proc//status`. + +## Build, run, observe + +```bash +[host]$ cd examples/22-scaling-the-server && ./demo.sh cpp +``` + +The runner (`python3 scripts/test-all-examples.py --only 22-scaling-the-server`) +reports `PASS PASS PASS` for cpp, go, rust (3 passed, 0 failed, 0 skipped). To +drive it by hand the way this chapter's evidence was produced, start an epoll +daemon on an ephemeral port, listen as `bob`, and send as `alice`: + +```bash +[host]$ ./cpp/build/release/app serve --engine epoll --port 0 & +chatterd: serving engine=epoll on 127.0.0.1:45829 +[host]$ ./cpp/build/release/app listen --port 45829 bob --count 1 & +[host]$ ./cpp/build/release/app send --port 45829 alice "hello, chatterd" +alice: hello, chatterd +``` + +`bob`'s terminal prints the same `alice: hello, chatterd` line, because every +`MSG` is delivered to every connection including the sender — seeing your own +echo is how `send` knows it was accepted. On `SIGTERM` the daemon prints +`chatterd: stopped engine=epoll messages=1 peak_conns=2`. The load case is +`flood`, which opens N concurrent connections, waits for all N `WELCOME`s, has +one of them broadcast, and counts deliveries: + +```bash +[host]$ ./cpp/build/release/app flood --port 45829 50 +flood: connected 50 +flood: delivered 50 +``` + +Fifty connections, one broadcast, fifty deliveries, no dropped frames — and the +daemon's summary reads `messages=1 peak_conns=50`. The threaded engine and all +three languages produce byte-identical output; that identity is what lets one +`verify.lua` cover them. + +## Cross-check: one thread, one process, one epoll_wait + +Output equality could hide a server that quietly spun up a thread per client, so +the cross-checks watch the mechanism. **First, the shape of the process.** With +50 `listen` clients held open against each engine, `/proc//status` and +`ss -tnp` agree: + +```console +[host]$ grep Threads /proc//status ; ss -tnp state established | grep -c '"app",pid=,' +Threads: 1 # epoll engine (Rust: also 1) +50 +Threads: 51 # threaded engine — one thread per connection +50 +Threads: 7 # Go — netpoller, a bounded thread pool +50 +``` + +All fifty established sockets belong to the *one* daemon pid in every case +(`users:(("app",pid=…,fd=12))` … `fd=49`); the engines differ only in how many +threads sit behind that one process. **Second, what drives the epoll engine.** +Tracing a two-party exchange, one thread does everything: + +```console +[host]$ strace -f -e trace=epoll_wait,accept4,epoll_ctl -p +2825978 epoll_wait(5, [{events=EPOLLIN, data=0}], 64, -1) = 1 +2825978 accept4(3, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 6 +2825978 epoll_ctl(5, EPOLL_CTL_ADD, 6, {events=EPOLLIN, data=0x2}) = 0 +2825978 accept4(3, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable) +2825978 epoll_wait(5, [{events=EPOLLIN, data=0x2}], 64, -1) = 1 +2825978 epoll_ctl(5, EPOLL_CTL_DEL, 6, NULL) = 0 +``` + +`data=0` is the listener token, `data=0x2`/`0x3` the two client tokens — exactly +the integers `accept_ready` assigned — and `accept4` loops to `EAGAIN` before the +next `epoll_wait`. Under a 50-connection flood, `strace -c` counts just **4** +`epoll_wait` calls covering 51 `accept4`, 100 `sendto`, and 100 `epoll_ctl` for +457 syscalls total: one wait multiplexes an entire batch. **Third, Go's hidden +epoll.** Tracing the Go binary shows the syscalls our code never wrote: + +```console +[host]$ strace -f -e trace=epoll_ctl,epoll_pwait -p +2826032 epoll_pwait(6, [{events=EPOLLIN, data=0x3ffafa0b2e000002}], 128, -1, NULL, 0) = 1 +2826032 epoll_ctl(6, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data=0x3ffafa0b3d000002}) = 0 +2826067 epoll_pwait(6 +``` + +The runtime registers sockets **edge-triggered** (`EPOLLET`) with an opaque +64-bit cookie where we used token integers, and issues the calls from *several* +threads (2826032, 2826067, 2826064): 165 `epoll_pwait` calls for the same flood +our C++ loop served in 4 `epoll_wait`s, the rest of its syscalls dominated by +`futex` and `nanosleep` from the scheduler. **Fourth, the C10K cost made +concrete.** Ten floods of 50 (500 connections) under `strace -c`: + +```console +engine=epoll : clone3 0 epoll_wait 40 accept4 510 total 4570 (0.017s in syscalls) +engine=threaded : clone3 500 accept4 500 total 8546 (1.85s in syscalls) +``` + +The threaded engine pays **one `clone3` per connection** — 500 thread creations — +and two orders of magnitude more time in the kernel; the epoll engine creates +zero threads and multiplexes the lot through 40 `epoll_wait` calls. That single +counter, `clone3: 500` versus `clone3: 0`, *is* the C10K problem in one line. + +> **On the lab VM** unverified — +> the kernel-side cost view — `perf stat` context-switches under load (this +> host's `perf_event_paranoid=2` restricts the counter to userspace, reading 0 +> without privilege), `offcputime` flame graphs showing threaded workers parked +> off-CPU, and `tcplife`/`tcpconnect` tracing every connection's lifetime across +> the fleet — needs root and a disposable kernel, so it runs on the +> `systems-target` VM in Part 8, not here. + +## What you learned + +- The C10K escape is nonblocking sockets plus one `epoll_wait`: `EAGAIN` is not an + error but "go wait", and a connection becomes *state* (`inbuf`, `outbuf`) that + survives across wakeups rather than a thread blocked in `recv`. +- Backpressure is the outbound queue plus `EPOLLOUT`: a slow reader fills its own + `outbuf`, we arm write-interest, and the next `epoll_wait` drains it — no thread + blocks, no frame drops, no other client is affected. Level-triggered epoll from + chapter 9 makes this safe with only "consume what woke you" discipline. +- Go's goroutine-per-connection code is the same epoll engine with the loop + hidden: the netpoller runs `epoll_pwait` (edge-triggered) underneath and parks + goroutines, not threads — 1 vs 51 vs 7 threads in `/proc`, all serving 50 + connections from one process. +- The mechanism is externally visible: `ss -tp` shows every connection on one + pid, `strace` shows `epoll_wait` (or the runtime's `epoll_pwait`) driving + everything, and `clone3: 0` versus `clone3: 500` measures the cost the whole + design exists to avoid. + +Next, **two hosts**: `chatterd` v2 leaves loopback for a real network — UDP +multicast discovery to find peers, then a TCP bridge carrying the same frames +between two VMs on one L2 segment. + +--- + +

verified — every number and +output excerpt above was produced on the Fedora 44 reference host (kernel +7.1.3-200.fc44, GCC 16.1.1, strace 7.1, 16 CPUs) this session: the runner printed +22-scaling-the-server PASS PASS PASS (3 passed, 0 failed, 0 +skipped; verify.lua 24 checks per language, 72 total); the hand-driven epoll +session produced the exact alice: hello, chatterd echo to both +sender and listener and the chatterd: stopped engine=epoll messages=1 +peak_conns=2 summary; flood 50 printed flood: connected +50 / flood: delivered 50 with peak_conns=50; with +50 connections held open, /proc/<pid>/status read +Threads: 1 (C++ and Rust epoll), 51 (C++ threaded), and +7 (Go), while ss -tnp showed all 50 established sockets +owned by the one daemon pid; the C++ epoll strace excerpt is real +trimmed output (single thread, tokens 0/0x2/0x3, accept4 to +EAGAIN, 4 epoll_wait calls for a 50-conn flood); the Go +trace showed epoll_ctl … EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET and 165 +epoll_pwait calls across several threads; and the 500-connection +strace -c comparison recorded clone3: 0 (epoll) versus +clone3: 500 (threaded). The perf/bcc-tools kernel-side +view is unverified as marked (host perf_event_paranoid=2) and is +exercised on systems-target in Part 8.

diff --git a/_docs/23-udp-and-peer-discovery.md b/_docs/23-udp-and-peer-discovery.md new file mode 100644 index 0000000..c6604bf --- /dev/null +++ b/_docs/23-udp-and-peer-discovery.md @@ -0,0 +1,391 @@ +--- +title: "UDP and peer discovery" +order: 23 +part: "Networking" +description: "chatterd learns to find peers it was never told about: UDP multicast beacons on 239.23.7.1, the IP_ADD_MEMBERSHIP / IP_MULTICAST_TTL setsockopt dance, a small gossip protocol deduped by name, and the bridge from a discovered datagram to the canonical chatterd TCP chat frame (introduced ch21) — proven across two libvirt guests with tcpdump, ss -uap, and both transcripts." +duration: "55 minutes" +--- + +Chapters 21 and 22 built `chatterd` on TCP: the canonical chatterd chat +frame — magic `"CH"`, version, type, a big-endian length, and a payload — +carried between two endpoints you already knew — you typed the peer's +address on the command line. This chapter removes that assumption. `chatterd +discover` finds peers it was never told about, and the primitive that makes +that possible is the one TCP cannot offer: **UDP multicast**, a single +`sendto` that reaches every listener in a group without the sender knowing +who — or how many — they are. The shape of the chapter is a bridge: a +connectionless UDP beacon announces "I exist, dial me here," a receiver hears +strangers and dedupes them, and the first time it meets a new name it opens +an ordinary TCP connection and speaks the *same canonical chat frame ch21 +introduced* — the header never changes; each version only adds message +*types*. Discovery is UDP; the conversation stays TCP. The star of the +run is real: two libvirt guests, `systems-target` and `systems-peer`, each +discovering the other across the `virbr0` bridge. + +The code is in `examples/23-udp-and-peer-discovery/`. `./demo.sh` there builds +all three implementations; its `README.md` specifies the CLI, the byte-exact +beacon and frame formats, and the exit codes all three languages share. + +{% include excalidraw.html + file="23-two-vm-topology" + alt="A libvirt virbr0 bridge band on 192.168.124.0/24 holds a central amber multicast-group box for 239.23.7.1 port 51824 TTL 1, with systems-target at 192.168.124.7 on the left and systems-peer at 192.168.124.95 on the right, each joined to the group on enp1s0 and holding udp fd 4 and tcp 9241; two dashed amber bidirectional beacon arrows (35 bytes from target, 34 bytes from peer) reach the group, and a solid bidirectional TCP arrow labelled one ch21 chat frame each way joins the two guests directly." + caption="Figure 23.1 — the two-guest lab topology: both VMs join 239.23.7.1, beacon to the group, and bridge each discovery to a direct TCP :9241 chat frame; every address, port, and byte-length here is from the two-host run later in the chapter" %} + +> **Tools used** — `ss`, `tcpdump`, `ip`, `strace` (host); `tcpdump`, `ss`, +> `ip maddr` (systems-target / systems-peer VMs, run live in this chapter); +> `tcplife`/`tcpconnect`/`offcputime` bcc-tools (lab VM, exercised in Part 8). +> Everything host-side is checked by `scripts/check-host.sh` or ships with +> Fedora; the VM tools ship in the lab cloud-init. + +## Why UDP, and why the datagram stays small + +TCP gives you a reliable, ordered byte *stream* over an established +connection; every one of those words is a cost. A connection is state on both +ends and a three-way handshake before the first byte; reliability means +retransmit timers and buffering; ordering means head-of-line blocking. For +discovery you want the opposite of all of it. You do not know who your peers +are, so there is nobody to connect *to*; you want one announcement to reach +many; and you do not care if a single beacon is lost, because another one +follows in 200 ms. That is UDP: connectionless, message-oriented, best-effort. +`sendto` hands the kernel one datagram, it goes on the wire as one IP packet +(or a fragmented set), and the receiver's `recv` returns it whole or not at +all — **message boundaries are preserved**, unlike a TCP stream where you must +re-frame yourself. Loss and reorder are possible and, for a beacon, harmless: +a dropped beacon just means "discovered one round later." + +The datagram stays small on purpose. `chatterd`'s beacon is one ASCII line — +`CHATTERD1 ` — 34 or 35 bytes on the wire in the +run below, far under the ~1472-byte non-fragmenting payload of a 1500-byte +MTU. Keeping a beacon inside one un-fragmented packet is a real discipline: +UDP has no reassembly guarantee beyond IP's, so a beacon that fragments turns +one lost fragment into a wholly lost beacon. Small, self-describing, and +idempotent is the whole design. + +**Multicast versus broadcast.** Broadcast (`255.255.255.255` or a subnet +directed broadcast) reaches *every* host on the segment whether it cares or +not, and routers rightly drop it. Multicast is opt-in: a datagram addressed +to a group in `239.0.0.0/8` (the administratively-scoped block) is delivered +only to sockets that have *joined* that group, via `IP_ADD_MEMBERSHIP`. Three +more socket options shape the send: `IP_MULTICAST_IF` pins which interface the +beacon leaves by (it matters on a multi-homed host), `IP_MULTICAST_TTL = 1` +keeps it on the local link so no router forwards it, and `IP_MULTICAST_LOOP` +controls whether *this host's* other sockets also receive it — the switch that +makes the local two-process demo work over loopback and is irrelevant across +two machines, where the packet is on the wire regardless. + +{% include excalidraw.html + file="23-discovery-sequence" + alt="Two horizontal bands. The upper blue band, UDP multicast discovery, holds three boxes left to right: announce loop sendto to group emitting CHATTERD1 target 9241; receive loop recv one datagram splitting four fields and checking magic; and an amber parse-and-dedup box that skips its own name and calls seen.first before dialing. An amber arrow labelled new peer bridge to TCP drops into the lower grey band, TCP bridge, whose boxes are connect with retry, write_frame slash read_frame with a 6-byte header (magic, version, type, 2-byte big-endian length) plus payload, and a dark stdout box printing discovered peer and peer says. A note reads loss and reorder tolerated." + caption="Figure 23.2 — one peer's path from a connectionless beacon to a connection-oriented frame: recv, dedup by name, then dial and speak the canonical chat frame ch21 introduced (JOIN out, DELIVER back)" %} + +## How the code works + +The multicast socket is the heart of the chapter, and all three languages run +the same four-option dance around it — join the group, pin the outgoing +interface, enable loopback, set TTL 1 — differing only in how thick the +library wrapper is. Here it is verbatim from +`examples/23-udp-and-peer-discovery/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +std::expected make_multicast_socket(in_addr group, in_addr iface, + std::uint16_t port) { + Fd fd(::socket(AF_INET, SOCK_DGRAM, 0)); + if (!fd) return std::unexpected(errno_msg("socket(udp)")); + int one = 1; + if (::setsockopt(fd.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) < 0) + return std::unexpected(errno_msg("SO_REUSEADDR")); + + sockaddr_in bind_addr{}; + bind_addr.sin_family = AF_INET; + bind_addr.sin_addr.s_addr = htonl(INADDR_ANY); + bind_addr.sin_port = htons(port); + if (::bind(fd.get(), reinterpret_cast(&bind_addr), sizeof(bind_addr)) < 0) + return std::unexpected(errno_msg("bind(udp)")); + + ip_mreq mreq{}; + mreq.imr_multiaddr = group; + mreq.imr_interface = iface; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) < 0) + return std::unexpected(errno_msg("IP_ADD_MEMBERSHIP")); + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_IF, &iface, sizeof(iface)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_IF")); + unsigned char loop = 1; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_LOOP, &loop, sizeof(loop)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_LOOP")); + unsigned char ttl = 1; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_TTL, &ttl, sizeof(ttl)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_TTL")); + set_rcv_timeout(fd.get(), 200); + return fd; +} +``` + +```go +func makeSender(ip4 [4]byte) (int, error) { + fd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM, 0) + if err != nil { + return -1, fmt.Errorf("socket: %w", err) + } + if err := unix.SetsockoptInet4Addr(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_IF, ip4); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_IF: %w", err) + } + if err := unix.SetsockoptByte(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_LOOP, 1); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_LOOP: %w", err) + } + if err := unix.SetsockoptByte(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_TTL, 1); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_TTL: %w", err) + } + return fd, nil +} +``` + +```rust +fn make_multicast(cfg: &Config) -> Fallible { + let sock = Socket::new(Domain::IPV4, Type::DGRAM, Some(Protocol::UDP))?; + sock.set_reuse_address(true)?; + let bind = SocketAddrV4::new(Ipv4Addr::UNSPECIFIED, cfg.port); + sock.bind(&SockAddr::from(bind))?; + sock.join_multicast_v4(&cfg.group, &cfg.iface)?; + sock.set_multicast_if_v4(&cfg.iface)?; + sock.set_multicast_loop_v4(true)?; + sock.set_multicast_ttl_v4(1)?; + sock.set_read_timeout(Some(Duration::from_millis(200)))?; + Ok(sock.into()) +} +``` + +Three spellings, one syscall sequence. The C++ version is the raw form and +shows every step: bind to `INADDR_ANY` on the group port (you bind the +*port*, not the group address, so every group joined on that port arrives), +then the four `setsockopt` calls. `SO_REUSEADDR` matters because two +`chatterd` processes on one host must both bind the same UDP port — without it +the second `bind` fails `EADDRINUSE`. The `ip_mreq` carries two addresses: +`imr_multiaddr` is the group to join, `imr_interface` is the local interface +address to join it *on* — get that wrong on a multi-homed box and the kernel +drops group traffic before your socket ever sees it. Rust's `socket2` collapses +the same calls into named methods (`join_multicast_v4`, `set_multicast_ttl_v4`) +and then `into()`s the socket to a plain `std::net::UdpSocket` — an `OwnedFd` +it can share across threads. Go splits the roles: the *receiver* is built by +`net.ListenMulticastUDP` (which does the join for you), while the beacon +*sender* is this separate raw `unix.Socket` with the three send-side options +set explicitly — the standard library has no one-call multicast sender, so it +drops to `golang.org/x/sys/unix`. Note the `SO_RCVTIMEO` / `set_read_timeout` +of 200 ms on every receive socket: it is not about beacons, it is the escape +hatch that lets each blocking `recv` wake periodically to re-check the stop +flag at shutdown. + +The rest is small. The **announce loop** (in `main`/`discover`) formats the +beacon once — `CHATTERD1 ` — and `sendto`s it to the +group `--rounds` times, `--announce-ms` apart. The **receive loop** reads a +datagram, splits on whitespace, requires exactly four fields with `tok[0] == +"CHATTERD1"`, ignores any beacon whose name equals its own, parses the peer's +port and IP, and consults a dedup set so a peer is announced *once* no matter +how many of its beacons arrive. On first sight it prints `discovered peer + at :`, then bridges to TCP: `dial` opens a connection to the +advertised `host-ip:tcp-port` — with a bounded retry, because the beacon can +arrive a hair before the peer's listener is up — `write_frame` sends a +canonical **JOIN** frame whose payload is its own name, `read_frame` reads +the peer's **DELIVER** reply (payload `\0` — name, a NUL byte, +then the greeting text), splits it on the NUL, and prints `peer says: +`. The **accept loop** on the TCP listener is the mirror image: read +the dialer's JOIN frame, reply with a DELIVER frame carrying its own name + +NUL + greeting text, close. Both peers discover each other independently, so +each dials once and each prints exactly one `peer … says:` line. The +`` field is the one fragile +convention worth flagging: a peer advertises the address it wants to be dialed +at (its `--iface`), so behind NAT or on a different subnet it would advertise +an unreachable address — a limitation a real gossip protocol solves with +observed source addresses, out of scope here. + +## Errors, three ways + +The contract matches every `chatterd` version: exit 0 on success, 1 on a +runtime failure with one diagnostic line, 2 on a usage error — and +`verify.lua` asserts all three per language. The split is about *when* a bad +input is caught. A malformed `--group`/`--iface`, an unknown flag, or a +missing subcommand is a pure usage error: parsing rejects it before any socket +exists, prints `usage: chatterd discover …`, and exits 2 (C++ returns +`std::unexpected(2)`, Go returns `ok=false` with code 2, Rust returns +`Err(2)`). But a *syntactically valid* `--iface` that no interface owns is +only discoverable at `bind` time, so it fails later as a runtime error, exit +1 — `chatterd: error: bind(tcp): Cannot assign requested address` and its Go/ +Rust equivalents. The dial path is deliberately forgiving: it retries up to 40 +times at 50 ms before giving up with `connect … giving up` (exit-neutral — it +logs to stderr and the loop continues), because a beacon legitimately races +its sender's own listener. And `send`/`recv` guard `EINTR` with a retry in the +C++ frame I/O, which the Go and Rust runtimes handle beneath their standard +libraries. + +## Concurrency lens + +Three flows of control run per process: the announce loop (on `main`), the TCP +accept loop, and the UDP receive loop. The receive loop is the only writer of +the dedup set, and each language reflects that differently. C++ wraps the set +in a `Seen` struct with a `std::mutex` — defensive, since a future second +reader would need it — while Go confines its `map[string]bool` to the +`recvLoop` goroutine and Rust confines its `HashSet` to the `recv_loop` +thread, so neither needs a lock at all: the state is owned by one flow, which +is Chapter 6's "share memory by communicating" reflex applied to sockets. +Shutdown is the interesting coordination. There is no natural end to a +`recv` loop, so each waits on a stop signal *and* a 200 ms socket read +timeout: the timeout wakes the blocked `recv` often enough to observe the +signal. C++ uses `std::jthread` and each loop's `stop_token`; Go cancels a +`context` and `Wait`s a `sync.WaitGroup`; Rust flips an `AtomicBool` and +`join`s the threads. The 800 ms grace window after the last beacon is the +subtle part: it lets an exchange that a *final* beacon just triggered finish +before the listen threads are told to stop, so a discovery on the last round +still completes its frame. + +## Build, run, observe + +```bash +[host]$ cd examples/23-udp-and-peer-discovery && ./demo.sh build +``` + +The runner (`python3 scripts/test-all-examples.py --only +23-udp-and-peer-discovery --mode vm`) reports `PASS PASS PASS` for cpp, go, +rust (3 passed, 0 failed, 0 skipped). Its `verify.lua` clears `TARGET` so the +assertion runs *locally* on loopback multicast: two instances discover each +other and exchange a frame. Driving that by hand the way this chapter's +loopback evidence was produced (distinct names, distinct TCP ports): + +```console +[host]$ ./cpp/build/release/app discover --group 239.23.7.1 --port 51823 --name alice --tcp-port 9231 --iface 127.0.0.1 & +[host]$ ./cpp/build/release/app discover --group 239.23.7.1 --port 51823 --name bob --tcp-port 9232 --iface 127.0.0.1 +chatterd: announcing as bob on 239.23.7.1:51823 (tcp 127.0.0.1:9232) +discovered peer alice at 127.0.0.1:9231 +peer alice says: hello from alice +``` + +and `alice`'s stdout is the mirror: `discovered peer bob at 127.0.0.1:9232` / +`peer bob says: hello from bob`. The socket is a joined, unconnected UDP +endpoint — `ss -uap` on the host during the run shows exactly that, one +`UNCONN` socket bound to all addresses on the group port, owned by our binary: + +```console +[host]$ ss -uap | grep 51833 +UNCONN 0 0 0.0.0.0:51833 0.0.0.0:* users:(("app",pid=2818975,fd=4)) +[host]$ ip maddr show lo | grep 239.23.7.1 + inet 239.23.7.1 +``` + +Loopback multicast is the one quirk to know: `lo` is not flagged `MULTICAST`, +but the IPv4 stack still loops group traffic when the multicast interface is +set to `127.0.0.1`, which is why the local demo passes `--iface 127.0.0.1`. +Output equality alone could be faked by a program that quietly skipped the +network, so the real proof is on two machines watching the wire. + +## Cross-check: two guests, the beacons on the wire, and both transcripts + +The star run is `systems-target` (192.168.124.7) and `systems-peer` +(192.168.124.95), each launched with its own `--iface`. First, the beacons +themselves — `tcpdump` on the peer guest sees both guests' multicast, decoded +byte for byte: + +```console +[peer]$ sudo tcpdump -i enp1s0 -n -X udp port 51824 -c 8 +15:09:31.986449 IP 192.168.124.95.51824 > 239.23.7.1.51824: UDP, length 34 + 0x0000: 4500 003e e3df 4000 0111 62af c0a8 7c5f E..>..@...b...|_ + 0x0020: 5445 5244 3120 7065 6572 2039 3234 3120 TERD1.peer.9241. + 0x0030: 3139 322e 3136 382e 3132 342e 3935 192.168.124.95 +15:09:32.531220 IP 192.168.124.7.51824 > 239.23.7.1.51824: UDP, length 35 + 0x0000: 4500 003f 89b9 4000 0111 bd2c c0a8 7c07 E..?..@....,..|. + 0x0020: 5445 5244 3120 7461 7267 6574 2039 3234 TERD1.target.924 + 0x0030: 3120 3139 322e 3136 382e 3132 342e 37 1.192.168.124.7 +``` + +Everything in the chapter is on the wire. The destination is the group +`239.23.7.1.51824`; the ASCII column reads `CHATTERD1 peer 9241 +192.168.124.95` (34 bytes) and `CHATTERD1 target 9241 192.168.124.7` (35 +bytes) — the length difference is exactly the shorter name and IP. In the IP +header, the byte after `4000` is `01`: TTL 1, our `IP_MULTICAST_TTL`, the +router-stop that keeps beacons link-local. Second, the joined socket and group +membership on the peer guest, while `chatterd` runs: + +```console +[peer]$ sudo ss -uanp | grep 51824 +UNCONN 0 0 0.0.0.0:51824 0.0.0.0:* users:(("chatterd",pid=2105,fd=4)) +[peer]$ ip maddr show enp1s0 + inet 239.23.7.1 +``` + +`ip maddr` proves the kernel actually joined the group on `enp1s0` — the +observable effect of `IP_ADD_MEMBERSHIP` — and `ss -uap` shows the same +`UNCONN` fd-4 UDP socket as on the host. Third, the payoff: both guests' +stdout, quoted verbatim from the two-host run: + +```console +[vm]$ /home/fedora/chatterd discover --group 239.23.7.1 --port 51824 --name target --tcp-port 9241 --iface 192.168.124.7 +discovered peer peer at 192.168.124.95:9241 +peer peer says: hello from peer +``` + +```console +[peer]$ /home/fedora/chatterd discover --group 239.23.7.1 --port 51824 --name peer --tcp-port 9241 --iface 192.168.124.95 +discovered peer target at 192.168.124.7:9241 +peer target says: hello from target +``` + +Each guest discovered the other exactly once and exchanged one TCP frame — the +message genuinely crossed the network. No libvirt workaround was needed: both +guests sit on the same `virbr0` L2 segment, so a TTL-1 multicast beacon is +delivered by the bridge as ordinary link-local traffic; libvirt's NAT only +governs traffic *leaving* the host toward the internet, which discovery never +does. + +> **On the lab VM** unverified — +> the kernel-side view of the UDP→TCP bridge — watching the dial land with +> bcc-tools `tcpconnect`, the short-lived chat connection appear and close in +> `tcplife`, and the receive loops' off-CPU parking in `offcputime` — needs +> root and the bcc stack, which are not run on this host; the Debugging part +> (Part 8) exercises exactly that on `systems-target`. + +## What you learned + +- UDP is connectionless, message-oriented, best-effort: `sendto` emits one + datagram whose boundaries `recv` preserves, and loss/reorder are tolerable + for a beacon that repeats — which is why discovery is UDP and the + conversation is TCP. +- Multicast is opt-in delivery to a joined group: `IP_ADD_MEMBERSHIP` joins, + `IP_MULTICAST_IF` pins the send interface, `IP_MULTICAST_TTL = 1` keeps it + link-local, and `IP_MULTICAST_LOOP` controls local delivery — verified live + by `ip maddr show` and the TTL-1 byte in `tcpdump`. +- Beacons stay small and self-describing (34–35 bytes here) to fit one + un-fragmented packet, and a receiver dedupes by name so one peer is + announced once regardless of beacon count. +- The bridge from discovery to conversation is a plain `connect` to the + advertised endpoint, reusing the canonical chat frame ch21 introduced (a + JOIN out, a DELIVER back) — proven across two guests, each printing one + `discovered peer` and one `peer … says:` line. + +Next, **time, timers, and deadlines**: the clocks a networked program must +choose between, and how `timerfd` and deadline discipline keep a beacon loop +accurate about *when* it fires. + +--- + +

verified — evidence produced +this session on the Fedora 44 reference host (kernel 7.1.3-200.fc44) and the +two lab guests systems-target (192.168.124.7) / +systems-peer (192.168.124.95): the runner printed +23-udp-and-peer-discovery  PASS  PASS  PASS +(3 passed, 0 failed, 0 skipped) under --mode vm; the host +loopback run produced the exact alice/bob +discovered peer and peer … says: lines, with +ss -uap showing the UNCONN 0.0.0.0:51833 fd-4 socket +and ip maddr show lo the joined group. On the two guests, +tcpdump -X on systems-peer captured both beacons to +239.23.7.1.51824 (34 B CHATTERD1 peer 9241 +192.168.124.95 and 35 B CHATTERD1 target 9241 192.168.124.7, +IP-header TTL byte 01); ss -uanp showed the joined +chatterd fd-4 UDP socket and ip maddr show enp1s0 the +239.23.7.1 membership; and both guests' stdout printed one +discovered peer and one peer … says: line as quoted. +No libvirt NAT workaround was required — the guests share one virbr0 +L2 segment. The bcc-tools tcpconnect/tcplife/offcputime +callout is unverified as marked and is exercised in Part 8.

diff --git a/_docs/24-time-timers-deadlines.md b/_docs/24-time-timers-deadlines.md new file mode 100644 index 0000000..4922866 --- /dev/null +++ b/_docs/24-time-timers-deadlines.md @@ -0,0 +1,460 @@ +--- +title: "Time, timers, and deadlines" +order: 24 +part: "Networking" +description: "chatterd v3 learns to survive a peer that vanishes: the POSIX clock taxonomy, clock_gettime's vDSO fast path (no syscall — proven by its absence under strace), timerfd as a pollable heartbeat, the per-language deadline idiom, why liveness must be measured on a monotonic clock, and jittered exponential backoff." +duration: "45 minutes" +--- + +Chapters 21–23 built `chatterd` into a peer-to-peer chat daemon that links two +hosts over TCP and passes framed messages. It has one fatal naïveté: it assumes +the peer stays. Pull the cable, kill the process, or let a switch drop the +flow, and a `chatterd` peer sits forever in a `read` that will never return — +TCP alone will not tell you the far end is gone in any useful window. Version 3 +fixes this with *time*: a periodic **heartbeat** proves the link is alive, a +**deadline** measured on silence declares it dead, and a **jittered +exponential backoff** brings it back without hammering. The one new idea under +all three is that not all clocks are the same clock — and picking the wrong one +turns a heartbeat into a liability the first time NTP corrects your wall clock. + +The code is in `examples/24-time-timers-deadlines/`. `./demo.sh` there builds +all three implementations; its `README.md` specifies the CLI, the wire format +(the same canonical frame chapters 21–23 used, now carrying a `PING` TYPE this +chapter adds), and the exit codes all three languages share. + +{% include excalidraw.html + file="24-clock-taxonomy" + alt="Top band: the four clocks clock_gettime can read as boxes — CLOCK_REALTIME (dashed, wall time, settable, can step backward), CLOCK_MONOTONIC (amber, since a boot epoch, never steps, steady_clock/Instant), CLOCK_BOOTTIME (monotonic plus suspend time), and PROCESS_CPUTIME_ID (this process's on-CPU time). Bottom band: while a peer is silent and ntpd corrects the wall clock, a dashed REALTIME box shows a wall-clock deadline rewinding so a dead peer looks alive, beside an amber MONOTONIC box where elapsed is always non-negative and the timeout fires on real silence; an amber arrow from CLOCK_MONOTONIC points to it labeled chatterd reads this." + caption="Figure 24.1 — the clock taxonomy and the monotonic-vs-wall bug: a wall-clock deadline slips when the clock is stepped; a CLOCK_MONOTONIC deadline does not" %} + +> **Tools used** — `strace`, `python3` (host); `cyclictest` (systems-target +> VM, exercised in Part 8). Everything host-side is checked by +> `scripts/check-host.sh` or ships with Fedora. + +## The clock taxonomy + +`clock_gettime(2)` takes a *clock id*, and the choice is the whole chapter. +Four ids matter here. **`CLOCK_REALTIME`** is the wall clock — seconds since the +Unix epoch, the thing `date` prints. It is *settable* (`settimeofday`, +`adjtimex`) and *disciplined by NTP*, which means it can jump forward, and — the +part that bites — jump *backward*. **`CLOCK_MONOTONIC`** counts from an +unspecified epoch (near boot) and has one guarantee the wall clock cannot give: +it never steps and never goes backward. NTP may gently *slew* its rate, but two +successive reads always satisfy `t1 >= t0`. **`CLOCK_BOOTTIME`** is monotonic +*and* keeps counting across system suspend, which `CLOCK_MONOTONIC` does not — +the right clock for "has 30 s of wall time passed even if the laptop slept." +**`CLOCK_PROCESS_CPUTIME_ID`** measures only the CPU time *this process* burned, +useful for profiling, useless for wall deadlines. + +The rule that follows is absolute: **any duration or deadline must be measured +on a monotonic clock.** A heartbeat timeout is a duration — "no traffic for +3 seconds" — so it reads `CLOCK_MONOTONIC`, never `CLOCK_REALTIME`. Figure 24.1 +is the bug you avoid by obeying this. Suppose you computed a deadline as +`wall_now() + 3s` and stored it. Halfway through, `ntpd` decides the clock is +0.5 s fast and steps it *back*. Now `wall_now()` reads earlier than when you +started; your deadline recedes into the future; a peer that died stays "alive" +for an extra half second — or, on a backward step large enough, your `select` +timeout goes negative and a healthy peer is declared dead instantly. Every +language in this chapter routes liveness through a monotonic reading: +`std::chrono::steady_clock` in C++, `std::time::Instant` in Rust, and Go's +`time.Now`, whose returned value carries a hidden monotonic component precisely +so `time.Since` cannot be fooled by a wall-clock step. + +## clock_gettime and the vDSO — no syscall at all + +Reading the clock happens constantly — on every loop iteration `chatterd` +recomputes how long until the deadline — so it had better be cheap. It is, and +the reason is chapter 4's vDSO. `CLOCK_MONOTONIC` and `CLOCK_REALTIME` reads do +not trap into the kernel: the kernel maps a shared page (the vDSO) into every +process, updates a timekeeping structure there on each tick, and the libc +`clock_gettime` wrapper reads it in userspace with no `syscall` instruction. +The proof is negative and we run it later: `strace` a program that reads the +monotonic clock four times and you will not see a single `clock_gettime` line, +because there was no syscall to trace. Sleeping is the opposite — you cannot +sleep in userspace, so `nanosleep`/`clock_nanosleep` and `timerfd_settime` do +trap, and those *do* show up. + +## How the code works + +`clockprobe` is a standalone subcommand that makes the abstractions concrete by +measuring them. It reads the monotonic clock's resolution with `clock_getres`, +times a 1 ms `nanosleep`, and times a 1 ms one-shot `timerfd` — each timed on a +steady clock — then prints one line. Here it is in all three languages, +verbatim from `examples/24-time-timers-deadlines/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +int run_clockprobe() { + timespec res{}; + ::clock_getres(CLOCK_MONOTONIC, &res); + const long res_ns = res.tv_sec * 1'000'000'000L + res.tv_nsec; + + const auto t0 = steady_clock::now(); + timespec req{0, 1'000'000}; // 1 ms + timespec rem{}; + while (::nanosleep(&req, &rem) != 0 && errno == EINTR) { + req = rem; + } + const auto sleep_us = duration_cast(steady_clock::now() - t0).count(); + + UniqueFd tfd(::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC)); + if (!tfd.valid()) { + die("timerfd_create", errno_ec()); + } + itimerspec its{}; + its.it_value.tv_nsec = 1'000'000; // one-shot 1 ms + if (::timerfd_settime(tfd.get(), 0, &its, nullptr) < 0) { + die("timerfd_settime", errno_ec()); + } + const auto t1 = steady_clock::now(); + std::uint64_t ticks = 0; + [[maybe_unused]] auto rd = ::read(tfd.get(), &ticks, sizeof ticks); + const auto tfd_us = duration_cast(steady_clock::now() - t1).count(); + + std::println("clockprobe: CLOCK_MONOTONIC res={}ns nanosleep(1ms) actual={}us " + "timerfd(1ms) actual={}us", + res_ns, sleep_us, tfd_us); + return 0; +} +``` + +```go +func runClockprobe() int { + var res unix.Timespec + if err := unix.ClockGetres(unix.CLOCK_MONOTONIC, &res); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: clock_getres: %v\n", err) + return 1 + } + resNs := res.Sec*1_000_000_000 + int64(res.Nsec) + + t0 := time.Now() + req := unix.NsecToTimespec(1_000_000) // 1 ms + for unix.Nanosleep(&req, &req) == unix.EINTR { + } + sleepUs := time.Since(t0).Microseconds() + + tfd, err := unix.TimerfdCreate(unix.CLOCK_MONOTONIC, unix.TFD_CLOEXEC) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: timerfd_create: %v\n", err) + return 1 + } + defer unix.Close(tfd) + its := unix.ItimerSpec{Value: unix.Timespec{Nsec: 1_000_000}} // one-shot 1 ms + if err := unix.TimerfdSettime(tfd, 0, &its, nil); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: timerfd_settime: %v\n", err) + return 1 + } + t1 := time.Now() + var ticks uint64 + buf := (*[8]byte)(unsafe.Pointer(&ticks))[:] + _, _ = unix.Read(tfd, buf) + tfdUs := time.Since(t1).Microseconds() + + fmt.Printf("clockprobe: CLOCK_MONOTONIC res=%dns nanosleep(1ms) actual=%dus "+ + "timerfd(1ms) actual=%dus\n", resNs, sleepUs, tfdUs) + return 0 +} +``` + +```rust +fn run_clockprobe() -> i32 { + let res = clock_getres(ClockId::CLOCK_MONOTONIC).expect("clock_getres"); + let res_ns = res.tv_sec() as i64 * 1_000_000_000 + res.tv_nsec() as i64; + + let t0 = Instant::now(); + let req = TimeSpec::nanoseconds(1_000_000); // 1 ms + while clock_nanosleep(ClockId::CLOCK_MONOTONIC, ClockNanosleepFlags::empty(), &req) + == Err(Errno::EINTR) + {} + let sleep_us = t0.elapsed().as_micros(); + + let tfd = TimerFd::new(TfdClockId::CLOCK_MONOTONIC, TimerFlags::empty()).expect("timerfd"); + tfd.set(Expiration::OneShot(TimeSpec::nanoseconds(1_000_000)), TimerSetTimeFlags::empty()) + .expect("timerfd_settime"); + let t1 = Instant::now(); + tfd.wait().expect("timerfd wait"); + let tfd_us = t1.elapsed().as_micros(); + + println!( + "clockprobe: CLOCK_MONOTONIC res={res_ns}ns nanosleep(1ms) actual={sleep_us}us \ + timerfd(1ms) actual={tfd_us}us" + ); + 0 +} +``` + +The three are the same program in three dialects. `clock_getres` on this host +returns `1ns` — nanosecond nominal resolution — but resolution is not accuracy: +the two 1 ms sleeps *overshoot*, and that gap is the lesson. A `timerfd` is a +timer wrapped in a file descriptor: `timerfd_create(CLOCK_MONOTONIC, …)` makes +one, `timerfd_settime` arms it (an `it_value` with no `it_interval` is a +one-shot; both set makes it periodic), and it becomes *readable* when it +expires — a `read` returns a `uint64` count of expirations. That last property +is why it belongs in this chapter: a timer that is an fd drops straight into the +`poll`/`epoll` loop chapter 9 built, no signal handler required. + +The session loop is where the deadline lives. In the C++ `run_session`, every +iteration computes how long to block from the monotonic deadline, and hands +that to `poll` as its timeout: + +```cpp + for (;;) { + int poll_ms = 0; + { + const auto left = deadline - steady_clock::now(); + poll_ms = left <= 0ns + ? 0 + : static_cast(duration_cast(left).count()) + 1; + } +``` + +`deadline` is a `steady_clock::time_point` set to `now() + timeout` at session +start and *reset to a fresh `now() + timeout` on every received frame* — JOIN, +MSG, or PING, all count as traffic. The `poll` array holds three fds: the +`signalfd` shutdown doorbell, the periodic heartbeat `timerfd`, and the peer +socket. When `poll` returns 0, the timeout elapsed with nothing to read — the +peer went silent for `timeout_ms` — and the session returns `TimedOut`. When +the heartbeat fd fires, the loop reads its 8-byte tick and sends a `PING`. This +is the same one-loop-three-sources shape as chapter 9's `fwatch`, now serving +liveness instead of file events. The three languages express the deadline +differently — C++ subtracts two `time_point`s, Rust does the identical +arithmetic on `Instant`, and Go resets a `time.Timer` and `select`s on its +channel — but the invariant is one sentence: *the clock is monotonic and the +deadline is data.* + +Reconnection is the connector's job, and it is where jitter earns its keep. +After a drop, `run_connect` waits out a backoff before redialing: + +```cpp + // Jittered exponential backoff (equal jitter: half fixed, half random). + const long half = backoff / 2; + std::uniform_int_distribution jit(0, half > 0 ? half : 0); + const long delay = half + jit(rng); + std::println("chatterd: reconnecting to {} in {}ms", opts.peer_name, delay); + if (sleep_or_shutdown(delay, shutdown_fd.get())) { + std::println("chatterd: {} shutting down", opts.name); + return 0; + } + backoff = std::min(backoff * 2, opts.max_backoff_ms); +``` + +Two ideas are stacked here. **Exponential** — `backoff` doubles per failed +attempt (200 → 400 → 800 …) up to `--max-backoff-ms`, so a peer that is down +for a while is polled ever more gently instead of in a hot loop. **Jitter** — +the actual `delay` is `half + rand(0..half)`, the "equal jitter" strategy: half +the interval fixed, half random. Jitter matters because without it, a hundred +clients that all lost the same server would redial in lockstep on every doubling +boundary, a synchronized thundering herd that knocks the server back down the +instant it recovers. Randomizing each client's wait smears the retries across +the window. A fresh connection resets `backoff` to the base, so a brief blip is +never punished with a long wait. The wait itself is `sleep_or_shutdown` — a +`poll` on the signalfd with the backoff as its timeout — so a `SIGTERM` +mid-backoff wakes instantly instead of stranding the process for seconds. + +## Errors, three ways + +The contract is chapter-wide: clean exit 0, runtime failures exit 1 with one +diagnostic, usage errors exit 2. The interesting failures here are temporal. +The most important one is a *non-error*: when the peer socket returns EOF (0 +bytes) because the far end closed, `chatterd` does **not** immediately declare +the peer gone. It sets `sock_dead`, stops polling the socket, and lets the +*deadline* fire — so death is always declared by the same monotonic timeout +whether the peer crashed silently or closed politely, and the observable line +is identical. Sending a `PING` into a half-closed socket can raise `EPIPE`; C++ +passes `MSG_NOSIGNAL` so it never becomes a fatal `SIGPIPE`, and the failed send +is deliberately ignored — "a broken pipe is left to the deadline," as the +comment says, because the deadline is the single source of truth about +liveness. `EINTR` on `poll` is retried in C++ and Rust and handled beneath the +runtime in Go. Bad numeric flags (`--timeout-ms abc`) and missing required +flags exit 2 through each language's parser — `std::expected` chains in C++, +error returns in Go, `Result` in Rust — all printing the same `usage:` block. + +## Concurrency lens + +The C++ and Rust sessions are single-threaded: one `poll` loop multiplexes the +socket, the heartbeat timerfd, and the shutdown signalfd, so there is no shared +state to lock — the timer and the deadline are just two more fds and one +`time_point`. Go inverts it, exactly as chapter 9 did: a per-connection reader +goroutine turns the socket into a `chan frame`, and the session is a `select` +over four cases — `ticker.C` (heartbeat), `deadline.C` (liveness), the frame +channel, and `ctx.Done()` (shutdown from `signal.NotifyContext`). The subtlety +Go forces you to get right is `time.Timer` reset: before `deadline.Reset`, the +code drains a possibly-already-fired timer channel, because a `Timer` that +expired between `select` iterations leaves a value buffered in `C` that would +otherwise make the *next* `select` see a phantom timeout. C++ and Rust sidestep +this entirely by recomputing the `poll` timeout from the deadline each +iteration — there is no timer object to get out of sync, just arithmetic on a +monotonic reading. Both designs reach the same behavior; the single-threaded +one has less to synchronize, the Go one reads more like the protocol. + +## Build, run, observe + +```bash +[host]$ cd examples/24-time-timers-deadlines && ./demo.sh build +``` + +The runner (`python3 scripts/test-all-examples.py --only +24-time-timers-deadlines`) reports `PASS PASS PASS` for cpp, go, rust — +`verify.lua` asserts observable behavior, not exit 0. First, `clockprobe` on +this host: + +```console +[host]$ ./cpp/build/release/app clockprobe +clockprobe: CLOCK_MONOTONIC res=1ns nanosleep(1ms) actual=1056us timerfd(1ms) actual=1013us +``` + +All three languages agree to within noise — Go printed `actual=1066us` / +`1005us`, Rust `1055us` / `1002us`. The number to sit with is the overshoot: a +1 ms sleep took ~1056 µs, ~56 µs long. That gap is not the clock's resolution +(1 ns) — it is the *scheduler's wake-up latency*, the time between the timer +expiring and your thread actually getting a CPU. On a general-purpose kernel +this is tens of microseconds and variable; the timerfd path is a touch tighter +(~1013 µs) because it wakes straight out of `read` rather than through +`nanosleep`'s return. Now the heartbeat scenario, driven by hand — a listener +`A` and a connector `B` on loopback, then `A` killed: + +```console +[host]$ ./cpp/build/release/app listen --name A --addr 127.0.0.1:0 \ + --heartbeat-ms 300 --timeout-ms 1000 --message "hi from A" & +chatterd: A listening on 127.0.0.1:35145 +[host]$ ./cpp/build/release/app connect --name B --peer A@127.0.0.1:35145 \ + --heartbeat-ms 300 --timeout-ms 1000 --backoff-ms 200 --message "hi from B" --seed 7 +chatterd: B connecting to A at 127.0.0.1:35145 +chatterd: B linked with A +chatterd: B message from A: hi from A +[host]$ kill -9 +chatterd: peer A timed out +chatterd: reconnecting to A in 107ms +chatterd: reconnecting to A in 245ms +chatterd: reconnecting to A in 712ms +chatterd: reconnecting to A in 1055ms +chatterd: B shutting down # after SIGTERM +``` + +The backoff line is the whole design on one screen. Base is 200 ms, so the +first `delay` is `100 + rand(0..100)` = 107 ms; then the ceiling doubles and the +window grows — 245 (from base 400), 712 (from 800), 1055 (from 1600) — each a +jittered draw inside its window, never a bare power of two. `SIGTERM` +mid-backoff prints `B shutting down` and exits 0, waking out of the `poll` +immediately. Figure 24.2 is this sequence. + +{% include excalidraw.html + file="24-heartbeat-timeout-seq" + alt="Two lifelines: connector B (amber) on the left, listener A (grey, ending early) on the right. Top: JOIN 'B' from B to A and JOIN 'A' back, then a note that both print linked and exchange messages. Middle: amber PING arrows every 300 ms in both directions, each resetting B's deadline, with a note that any frame sets deadline = mono_now() + timeout. Then A killed, socket EOF, and a dashed PING from B that is dropped on the broken pipe. Bottom on B's side: an amber box 'deadline fires on silence — chatterd: peer A timed out', then three stacked reconnect boxes 107 ms, 245 ms, 712 ms chained by amber arrows, beside a note explaining equal-jitter backoff delay = B/2 + rand(0..B/2) with B doubling per attempt and capped at max-backoff." + caption="Figure 24.2 — chatterd v3 from link to loss to reconnect: the monotonic deadline fires on silence, then jittered exponential backoff (real 107/245/712 ms from this run) paces the redials" %} + +## Cross-check: the vDSO's absence, and the heartbeat live under strace + +Two independent traces confirm the mechanism. First, the vDSO claim. Ask +`strace` for *only* `clock_gettime` while running `clockprobe`, which reads the +monotonic clock four times (`t0`, after the sleep, `t1`, after the timerfd) plus +`clock_getres`: + +```console +[host]$ strace -e trace=clock_gettime ./cpp/build/release/app clockprobe 2>&1 | grep -c clock_gettime +0 +``` + +Zero lines. Not because the program never reads the clock, but because those +reads never became syscalls — they were served from the vDSO page in userspace. +What *does* trap is visible when you trace the sleeping calls: + +```console +[host]$ strace -c ./cpp/build/release/app clockprobe +% time seconds usecs/call calls errors syscall +------ ----------- ----------- --------- --------- ---------------- + 1.20 0.000008 8 1 clock_nanosleep + 0.75 0.000005 5 1 timerfd_create + 0.30 0.000002 2 1 timerfd_settime +``` + +`clock_nanosleep` (glibc implements `nanosleep` on top of it), `timerfd_create`, +`timerfd_settime` — each once, exactly as the source calls them — and no +`clock_gettime` row at all. The Rust binary shows +`clock_nanosleep(CLOCK_MONOTONIC, …)` because it calls that clock explicitly; +the C++ one shows `CLOCK_REALTIME` because that is glibc's `nanosleep` default — +a fine reminder that the *sleep* clock and the *deadline* clock are chosen +independently. + +Second, the heartbeat itself. Tracing the connector's `poll`, `read`, `sendto`, +and `timerfd_settime` during a live session lays the whole loop bare: + +```console +[host]$ strace -e trace=poll,read,timerfd_settime,sendto ./cpp/.../app connect --name B ... +timerfd_settime(5, 0, {it_interval={tv_sec=0, tv_nsec=300000000}, ...}, NULL) = 0 +sendto(4, "CH\1\1\0\1B", 7, MSG_NOSIGNAL, NULL, 0) = 7 +poll([{fd=3,...}, {fd=5,...}, {fd=4,...}], 3, 1200) = 1 ([{fd=4, revents=POLLIN}]) +poll([{fd=3,...}, {fd=5,...}, {fd=4,...}], 3, 1200) = 1 ([{fd=5, revents=POLLIN}]) +read(5, "\1\0\0\0\0\0\0\0", 8) = 8 +sendto(4, "CH\1\5\0\0", 6, MSG_NOSIGNAL, NULL, 0) = 6 +poll([{fd=3,...}, {fd=5,...}, {fd=4,...}], 3, 900) = 1 ([{fd=4, revents=POLLIN}]) +``` + +Every claim in the chapter is on those lines. `timerfd_settime` arms the +heartbeat with a 300 ms interval. The `sendto(4, "CH\1\1\0\1B", 7, …)` is the +7-byte JOIN frame — `43 48 01 01 00 01 42`, the canonical header (`CH`, +version 1, type 1 = JOIN) plus the name `B` — matching the wire format byte +for byte, the same header chapters 21–23 used; only the TYPE byte and the +engine around it differ by chatterd version. When fd 5 (the timerfd) fires, +the loop reads its 8-byte tick (`\1\0\0\0\0\0\0\0` — one expiration) and +sends the 6-byte PING `CH\1\5\0\0` (`43 48 01 05 00 00`). And +the `poll` timeout *is* the deadline countdown: `1200` right after receiving a +frame (the deadline was just reset), then `900` after a heartbeat-only wakeup — +the monotonic deadline shrinking exactly as `deadline - steady_clock::now()` +says it should. The trace and the printed output tell one story: the timer +drives the PINGs, the monotonic deadline drives the timeout. + +> **On the lab VM** unverified — +> the scheduler wake-up latency that `clockprobe` measures as ~56 µs of +> overshoot has a proper tool, `cyclictest`, which stresses the timer path and +> reports latency percentiles; and comparing a `PREEMPT` versus `PREEMPT_RT` +> kernel needs a disposable box. Both need root and a kernel we can perturb, so +> they run on `systems-target` in the Debugging part (Part 8), not on this host. + +## What you learned + +- The clock id is the decision: `CLOCK_REALTIME` is the settable, NTP-steppable + wall clock; `CLOCK_MONOTONIC` never steps and is the only correct clock for a + duration or deadline; `CLOCK_BOOTTIME` adds suspend time; + `CLOCK_PROCESS_CPUTIME_ID` measures CPU, not wall time. `chatterd` measures + silence on the monotonic clock so an NTP correction can never fake a timeout. +- `clock_gettime` on the common clocks is a vDSO read, not a syscall — `strace` + showed **zero** `clock_gettime` lines for a program that read the clock four + times, while `clock_nanosleep` and `timerfd_settime` (which must trap) each + appeared once. +- A `timerfd(CLOCK_MONOTONIC)` is a timer you can `poll` — it drops the + heartbeat into the same event loop as the socket and the shutdown signalfd, + and `clockprobe` measured a 1 ms timer landing at ~1013 µs, the ~13 µs + overshoot being scheduler wake-up latency, not clock resolution. +- Jittered exponential backoff (`delay = B/2 + rand(0..B/2)`, `B` doubling to a + cap, reset on a fresh link) paces reconnects and de-synchronizes a fleet of + clients — the live run drew 107, 245, 712, 1055 ms, growing but never a bare + power of two. + +Next, **shared state and the futex**: `chatterd`'s peers coordinated by passing +bytes; the next chapter coordinates threads that share one address space, where +a lock is a userspace atomic that only touches the kernel when it must block. + +--- + +

verified — every number and +output excerpt above was produced on the Fedora 44 reference host (kernel +7.1.3-200.fc44, strace 7.1) this session: the runner printed +24-time-timers-deadlines PASS PASS PASS (3 passed, 0 failed, 0 +skipped), and the orchestrator was re-run to confirm the timing scenario is not +flaky. clockprobe printed res=1ns with +nanosleep(1ms) actual=1056us timerfd(1ms) actual=1013us for C++ +(Go 1066/1005, Rust 1055/1002). The hand-driven loopback session produced the +exact linked, message from A, peer A timed +out, and jittered backoff lines 107/245/712/1055ms (base 200, +seed 7), with SIGTERM printing B shutting down and +exit 0. strace -e trace=clock_gettime on clockprobe +matched 0 lines (vDSO), while strace -c showed +clock_nanosleep, timerfd_create, and +timerfd_settime once each and no clock_gettime row; +the live connector trace showed the 300 ms timerfd_settime, the +7-byte JOIN CH\1\1\0\1B, the 8-byte timerfd tick, the 6-byte PING +CH\1\5\0\0, and the poll timeout counting 1200→900 as +the monotonic deadline. The cyclictest / PREEMPT_RT +callout is unverified as marked and is exercised in Part 8.

diff --git a/_docs/25-shared-state-and-the-futex.md b/_docs/25-shared-state-and-the-futex.md new file mode 100644 index 0000000..d00372b --- /dev/null +++ b/_docs/25-shared-state-and-the-futex.md @@ -0,0 +1,421 @@ +--- +title: "Shared state and the futex" +order: 25 +part: "Concurrency in Depth" +description: "workq builds a bounded MPMC job queue three ways — std::mutex + condvars, a buffered channel, Mutex + Condvar — and we look under all of them at the one primitive doing the work: the futex, whose uncontended fast path is a userspace CAS with no syscall and whose slow path is FUTEX_WAIT/WAKE, proven by strace scaling from 5,186 to 488,818 calls, then broken on purpose under ThreadSanitizer and go -race while Rust refuses to compile the race at all." +duration: "50 minutes" +--- + +Every chapter since Part 3 has reached for a mutex without asking what one +*is*. `pmon` guarded its supervisor state with one; Chapter 20's `shmkv` +went a layer lower and called `futex(2)` by hand on a word in a mapped file. +This chapter closes the gap between those two: the `std::mutex`, +`sync.Mutex`, and `std::sync::Mutex` you have been using are **futexes +wearing a library**, and once you can see the futex underneath, the whole +behaviour of contended code — why an uncontended lock is nearly free, why a +contended one is expensive, why a lost wakeup hangs a program forever — +stops being folklore. The example is `workq`, a bounded multi-producer / +multi-consumer job queue with a worker pool, written three times with each +language's native shared-state kit but one observable contract, so a single +`verify.lua` covers all three. Then we break it: `--buggy` routes every +consumer into one unsynchronised counter, and we watch three tools react. + +The code is in `examples/25-shared-state-and-the-futex/`. `./demo.sh` there +builds all three implementations; its `README.md` specifies the CLI, the +deterministic value contract, and the exit codes all three languages share. + +{% include excalidraw.html + file="25-mpmc-queue" + alt="Left to right: three producer boxes (producer 0 owns i=0 mod P, producer 1 owns i=1, a dashed producer P-1) feed push arrows into a central amber bounded-queue box labeled cap K, mutex plus not_full and not_empty condvars, equals buffered chan in Go, produced is atomic long; pop arrows fan out to three consumer boxes each holding a local (count, sum), the dashed one being consumer C-1; an amber dashed XOR-combine-after-join arrow runs from the consumer stack down to a dark box reading XOR-fold locals after join, checksum equals 42b3746c6cee7465, independent of P, C, cap. A caption note says one mutex serializes every push and pop, and XOR is commutative so drain order cannot change the result." + caption="Figure 25.1 — workq's shape: P producers, one bounded queue, C consumers folding into per-consumer accumulators combined by XOR — a program whose correct output is one fixed constant no matter how the threads interleave" %} + +> **Tools used** — `strace`, `perf`, `/usr/bin/time`, `cat` +> (`/proc//task/*/status`), `cmake`/`g++` with `-fsanitize=thread`, +> `go build -race`, `rustc` (host); bcc-tools `offcputime` (lab VM, exercised +> in Part 8). Everything host-side is checked by `scripts/check-host.sh` or +> ships with Fedora. + +## The mutex is a futex + +A **futex** — fast userspace mutex — is a 32-bit word in your address space +plus one syscall, `futex(2)`, that the kernel only ever touches when threads +actually collide. The design splits every lock operation into two paths. On +the **fast path**, the lock word is `0` (free); acquiring it is a single +atomic compare-and-swap from `0` to `1`, executed entirely in userspace by +one `cmpxchg` instruction. No syscall, no mode switch, no scheduler — the +kernel does not even know a lock was taken. On the **slow path**, the CAS +loses because the word is already `1`: only *now* does the thread call +`futex(FUTEX_WAIT, &word, 1)`, which asks the kernel to park it on a wait +queue keyed by the physical address of the word until someone else calls +`futex(FUTEX_WAKE, &word)` on unlock. This is the whole reason locks are +cheap: contention is what costs, not locking. + +{% include excalidraw.html + file="25-futex-fast-slow-path" + alt="Two horizontal bands. The upper user-space band (the lock word lives here, glibc pthread / libstdc++ / std) shows on the left Thread A: lock() with word == 0, an arrow labeled cmpxchg into an amber CAS word 0 to 1 box (one atomic instruction, returns held), and a note Fast path: no syscall, ~98% of lock ops in the 1x1 run stay here. On the right Thread B: lock() finds word == 1 and its CAS loses, and Thread A: unlock() sets word 1 to 0. The lower kernel band (entered only when the CAS loses) holds an amber futex wait queue box keyed by (mm, and word) with Thread B parked; an amber futex(FUTEX_WAIT) arrow descends from Thread B into it, an amber futex(FUTEX_WAKE) arrow descends from Thread A's unlock, and a dashed runnable arrow returns Thread B to user space. A footer note reads voluntary context switches track the parking: 302 in the 1x1 run to 139,565 in the 8x8 run, from /usr/bin/time -v." + caption="Figure 25.2 — one lock, two paths: the uncontended CAS never leaves userspace; only a lost CAS descends into FUTEX_WAIT, and the numbers are this chapter's measured strace and context-switch scaling" %} + +The **condition variable** is the same primitive with a subtlety that has +sunk more programs than any other in this book: the **lost wakeup**. A +condvar's `wait` must (1) release the mutex, (2) block, and (3) reacquire — +and if a `notify` lands in the gap between a thread deciding to wait and +actually parking, a naive implementation sleeps through it forever. Two +mechanisms defend against it. First, `futex(FUTEX_WAIT, &word, expected)` +takes the value the caller *expects*; the kernel re-checks it under the wait +queue lock and returns `EAGAIN` if the word already changed, refusing to +park on stale state — in the 1×1 run below, 1,796 of 4,583 futex calls were +exactly this `EAGAIN` re-check. Second, and this is the discipline *your* +code owes, a condvar wait must always sit inside a **predicate loop**, never +a bare `if` — re-test the condition on every wake, because wakeups can be +spurious and the world can change between wake and reacquire. Every queue in +this chapter obeys that rule, and it is the single most important line in +each. + +Higher-level wrappers are the same futex again. C++20 added +`std::atomic::wait`/`notify_one`, a futex on a bare atomic with no mutex; +Go's `sync.Mutex` spins briefly, then parks on the runtime's futex +equivalent, and since Go 1.9 flips into a **starvation mode** that hands the +lock directly to the longest waiter once anyone has waited over 1 ms, trading +throughput for fairness; Rust's `parking_lot::Mutex` is one word and often +beats `std` under contention, though `std::sync::Mutex` on Linux has itself +been a thin futex since 1.62. Different vocabularies, one kernel object. + +## How the code works + +The queue is the heart of it, and it is where the three languages diverge +most, so it gets the tabs. Each is the real blocking-queue primitive, +verbatim from `examples/25-shared-state-and-the-futex/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp + void Push(std::uint64_t v) { + std::unique_lock lock(mu_); + not_full_.wait(lock, [&] { return q_.size() < cap_; }); + q_.push_back(v); + not_empty_.notify_one(); + } + + // Pop one item; returns false once the queue is closed and drained. + bool Pop(std::uint64_t& out) { + std::unique_lock lock(mu_); + not_empty_.wait(lock, [&] { return !q_.empty() || closed_; }); + if (q_.empty()) return false; // closed_ is true here + out = q_.front(); + q_.pop_front(); + not_full_.notify_one(); + return true; + } + + void Close() { + { + std::unique_lock lock(mu_); + closed_ = true; + } + not_empty_.notify_all(); + } +``` + +```go + // Closer: end the consumers' range once every item has been pushed. + go func() { + producers.Wait() + close(queue) + }() + + var consumers sync.WaitGroup + for idx := 0; idx < c.consumers; idx++ { + consumers.Add(1) + go func(idx int) { + defer consumers.Done() + for v := range queue { + if c.buggy { + sharedConsumed++ // DATA RACE: unsynchronized shared counter + sharedChecksum ^= v // DATA RACE: unsynchronized shared checksum + } else { + localCounts[idx]++ + localSums[idx] ^= v + } + } + }(idx) + } + consumers.Wait() +``` + +```rust + fn push(&self, v: u64) { + let mut g = self.inner.lock().unwrap(); + while g.q.len() >= self.cap { + g = self.not_full.wait(g).unwrap(); + } + g.q.push_back(v); + drop(g); + self.not_empty.notify_one(); + } + + /// Pop one item; returns None once the queue is closed and drained. + fn pop(&self) -> Option { + let mut g = self.inner.lock().unwrap(); + loop { + if let Some(v) = g.q.pop_front() { + drop(g); + self.not_full.notify_one(); + return Some(v); + } + if g.closed { + return None; + } + g = self.not_empty.wait(g).unwrap(); + } + } + + fn close(&self) { + let mut g = self.inner.lock().unwrap(); + g.closed = true; + drop(g); + self.not_empty.notify_all(); + } +``` + +The C++ and Rust queues are the same design in two spellings: one +`std::mutex`/`Mutex` guards a `std::deque`/`VecDeque`, two condition +variables split the two blocking reasons apart — producers park on +`not_full`, consumers on `not_empty` — and `Close` sets `closed_` under the +lock then `notify_all`s so every drained consumer wakes, sees the queue empty +*and* closed, and returns cleanly. The predicate is the correctness: C++'s +`not_empty_.wait(lock, [&]{ return !q_.empty() || closed_; })` re-tests on +every wake, and Rust spells the identical loop by hand with +`while ... { g = self.not_empty.wait(g).unwrap(); }`. Notice both `notify` +*after* releasing (Rust's explicit `drop(g)`; C++'s scope) so the woken +thread does not immediately re-block on a lock the notifier still holds — the +classic "wake-then-stall" pessimisation. Go throws the whole apparatus away: +a **buffered channel** `make(chan uint64, c.cap)` *is* the bounded blocking +queue — send blocks when full, receive blocks when empty, both on the +runtime's own futex parks — and closing the channel is the drain signal that +ends every consumer's `range`. A separate closer goroutine waits on the +producers' `WaitGroup` and calls `close(queue)`; without that indirection, +whoever closed early would panic a producer still sending. + +The accumulator design is the quiet star. On the correct path each consumer +folds into *its own* `localSums[idx]` slot — no shared write on the hot path +at all — and the per-consumer sums are XOR-combined only after every thread +has joined, an edge the join itself establishes. That is deliberate: because +each item's `payload(seed, i)` is a pure function of its index and XOR is +commutative and associative, the final checksum is one fixed constant +(`42b3746c6cee7465` for the default seed at N=100000) no matter how items +split across producers or which consumer drains which. The only shared write +in flight is the `produced` counter, and it is a `std::atomic` / +`atomic.AddInt64` / `AtomicI64` incremented with **relaxed** ordering, +because it needs atomicity but no happens-before edge — the mutex and channel +already supply all the ordering the data needs. + +## Errors, three ways + +The argument contract is the same shape as every example in the book, and +all three languages produce byte-identical diagnostics. A missing required +flag prints the usage line alone and exits 2; `--foo` prints +`workq: unknown flag: --foo` then usage; `--items x` prints +`workq: not an integer: x`; `--producers 0` prints +`workq: --producers and --consumers must be >= 1`; and a value-less +`--producers` at end of line prints `workq: --producers needs a value`. +`verify.lua` asserts all of these, plus the degenerate `--items 0` draining +to `produced=0 consumed=0 checksum=0000000000000000`, per language — 25 +checks each. The split of responsibility mirrors the languages' Chapter 5 +policies: C++ threads `std::expected` through the parser, +Go returns `(config, msg, ok)`, Rust returns `Result>` +where `Err(None)` means "usage only" — three encodings of the same state +machine, one observable behaviour. + +## Concurrency lens + +The interesting failures here are not error codes, they are **data races**, +and `--buggy` manufactures a real one: every consumer abandons its private +slot and mutates one shared `shared_consumed` / `shared_checksum` with no +lock. Two threads running `counter += 1` interleave load/add/store and lose +updates, so `consumed` comes out *below* N and the checksum is garbage — the +canonical race. What each language does about it is the lesson. C++ and Go let +you *write* the race and rely on a sanitizer to catch it at runtime; Rust +will not let the racing version compile at all, because a shared `&mut` across +threads violates the borrow checker — the buggy path has to smuggle a raw +`*mut` through a hand-written `unsafe impl Send`/`Sync` wrapper (`Racy`) +just to express what the other two write by accident. The type system is not +being pedantic; it is refusing the exact bug the sanitizers exist to find. We +prove all three below. + +## Build, run, observe + +```bash +[host]$ cd examples/25-shared-state-and-the-futex && ./demo.sh build +``` + +The runner agrees on the correct path: `python3 scripts/test-all-examples.py +--only 25-shared-state-and-the-futex` reports `PASS PASS PASS` (3 passed, 0 +failed, 0 skipped), 25 `verify.lua` checks per language, Go built under +`-race`. A hand run confirms the shared constant regardless of shape: + +```console +[host]$ ./cpp/build/release/app --producers 8 --consumers 8 --items 100000 --cap 8 +workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms=127 +``` + +Now break it. **C++ under ThreadSanitizer** (`cmake --preset tsan && cmake +--build --preset tsan`) names the two racing lines precisely: + +```console +[host]$ ./cpp/build/tsan/app --producers 4 --consumers 4 --items 100000 --buggy +WARNING: ThreadSanitizer: data race (pid=2802859) + Read of size 8 at 0x7ffe96951458 by thread T6: + #0 operator() .../cpp/src/main.cpp:209 (app+0x400da1) + Previous write of size 8 at 0x7ffe96951458 by thread T5: + #0 operator() .../cpp/src/main.cpp:209 (app+0x400db0) +SUMMARY: ThreadSanitizer: data race .../cpp/src/main.cpp:209 in operator() +... +workq: produced=100000 consumed=80629 checksum=a27c6c5d34a3e44a ms=4177 +``` + +Lines 209 and 210 are exactly `shared_consumed += 1;` and +`shared_checksum ^= v;`, and `consumed=80629` shows ~19,000 lost updates. +**Go under `-race`** finds the same thing at `main.go:160`, prints +`WARNING: DATA RACE` for each pair of colliding goroutines, and exits nonzero: + +```console +[host]$ ./go/bin/app --producers 4 --consumers 4 --items 100000 --buggy +WARNING: DATA RACE +Read at 0x00c000018198 by goroutine 13: + main.run.func3() .../go/main.go:160 +0x23d +Previous write at 0x00c000018198 by goroutine 16: + main.run.func3() .../go/main.go:160 +0x252 +... +workq: produced=100000 consumed=95400 checksum=46376190c7e5d451 ms=3122 +Found 4 data race(s) +``` + +**Rust** needs no sanitizer — safe Rust rejects the race outright. Reduce the +buggy accumulator to its essence and `rustc` refuses it: + +```console +[host]$ rustc --edition 2024 racetest.rs +error[E0499]: cannot borrow `consumed` as mutable more than once at a time + 6 | s.spawn(|| { + | - ^^ `consumed` was mutably borrowed here in the previous iteration of the loop + 7 | consumed += 1; + | -------- borrows occur due to use of `consumed` in closure +``` + +The example's `--buggy` only compiles because it opts out through `unsafe`, +and when it runs the corruption wanders every time — `consumed=97094`, +`97633`, `97301`, `95632` across four runs, never the correct `100000` — the +empirical shadow of the guarantee the compiler was offering. + +## Cross-check: the futex under the microscope + +The claim that an uncontended lock avoids the kernel entirely is checkable +with `strace -c`, counting `futex(2)` calls as we turn contention up. Two +runs of the same 100,000-item workload, first with one producer and one +consumer over a roomy 256-slot queue, then with eight of each over an +8-slot queue: + +```console +[host]$ strace -f -c -e trace=futex ./cpp/build/release/app --producers 1 --consumers 1 --items 100000 --cap 256 +% time seconds usecs/call calls errors syscall +100.00 0.021349 4 4583 1796 futex +[host]$ strace -f -c -e trace=futex ./cpp/build/release/app --producers 8 --consumers 8 --items 100000 --cap 8 +% time seconds usecs/call calls errors syscall +100.00 33.008779 67 491514 124803 futex +``` + +The 1×1 run performs roughly 200,000 lock acquisitions (a `Push` and a `Pop` +per item, each taking the mutex) yet enters the kernel only **4,583** times — +about 98% of all lock operations resolved as a pure userspace CAS, exactly +the fast path. Crank contention to 8×8 with a tight buffer and the count +explodes to **491,514**: every collision that loses its CAS now pays a +`FUTEX_WAIT`, and every unlock with a waiter pays a `FUTEX_WAKE` (the exact +counts drift a few percent per run — a repeat 1×1 gave 4,623, an 8×8 gave +488,818 — but the two-orders-of-magnitude gap is invariant). Watching +the raw calls confirms both operations, `2` being the mutex's contended state +and `2147483647` a condvar `notify_all`: + +```console +[pid ...] futex(0x7ffe96160f70, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 +[pid ...] futex(0x7ffe96160f70, FUTEX_WAKE_PRIVATE, 1) = 1 + futex(0x7f9c51aaf6bc, FUTEX_WAKE_PRIVATE, 2147483647) = 0 +``` + +An independent surface tells the same story. Every `FUTEX_WAIT` that actually +parks is a **voluntary context switch** — the thread yielding the CPU because +it chose to block — and `/usr/bin/time -v` counts them from `getrusage`: + +```console +[host]$ /usr/bin/time -v ./cpp/build/release/app --producers 1 --consumers 1 --items 100000 --cap 256 + Voluntary context switches: 302 + Involuntary context switches: 0 +[host]$ /usr/bin/time -v ./cpp/build/release/app --producers 8 --consumers 8 --items 100000 --cap 8 + Voluntary context switches: 139565 + Involuntary context switches: 645 +``` + +302 versus 139,565 — the futex-call scaling and the context-switch scaling +are two views of one fact: contention turns lock operations into kernel +sleeps. The kernel exposes the very same counters live per thread; summing +`voluntary_ctxt_switches` across `/proc//task/*/status` mid-run on a +larger job gave 5,551 for the 1×1 shape (3 tasks) against 491,926 for the 8×8 +shape (17 tasks). (`perf stat` reads `task-clock:u` freely on this host, but +the software `context-switches` event is a kernel counter gated by +`perf_event_paranoid=2`, so the `getrusage`/`/proc` route is the +unprivileged one — the same numbers without lowering the paranoia knob.) + +> **On the lab VM** unverified — +> the fleet view of this is bcc-tools `offcputime`, which attributes every +> thread's *off-CPU* time to the stack that blocked it, so a contended +> `workq` shows its wall-clock time piling up under `futex_wait` → +> `pthread_cond_wait` with no `strace` slowdown. That needs `CAP_BPFMON` and +> a disposable kernel, so it is not run here; the Debugging part (Part 8) +> exercises `offcputime` on the `systems-target` VM. + +## What you learned + +- A mutex is a **futex**: an uncontended lock is a userspace CAS with no + syscall (4,583 futex calls for ~200,000 lock ops in the 1×1 run), and only + a lost CAS descends into `FUTEX_WAIT`/`FUTEX_WAKE` — which is why + contention, not locking, is what costs. +- A condition variable must wait inside a **predicate loop**, never a bare + `if`; the kernel's `EAGAIN` value re-check and your re-test on every wake + are the two halves of the lost-wakeup defence. +- The correct queue keeps shared writes off the hot path (per-consumer slots, + XOR-combined after join; the only shared word is a relaxed atomic counter), + so a mutex or channel supplies every happens-before edge the data needs. +- A data race is undefined behaviour, and the three languages meet it + differently: ThreadSanitizer and `go -race` catch it at runtime (`consumed` + falling to 80,629 / 95,400), while Rust's borrow checker refuses to compile + the racing version at all — the bug the sanitizers hunt is the bug the type + system forbids. + +Next, **atomics and lock-free**: `spscring` drops the lock entirely and +coordinates a ring buffer with two atomic counters, where acquire/release +ordering — not a futex — is the whole game. + +--- + +

verified — every number and +excerpt above was produced on the Fedora 44 reference host (kernel +7.1.3-200.fc44, GCC 16.1.1, go1.26.5, rustc 1.97.1, strace 7.1) this session. +The runner printed 25-shared-state-and-the-futex PASS PASS PASS +(3 passed, 0 failed, 0 skipped; verify.lua 25/25 per language, Go under +-race); the hand run printed the +checksum=42b3746c6cee7465 line; ThreadSanitizer reported the data +race at main.cpp:209 with consumed=80629; +go -race reported DATA RACE at main.go:160, +Found 4 data race(s), consumed=95400; rustc +--edition 2024 rejected the reduced racing program with +error[E0499], and the --buggy Rust binary printed +consumed=97094/97633/97301/95632 across four runs; strace +-c counted 4583 futex calls (1796 EAGAIN) for the 1×1 +cap-256 run and 491514 (124803 EAGAIN) for the 8×8 cap-8 run +(counts drift a few percent per run), with real +FUTEX_WAIT_PRIVATE/FUTEX_WAKE_PRIVATE lines +observed; and /usr/bin/time -v reported 302 versus +139565 voluntary context switches, matched by +voluntary_ctxt_switches summed across +/proc/<pid>/task/*/status (5,551 vs 491,926 mid-run). The +offcputime bcc-tools callout is unverified as marked and is +exercised in Part 8.

diff --git a/_docs/26-atomics-and-lock-free.md b/_docs/26-atomics-and-lock-free.md new file mode 100644 index 0000000..e5607f2 --- /dev/null +++ b/_docs/26-atomics-and-lock-free.md @@ -0,0 +1,438 @@ +--- +title: "Atomics and lock-free: memory ordering, the SPSC ring, and false sharing" +order: 26 +part: "Concurrency in Depth" +description: "spscring drops the lock entirely: a single-producer/single-consumer ring coordinated by two atomic counters, with acquire/release explained on the real head/tail, why a non-atomic ring tears, the ABA problem SPSC sidesteps, and false sharing measured with perf — a padded line beats an unpadded one by ~34% on this host." +duration: "55 minutes" +--- + +Every coordination primitive so far in this book has ultimately been a *lock*: +`parhash` handed work across a mutex-guarded queue, and `shmkv` reached all +the way down to the futex the mutex is built from. A lock is a promise that +only one thread runs the critical section at a time — correct, general, and +never free. This chapter removes it. `spscring` is a bounded ring buffer with +exactly one producer thread and one consumer thread, and it moves ten million +values between them with no mutex, no futex, and no `unsafe` data race — only +two atomic counters and a careful choice of *memory ordering*. That last +phrase is the one new idea: on a modern multicore CPU the order in which one +thread's writes become visible to another is not "the order you wrote them," +and the whole discipline of lock-free code is naming exactly how much ordering +you need and paying for no more. + +The code is in `examples/26-atomics-and-lock-free/`. `./demo.sh` builds all +three implementations; its `README.md` specifies the CLI, the one-line output, +and the exit codes all three languages share. + +{% include excalidraw.html + file="26-spsc-ring" + alt="Two thread boxes over a shared-ring band. The producer on core 0 owns tail and reads head, does buf[t%K]=v then tail.store(t+1, Release); the consumer on core 2 owns head and reads tail, does v=buf[h%K] then head.store(h+1, Release). Two accent boxes in the middle are the tail atomic (producer-only writer) and head atomic (consumer-only writer). Amber arrows show each thread's Release store to its own counter and Acquire load of the other's; a dashed amber arrow from tail down into buf[0] is labelled tail Release publishes buf[t%K]. A ring of six slots buf[0..5] shows buf[0]=3 and buf[1]=4 filled, the rest free; a note reads occupancy = tail - head = 2, full when tail - head == K, empty when head == tail." + caption="Figure 26.1 — the Lamport SPSC ring: each counter has exactly one writer, and the Release store on that counter publishes the slot the other thread is about to read" %} + +> **Tools used** — `perf stat`, `perf c2c`, `objdump`, `go tool objdump`, +> `taskset`, `python3` (host). `perf`/`objdump`/`taskset` are in +> `scripts/check-host.sh`; `perf c2c` needs root and is deferred to the lab VM +> (Part 8). + +## Memory ordering, on the ring's own counters + +The ring is two monotonic `u64` counters. `tail` counts values the producer +has pushed; `head` counts values the consumer has popped. The physical slot +for logical position `p` is `buf[p % K]`, occupancy is `tail - head`, the ring +is **full** at `tail - head == K` and **empty** at `head == tail`. The single +rule that makes this safe without a lock is *one writer per counter*: the +producer writes `tail` and only reads `head`; the consumer writes `head` and +only reads `tail`. No location is ever written by two threads, so there is no +write-write race to arbitrate — which is exactly what a lock would have +arbitrated. + +What is left is a *publication* problem, and it is subtler than it looks. +When the producer runs `buf[t % cap] = i` and then `tail.store(t + 1)`, it has +performed two writes on core 0. The consumer, spinning on core 2 until +`tail` moves, will read the slot the instant it sees the new `tail`. For that +read to be valid, the slot write must be visible *before* the counter write — +and nothing about program order guarantees that across cores. Both the +compiler and the CPU are free to reorder independent stores; core 2 may +observe the new `tail` while the slot still holds stale bytes. That is a +*torn* hand-off, and a naive ring built on plain `uint64_t` counters exhibits +it: the sum comes out wrong, sporadically, on exactly the runs you did not +test. Atomics fix it not merely by being indivisible but by carrying an +**ordering**: + +- **Release** on a store says "every write I did before this one is visible to + any thread that acquires this value." `tail.store(t + 1, release)` publishes + the slot write that precedes it. +- **Acquire** on a load says "every write the releasing thread did before its + release is now visible to me." `tail.load(acquire)` in the consumer observes + the slot. Together they form a *happens-before* edge across the two cores. +- **Relaxed** promises atomicity and nothing about ordering. Rust marks the + slot accesses `Ordering::Relaxed` precisely because they need no edge of + their own — the `tail` release/acquire pair already covers them. +- **Sequential consistency** is the strongest: one global total order over all + such operations. It is the *only* ordering Go's `sync/atomic` exposes, and it + is strictly more than this ring needs. + +{% include excalidraw.html + file="26-memory-ordering" + alt="Three stacked bands, each a memory ordering paired with its x86 code generation. Top, sequential consistency (Go's only setting): tail.Store(t) compiles to XCHGQ, implicitly locked, a full fence — the store pays for global order. Middle, acquire/release (C++ and Rust head/tail): a Release store to Acquire load edge compiles to a plain MOV, no lock and no fence, because x86 TSO already orders store-to-store and load-to-load. Bottom, relaxed (Rust ring slots): no cross-thread ordering promise, buf[t%K].store(v, Relaxed), compiles to the same plain MOV as release, safe only because tail's edge covers it. A footnote says on x86 the ordering is nearly free — the difference is the promise, not the instruction; on ARM or POWER the movs would need real barriers." + caption="Figure 26.2 — the same hand-off under three orderings and what each becomes on x86-64; the instructions are from the objdump in the cross-check" %} + +Here is the caution the brief demands: **do not read this ring's `Relaxed` +slots as a portability claim.** They are correct here because the acquire/release +edge on `tail` synchronises them, and because the single-writer rule means no +slot is ever concurrently written. Change the shape — more producers, a slot +read-modified in place — and `Relaxed` is a bug. Relaxed orderings are a +scalpel for a proof you have already written, not a default. + +## Why SPSC, and the ABA problem it sidesteps + +The reason this ring can be so simple is that it is *single*-producer, +*single*-consumer. The famous hazard of lock-free stacks and queues is **ABA**: +a thread reads a pointer value `A`, is preempted, and by the time it runs its +compare-and-swap the location has gone `A → B → A` — the CAS succeeds on a +value that is numerically identical but semantically stale, and the structure +corrupts. ABA is a property of designs that *compare-and-swap a reused value*. +`spscring` never does. Its counters are **monotonic** — they only ever +increase, for the life of the run — so no `tail` or `head` value ever recurs, +and there is no CAS at all: each counter has one writer that simply `store`s +the next number. Using unbounded monotonic counters rather than wrapped +indices is also what makes full-versus-empty unambiguous without wasting a +slot. SPSC is the foundation the rest of lock-free programming builds on +precisely because it makes ABA structurally impossible; the moment you add a +second producer you inherit CAS, ABA, and the tag-counter or hazard-pointer +machinery that fights it. + +## How the code works + +The heart of all three programs is the same two loops — a producer that spins +while the ring is full and a consumer that spins while it is empty — and the +ordering annotations are the whole story. The excerpts below are verbatim; read +the C++ and Rust ordering arguments against Go's conspicuous absence of them. + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp + // Producer: push 0..items-1. jthread joins on scope exit. + std::jthread producer([&] { + std::uint64_t t = 0; + for (std::uint64_t i = 0; i < items; ++i) { + // Wait until the ring has a free slot (consumer advanced head). + while (t - head.load(std::memory_order_acquire) == cap) { + // spin + } + buf[t % cap] = i; + ++t; + tail.store(t, std::memory_order_release); + } + }); + + // Consumer runs on this thread. + std::uint64_t sum = 0; + std::uint64_t h = 0; + for (std::uint64_t i = 0; i < items; ++i) { + while (h == tail.load(std::memory_order_acquire)) { + // spin: ring empty + } + sum += buf[h % cap]; + ++h; + head.store(h, std::memory_order_release); + } +``` + +```go + done := make(chan struct{}) + go func() { // producer + var t uint64 + for i := uint64(0); i < items; i++ { + for t-head.Load() == capacity { // ring full: spin + } + buf[t%capacity] = i + t++ + tail.Store(t) + } + close(done) + }() + + // consumer (main goroutine) + var h uint64 + for i := uint64(0); i < items; i++ { + for h == tail.Load() { // ring empty: spin + } + sum += buf[h%capacity] + h++ + head.Store(h) + } + + <-done +``` + +```rust + let sum = thread::scope(|s| { + // Producer: push 0..items-1. + s.spawn(move || { + let mut t: u64 = 0; + for i in 0..items { + while t - head.load(Ordering::Acquire) == cap { + std::hint::spin_loop(); + } + buf[(t % cap) as usize].store(i, Ordering::Relaxed); + t += 1; + tail.store(t, Ordering::Release); + } + }); + + // Consumer on this thread. + let mut h: u64 = 0; + let mut sum: u64 = 0; + for _ in 0..items { + while h == tail.load(Ordering::Acquire) { + std::hint::spin_loop(); + } + sum = sum.wrapping_add(buf[(h % cap) as usize].load(Ordering::Relaxed)); + h += 1; + head.store(h, Ordering::Release); + } + sum + }); +``` + +Three things distinguish the spellings. C++ names the ordering on every atomic +operation (`memory_order_acquire` on the loads, `memory_order_release` on the +stores) and leaves the slot a plain `std::vector` — the release/acquire +edge is what makes the plain slot write race-free. Rust makes the slot itself +an `AtomicU64` accessed `Ordering::Relaxed`, so the absence of a data race is +proven *by the type system* rather than asserted; the `spin_loop()` hint emits +the CPU's `PAUSE` instruction so a spinning core does not starve its +hyperthread sibling. Go carries **no ordering argument at all**: its +`sync/atomic` `Load`/`Store` are sequentially consistent, full stop. You cannot +relax them, but you also cannot get them wrong — the Go memory model guarantees +that an atomic write is *synchronized-before* the read that observes it, which +is exactly the acquire/release edge this ring needs, wearing a stronger +guarantee than it strictly requires. The Go binary is built and shipped under +the race detector (`go/demo.sh` passes `-race`); the detector understands the +atomic edge and stays silent on the plain `buf[]` slot writes, which is direct +evidence that the lock-free hand-off is race-free rather than merely lucky. + +The `sum` field is a correctness oracle. The pushed values are `0 .. N-1`, so a +run that delivers every value exactly once must total `N·(N-1)/2`. For ten +million items that is `49999995000000`; any dropped, duplicated, or torn +hand-off would move it. It is checked on every run below. + +## Errors, three ways + +`spscring` has a deliberately tiny failure surface — it is compute, not I/O — +and it is all in argument parsing. A missing `--capacity` or `--items`, a +non-numeric value, a zero capacity, or an unknown `--pad` word all resolve to +one behaviour, identical across the three languages: print +`usage: spscring --capacity K --items N [--pad on|off]` to stderr and exit **2**. +C++ threads the fallible parse through `std::optional` and returns the +usage line on `nullopt`; Go returns `(args, false)` and `os.Exit(2)`; Rust +returns `Result` and `ExitCode::from(2)`. A successful run prints its +one line and exits **0**. There is no exit-1 runtime-error path, because once +the arguments are valid the program cannot fail — there is no file, socket, or +allocation that can refuse it beyond the initial `Vec`/`vector` reservation. +`verify.lua` asserts all of this: 15 checks per language, including that the +one-slot ring (`--capacity 1`, maximum head/tail contention) still sums +correctly, and that the no-argument invocation produces the exact usage string +and exit 2. + +## Concurrency lens + +This is the most concurrent program in the book and the one with the least +synchronisation machinery: two threads, zero locks, two atomics. The safety +argument is entirely the single-writer rule plus the release/acquire edge, and +it is worth stating what would break each way. Drop the atomicity and the +counter loads can tear on a 32-bit split (not on this 64-bit host, but the type +is doing real work on others). Drop the *ordering* — use relaxed on `tail` — +and the slot publication edge vanishes: the consumer can read a slot the +producer has not finished writing, and the sum corrupts. Add a second producer +and the single-writer rule collapses, taking the whole lock-free proof with it. +The spin loops are the price of going lock-free: a blocked consumer *burns* its +core rather than sleeping, which is the right trade only when the queue is +rarely empty and latency matters more than a spare core. A production ring pairs +this fast path with a futex or `condvar` slow path for when it idles — which is +exactly the chapter-20 futex, reappearing as the fallback beneath the atomics +rather than the mechanism on top of them. + +## Cache lines and false sharing + +The `--pad` flag exposes a hardware effect that no amount of correct ordering +addresses. Caches move memory in 64-byte **lines**, and coherency is tracked +per line, not per variable. When `head` and `tail` sit in the *same* line — the +default, `--pad off` — the consumer's `head.store` invalidates the producer's +cached copy of the whole line, `tail` included, even though the producer never +touched `head`. On the next iteration the producer must re-fetch a line it +already had, and the line ping-pongs between the two cores on every hand-off. +This is **false sharing**: two threads contending for one cache line while +touching logically disjoint data. `--pad on` places each counter on its own +line and the contention disappears. + +{% include excalidraw.html + file="26-false-sharing" + alt="Two panels. Left, pad=off: cores 0 and 2 both point with bidirectional amber invalidate arrows at one cache line #1 holding head and tail adjacent; annotated L1 load-misses about 128 million, median 4.64 Mops at capacity 1 with 10 million items. Right, pad=on: core 0 points locally down to line #1 holding head plus 56 bytes of padding, core 2 points locally to line #2 holding tail plus 56 bytes; annotated L1 load-misses about 108 million, median 6.20 Mops, about 34 percent faster." + caption="Figure 26.3 — false sharing measured: one shared line ping-pongs between cores (left), two padded lines each stay core-local (right); the miss counts and throughputs are the perf numbers from the cross-check" %} + +The C++ control block makes the choice a type: + +```cpp +template <> +struct Control { + alignas(64) std::atomic head{0}; + alignas(64) std::atomic tail{0}; +}; + +template <> +struct Control { + std::atomic head{0}; + std::atomic tail{0}; +}; +``` + +Go inserts `[56]byte` after each counter; Rust wraps each in +`#[repr(align(64))]`. Same bytes, same effect, three spellings. + +## Build, run, observe + +```bash +[host]$ cd examples/26-atomics-and-lock-free && ./demo.sh build +``` + +The runner (`python3 scripts/test-all-examples.py --only +26-atomics-and-lock-free`) reports `PASS PASS PASS` for cpp, go, rust — 15 +behavioural checks per language, Go under `-race`. Driven by hand on this host, +pinned to two separate physical cores with `taskset` so the producer and +consumer land on cores 0 and 2: + +```console +[host]$ taskset -c 0,2 ./cpp/build/release/app --capacity 1 --items 10000000 --pad off +spscring: items=10000000 sum=49999995000000 throughput_mops=5.51 pad=off +[host]$ taskset -c 0,2 ./cpp/build/release/app --capacity 1 --items 10000000 --pad on +spscring: items=10000000 sum=49999995000000 throughput_mops=6.42 pad=on +``` + +Both modes produce the identical, correct `sum=49999995000000`; only the +throughput differs. A one-slot ring forces a lockstep hand-off — every push +waits for the previous pop — which is the worst case for false sharing and so +the clearest place to measure it. Across eleven runs each, the median was +**4.64 Mops** padded off versus **6.20 Mops** padded on: padding buys roughly +**34%** on this Tiger Lake host. (At `--capacity 1024` the hand-off decouples, +throughput jumps into the tens of Mops, and the powersave governor's frequency +swings swamp the effect — the small ring is the reliable measurement surface, so +that is where the chapter takes its numbers.) + +## Cross-check: perf counts the false sharing, objdump shows the ordering + +Two independent tools confirm the two claims. First, `perf stat` counts the +cache traffic directly. Same binary, same pinning, `--capacity 1 --items +10000000`: + +```console +[host]$ taskset -c 0,2 perf stat -e cache-misses,L1-dcache-load-misses ./cpp/build/release/app --capacity 1 --items 10000000 --pad off +spscring: items=10000000 sum=49999995000000 throughput_mops=5.51 pad=off + 12,493 cache-misses:u + 128,086,713 L1-dcache-load-misses:u + 1.817560572 seconds time elapsed +[host]$ taskset -c 0,2 perf stat -e cache-misses,L1-dcache-load-misses ./cpp/build/release/app --capacity 1 --items 10000000 --pad on +spscring: items=10000000 sum=49999995000000 throughput_mops=6.42 pad=on + 10,456 cache-misses:u + 107,745,083 L1-dcache-load-misses:u + 1.558742655 seconds time elapsed +``` + +The signature is in `L1-dcache-load-misses`: **~128 million** unpadded versus +**~108 million** padded, a ~20-million-miss gap that is the ring re-fetching a +line the other core just invalidated. Each spinning load on the shared line +misses in L1 because the line keeps bouncing away; separating the counters lets +each core keep its own line warm. The medians over eleven runs held the same +direction (~127 M vs ~108 M). That extra miss stream is the entire mechanism +behind the throughput gap the previous section measured. + +Second, `objdump` settles what the three orderings actually *cost* on x86. The +C++ producer's release store and its acquire load of `head` compile to plain +`mov` (trimmed to the loop body): + +```console +[host]$ objdump -d --no-show-raw-insn cpp/build/release/app | grep -A30 'run_benchILb1EE.*_M_runEv>:' + 4017c0: mov 0x10(%rdi),%rax + 4017c4: mov (%rax),%rdx + ... + 4017e8: mov %rsi,(%rax,%rdx,8) + 4017f0: add $0x1,%rsi + 4017f4: mov %rsi,(%rax) +``` + +`4017c4` is `head.load(Acquire)`, `4017e8` is `buf[t%cap] = i`, and `4017f4` is +`tail.store(t, Release)` — three plain `mov`s, no `lock` prefix and no `mfence` +anywhere in the function (a `grep -c` for those over the whole producer symbol +returns 0). On x86-64's Total Store Order model, every load already has acquire +semantics and every store already has release semantics, so acquire/release is +*free* — the ordering lives in the promise, not the instruction. Rust's +`tail.load(Acquire)` in the consumer is likewise a bare `mov 0x0(%rbp),%rax` +right after the `pause`. Go's sequentially-consistent store is the contrast +that proves the point. Rebuilt without `-race` so the atomic inlines rather than +calling the detector's shim, the same `tail.Store(t)` is not a `mov`: + +```console +[host]$ go build -o /tmp/app-norace ./... && go tool objdump -s 'main\.bench' /tmp/app-norace | grep -iE 'XCHG|main.go:135' + main.go:135 XCHGQ R11, 0(R8) +``` + +`XCHGQ` is implicitly `lock`ed and carries a full barrier — the price of +sequential consistency's global order, paid on every push. That single +instruction is why "you cannot get Go's atomics wrong" and "Go's atomics are +the strongest" are the same sentence: seq-cst asks for more ordering than this +ring needs, and on x86 the store, not the load, is where you feel it. + +> **On the lab VM** unverified — +> `perf c2c`, which samples HITM (hit-modified) events to name the exact cache +> line and the two instructions trading it, needs kernel profiling this host's +> `perf_event_paranoid=2` refuses without root; Part 8 runs it on +> `systems-target` to pin the false-sharing line to the `Control` struct +> address directly, alongside `offcputime` confirming the spin loops are on-CPU +> (never sleeping, so nothing to see) — the flip side of the lock this chapter +> removed. + +## What you learned + +- A lock-free SPSC ring is two monotonic atomic counters with one writer each; + the safety comes from that single-writer rule plus a Release-store / + Acquire-load edge on each counter that *publishes* the slot write before the + other thread reads it. A non-atomic ring tears because plain stores may become + visible out of order across cores. +- Memory ordering is a spectrum: sequential consistency (Go's only setting) > + acquire/release (C++ and Rust head/tail) > relaxed (Rust slots). On x86 TSO, + acquire/release and relaxed are both plain `mov` while seq-cst's store is a + locked `XCHGQ` — but that instruction equality is *not* portability: on ARM + the same relaxed slot would need a real barrier, so relaxed is a scalpel for a + proof you have written, never a default. +- SPSC sidesteps ABA structurally: monotonic counters never recur and there is + no compare-and-swap, so the reused-value hazard that plagues multi-producer + lock-free queues cannot arise here. +- False sharing is a cache-line effect, not a correctness bug: `head` and `tail` + in one 64-byte line ping-pong between cores, and `perf stat` shows it as + ~128 M vs ~108 M `L1-dcache-load-misses` — worth a real ~34% here, fixed by + `alignas(64)` / `#[repr(align(64))]` / `[56]byte` padding. + +Next, **async runtimes**: the same epoll and futex machinery re-composed as +C++ coroutines, the Go scheduler, and tokio — and when plain threads still win. + +--- + +

verified — every number and +output excerpt above was produced on the Fedora 44 reference host (kernel +7.1.3-200.fc44, 11th-gen Intel i7-11800H, 16 CPUs, powersave governor) this +session: the runner printed 26-atomics-and-lock-free PASS PASS +PASS (3 passed, 0 failed, 0 skipped; verify.lua 15 checks per language, +Go under -race); the hand-driven taskset -c 0,2 runs +printed the exact sum=49999995000000 for both pad=off +(throughput 5.51) and pad=on (6.42), and eleven-run batches gave +medians 4.64 vs 6.20 Mops; perf stat reported +128,086,713 vs 107,745,083 L1-dcache-load-misses:u +(medians ~127 M vs ~108 M); objdump showed the C++ release store +and acquire load as plain mov (4017f4: mov %rsi,(%rax)) +with no lock/mfence, Rust's acquire load as a plain +mov after pause, and go tool objdump on a +non--race build showed tail.Store as +XCHGQ R11, 0(R8). The perf c2c HITM analysis and +offcputime are unverified as marked — perf c2c failed +with perf_event_paranoid=2 here and is exercised on the lab VM in +Part 8.

diff --git a/_docs/27-async-runtimes-and-coroutines.md b/_docs/27-async-runtimes-and-coroutines.md new file mode 100644 index 0000000..3c38644 --- /dev/null +++ b/_docs/27-async-runtimes-and-coroutines.md @@ -0,0 +1,434 @@ +--- +title: "Async runtimes and coroutines" +order: 27 +part: "Concurrency in Depth" +description: "chatterd v4 tells the same broadcast three async ways over the same OS threads: hand-rolled C++20 coroutines suspending on an epoll reactor, Go's G-M-P scheduler and netpoller that needed no rewrite, and tokio's tasks-and-Wakers — with /proc//task thread counts (1 vs 17 vs dynamic) and a single-epoll-loop strace showing how each spends threads, plus the framework for when plain threads still win." +duration: "60 minutes" +--- + +Chapters 25 and 26 took concurrency down to the metal — the futex under a +mutex, atomics and a lock-free ring. This chapter goes the other direction: +up to the abstractions that let one program juggle tens of thousands of +connections without one thread each. `chatterd` — the book's peer-to-peer +chat daemon — gets a v4 **async rewrite** whose observable behaviour is +byte-for-byte unchanged: a message from one client is broadcast to every +other. What changes is the connection engine, selected by `serve --engine +thread|async`. The `thread` engine (one OS thread per client, the chapter 21 +baseline) stays, so you can put the two side by side. The `async` engine is +a different story in each language: C++ grows a hand-rolled C++20 coroutine +reactor over `epoll`; Rust switches to **tokio**; and Go's async engine is +its *thread* engine, because goroutines were already the answer. The one new +idea is that all three suspend a *unit of work* — a coroutine frame, a +goroutine, a task — instead of parking an OS thread, and multiplex many of +them onto few threads. The concurrency lens is not a section here. It is the +whole chapter. + +The code is in `examples/27-async-runtimes-and-coroutines/`. `./demo.sh` +builds all three; its `README.md` specifies the CLI, the canonical chatterd +chat frame (magic `"CH"` + version + type + big-endian length, unchanged +since ch21 — this chapter's engines use only its JOIN/MSG/DELIVER types), +and the exit codes all three languages share. + +{% include excalidraw.html + file="27-coroutine-suspension" + alt="Three horizontal bands. In the user band, a connection() coroutine runs four boxes left to right: step 1 co_await fill() with recv returning EAGAIN, step 2 the accent-highlighted co_await ReadAwaitable calling await_suspend(h), step 3 a dashed frame-suspended box where control returns down, and step 6 h.resume() where recv returns bytes and decodes. In the reactor band (the single OS thread, Reactor::run) sit wait_readable(fd,h) which stores waiters_[fd]=h and arms the fd, and epoll_wait(-1) which resumes the parked handle. In the kernel band an EPOLLIN | EPOLLONESHOT box marks the fd ready via events[i].data.fd. An amber suspend arrow runs down from step 2 to wait_readable, an epoll_ctl arrow arms the kernel fd, an amber ready arrow returns to epoll_wait, and an amber resume arrow returns up to step 6." + caption="Figure 27.1 — chatterd v4's C++20 coroutine engine: co_await ReadAwaitable suspends the connection frame and parks its handle in the reactor; epoll_wait resumes the exact handle on readiness. Suspension is a return, not a blocked thread." %} + +> **Tools used** — `strace`, `ls`/`/proc//task`, `nproc`, `python3` +> (host); `offcputime`, `tcplife`, `tcpconnect` from bcc-tools (lab VM, +> exercised in Part 8). Everything host-side is checked by +> `scripts/check-host.sh` or ships with Fedora. + +## One machine, three stories over the same threads + +Every async runtime solves one problem: a blocking `recv` wastes a whole OS +thread while it waits, and threads are expensive — a megabyte of stack, a +scheduler slot, a context-switch each time the kernel changes your mind. The +readiness model from chapter 9 already fixed the *waiting* (nonblocking fds +plus one `epoll_wait`), but it left the *code* inside out: a hand-written +state machine per connection, dispatching on a token. Coroutines, goroutines, +and tasks all buy back the straight-line shape — you write +`read; broadcast; read` and the runtime slices it at each suspension point, +storing "where was I" somewhere cheaper than a thread. The three differ only +in *where that somewhere lives* and *who drives the resume*. + +## How the code works + +**C++: a coroutine is a struct the compiler writes for you.** When a function +contains `co_await`, `co_return`, or `co_yield`, the compiler transforms it +into a state machine whose locals live in a heap-allocated *coroutine frame*, +and it hands control to a `promise_type` you supply. chatterd defines two. +`Detached` is fire-and-forget — `initial_suspend` and `final_suspend` both +return `std::suspend_never`, so it starts eagerly and destroys its own frame +when it finishes; the acceptor and every per-connection coroutine are +`Detached`. `Task` is the lazy, value-returning kind, and its +`final_suspend` is where the interesting trick lives: + +```cpp + struct Final { + bool await_ready() noexcept { return false; } + std::coroutine_handle<> await_suspend( + std::coroutine_handle h) noexcept { + auto c = h.promise().cont_; + return c ? c : std::noop_coroutine(); + } + void await_resume() noexcept {} + }; +``` + +Returning a handle from `await_suspend` is **symmetric transfer**: instead of +returning to the caller and unwinding the stack, the finished task resumes its +awaiter directly, in the same stack frame — no recursion, no stack growth even +across thousands of chained resumptions. The `co_await` operator on `Task` +does the mirror image: `await_suspend` stashes the caller's handle in +`cont_` and returns the task's own handle to begin it. The awaitable that +connects all this to the kernel is tiny — it is the whole point of the +figure. `ReadAwaitable::await_suspend` hands the *current* coroutine handle to +the reactor, which stores it against the fd and arms `epoll`; when the fd +fires, `Reactor::run` calls `h.resume()` and the `co_await` expression +completes as if `recv` had simply blocked: + +```cpp + void wait_readable(int fd, std::coroutine_handle<> h) { + waiters_[fd] = h; + epoll_event ev{}; + ev.events = EPOLLIN | EPOLLONESHOT; + ev.data.fd = fd; + if (armed_.contains(fd)) { + ::epoll_ctl(ep_.get(), EPOLL_CTL_MOD, fd, &ev); + } else { + ::epoll_ctl(ep_.get(), EPOLL_CTL_ADD, fd, &ev); + armed_.insert(fd); + } + } +``` + +`EPOLLONESHOT` is chapter 9's multi-consumer safeguard reused for a different +reason: each `co_await` re-arms exactly once, so a readable fd fires for +exactly the one coroutine parked on it and nothing double-resumes. `data.fd` +carries the fd back, `waiters_[fd]` maps it to the handle, and `h.resume()` +runs the frame from its suspension point. That is the entire runtime — no +external async library is linked. + +Below is the suspension point itself, beside the two runtimes that hide the +same mechanism. Each excerpt is the *wait-for-the-next-event-and-dispatch* +heart of its engine, verbatim from +`examples/27-async-runtimes-and-coroutines/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +// Pull at least one more byte into `buf`. false => EOF / error. +Task fill(Reactor& reactor, int fd, std::string& buf) { + for (;;) { + char tmp[4096]; + ssize_t n = ::recv(fd, tmp, sizeof tmp, 0); + if (n > 0) { + buf.append(tmp, static_cast(n)); + co_return true; + } + if (n == 0) { + co_return false; // peer closed + } + if (errno == EAGAIN || errno == EWOULDBLOCK) { + co_await ReadAwaitable{reactor, fd}; + continue; + } + if (errno == EINTR) { + continue; + } + co_return false; + } +} +``` + +```go +func (h *hub) run(ctx context.Context) { + clients := make(map[*client]struct{}) + for { + select { + case c := <-h.register: + clients[c] = struct{}{} + case c := <-h.unregister: + if _, ok := clients[c]; ok { + delete(clients, c) + close(c.out) + } + case b := <-h.broadcast: + for c := range clients { + if c != b.from { + c.out <- b.frame // buffered; delivery is guaranteed + } + } + case <-ctx.Done(): + for c := range clients { + c.conn.Close() // unblock the reader goroutine + close(c.out) + } + return + } + } +} +``` + +```rust + loop { + tokio::select! { + // Every later frame is MSG: its payload is the message text. + r = read_frame(&mut rd) => match r { + Ok((_typ, body)) => { + let frame = encode_frame(TYPE_DELIVER, &format!("{nick}\0{body}")); + let map = clients.lock().unwrap(); + for (cid, tx) in map.iter() { + if *cid != id { + let _ = tx.send(frame.clone()); + } + } + } + Err(_) => break, + }, + _ = shutdown.recv() => break, + } + } +``` + +**Go: nothing to rewrite.** The Go excerpt has no `epoll`, no coroutine, no +`.await` — and that is the lesson. `serve --engine async` runs the *same* +goroutine-per-connection hub as `--engine thread`; the flag exists only for +parity. Each `handleConn` is its own goroutine calling a plain blocking +`readFrame`; when that read would block, the runtime parks the *goroutine* on +its **netpoller** (an `epoll` instance the runtime owns) and runs another on +the same OS thread. A single `hub` goroutine owns the client set and fans out +broadcasts over buffered channels, so no lock guards the map — chapter 6's +"share memory by communicating." Go absorbed the async model in 2012; the +`select` above *is* an event loop, one level up, with channels where the +kernel has fds. + +**Rust: tokio, `.await` as a compiler transform.** An `async fn` compiles to +a state machine implementing `Future`; `.await` desugars to a loop that calls +`poll(cx)` and, on `Poll::Pending`, returns — after squirrelling away the +`Waker` from `cx`. tokio's `serve` builds a multi-threaded runtime and +`tokio::spawn`s one `handle_conn` task per connection; `read_frame(&mut rd)` +inside the `select!` is where the task yields. When the socket has nothing, +tokio's reactor (built on `mio`, itself `epoll`) holds the task's Waker; on +readiness it calls `wake()`, which re-enqueues the task for a worker thread to +`poll` again. The fan-out uses a per-client `mpsc` channel drained by a small +writer task — the same decoupling as Go's `c.out`, spelled in tokio's types. + +A forward glance for C++: `std::execution` (senders/receivers, P2300) was +merged into the C++26 working draft and points at a *standard* vocabulary for +exactly this — a portable way to describe async work and the executor that +runs it, so you would not hand-roll `Task` and `Reactor` at all. It is +genuinely forward-looking; this chapter builds the machinery by hand precisely +to show what a runtime does underneath such an abstraction. + +## Errors, three ways + +The CLI contract is the book's usual: clean shutdown exits 0, runtime failure +exits 1 with one diagnostic, usage errors exit 2 — and `verify.lua` asserts +all of it (42 checks: 14 per language across both engines plus the CLI cases). +The async-specific error is *cancellation*. On `SIGTERM` each engine must +unwind in-flight work without a leak. C++ flips an `std::atomic` in an +async-signal-safe handler and writes to an `eventfd` registered in the +reactor, so the blocked `epoll_wait` returns, the loop guard sees the flag, +and every self-owning `Detached` frame is dropped as its fd closes. Go uses +`signal.NotifyContext`, so `<-ctx.Done()` becomes another `select` arm that +closes each `c.out` and the listener. tokio makes the signal a stream — +`sigint.recv()`/`sigterm.recv()` are arms of the accept `select!`, and a +`broadcast` channel tells every connection task to `break`. Three spellings of +the same discipline: turn the signal into just another event the loop already +knows how to wait on. All three then print `chatterd: shutdown` and exit 0. + +## Concurrency lens + +This is the whole chapter, so quantify it. The observable difference between +the three engines is *how many OS threads they spend to hold the same +connections* — and `/proc//task` counts them directly. Serving 50 held +connections on this 16-core host (`nproc` = 16, `GOMAXPROCS` = 16): + +| engine | idle threads | with 50 clients | model | +|---|---|---|---| +| C++ `async` (coroutines) | 1 | **1** | one reactor thread, 50 frames | +| Rust `async` (tokio) | 17 | **17** | 1 main + 16 `tokio-rt-worker` (= cores) | +| Go `async` (goroutines) | 7 | **11** | M threads grown on demand, ≤ P | +| C++ `thread` | 1 | **51** | one OS thread per client | +| Rust `thread` | 1 | **51** | one OS thread per client | + +The three async engines pin three philosophies. The C++ coroutine reactor is +resolutely **single-threaded** — 50 connections, one thread — so, exactly like +chapter 9's epoll loop, it needs no locks on the reactor's own state and no +`EPOLLONESHOT` re-arm race beyond the one it already runs. tokio commits to +**worker-threads-equal-cores**: a fixed 17 (one `app` main thread plus 16 +`tokio-rt-worker` threads, confirmed by their `comm` names), independent of +connection count — tasks migrate between workers by work-stealing. Go sits +between: goroutines are scheduled onto at most `GOMAXPROCS` logical processors +(P), but the runtime only *materialises* OS threads (M) as work demands, so +the count drifts (7 idle, 11 under load) rather than tracking a constant. The +two thread engines are the control: 1 + 50 = 51, the cost the async engines +exist to avoid. + +Two runtime mechanisms deserve names. **Go's preemption**: before Go 1.14 a +goroutine in a tight loop with no function calls could starve its P forever; +now the runtime's monitor sends **`SIGURG`** (chapter 12's async-preemption +signal) to an M whose goroutine has run past ~10 ms, and the handler forces a +yield — cooperative scheduling with an involuntary backstop. **tokio's +Waker**: a task that returns `Poll::Pending` is *inert* until something calls +`wake()` on the Waker it registered; the reactor is that something, calling it +from the `epoll_wait` that noticed the fd. Miss a `wake()` and the task hangs +forever — the async analogue of a lost wakeup. When a tokio program *does* +stall, `tokio-console` is the tool: it shows per-task poll counts and the tasks +that never yield, the way `top` shows busy threads. + +{% include excalidraw.html + file="27-gmp-scheduler" + alt="Three bands. The top band holds goroutines G — conn 1, conn 2, conn 3, and G times N — plus a dashed G blocked in recv, parked on the netpoller not on an M. The middle band is the scheduler: P0 and P1 each a logical processor bound to an OS thread M with a local run queue, then ellipsis P15 marked equals 16 cores, and an accent netpoller box, one epoll instance, running epoll_wait for the whole runtime. The bottom kernel band shows an async-preemption box where the runtime sends SIGURG to an M so a G running over 10 ms yields, and an epoll interest list where a socket readable puts a G back on a run queue. Amber arrows show the blocked G parking on the netpoller and readiness returning it to runnable; grey arrows show goroutines running on Ps." + caption="Figure 27.2 — Go's G-M-P scheduler and netpoller: goroutines suspend on I/O against one runtime-owned epoll instance, SIGURG preempts CPU hogs, and no async rewrite was ever needed." %} + +{% include excalidraw.html + file="27-tokio-task-lifecycle" + alt="Three bands. The top band is tokio tasks — an async fn per connection compiled to a state machine: tokio::spawn(handle_conn), an accent .await arrow to poll(cx) returning Poll::Pending, a dashed task-parked box with the Waker held by the reactor, and poll(cx) again returning Poll::Ready and sending the frame. The middle band is 16 tokio-rt-worker threads equal to cores, work-stealing schedulers, with a worker thread that drives poll and a run queue where Waker::wake re-enqueues the task. The bottom band is the mio reactor, one epoll instance, whose epoll_wait on a readable fd calls the registered Waker. Amber arrows trace poll, wake, and re-poll around the cycle." + caption="Figure 27.3 — tokio's task lifecycle: 20k tasks share 16 worker threads; .await parks a task and registers a Waker, epoll readiness fires that Waker, and the task is re-polled to completion." %} + +### The threads-vs-async decision + +The table is also the decision framework. Reach for **plain threads** when +connections are few or the work per connection is CPU-bound and long — a +thread *is* a coroutine the kernel schedules, and you get preemption and +parallelism for free with none of the coloured-function friction. Reach for +**async** when you have many mostly-idle connections (a chat fan-out, a proxy, +a poller): 10k threads cost 10 GB of stacks and murder the scheduler, while +10k coroutine frames cost a few hundred bytes each. Between them: if you are in +Go, the choice is made — goroutines are cheap enough to thread-per-connection +*and* the runtime multiplexes them, so you write blocking code and get async +behaviour. In C++ or Rust the choice is real, and the practical tie-breaker is +that async adds a runtime you must understand to debug — which is why the rest +of this book's servers stayed single-threaded-epoll or thread-per-client until +a chapter, this one, whose subject *is* the runtime. + +## Build, run, observe + +```bash +[host]$ cd examples/27-async-runtimes-and-coroutines && ./demo.sh build +``` + +The runner (`python3 scripts/test-all-examples.py --only +27-async-runtimes-and-coroutines`) reports `PASS PASS PASS` for cpp, go, rust. +Driving one engine by hand the way the evidence above was produced: + +```bash +[host]$ ./cpp/build/release/app serve --engine async --port 47100 & +chatterd: listening on 127.0.0.1:47100 engine=async +[host]$ ./cpp/build/release/app loadtest --port 47100 --clients 20 +loadtest: delivered 20/20 +[host]$ kill -TERM %1 +chatterd: shutdown +``` + +`loadtest` opens 20 receivers plus a sender, broadcasts `hello world`, and +checks every receiver got `sender: hello world`; it exits 0 iff all 20 land. +Swap `--engine thread` and the transcript is identical, byte for byte on the +wire — the async rewrite changed the *how*, never the *what*. + +## Cross-check: count the threads, trace the loop + +**Thread counts, from `/proc`.** The table above is a live measurement: start +an engine, hold 50 connections open with a small Python client, and count +`/proc//task`: + +```console +[host]$ ls /proc/$SRV/task | wc -l # cpp async, 50 clients held +1 +[host]$ ls /proc/$SRV/task | wc -l # tokio, 50 clients held +17 +[host]$ for t in /proc/$SRV/task/*; do cat $t/comm; done | sort | uniq -c + 1 app + 16 tokio-rt-worker +``` + +One thread for the coroutine engine regardless of load; seventeen for tokio, +sixteen of them named `tokio-rt-worker` — worker-threads-equal-cores made +visible. + +**One epoll loop, under `strace`.** Tracing the C++ coroutine engine while it +serves the 20-client loadtest, the syscall summary proves the reactor is a +single epoll loop with no threads spawned at all: + +```console +[host]$ strace -f -c ./cpp/build/release/app serve --engine async --port 48403 + calls errors syscall (time/usecs columns trimmed) +--------- --------- ---------------- + 65 22 recvfrom + 46 epoll_ctl + 22 1 accept4 + 20 sendto + 6 1 epoll_wait + 1 epoll_create1 + 1 eventfd2 + 1 futex +``` + +Read it against the code: **one** `epoll_create1` (one interest list), **zero** +`clone`/`clone3` (no thread ever created — a single `futex` call, uncontended, +confirms it), 22 `accept4` for the 21 connections plus the trailing `EAGAIN`, +65 `recvfrom` with 22 `EAGAIN`s (each `EAGAIN` is a `co_await ReadAwaitable` +suspension), 46 `epoll_ctl` (the `EPOLLONESHOT` re-arm on every resumption), +and **20** `sendto` — the broadcast reaching all 20 receivers. Every mechanism +in Figure 27.1 is one line of that table. The tokio and Go engines, by +contrast, show `clone3` calls at startup (their worker/monitor threads) and +their epoll traffic issued from threads your code never wrote — the same +inversion chapter 9 saw with Go's netpoller, now with a tokio runtime beside +it. + +> **On the lab VM** unverified — +> the kernel-side view of *where the threads actually go* needs bcc-tools and +> root, so it is deferred to Part 8 on `systems-target`: `offcputime` folds an +> off-CPU flamegraph showing the coroutine reactor blocked in one `epoll_wait` +> while tokio's workers park in `futex`, and `tcplife`/`tcpconnect` tally the +> connection lifetimes each engine handles. Those are not run on this host. + +## What you learned + +- Async runtimes all suspend a *unit of work* — a coroutine frame, a + goroutine, a task — instead of an OS thread, and multiplex many onto few. + C++20 `co_await`/`co_return` compile to a heap frame plus a `promise_type`; + `ReadAwaitable::await_suspend` parks the `std::coroutine_handle` in an epoll + reactor that resumes it on readiness, with `EPOLLONESHOT` making the re-arm + race-free. +- Go needed no rewrite: goroutines already suspend on I/O against a + runtime-owned netpoller (epoll), scheduled onto `GOMAXPROCS` processors with + `SIGURG` async preemption as the backstop — `select` is the event loop one + level up. +- tokio desugars `.await` to `poll(cx)` returning `Poll::Pending` and + registering a `Waker`; the mio reactor calls that Waker on readiness to + re-enqueue the task onto one of its worker-threads-equal-cores; + `tokio-console` is the debugger for stalled tasks. +- The measured cost tells you which to pick: for 50 held connections, C++ + coroutines spent **1** thread, tokio **17**, Go **7–11**, and the + thread-per-client engines **51** — async wins on many idle connections, + plain threads win on few or CPU-bound ones. + +Next, the Debugging part opens with **gdb** — from a core dump on your desk to +a remote `gdbserver` in the lab VM, learning to read the state these runtimes +build at rest. + +--- + +

verified — all evidence here +was produced on the Fedora 44 reference host (kernel 7.1.3-200.fc44, GCC +16.1.1, Go 1.26.5, rustc 1.97.1, tokio 1.53.0, strace 7.1, 16 cores, +GOMAXPROCS 16) this session: the runner printed +27-async-runtimes-and-coroutines PASS PASS PASS (3 passed, 0 +failed, 0 skipped; verify.lua 14 checks per language over both engines plus the +CLI cases); the hand-driven session produced the exact +listening … engine=async, loadtest: delivered 20/20, +and chatterd: shutdown lines; the thread-count table is live +/proc/<pid>/task counts with 50 held connections (cpp async +1, tokio 17 with 16 tokio-rt-worker threads by comm, +go 7 idle / 11 under load, both thread engines 51); and the C++ async +strace -c summary is real trimmed output (1 epoll_create1, +6 epoll_wait, 46 epoll_ctl, 22 accept4, +65 recvfrom with 22 EAGAIN, 20 sendto, 1 +futex, no clone). The forward-looking +std::execution/P2300 note is marked as such. The "On the lab VM" +bcc-tools callout (offcputime/tcplife/tcpconnect) is unverified as marked and +is exercised in Part 8.

diff --git a/assets/diagrams/21-frame-format.excalidraw b/assets/diagrams/21-frame-format.excalidraw new file mode 100644 index 0000000..8a8e464 --- /dev/null +++ b/assets/diagrams/21-frame-format.excalidraw @@ -0,0 +1,1559 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r355323462", + "type": "rectangle", + "x": 30, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1602720615, + "versionNonce": 242094043, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1043636114", + "type": "text", + "x": 40, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 163657303, + "versionNonce": 1884274317, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0f", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0f", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r569960258", + "type": "rectangle", + "x": 76, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1165656643, + "versionNonce": 1273399709, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1779695074", + "type": "text", + "x": 86, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 723555957, + "versionNonce": 403124437, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "00", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "00", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r837576106", + "type": "rectangle", + "x": 122, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 602767469, + "versionNonce": 1726959786, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t505149137", + "type": "text", + "x": 132, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1989772075, + "versionNonce": 862743893, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "00", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "00", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1143362342", + "type": "rectangle", + "x": 168, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1130575963, + "versionNonce": 419255562, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1502370182", + "type": "text", + "x": 178, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1863059643, + "versionNonce": 1564351672, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "00", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "00", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r46129755", + "type": "rectangle", + "x": 214, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1245233848, + "versionNonce": 1665354436, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1645055265", + "type": "text", + "x": 224, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1640940878, + "versionNonce": 1077742160, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "61", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "61", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1445304028", + "type": "rectangle", + "x": 260, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1630971835, + "versionNonce": 1348142933, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1942224931", + "type": "text", + "x": 270, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 523884829, + "versionNonce": 1912421591, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6c", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6c", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1779354742", + "type": "rectangle", + "x": 306, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1577766736, + "versionNonce": 798467599, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t772344532", + "type": "text", + "x": 316, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 858199101, + "versionNonce": 455865384, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "69", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "69", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r500879135", + "type": "rectangle", + "x": 352, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1119634838, + "versionNonce": 1003036056, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1792675756", + "type": "text", + "x": 362, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 478041693, + "versionNonce": 1007592016, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "63", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "63", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1692073164", + "type": "rectangle", + "x": 398, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 476995592, + "versionNonce": 582865421, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1284392896", + "type": "text", + "x": 408, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 319819465, + "versionNonce": 1230670428, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "65", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "65", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r612112355", + "type": "rectangle", + "x": 444, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1797231865, + "versionNonce": 748950489, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t991623095", + "type": "text", + "x": 454, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1142304031, + "versionNonce": 531746598, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "00", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "00", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r848696820", + "type": "rectangle", + "x": 490, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 934313729, + "versionNonce": 76803152, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t233811636", + "type": "text", + "x": 500, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 174902461, + "versionNonce": 776114968, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "68", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "68", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r941274773", + "type": "rectangle", + "x": 536, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1620033700, + "versionNonce": 1669931839, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t728259141", + "type": "text", + "x": 546, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1292290809, + "versionNonce": 774232038, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "65", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "65", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1494004595", + "type": "rectangle", + "x": 582, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 47125302, + "versionNonce": 1421785583, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t791225550", + "type": "text", + "x": 592, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1959223781, + "versionNonce": 1147116519, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6c", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6c", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1201780637", + "type": "rectangle", + "x": 628, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1067033888, + "versionNonce": 706586474, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t576587038", + "type": "text", + "x": 638, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1497884788, + "versionNonce": 351940467, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6c", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6c", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1734789853", + "type": "rectangle", + "x": 674, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1561980282, + "versionNonce": 1735771978, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t27713636", + "type": "text", + "x": 684, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 236731265, + "versionNonce": 88286843, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6f", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6f", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1325702666", + "type": "rectangle", + "x": 720, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 426904730, + "versionNonce": 313839890, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1895763156", + "type": "text", + "x": 730, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 786712448, + "versionNonce": 1851821603, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "20", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "20", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1952821908", + "type": "rectangle", + "x": 766, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 121618534, + "versionNonce": 730178540, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1959655725", + "type": "text", + "x": 776, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1259470220, + "versionNonce": 601361143, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "62", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "62", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r317312795", + "type": "rectangle", + "x": 812, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1940282094, + "versionNonce": 1160340324, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t776444066", + "type": "text", + "x": 822, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1468379712, + "versionNonce": 582847057, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6f", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6f", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r371117561", + "type": "rectangle", + "x": 858, + "y": 118, + "width": 44, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1269850521, + "versionNonce": 1109805414, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t741837564", + "type": "text", + "x": 868, + "y": 126, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 872939725, + "versionNonce": 1467105347, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "62", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "62", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1807946953", + "type": "text", + "x": 460, + "y": 30, + "width": 550.5500000000001, + "height": 16.25, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1952536611, + "versionNonce": 1270382292, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "One length-prefixed frame: alice sending \"hello bob\" (19 bytes on the wire)", + "fontSize": 13, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "One length-prefixed frame: alice sending \"hello bob\" (19 bytes on the wire)", + "lineHeight": 1.25, + "baseline": 13 + }, + { + "id": "t1661812396", + "type": "text", + "x": 121.0, + "y": 90, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 503045657, + "versionNonce": 1378615778, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "u32 length (LE) = 15", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "u32 length (LE) = 15", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t478549773", + "type": "text", + "x": 328.0, + "y": 90, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1736147349, + "versionNonce": 32531758, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "name = alice", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "name = alice", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t712824433", + "type": "text", + "x": 466.0, + "y": 90, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1849393444, + "versionNonce": 223807561, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "NUL", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "NUL", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t55676143", + "type": "text", + "x": 696.0, + "y": 90, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1703891076, + "versionNonce": 1100684621, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "text = hello bob", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "text = hello bob", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1725172666", + "type": "text", + "x": 460, + "y": 195, + "width": 521.4000000000001, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 832897374, + "versionNonce": 627768349, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0f 00 00 00 = 0x0000000f = 15 = len(\"alice\") + 1 (NUL) + len(\"hello bob\")", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0f 00 00 00 = 0x0000000f = 15 = len(\"alice\") + 1 (NUL) + len(\"hello bob\")", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t1817663814", + "type": "text", + "x": 30, + "y": 240, + "width": 382.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1039820349, + "versionNonce": 828989475, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Two frame kinds, told apart only by whether text is empty:", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Two frame kinds, told apart only by whether text is empty:", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t428549704", + "type": "text", + "x": 40, + "y": 264, + "width": 496.1, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 436261475, + "versionNonce": 1028034046, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "JOIN name NUL (empty text) -- client sends exactly one at connect", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "JOIN name NUL (empty text) -- client sends exactly one at connect", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1202574355", + "type": "text", + "x": 40, + "y": 284, + "width": 459.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 309633789, + "versionNonce": 1023744, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CHAT name NUL text -- one per non-empty stdin line", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CHAT name NUL text -- one per non-empty stdin line", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t55983301", + "type": "text", + "x": 30, + "y": 318, + "width": 520.3000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1951062097, + "versionNonce": 1989729868, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "len == 0 and len > 1048576 (1 MiB) are rejected; the 4 header bytes are never counted.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "len == 0 and len > 1048576 (1 MiB) are rejected; the 4 header bytes are never counted.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/21-frame-format.svg b/assets/diagrams/21-frame-format.svg new file mode 100644 index 0000000..1e72d3a --- /dev/null +++ b/assets/diagrams/21-frame-format.svg @@ -0,0 +1,51 @@ + + + +0f + +00 + +00 + +00 + +61 + +6c + +69 + +63 + +65 + +00 + +68 + +65 + +6c + +6c + +6f + +20 + +62 + +6f + +62 +One length-prefixed frame: alice sending "hello bob" (19 bytes on the wire) +u32 length (LE) = 15 +name = alice +NUL +text = hello bob +0f 00 00 00 = 0x0000000f = 15 = len("alice") + 1 (NUL) + len("hello bob") +Two frame kinds, told apart only by whether text is empty: +JOIN name NUL (empty text) -- client sends exactly one at connect +CHAT name NUL text -- one per non-empty stdin line +len == 0 and len > 1048576 (1 MiB) are rejected; the 4 header bytes are never counted. + \ No newline at end of file diff --git a/assets/diagrams/21-server-lifecycle.excalidraw b/assets/diagrams/21-server-lifecycle.excalidraw new file mode 100644 index 0000000..bc0b363 --- /dev/null +++ b/assets/diagrams/21-server-lifecycle.excalidraw @@ -0,0 +1,1432 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1152892381", + "type": "rectangle", + "x": 20, + "y": 55, + "width": 900, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 561769319, + "versionNonce": 526625400, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1622342454", + "type": "text", + "x": 32, + "y": 61, + "width": 178.20000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1394967220, + "versionNonce": 138737204, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "serve \u2014 one listener thread", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "serve \u2014 one listener thread", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r40997284", + "type": "rectangle", + "x": 20, + "y": 250, + "width": 900, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1875329, + "versionNonce": 821317803, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t258310001", + "type": "text", + "x": 32, + "y": 256, + "width": 435.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1902858021, + "versionNonce": 1206211039, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one std::jthread / goroutine / std::thread per accepted connection", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one std::jthread / goroutine / std::thread per accepted connection", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1375257014", + "type": "rectangle", + "x": 35, + "y": 95, + "width": 158, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 707033999, + "versionNonce": 414963334, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1843333063", + "type": "text", + "x": 45, + "y": 103, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1758596935, + "versionNonce": 176644705, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "socket()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "socket()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1321293881", + "type": "text", + "x": 45, + "y": 123, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 878985760, + "versionNonce": 1918319224, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "AF_INET", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "AF_INET", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t73153247", + "type": "text", + "x": 45, + "y": 138, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 167926207, + "versionNonce": 166740661, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SOCK_STREAM|CLOEXEC", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SOCK_STREAM|CLOEXEC", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r42238394", + "type": "rectangle", + "x": 210, + "y": 95, + "width": 158, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 522954379, + "versionNonce": 1080153431, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t740741809", + "type": "text", + "x": 220, + "y": 103, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1626076342, + "versionNonce": 451424622, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "setsockopt", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "setsockopt", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1356787510", + "type": "text", + "x": 220, + "y": 123, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 582133400, + "versionNonce": 873265121, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SO_REUSEADDR = 1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SO_REUSEADDR = 1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t416669672", + "type": "text", + "x": 220, + "y": 138, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 561230703, + "versionNonce": 1132679848, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "rebind past TIME_WAIT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "rebind past TIME_WAIT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r802873671", + "type": "rectangle", + "x": 385, + "y": 95, + "width": 158, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1134824324, + "versionNonce": 926284288, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1546448200", + "type": "text", + "x": 395, + "y": 103, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1878224708, + "versionNonce": 1087038138, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bind()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bind()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1044171797", + "type": "text", + "x": 395, + "y": 123, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 395928954, + "versionNonce": 1116438756, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sockaddr_in", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sockaddr_in", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1360582341", + "type": "text", + "x": 395, + "y": 138, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1740672493, + "versionNonce": 359454408, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "127.0.0.1:9000", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "127.0.0.1:9000", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r496992635", + "type": "rectangle", + "x": 560, + "y": 95, + "width": 158, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 457452243, + "versionNonce": 1458458236, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1320352866", + "type": "text", + "x": 570, + "y": 103, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1346532489, + "versionNonce": 1396356495, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "listen(128)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "listen(128)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1338148112", + "type": "text", + "x": 570, + "y": 123, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 784639339, + "versionNonce": 1559001533, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "passive socket", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "passive socket", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1738044244", + "type": "text", + "x": 570, + "y": 138, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 530560248, + "versionNonce": 497313501, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fd 3", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fd 3", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r714698727", + "type": "rectangle", + "x": 735, + "y": 95, + "width": 165, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 458405048, + "versionNonce": 1783989981, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t584672834", + "type": "text", + "x": 745, + "y": 103, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 220449817, + "versionNonce": 1433683366, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "accept4() loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "accept4() loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t789265069", + "type": "text", + "x": 745, + "y": 123, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1580424683, + "versionNonce": 1101474364, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SOCK_CLOEXEC", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SOCK_CLOEXEC", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1673764552", + "type": "text", + "x": 745, + "y": 138, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1216765389, + "versionNonce": 636362051, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "-> new fd 4, 5, ...", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "-> new fd 4, 5, ...", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r565959166", + "type": "rectangle", + "x": 300, + "y": 288, + "width": 190, + "height": 74, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1026212348, + "versionNonce": 721528277, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t513360", + "type": "text", + "x": 310, + "y": 296, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 809597906, + "versionNonce": 1753106208, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "serve_client fd 4", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "serve_client fd 4", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t978312840", + "type": "text", + "x": 310, + "y": 316, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 687503755, + "versionNonce": 646411432, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "read_frame -> broadcast", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "read_frame -> broadcast", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t251733069", + "type": "text", + "x": 310, + "y": 331, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 247994664, + "versionNonce": 737519155, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bob", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bob", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r118395181", + "type": "rectangle", + "x": 540, + "y": 288, + "width": 190, + "height": 74, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 155362271, + "versionNonce": 1529878387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t313424056", + "type": "text", + "x": 550, + "y": 296, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 433812282, + "versionNonce": 1521518676, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "serve_client fd 5", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "serve_client fd 5", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1849346317", + "type": "text", + "x": 550, + "y": 316, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1897274707, + "versionNonce": 1256135218, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "read_frame -> broadcast", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "read_frame -> broadcast", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t687904505", + "type": "text", + "x": 550, + "y": 331, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 752319277, + "versionNonce": 1406064568, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "alice", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "alice", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a392052071", + "type": "arrow", + "x": 193, + "y": 128, + "width": 17, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 135560859, + "versionNonce": 1830519033, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 17, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1591623972", + "type": "arrow", + "x": 368, + "y": 128, + "width": 17, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 892903431, + "versionNonce": 561955350, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 17, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a335984423", + "type": "arrow", + "x": 543, + "y": 128, + "width": 17, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 731021195, + "versionNonce": 875309504, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 17, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1137753919", + "type": "arrow", + "x": 718, + "y": 128, + "width": 17, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 147384842, + "versionNonce": 1297086757, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 17, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1194426924", + "type": "arrow", + "x": 790, + "y": 161, + "width": 360, + "height": 127, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1882058582, + "versionNonce": 813410418, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -360, + 127 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1140968120", + "type": "text", + "x": 610.0, + "y": 210.5, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 957511843, + "versionNonce": 1609521348, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spawn thread", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spawn thread", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a706050784", + "type": "arrow", + "x": 830, + "y": 161, + "width": 190, + "height": 127, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1711051505, + "versionNonce": 736988179, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -190, + 127 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1856985279", + "type": "text", + "x": 735.0, + "y": 210.5, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 219553790, + "versionNonce": 767877386, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spawn thread", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spawn thread", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1301751269", + "type": "text", + "x": 470, + "y": 222, + "width": 574.75, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1962168425, + "versionNonce": 713742179, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SIGINT -> g_stop = 1; poll(2) wakes -> stop accepting; listener fd closes; threads join; exit 0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SIGINT -> g_stop = 1; poll(2) wakes -> stop accepting; listener fd closes; threads join; exit 0", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/21-server-lifecycle.svg b/assets/diagrams/21-server-lifecycle.svg new file mode 100644 index 0000000..1994e07 --- /dev/null +++ b/assets/diagrams/21-server-lifecycle.svg @@ -0,0 +1,46 @@ + + + +serve — one listener thread + +one std::jthread / goroutine / std::thread per accepted connection + +socket() +AF_INET +SOCK_STREAM|CLOEXEC + +setsockopt +SO_REUSEADDR = 1 +rebind past TIME_WAIT + +bind() +sockaddr_in +127.0.0.1:9000 + +listen(128) +passive socket +fd 3 + +accept4() loop +SOCK_CLOEXEC +-> new fd 4, 5, ... + +serve_client fd 4 +read_frame -> broadcast +bob + +serve_client fd 5 +read_frame -> broadcast +alice + + + + + +spawn thread +spawn thread + +spawn thread +spawn thread +SIGINT -> g_stop = 1; poll(2) wakes -> stop accepting; listener fd closes; threads join; exit 0 + \ No newline at end of file diff --git a/assets/diagrams/22-connection-state-machine.excalidraw b/assets/diagrams/22-connection-state-machine.excalidraw new file mode 100644 index 0000000..08794d5 --- /dev/null +++ b/assets/diagrams/22-connection-state-machine.excalidraw @@ -0,0 +1,1651 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r502771003", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 900, + "height": 180, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1301704919, + "versionNonce": 1073244193, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t411480817", + "type": "text", + "x": 32, + "y": 26, + "width": 356.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1240642641, + "versionNonce": 674334343, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "READ PATH \u2014 recv to EAGAIN, reassemble frames, fan out", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "READ PATH \u2014 recv to EAGAIN, reassemble frames, fan out", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1946108064", + "type": "rectangle", + "x": 20, + "y": 215, + "width": 900, + "height": 235, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 689809039, + "versionNonce": 1635579143, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t170701042", + "type": "text", + "x": 32, + "y": 221, + "width": 376.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 84931381, + "versionNonce": 1745464436, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "WRITE PATH + BACKPRESSURE \u2014 outbuf queue, EPOLLOUT re-arm", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "WRITE PATH + BACKPRESSURE \u2014 outbuf queue, EPOLLOUT re-arm", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r693109731", + "type": "rectangle", + "x": 45, + "y": 70, + "width": 165, + "height": 66, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 616087179, + "versionNonce": 23065433, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t569465270", + "type": "text", + "x": 55, + "y": 78, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1261484789, + "versionNonce": 579203339, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "recv() loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "recv() loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1151281789", + "type": "text", + "x": 55, + "y": 98, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 344483913, + "versionNonce": 689069803, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "drain nonblocking", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "drain nonblocking", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1217644751", + "type": "text", + "x": 55, + "y": 113, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1363144429, + "versionNonce": 1293318258, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "until EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "until EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1664830937", + "type": "rectangle", + "x": 255, + "y": 70, + "width": 165, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1716237505, + "versionNonce": 1211109553, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1989101733", + "type": "text", + "x": 265, + "y": 78, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1456998076, + "versionNonce": 392861098, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inbuf", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inbuf", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t638496116", + "type": "text", + "x": 265, + "y": 98, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 549100285, + "versionNonce": 1180586254, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reassemble partial", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reassemble partial", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t818482012", + "type": "text", + "x": 265, + "y": 113, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1653288021, + "versionNonce": 1784670970, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "frames across reads", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "frames across reads", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1107565664", + "type": "rectangle", + "x": 465, + "y": 70, + "width": 165, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 877130475, + "versionNonce": 1429342143, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t843029223", + "type": "text", + "x": 475, + "y": 78, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1633893884, + "versionNonce": 303159542, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "take_frame()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "take_frame()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t40668276", + "type": "text", + "x": 475, + "y": 98, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1061313709, + "versionNonce": 1983722654, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one whole 6+LEN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one whole 6+LEN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1120345034", + "type": "text", + "x": 475, + "y": 113, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1300032016, + "versionNonce": 1675847030, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "frame? or wait", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "frame? or wait", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1528108926", + "type": "rectangle", + "x": 690, + "y": 70, + "width": 205, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1766813957, + "versionNonce": 102410430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t418509990", + "type": "text", + "x": 700, + "y": 78, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 628263894, + "versionNonce": 1620044670, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "MSG -> broadcast()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "MSG -> broadcast()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1238245217", + "type": "text", + "x": 700, + "y": 98, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 369363052, + "versionNonce": 479419385, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "append frame to EVERY", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "append frame to EVERY", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t883580252", + "type": "text", + "x": 700, + "y": 113, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 533458096, + "versionNonce": 828126859, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "conn's outbuf", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "conn's outbuf", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r774824219", + "type": "rectangle", + "x": 690, + "y": 255, + "width": 205, + "height": 62, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1730645040, + "versionNonce": 1094891869, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1072571025", + "type": "text", + "x": 700, + "y": 263, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1372247031, + "versionNonce": 1446269650, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "queue() + flush()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "queue() + flush()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1098307215", + "type": "text", + "x": 700, + "y": 283, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1217465612, + "versionNonce": 668833180, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "send() what fits now", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "send() what fits now", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1840584891", + "type": "rectangle", + "x": 465, + "y": 250, + "width": 165, + "height": 72, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 981994539, + "versionNonce": 1092330822, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t298559307", + "type": "text", + "x": 475, + "y": 258, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1406899387, + "versionNonce": 1195217358, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "outbuf", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "outbuf", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t2077783", + "type": "text", + "x": 475, + "y": 278, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1635944810, + "versionNonce": 975327054, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the backpressure queue", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the backpressure queue", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t399767761", + "type": "text", + "x": 475, + "y": 293, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 437946699, + "versionNonce": 1938034987, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "remainder after EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "remainder after EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1063018489", + "type": "rectangle", + "x": 255, + "y": 255, + "width": 165, + "height": 62, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1345608272, + "versionNonce": 1924107982, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1386315135", + "type": "text", + "x": 265, + "y": 263, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1218113492, + "versionNonce": 1763326810, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel socket buffer", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel socket buffer", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1035087597", + "type": "text", + "x": 265, + "y": 283, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1823508584, + "versionNonce": 126920421, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "full -> send() EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "full -> send() EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1427285068", + "type": "rectangle", + "x": 45, + "y": 362, + "width": 185, + "height": 68, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1853248177, + "versionNonce": 518197366, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1278967485", + "type": "text", + "x": 55, + "y": 370, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 188527053, + "versionNonce": 1372726900, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_wait", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_wait", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t33852491", + "type": "text", + "x": 55, + "y": 390, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1901965387, + "versionNonce": 1363791292, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "later reports EPOLLOUT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "later reports EPOLLOUT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t477852060", + "type": "text", + "x": 55, + "y": 405, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 840469328, + "versionNonce": 1015111081, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "for this conn", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "for this conn", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1855612221", + "type": "arrow", + "x": 210, + "y": 103, + "width": 43, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 666968151, + "versionNonce": 780003234, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 43, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1602877133", + "type": "arrow", + "x": 420, + "y": 103, + "width": 43, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 161755099, + "versionNonce": 1003674819, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 43, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a603434943", + "type": "arrow", + "x": 630, + "y": 103, + "width": 58, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 153053064, + "versionNonce": 566336336, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 58, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a315097501", + "type": "arrow", + "x": 792, + "y": 136, + "width": 0, + "height": 117, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1829355649, + "versionNonce": 1922372049, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 117 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t357735818", + "type": "text", + "x": 792.0, + "y": 180.5, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 279372844, + "versionNonce": 220174417, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "for each conn", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "for each conn", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1835039372", + "type": "arrow", + "x": 690, + "y": 286, + "width": 58, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1408511656, + "versionNonce": 593830212, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -58, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1772041439", + "type": "arrow", + "x": 465, + "y": 286, + "width": 43, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 74479509, + "versionNonce": 1649725159, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -43, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t182235082", + "type": "text", + "x": 443.5, + "y": 272.0, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 391050649, + "versionNonce": 70756567, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "send() EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "send() EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a551838562", + "type": "arrow", + "x": 470, + "y": 322, + "width": 295, + "height": 36, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 466559719, + "versionNonce": 740616018, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -295, + 36 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1408463903", + "type": "text", + "x": 322.5, + "y": 326.0, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 601320537, + "versionNonce": 1055849824, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "full: arm EPOLLOUT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "full: arm EPOLLOUT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a818428432", + "type": "arrow", + "x": 232, + "y": 382, + "width": 456, + "height": 60, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1162954838, + "versionNonce": 1733621434, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 456, + -60 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1993295161", + "type": "text", + "x": 460.0, + "y": 338.0, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1137780846, + "versionNonce": 158932074, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "EPOLLOUT wakeup", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "EPOLLOUT wakeup", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t844737004", + "type": "text", + "x": 235, + "y": 405, + "width": 462.00000000000006, + "height": 12.5, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 663705995, + "versionNonce": 1851959826, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "level-triggered: EPOLLIN always armed; EPOLLOUT armed only while outbuf is non-empty", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "level-triggered: EPOLLIN always armed; EPOLLOUT armed only while outbuf is non-empty", + "lineHeight": 1.25, + "baseline": 10 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/22-connection-state-machine.svg b/assets/diagrams/22-connection-state-machine.svg new file mode 100644 index 0000000..325f691 --- /dev/null +++ b/assets/diagrams/22-connection-state-machine.svg @@ -0,0 +1,54 @@ + + + +READ PATH — recv to EAGAIN, reassemble frames, fan out + +WRITE PATH + BACKPRESSURE — outbuf queue, EPOLLOUT re-arm + +recv() loop +drain nonblocking +until EAGAIN + +inbuf +reassemble partial +frames across reads + +take_frame() +one whole 6+LEN +frame? or wait + +MSG -> broadcast() +append frame to EVERY +conn's outbuf + +queue() + flush() +send() what fits now + +outbuf +the backpressure queue +remainder after EAGAIN + +kernel socket buffer +full -> send() EAGAIN + +epoll_wait +later reports EPOLLOUT +for this conn + + + + +for each conn +for each conn + + +send() EAGAIN +send() EAGAIN + +full: arm EPOLLOUT +full: arm EPOLLOUT + +EPOLLOUT wakeup +EPOLLOUT wakeup +level-triggered: EPOLLIN always armed; EPOLLOUT armed only while outbuf is non-empty + \ No newline at end of file diff --git a/assets/diagrams/22-netpoller-vs-epoll.excalidraw b/assets/diagrams/22-netpoller-vs-epoll.excalidraw new file mode 100644 index 0000000..d198178 --- /dev/null +++ b/assets/diagrams/22-netpoller-vs-epoll.excalidraw @@ -0,0 +1,967 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1390279253", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 440, + "height": 430, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1365940760, + "versionNonce": 1888668320, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t914074625", + "type": "text", + "x": 32, + "y": 26, + "width": 244.20000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 116162596, + "versionNonce": 1054139080, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "C++ / Rust \u2014 you write the epoll loop", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "C++ / Rust \u2014 you write the epoll loop", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r979643017", + "type": "rectangle", + "x": 480, + "y": 20, + "width": 440, + "height": 430, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 610030074, + "versionNonce": 1007629437, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1677428427", + "type": "text", + "x": 492, + "y": 26, + "width": 356.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1955798609, + "versionNonce": 1440281122, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go \u2014 you write goroutines; the runtime hides the epoll", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go \u2014 you write goroutines; the runtime hides the epoll", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1393515539", + "type": "rectangle", + "x": 60, + "y": 70, + "width": 360, + "height": 64, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1419298340, + "versionNonce": 753293654, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t180651057", + "type": "text", + "x": 70, + "y": 78, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 66942632, + "versionNonce": 619321702, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "1 OS thread (Threads: 1)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "1 OS thread (Threads: 1)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1033830475", + "type": "text", + "x": 70, + "y": 98, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 613422962, + "versionNonce": 508498588, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_wait \u2014 4 calls for a 50-conn flood", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_wait \u2014 4 calls for a 50-conn flood", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1641243106", + "type": "rectangle", + "x": 60, + "y": 250, + "width": 360, + "height": 118, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1649300826, + "versionNonce": 802322953, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1370386193", + "type": "text", + "x": 70, + "y": 258, + "width": 231.00000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 723196902, + "versionNonce": 1162249955, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel eventpoll interest list", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel eventpoll interest list", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1256193513", + "type": "text", + "x": 70, + "y": 278, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 748485019, + "versionNonce": 1126243156, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tok 0 listener tok 1 signalfd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tok 0 listener tok 1 signalfd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t713703305", + "type": "text", + "x": 70, + "y": 293, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1003064836, + "versionNonce": 619442360, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tok 2..N every client socket, one process", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tok 2..N every client socket, one process", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r832814720", + "type": "rectangle", + "x": 520, + "y": 70, + "width": 360, + "height": 70, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 772238666, + "versionNonce": 580480374, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t794852119", + "type": "text", + "x": 530, + "y": 78, + "width": 300.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1131498448, + "versionNonce": 411467586, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "goroutine per conn (read + write) + hub", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "goroutine per conn (read + write) + hub", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t709031656", + "type": "text", + "x": 530, + "y": 98, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1494348958, + "versionNonce": 1249736235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "channels fan out \u2014 ~7 OS threads total", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "channels fan out \u2014 ~7 OS threads total", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r287451905", + "type": "rectangle", + "x": 520, + "y": 185, + "width": 360, + "height": 64, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 748888707, + "versionNonce": 412670383, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1058867696", + "type": "text", + "x": 530, + "y": 193, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1935263643, + "versionNonce": 373230909, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "runtime netpoller (hidden)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "runtime netpoller (hidden)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1117855148", + "type": "text", + "x": 530, + "y": 213, + "width": 284.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 18134566, + "versionNonce": 1378367471, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_pwait \u2014 165 calls, edge-triggered EPOLLET", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_pwait \u2014 165 calls, edge-triggered EPOLLET", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r173864000", + "type": "rectangle", + "x": 520, + "y": 300, + "width": 360, + "height": 64, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1400595557, + "versionNonce": 884220039, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1208447907", + "type": "text", + "x": 530, + "y": 308, + "width": 146.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 452041279, + "versionNonce": 459025685, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "same kernel sockets", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "same kernel sockets", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1047419941", + "type": "text", + "x": 530, + "y": 328, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 951699317, + "versionNonce": 1025150638, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one process, N conns (ss -tp)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one process, N conns (ss -tp)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1677074324", + "type": "arrow", + "x": 240, + "y": 250, + "width": 0, + "height": 114, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 9339970, + "versionNonce": 1575911041, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -114 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1944888087", + "type": "text", + "x": 240.0, + "y": 179.0, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 593623962, + "versionNonce": 1850067142, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "EPOLLIN/EPOLLOUT + token", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "EPOLLIN/EPOLLOUT + token", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1892089442", + "type": "arrow", + "x": 700, + "y": 185, + "width": 0, + "height": 43, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1497150968, + "versionNonce": 956492662, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -43 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t271891777", + "type": "text", + "x": 700.0, + "y": 149.5, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 980043827, + "versionNonce": 1197340968, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "park / resume goroutine", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "park / resume goroutine", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a441894709", + "type": "arrow", + "x": 700, + "y": 300, + "width": 0, + "height": 49, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 969291590, + "versionNonce": 1410798106, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -49 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t950958082", + "type": "text", + "x": 60, + "y": 148, + "width": 357.50000000000006, + "height": 12.5, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1968208069, + "versionNonce": 1955929748, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one thread multiplexes N sockets; state machine is yours to write", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one thread multiplexes N sockets; state machine is yours to write", + "lineHeight": 1.25, + "baseline": 10 + }, + { + "id": "t1290743861", + "type": "text", + "x": 60, + "y": 190, + "width": 236.50000000000003, + "height": 12.5, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1647881203, + "versionNonce": 833216433, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "backpressure = outbuf + EPOLLOUT (explicit)", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "backpressure = outbuf + EPOLLOUT (explicit)", + "lineHeight": 1.25, + "baseline": 10 + }, + { + "id": "t359870873", + "type": "text", + "x": 520, + "y": 395, + "width": 484.00000000000006, + "height": 12.5, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 987140449, + "versionNonce": 1474857800, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads", + "lineHeight": 1.25, + "baseline": 10 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/22-netpoller-vs-epoll.svg b/assets/diagrams/22-netpoller-vs-epoll.svg new file mode 100644 index 0000000..ba99d23 --- /dev/null +++ b/assets/diagrams/22-netpoller-vs-epoll.svg @@ -0,0 +1,33 @@ + + + +C++ / Rust — you write the epoll loop + +Go — you write goroutines; the runtime hides the epoll + +1 OS thread (Threads: 1) +epoll_wait — 4 calls for a 50-conn flood + +kernel eventpoll interest list +tok 0 listener tok 1 signalfd +tok 2..N every client socket, one process + +goroutine per conn (read + write) + hub +channels fan out — ~7 OS threads total + +runtime netpoller (hidden) +epoll_pwait — 165 calls, edge-triggered EPOLLET + +same kernel sockets +one process, N conns (ss -tp) + +EPOLLIN/EPOLLOUT + token +EPOLLIN/EPOLLOUT + token + +park / resume goroutine +park / resume goroutine + +one thread multiplexes N sockets; state machine is yours to write +backpressure = outbuf + EPOLLOUT (explicit) +backpressure = blocking channel send (implicit); threaded engine = same code, 51 threads + \ No newline at end of file diff --git a/assets/diagrams/23-discovery-sequence.excalidraw b/assets/diagrams/23-discovery-sequence.excalidraw new file mode 100644 index 0000000..527192f --- /dev/null +++ b/assets/diagrams/23-discovery-sequence.excalidraw @@ -0,0 +1,1256 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r563870478", + "type": "rectangle", + "x": 30, + "y": 55, + "width": 840, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1475158768, + "versionNonce": 574293742, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t95948033", + "type": "text", + "x": 42, + "y": 61, + "width": 488.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 347906429, + "versionNonce": 571111013, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "UDP multicast discovery \u2014 connectionless, one datagram, boundary preserved", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "UDP multicast discovery \u2014 connectionless, one datagram, boundary preserved", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1589429529", + "type": "rectangle", + "x": 30, + "y": 250, + "width": 840, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 17619008, + "versionNonce": 1710073951, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t40070191", + "type": "text", + "x": 42, + "y": 256, + "width": 534.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 226175673, + "versionNonce": 242549811, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "TCP bridge \u2014 connection-oriented, reliable, the canonical chatterd chat frame (JOIN/DELIVER)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "TCP bridge \u2014 connection-oriented, reliable, the canonical chatterd chat frame (JOIN/DELIVER)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1665246319", + "type": "rectangle", + "x": 55, + "y": 95, + "width": 190, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 873446804, + "versionNonce": 1677619890, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1602686408", + "type": "text", + "x": 65, + "y": 103, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1741322084, + "versionNonce": 1763731739, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "announce loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "announce loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1441847852", + "type": "text", + "x": 65, + "y": 123, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 660271143, + "versionNonce": 1379439653, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sendto \u2192 group", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sendto \u2192 group", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t103152240", + "type": "text", + "x": 65, + "y": 138, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 277776241, + "versionNonce": 244828993, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"CHATTERD1 target 9241 \u2026\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"CHATTERD1 target 9241 \u2026\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1806365739", + "type": "rectangle", + "x": 290, + "y": 95, + "width": 190, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 867994041, + "versionNonce": 1217652507, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t606949641", + "type": "text", + "x": 300, + "y": 103, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1625016573, + "versionNonce": 1508421392, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "receive loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "receive loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1674945331", + "type": "text", + "x": 300, + "y": 123, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 883962731, + "versionNonce": 1275580177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "recv() one datagram", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "recv() one datagram", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t106990463", + "type": "text", + "x": 300, + "y": 138, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 789966345, + "versionNonce": 1413553975, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "split 4 fields \u00b7 check magic", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "split 4 fields \u00b7 check magic", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1864319386", + "type": "rectangle", + "x": 525, + "y": 95, + "width": 200, + "height": 92, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1725980688, + "versionNonce": 376644726, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t222088417", + "type": "text", + "x": 535, + "y": 103, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 234026597, + "versionNonce": 849735301, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "parse + dedup", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "parse + dedup", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1198107617", + "type": "text", + "x": 535, + "y": 123, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1575975999, + "versionNonce": 134734310, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "name == self? skip", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "name == self? skip", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t801600686", + "type": "text", + "x": 535, + "y": 138, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 707021236, + "versionNonce": 596521733, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "seen.first(name)? \u2192 dial", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "seen.first(name)? \u2192 dial", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1547076147", + "type": "rectangle", + "x": 290, + "y": 288, + "width": 190, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 912211967, + "versionNonce": 638722200, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t690068618", + "type": "text", + "x": 300, + "y": 296, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 129520658, + "versionNonce": 347090707, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "connect()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "connect()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t97096724", + "type": "text", + "x": 300, + "y": 316, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1442569990, + "versionNonce": 1184958208, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "dial host-ip : tcp-port", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "dial host-ip : tcp-port", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1770195352", + "type": "text", + "x": 300, + "y": 331, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1845462855, + "versionNonce": 108514276, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "retry \u2264 40 \u00d7 50 ms", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "retry \u2264 40 \u00d7 50 ms", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1195623705", + "type": "rectangle", + "x": 525, + "y": 288, + "width": 190, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1115380195, + "versionNonce": 79561097, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1673641979", + "type": "text", + "x": 535, + "y": 296, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 23819927, + "versionNonce": 266174665, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "send_frame / read_frame", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "send_frame / read_frame", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t344693132", + "type": "text", + "x": 535, + "y": 316, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 606611699, + "versionNonce": 1374872312, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6B hdr + BE len, payload", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6B hdr + BE len, payload", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1522315852", + "type": "text", + "x": 535, + "y": 331, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 497064293, + "versionNonce": 564887194, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one frame each way", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one frame each way", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r171094651", + "type": "rectangle", + "x": 760, + "y": 288, + "width": 110, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 334188359, + "versionNonce": 166764079, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1831550014", + "type": "text", + "x": 770, + "y": 296, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 280165886, + "versionNonce": 1330479818, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stdout", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stdout", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t740555758", + "type": "text", + "x": 770, + "y": 316, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 854543063, + "versionNonce": 1591931599, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "discovered peer", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "discovered peer", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1638608757", + "type": "text", + "x": 770, + "y": 331, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1600475699, + "versionNonce": 1441590472, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "peer \u2026 says:", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "peer \u2026 says:", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a189750336", + "type": "arrow", + "x": 245, + "y": 141, + "width": 45, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 21354379, + "versionNonce": 1392918972, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 45, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1145107463", + "type": "text", + "x": 267.5, + "y": 127.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 418697925, + "versionNonce": 1563507793, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "multicast", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "multicast", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a938831104", + "type": "arrow", + "x": 480, + "y": 141, + "width": 45, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1555519024, + "versionNonce": 511273200, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 45, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a800466595", + "type": "arrow", + "x": 625, + "y": 187, + "width": 205, + "height": 101, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1407640154, + "versionNonce": 88343924, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -205, + 101 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t52947051", + "type": "text", + "x": 522.5, + "y": 223.5, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1028513214, + "versionNonce": 84262621, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "new peer \u2192 bridge to TCP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "new peer \u2192 bridge to TCP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a946256000", + "type": "arrow", + "x": 480, + "y": 334, + "width": 45, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 551251263, + "versionNonce": 140880993, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 45, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a68446944", + "type": "arrow", + "x": 715, + "y": 334, + "width": 45, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 214936869, + "versionNonce": 403439666, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 45, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t67653855", + "type": "text", + "x": 55, + "y": 214, + "width": 496.1, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 687743978, + "versionNonce": 1434484255, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "loss / reorder tolerated: a dropped beacon just means \"discovered one round later\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "loss / reorder tolerated: a dropped beacon just means \"discovered one round later\"", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/23-discovery-sequence.svg b/assets/diagrams/23-discovery-sequence.svg new file mode 100644 index 0000000..662ed1e --- /dev/null +++ b/assets/diagrams/23-discovery-sequence.svg @@ -0,0 +1,41 @@ + + + +UDP multicast discovery — connectionless, one datagram, boundary preserved + +TCP bridge — connection-oriented, reliable, the canonical chatterd chat frame (JOIN/DELIVER) + +announce loop +sendto → group +"CHATTERD1 target 9241 …" + +receive loop +recv() one datagram +split 4 fields · check magic + +parse + dedup +name == self? skip +seen.first(name)? → dial + +connect() +dial host-ip : tcp-port +retry ≤ 40 × 50 ms + +send_frame / read_frame +6B hdr + BE len, payload +one frame each way + +stdout +discovered peer +peer … says: + +multicast +multicast + + +new peer → bridge to TCP +new peer → bridge to TCP + + +loss / reorder tolerated: a dropped beacon just means "discovered one round later" + \ No newline at end of file diff --git a/assets/diagrams/23-two-vm-topology.excalidraw b/assets/diagrams/23-two-vm-topology.excalidraw new file mode 100644 index 0000000..62768fe --- /dev/null +++ b/assets/diagrams/23-two-vm-topology.excalidraw @@ -0,0 +1,778 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1303342699", + "type": "rectangle", + "x": 30, + "y": 60, + "width": 840, + "height": 310, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1096375269, + "versionNonce": 1508981242, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1494296421", + "type": "text", + "x": 42, + "y": 66, + "width": 673.2, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 743420340, + "versionNonce": 765859794, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "libvirt default network \u2014 virbr0 bridge \u00b7 192.168.124.0/24 \u00b7 one L2 segment (NAT only toward the host)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "libvirt default network \u2014 virbr0 bridge \u00b7 192.168.124.0/24 \u00b7 one L2 segment (NAT only toward the host)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1768248665", + "type": "rectangle", + "x": 350, + "y": 92, + "width": 200, + "height": 62, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 607713276, + "versionNonce": 808577167, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1757501526", + "type": "text", + "x": 360, + "y": 100, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1233790994, + "versionNonce": 931130880, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "multicast group", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "multicast group", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t959635849", + "type": "text", + "x": 360, + "y": 120, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 805584658, + "versionNonce": 716836802, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "239.23.7.1 : 51824 \u00b7 TTL 1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "239.23.7.1 : 51824 \u00b7 TTL 1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r180383849", + "type": "rectangle", + "x": 60, + "y": 245, + "width": 250, + "height": 108, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1001573733, + "versionNonce": 976758030, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1615896329", + "type": "text", + "x": 70, + "y": 253, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1246961162, + "versionNonce": 1884422987, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-target", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-target", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1752473053", + "type": "text", + "x": 70, + "y": 273, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1423521664, + "versionNonce": 1613277123, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "192.168.124.7", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "192.168.124.7", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t314273250", + "type": "text", + "x": 70, + "y": 288, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1564826307, + "versionNonce": 3475575, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "joined 239.23.7.1 on enp1s0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "joined 239.23.7.1 on enp1s0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t864252471", + "type": "text", + "x": 70, + "y": 303, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 721760974, + "versionNonce": 440457188, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "udp :51824 fd=4 \u00b7 tcp :9241", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "udp :51824 fd=4 \u00b7 tcp :9241", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r445728529", + "type": "rectangle", + "x": 590, + "y": 245, + "width": 250, + "height": 108, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 615744952, + "versionNonce": 637985634, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1039384173", + "type": "text", + "x": 600, + "y": 253, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1707201767, + "versionNonce": 56307813, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-peer", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-peer", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t550942610", + "type": "text", + "x": 600, + "y": 273, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 165485545, + "versionNonce": 1974836491, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "192.168.124.95", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "192.168.124.95", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t539088032", + "type": "text", + "x": 600, + "y": 288, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1192211015, + "versionNonce": 391760712, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "joined 239.23.7.1 on enp1s0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "joined 239.23.7.1 on enp1s0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1548220979", + "type": "text", + "x": 600, + "y": 303, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 994830044, + "versionNonce": 79013785, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "udp :51824 fd=4 \u00b7 tcp :9241", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "udp :51824 fd=4 \u00b7 tcp :9241", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a283105628", + "type": "arrow", + "x": 185, + "y": 245, + "width": 195, + "height": 91, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 177028493, + "versionNonce": 178627803, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 195, + -91 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1782658819", + "type": "text", + "x": 282.5, + "y": 185.5, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1140675234, + "versionNonce": 1796799121, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "beacon 35 B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "beacon 35 B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a131167378", + "type": "arrow", + "x": 715, + "y": 245, + "width": 195, + "height": 91, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1771646290, + "versionNonce": 279665361, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -195, + -91 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1636509537", + "type": "text", + "x": 617.5, + "y": 185.5, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 730995418, + "versionNonce": 1146277609, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "beacon 34 B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "beacon 34 B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a41558758", + "type": "arrow", + "x": 310, + "y": 320, + "width": 280, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1120269075, + "versionNonce": 1005827503, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 280, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1364646455", + "type": "text", + "x": 450.0, + "y": 306.0, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 329151083, + "versionNonce": 1122772177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "TCP :9241 \u2014 one ch21 chat frame each way", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "TCP :9241 \u2014 one ch21 chat frame each way", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1222357774", + "type": "text", + "x": 450, + "y": 378, + "width": 562.6500000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 353255382, + "versionNonce": 523545387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "each guest sends one beacon to the group and hears the other's \u2014 dedup by name, then dial TCP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "each guest sends one beacon to the group and hears the other's \u2014 dedup by name, then dial TCP", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/23-two-vm-topology.svg b/assets/diagrams/23-two-vm-topology.svg new file mode 100644 index 0000000..6b4dc97 --- /dev/null +++ b/assets/diagrams/23-two-vm-topology.svg @@ -0,0 +1,28 @@ + + + +libvirt default network — virbr0 bridge · 192.168.124.0/24 · one L2 segment (NAT only toward the host) + +multicast group +239.23.7.1 : 51824 · TTL 1 + +systems-target +192.168.124.7 +joined 239.23.7.1 on enp1s0 +udp :51824 fd=4 · tcp :9241 + +systems-peer +192.168.124.95 +joined 239.23.7.1 on enp1s0 +udp :51824 fd=4 · tcp :9241 + +beacon 35 B +beacon 35 B + +beacon 34 B +beacon 34 B + +TCP :9241 — one ch21 chat frame each way +TCP :9241 — one ch21 chat frame each way +each guest sends one beacon to the group and hears the other's — dedup by name, then dial TCP + \ No newline at end of file diff --git a/assets/diagrams/24-clock-taxonomy.excalidraw b/assets/diagrams/24-clock-taxonomy.excalidraw new file mode 100644 index 0000000..d4039f2 --- /dev/null +++ b/assets/diagrams/24-clock-taxonomy.excalidraw @@ -0,0 +1,1259 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1405318249", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 840, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 72601687, + "versionNonce": 396017252, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t486406008", + "type": "text", + "x": 32, + "y": 50, + "width": 303.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 657300889, + "versionNonce": 1540734984, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "The clocks clock_gettime(clk_id, ...) can read", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "The clocks clock_gettime(clk_id, ...) can read", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r223923208", + "type": "rectangle", + "x": 20, + "y": 214, + "width": 840, + "height": 236, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 16065504, + "versionNonce": 1057748747, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1189506071", + "type": "text", + "x": 32, + "y": 220, + "width": 343.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1212430426, + "versionNonce": 125396275, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Why a heartbeat deadline MUST read a monotonic clock", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Why a heartbeat deadline MUST read a monotonic clock", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1863707640", + "type": "rectangle", + "x": 35, + "y": 76, + "width": 190, + "height": 96, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 412227919, + "versionNonce": 1703177489, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1639105129", + "type": "text", + "x": 45, + "y": 84, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1961435820, + "versionNonce": 1620909235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CLOCK_REALTIME", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CLOCK_REALTIME", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t796080157", + "type": "text", + "x": 45, + "y": 104, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1303104579, + "versionNonce": 489107314, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wall time / UTC", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wall time / UTC", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t95838711", + "type": "text", + "x": 45, + "y": 119, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1942251272, + "versionNonce": 94085928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "settable + NTP-slewed", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "settable + NTP-slewed", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1008647747", + "type": "text", + "x": 45, + "y": 134, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 539081921, + "versionNonce": 986714968, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CAN STEP backward", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CAN STEP backward", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r539417637", + "type": "rectangle", + "x": 240, + "y": 76, + "width": 190, + "height": 96, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1961171696, + "versionNonce": 99727811, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t127991715", + "type": "text", + "x": 250, + "y": 84, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1641578708, + "versionNonce": 1725733516, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CLOCK_MONOTONIC", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CLOCK_MONOTONIC", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t68055603", + "type": "text", + "x": 250, + "y": 104, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1492312724, + "versionNonce": 1537093549, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "since a boot epoch", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "since a boot epoch", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1134857420", + "type": "text", + "x": 250, + "y": 119, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1022143113, + "versionNonce": 368674203, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "never steps, never back", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "never steps, never back", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1689699383", + "type": "text", + "x": 250, + "y": 134, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1969272107, + "versionNonce": 469630923, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "steady_clock / Instant", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "steady_clock / Instant", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1459533523", + "type": "rectangle", + "x": 445, + "y": 76, + "width": 190, + "height": 96, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1469929924, + "versionNonce": 183894987, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1078744376", + "type": "text", + "x": 455, + "y": 84, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 651327620, + "versionNonce": 296537773, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CLOCK_BOOTTIME", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CLOCK_BOOTTIME", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t370269786", + "type": "text", + "x": 455, + "y": 104, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1564430491, + "versionNonce": 1688407543, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "monotonic, and also", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "monotonic, and also", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1204437650", + "type": "text", + "x": 455, + "y": 119, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 853969492, + "versionNonce": 914697354, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "counts suspend time", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "counts suspend time", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r157716437", + "type": "rectangle", + "x": 650, + "y": 76, + "width": 190, + "height": 96, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 180413715, + "versionNonce": 1304997875, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t892989173", + "type": "text", + "x": 660, + "y": 84, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1726726478, + "versionNonce": 936210750, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PROCESS_CPUTIME_ID", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PROCESS_CPUTIME_ID", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1472879314", + "type": "text", + "x": 660, + "y": 104, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1858168484, + "versionNonce": 205797055, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "this process's", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "this process's", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1796686813", + "type": "text", + "x": 660, + "y": 119, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1882378739, + "versionNonce": 1155366442, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "on-CPU time only", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "on-CPU time only", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1363656016", + "type": "rectangle", + "x": 60, + "y": 300, + "width": 350, + "height": 132, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1540350244, + "versionNonce": 1867781588, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1371969670", + "type": "text", + "x": 70, + "y": 308, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 969589205, + "versionNonce": 854230569, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "deadline = wall_now() + timeout", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "deadline = wall_now() + timeout", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1424380106", + "type": "text", + "x": 70, + "y": 328, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 243819681, + "versionNonce": 1470347786, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "NTP steps the wall clock -0.5 s", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "NTP steps the wall clock -0.5 s", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t731793684", + "type": "text", + "x": 70, + "y": 343, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 660693535, + "versionNonce": 1622853275, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wall_now() jumps backward", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wall_now() jumps backward", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t803822596", + "type": "text", + "x": 70, + "y": 358, + "width": 199.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 553516209, + "versionNonce": 1265704260, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "dead peer looks alive (or a +step", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "dead peer looks alive (or a +step", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t199674681", + "type": "text", + "x": 70, + "y": 373, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1758988484, + "versionNonce": 861289213, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fires an instant false timeout)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fires an instant false timeout)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r296463249", + "type": "rectangle", + "x": 470, + "y": 300, + "width": 350, + "height": 132, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1147360645, + "versionNonce": 336531438, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t579976076", + "type": "text", + "x": 480, + "y": 308, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 451304180, + "versionNonce": 1872254785, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "deadline = mono_now() + timeout", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "deadline = mono_now() + timeout", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t15480502", + "type": "text", + "x": 480, + "y": 328, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1243640795, + "versionNonce": 287930077, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the step is invisible here", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the step is invisible here", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t994006610", + "type": "text", + "x": 480, + "y": 343, + "width": 199.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1655935431, + "versionNonce": 1959717840, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "elapsed = mono_now() - start >= 0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "elapsed = mono_now() - start >= 0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t931917505", + "type": "text", + "x": 480, + "y": 358, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1346139544, + "versionNonce": 361908679, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "timeout fires on real silence,", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "timeout fires on real silence,", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1523158849", + "type": "text", + "x": 480, + "y": 373, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1098784410, + "versionNonce": 1109638036, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "never on a clock correction", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "never on a clock correction", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1627080819", + "type": "arrow", + "x": 335, + "y": 172, + "width": 310, + "height": 128, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 32659589, + "versionNonce": 1199771812, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 310, + 128 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1927248392", + "type": "text", + "x": 490.0, + "y": 222.0, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1862248175, + "versionNonce": 154468074, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chatterd reads this", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chatterd reads this", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1924557932", + "type": "text", + "x": 440, + "y": 258, + "width": 369.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 894833452, + "versionNonce": 1694893583, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "peer goes silent; meanwhile ntpd corrects the wall clock", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "peer goes silent; meanwhile ntpd corrects the wall clock", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/24-clock-taxonomy.svg b/assets/diagrams/24-clock-taxonomy.svg new file mode 100644 index 0000000..54d4f2c --- /dev/null +++ b/assets/diagrams/24-clock-taxonomy.svg @@ -0,0 +1,41 @@ + + + +The clocks clock_gettime(clk_id, ...) can read + +Why a heartbeat deadline MUST read a monotonic clock + +CLOCK_REALTIME +wall time / UTC +settable + NTP-slewed +CAN STEP backward + +CLOCK_MONOTONIC +since a boot epoch +never steps, never back +steady_clock / Instant + +CLOCK_BOOTTIME +monotonic, and also +counts suspend time + +PROCESS_CPUTIME_ID +this process's +on-CPU time only + +deadline = wall_now() + timeout +NTP steps the wall clock -0.5 s +wall_now() jumps backward +dead peer looks alive (or a +step +fires an instant false timeout) + +deadline = mono_now() + timeout +the step is invisible here +elapsed = mono_now() - start >= 0 +timeout fires on real silence, +never on a clock correction + +chatterd reads this +chatterd reads this +peer goes silent; meanwhile ntpd corrects the wall clock + \ No newline at end of file diff --git a/assets/diagrams/24-heartbeat-timeout-seq.excalidraw b/assets/diagrams/24-heartbeat-timeout-seq.excalidraw new file mode 100644 index 0000000..9d995f8 --- /dev/null +++ b/assets/diagrams/24-heartbeat-timeout-seq.excalidraw @@ -0,0 +1,1315 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r290108488", + "type": "rectangle", + "x": 178, + "y": 92, + "width": 4, + "height": 462, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#e8870c", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 765037328, + "versionNonce": 1300822289, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "r764748586", + "type": "rectangle", + "x": 638, + "y": 92, + "width": 4, + "height": 214, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#bbbbbb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 105868517, + "versionNonce": 1344930945, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "r801225739", + "type": "rectangle", + "x": 108, + "y": 42, + "width": 144, + "height": 46, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1312024797, + "versionNonce": 1231780411, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1661877247", + "type": "text", + "x": 118, + "y": 50, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1958687655, + "versionNonce": 1110982583, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "connector B", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "connector B", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1335243729", + "type": "text", + "x": 118, + "y": 70, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 140659489, + "versionNonce": 1364896721, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "dials + reconnects", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "dials + reconnects", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1043439127", + "type": "rectangle", + "x": 568, + "y": 42, + "width": 144, + "height": 46, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1166971306, + "versionNonce": 964094010, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t67460306", + "type": "text", + "x": 578, + "y": 50, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 231460540, + "versionNonce": 696430367, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "listener A", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "listener A", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1608380919", + "type": "text", + "x": 578, + "y": 70, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 163639921, + "versionNonce": 1842888717, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "accept loop", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "accept loop", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1973500363", + "type": "rectangle", + "x": 70, + "y": 356, + "width": 236, + "height": 44, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1762431647, + "versionNonce": 207826304, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t911613345", + "type": "text", + "x": 80, + "y": 364, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1215956957, + "versionNonce": 408664157, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "deadline fires on silence", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "deadline fires on silence", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1764235330", + "type": "text", + "x": 80, + "y": 384, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1150055282, + "versionNonce": 1038545390, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chatterd: peer A timed out", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chatterd: peer A timed out", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r802922679", + "type": "rectangle", + "x": 70, + "y": 414, + "width": 236, + "height": 32, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1755644737, + "versionNonce": 667644108, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1742009191", + "type": "text", + "x": 80, + "y": 422, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1571744306, + "versionNonce": 550858485, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reconnecting to A in 107 ms", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reconnecting to A in 107 ms", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1996967745", + "type": "rectangle", + "x": 70, + "y": 452, + "width": 236, + "height": 32, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1375070583, + "versionNonce": 524919901, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t501618453", + "type": "text", + "x": 80, + "y": 460, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 428379308, + "versionNonce": 689924613, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reconnecting to A in 245 ms", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reconnecting to A in 245 ms", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1290036948", + "type": "rectangle", + "x": 70, + "y": 490, + "width": 236, + "height": 32, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 938537001, + "versionNonce": 77230175, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t105134336", + "type": "text", + "x": 80, + "y": 498, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 652007847, + "versionNonce": 1688228930, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reconnecting to A in 712 ms", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reconnecting to A in 712 ms", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "a1532853731", + "type": "arrow", + "x": 182, + "y": 120, + "width": 454, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1284708268, + "versionNonce": 1343667061, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 454, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1760373463", + "type": "text", + "x": 409.0, + "y": 106.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1094380302, + "versionNonce": 36394648, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "HELLO 'B'", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "HELLO 'B'", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1735228808", + "type": "arrow", + "x": 636, + "y": 150, + "width": 454, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 688086807, + "versionNonce": 1066145933, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -454, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t341163561", + "type": "text", + "x": 409.0, + "y": 136.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1819734549, + "versionNonce": 1213306054, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "HELLO 'A'", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "HELLO 'A'", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a925548027", + "type": "arrow", + "x": 182, + "y": 210, + "width": 454, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 11470473, + "versionNonce": 500240158, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 454, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1041468695", + "type": "text", + "x": 409.0, + "y": 196.0, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 46128, + "versionNonce": 368822375, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PING every 300 ms", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PING every 300 ms", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a985642152", + "type": "arrow", + "x": 636, + "y": 240, + "width": 454, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 847473332, + "versionNonce": 578985287, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -454, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t333254620", + "type": "text", + "x": 409.0, + "y": 226.0, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 531623026, + "versionNonce": 975893323, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PING -> resets B deadline", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PING -> resets B deadline", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a585307817", + "type": "arrow", + "x": 182, + "y": 330, + "width": 454, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1974961421, + "versionNonce": 1102931575, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 454, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1089799011", + "type": "text", + "x": 409.0, + "y": 316.0, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 647833847, + "versionNonce": 798850817, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PING (broken pipe, dropped)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PING (broken pipe, dropped)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a11152297", + "type": "arrow", + "x": 188, + "y": 400, + "width": 0, + "height": 14, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 47928519, + "versionNonce": 428070297, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 14 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a523328902", + "type": "arrow", + "x": 188, + "y": 446, + "width": 0, + "height": 6, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 399419655, + "versionNonce": 1440734224, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 6 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a313793132", + "type": "arrow", + "x": 188, + "y": 484, + "width": 0, + "height": 6, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 684944375, + "versionNonce": 656105005, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 6 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t809181183", + "type": "text", + "x": 410, + "y": 170, + "width": 296.45000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 447263778, + "versionNonce": 141071126, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "both print 'linked'; --message delivered each way", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "both print 'linked'; --message delivered each way", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1329757031", + "type": "text", + "x": 410, + "y": 268, + "width": 308.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 822793560, + "versionNonce": 1745668551, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "any frame: deadline = mono_now() + timeout (Reset)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "any frame: deadline = mono_now() + timeout (Reset)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1317438331", + "type": "text", + "x": 640, + "y": 290, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1023400994, + "versionNonce": 1638321682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "A killed -> socket EOF", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "A killed -> socket EOF", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t503205463", + "type": "text", + "x": 330, + "y": 420, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 365772642, + "versionNonce": 1588355555, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "equal-jitter backoff:", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "equal-jitter backoff:", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1656788617", + "type": "text", + "x": 330, + "y": 438, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 159553336, + "versionNonce": 693854290, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "delay = B/2 + rand(0..B/2)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "delay = B/2 + rand(0..B/2)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t795266433", + "type": "text", + "x": 330, + "y": 456, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 236549455, + "versionNonce": 1147429902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "B doubles per attempt: 200, 400, 800 ...", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "B doubles per attempt: 200, 400, 800 ...", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t774417021", + "type": "text", + "x": 330, + "y": 474, + "width": 296.45000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1627780377, + "versionNonce": 1472188834, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "capped at --max-backoff-ms; reset on a fresh link", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "capped at --max-backoff-ms; reset on a fresh link", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/24-heartbeat-timeout-seq.svg b/assets/diagrams/24-heartbeat-timeout-seq.svg new file mode 100644 index 0000000..ec39070 --- /dev/null +++ b/assets/diagrams/24-heartbeat-timeout-seq.svg @@ -0,0 +1,45 @@ + + + + + +connector B +dials + reconnects + +listener A +accept loop + +deadline fires on silence +chatterd: peer A timed out + +reconnecting to A in 107 ms + +reconnecting to A in 245 ms + +reconnecting to A in 712 ms + +HELLO 'B' +HELLO 'B' + +HELLO 'A' +HELLO 'A' + +PING every 300 ms +PING every 300 ms + +PING -> resets B deadline +PING -> resets B deadline + +PING (broken pipe, dropped) +PING (broken pipe, dropped) + + + +both print 'linked'; --message delivered each way +any frame: deadline = mono_now() + timeout (Reset) +A killed -> socket EOF +equal-jitter backoff: +delay = B/2 + rand(0..B/2) +B doubles per attempt: 200, 400, 800 ... +capped at --max-backoff-ms; reset on a fresh link + \ No newline at end of file diff --git a/assets/diagrams/25-futex-fast-slow-path.excalidraw b/assets/diagrams/25-futex-fast-slow-path.excalidraw new file mode 100644 index 0000000..71911af --- /dev/null +++ b/assets/diagrams/25-futex-fast-slow-path.excalidraw @@ -0,0 +1,1115 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1777253991", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 900, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 732874807, + "versionNonce": 404013983, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1276374550", + "type": "text", + "x": 32, + "y": 50, + "width": 468.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1271905320, + "versionNonce": 786193739, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "user space \u2014 the lock word lives here (glibc pthread / libstdc++ / std)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "user space \u2014 the lock word lives here (glibc pthread / libstdc++ / std)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r893443186", + "type": "rectangle", + "x": 20, + "y": 250, + "width": 900, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 780807474, + "versionNonce": 1250467558, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1420205592", + "type": "text", + "x": 32, + "y": 256, + "width": 264.0, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1163251667, + "versionNonce": 1319508632, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel \u2014 entered only when the CAS loses", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel \u2014 entered only when the CAS loses", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1449513479", + "type": "rectangle", + "x": 48, + "y": 80, + "width": 150, + "height": 46, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 483526917, + "versionNonce": 1424062934, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1832916626", + "type": "text", + "x": 58, + "y": 88, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1062635193, + "versionNonce": 695830494, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Thread A: lock()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Thread A: lock()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1872236468", + "type": "text", + "x": 58, + "y": 108, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1117846676, + "versionNonce": 1401618492, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "word == 0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "word == 0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r509346989", + "type": "rectangle", + "x": 248, + "y": 72, + "width": 170, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 97606047, + "versionNonce": 1907422764, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t486794987", + "type": "text", + "x": 258, + "y": 80, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 442598117, + "versionNonce": 212536235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CAS word 0 -> 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CAS word 0 -> 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t580130818", + "type": "text", + "x": 258, + "y": 100, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1560817650, + "versionNonce": 1190567285, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one atomic instruction", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one atomic instruction", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1691007161", + "type": "text", + "x": 258, + "y": 115, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 743304163, + "versionNonce": 259089397, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "returns: held", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "returns: held", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r761985924", + "type": "rectangle", + "x": 500, + "y": 80, + "width": 150, + "height": 46, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 516661483, + "versionNonce": 1635344417, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t236938003", + "type": "text", + "x": 510, + "y": 88, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 667119428, + "versionNonce": 1190247739, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Thread B: lock()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Thread B: lock()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t500321632", + "type": "text", + "x": 510, + "y": 108, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 623136198, + "versionNonce": 486996186, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "word == 1, CAS loses", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "word == 1, CAS loses", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r456431596", + "type": "rectangle", + "x": 720, + "y": 80, + "width": 160, + "height": 46, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1676945099, + "versionNonce": 1151779135, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1383653844", + "type": "text", + "x": 730, + "y": 88, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1627804659, + "versionNonce": 730026752, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Thread A: unlock()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Thread A: unlock()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t776054770", + "type": "text", + "x": 730, + "y": 108, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 318900163, + "versionNonce": 611473098, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "word 1 -> 0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "word 1 -> 0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1515288319", + "type": "rectangle", + "x": 470, + "y": 288, + "width": 210, + "height": 74, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 729053136, + "versionNonce": 1098046696, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1574698013", + "type": "text", + "x": 480, + "y": 296, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1137307911, + "versionNonce": 990681891, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "futex wait queue", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "futex wait queue", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1747446453", + "type": "text", + "x": 480, + "y": 316, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 826385999, + "versionNonce": 1997707172, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "keyed by (mm, &word)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "keyed by (mm, &word)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1417842549", + "type": "text", + "x": 480, + "y": 331, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 638647542, + "versionNonce": 375930871, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Thread B parked", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Thread B parked", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1014081597", + "type": "arrow", + "x": 198, + "y": 103, + "width": 50, + "height": 2, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 329940359, + "versionNonce": 1657317903, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 50, + 2 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t347840025", + "type": "text", + "x": 223.0, + "y": 90.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 312443274, + "versionNonce": 1972232872, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cmpxchg", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cmpxchg", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a223618254", + "type": "arrow", + "x": 545, + "y": 126, + "width": 25, + "height": 162, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1866946353, + "versionNonce": 642594912, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -25, + 162 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1053914789", + "type": "text", + "x": 532.5, + "y": 193.0, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 917730269, + "versionNonce": 611743112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "futex(FUTEX_WAIT)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "futex(FUTEX_WAIT)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1600895699", + "type": "arrow", + "x": 800, + "y": 126, + "width": 140, + "height": 162, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 399894615, + "versionNonce": 1433279350, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -140, + 162 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1446792408", + "type": "text", + "x": 730.0, + "y": 193.0, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1489945896, + "versionNonce": 832311613, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "futex(FUTEX_WAKE)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "futex(FUTEX_WAKE)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a217148335", + "type": "arrow", + "x": 640, + "y": 288, + "width": 40, + "height": 160, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1354843987, + "versionNonce": 1484817760, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -40, + -160 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t299151513", + "type": "text", + "x": 620.0, + "y": 194.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 155543623, + "versionNonce": 1509834451, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "runnable", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "runnable", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t260742412", + "type": "text", + "x": 48, + "y": 162, + "width": 393.25000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 688195908, + "versionNonce": 781951083, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Fast path: no syscall. ~98% of lock ops in the 1x1 run stay here.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Fast path: no syscall. ~98% of lock ops in the 1x1 run stay here.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t684048793", + "type": "text", + "x": 470, + "y": 378, + "width": 496.1, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1109949422, + "versionNonce": 1968927114, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Slow path: a syscall each way. 1x1 cap256 = 4,583 futex calls; 8x8 cap8 = 491,514.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Slow path: a syscall each way. 1x1 cap256 = 4,583 futex calls; 8x8 cap8 = 491,514.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1150024116", + "type": "text", + "x": 20, + "y": 414, + "width": 580.8000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 88594384, + "versionNonce": 473502904, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "voluntary context switches track the parking: 302 (1x1) -> 139,565 (8x8), from /usr/bin/time -v.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "voluntary context switches track the parking: 302 (1x1) -> 139,565 (8x8), from /usr/bin/time -v.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/25-futex-fast-slow-path.svg b/assets/diagrams/25-futex-fast-slow-path.svg new file mode 100644 index 0000000..86d968f --- /dev/null +++ b/assets/diagrams/25-futex-fast-slow-path.svg @@ -0,0 +1,39 @@ + + + +user space — the lock word lives here (glibc pthread / libstdc++ / std) + +kernel — entered only when the CAS loses + +Thread A: lock() +word == 0 + +CAS word 0 -> 1 +one atomic instruction +returns: held + +Thread B: lock() +word == 1, CAS loses + +Thread A: unlock() +word 1 -> 0 + +futex wait queue +keyed by (mm, &word) +Thread B parked + +cmpxchg +cmpxchg + +futex(FUTEX_WAIT) +futex(FUTEX_WAIT) + +futex(FUTEX_WAKE) +futex(FUTEX_WAKE) + +runnable +runnable +Fast path: no syscall. ~98% of lock ops in the 1x1 run stay here. +Slow path: a syscall each way. 1x1 cap256 = 4,583 futex calls; 8x8 cap8 = 491,514. +voluntary context switches track the parking: 302 (1x1) -> 139,565 (8x8), from /usr/bin/time -v. + \ No newline at end of file diff --git a/assets/diagrams/25-mpmc-queue.excalidraw b/assets/diagrams/25-mpmc-queue.excalidraw new file mode 100644 index 0000000..91b6dd2 --- /dev/null +++ b/assets/diagrams/25-mpmc-queue.excalidraw @@ -0,0 +1,1342 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1016620709", + "type": "rectangle", + "x": 40, + "y": 66, + "width": 150, + "height": 44, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1543031170, + "versionNonce": 814670781, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t98033249", + "type": "text", + "x": 50, + "y": 74, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1421377175, + "versionNonce": 928769964, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "producer 0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "producer 0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t31862209", + "type": "text", + "x": 50, + "y": 94, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1112111286, + "versionNonce": 1101148308, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "owns i = 0 (mod P)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "owns i = 0 (mod P)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r229073170", + "type": "rectangle", + "x": 40, + "y": 122, + "width": 150, + "height": 44, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 305671174, + "versionNonce": 1499152978, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1607450913", + "type": "text", + "x": 50, + "y": 130, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 706199539, + "versionNonce": 270437737, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "producer 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "producer 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1928158854", + "type": "text", + "x": 50, + "y": 150, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1525509552, + "versionNonce": 287150398, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "owns i = 1 (mod P)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "owns i = 1 (mod P)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r266557505", + "type": "rectangle", + "x": 40, + "y": 178, + "width": 150, + "height": 44, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 845041150, + "versionNonce": 578601943, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1360963631", + "type": "text", + "x": 50, + "y": 186, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 905578798, + "versionNonce": 772809896, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "...", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "...", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t135613689", + "type": "text", + "x": 50, + "y": 206, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 762914231, + "versionNonce": 20104328, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "producer P-1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "producer P-1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r736402756", + "type": "rectangle", + "x": 360, + "y": 96, + "width": 230, + "height": 100, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1484784674, + "versionNonce": 607478937, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1478105825", + "type": "text", + "x": 370, + "y": 104, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1506073693, + "versionNonce": 1666831614, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bounded queue, cap K", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bounded queue, cap K", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1683725873", + "type": "text", + "x": 370, + "y": 124, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1173100528, + "versionNonce": 707461150, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "mutex + not_full / not_empty", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "mutex + not_full / not_empty", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1005556013", + "type": "text", + "x": 370, + "y": 139, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 469223082, + "versionNonce": 979348576, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(= buffered chan in Go)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(= buffered chan in Go)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1046367660", + "type": "text", + "x": 370, + "y": 154, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 924579701, + "versionNonce": 1279620275, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "produced: atomic", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "produced: atomic", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r593511850", + "type": "rectangle", + "x": 780, + "y": 66, + "width": 170, + "height": 44, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 816714592, + "versionNonce": 574499540, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t721211092", + "type": "text", + "x": 790, + "y": 74, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1831022196, + "versionNonce": 1542386927, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "consumer 0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "consumer 0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t760653906", + "type": "text", + "x": 790, + "y": 94, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 735321565, + "versionNonce": 795930579, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local (count, sum)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local (count, sum)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r151759327", + "type": "rectangle", + "x": 780, + "y": 122, + "width": 170, + "height": 44, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1658270998, + "versionNonce": 1980596491, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t785843241", + "type": "text", + "x": 790, + "y": 130, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1785251781, + "versionNonce": 246405592, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "consumer 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "consumer 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1544020194", + "type": "text", + "x": 790, + "y": 150, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 652870104, + "versionNonce": 1001360099, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local (count, sum)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local (count, sum)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r913695258", + "type": "rectangle", + "x": 780, + "y": 178, + "width": 170, + "height": 44, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 980228794, + "versionNonce": 1142309030, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1000685719", + "type": "text", + "x": 790, + "y": 186, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 487014724, + "versionNonce": 1269058459, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "...", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "...", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1461247949", + "type": "text", + "x": 790, + "y": 206, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 772433702, + "versionNonce": 1307900430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "consumer C-1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "consumer C-1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r853999115", + "type": "rectangle", + "x": 560, + "y": 320, + "width": 300, + "height": 62, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 126173798, + "versionNonce": 1840411714, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1575304688", + "type": "text", + "x": 570, + "y": 328, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 36103301, + "versionNonce": 153401790, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "XOR-fold locals after join", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "XOR-fold locals after join", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t525272681", + "type": "text", + "x": 570, + "y": 348, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1508083760, + "versionNonce": 491717810, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "checksum = 42b3746c6cee7465", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "checksum = 42b3746c6cee7465", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1352611445", + "type": "text", + "x": 570, + "y": 363, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1219168397, + "versionNonce": 1959669289, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "independent of P, C, cap", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "independent of P, C, cap", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1787406287", + "type": "arrow", + "x": 190, + "y": 88, + "width": 170, + "height": 42, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 461388060, + "versionNonce": 1593542167, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 170, + 42 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1912079201", + "type": "text", + "x": 275.0, + "y": 95.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1015160021, + "versionNonce": 1470949693, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "push()", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "push()", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a29123230", + "type": "arrow", + "x": 190, + "y": 144, + "width": 170, + "height": 6, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1740666710, + "versionNonce": 566606275, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 170, + 6 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a17679165", + "type": "arrow", + "x": 190, + "y": 200, + "width": 170, + "height": 24, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 542005000, + "versionNonce": 264513638, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 170, + -24 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a6519333", + "type": "arrow", + "x": 590, + "y": 130, + "width": 190, + "height": 42, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 339564319, + "versionNonce": 1677326201, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 190, + -42 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t33241989", + "type": "text", + "x": 685.0, + "y": 95.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 69145172, + "versionNonce": 305030316, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pop()", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pop()", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1952066238", + "type": "arrow", + "x": 590, + "y": 150, + "width": 190, + "height": 6, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 552399547, + "versionNonce": 311754427, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 190, + -6 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1602128337", + "type": "arrow", + "x": 590, + "y": 176, + "width": 190, + "height": 24, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 502304471, + "versionNonce": 1149960621, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 190, + 24 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a528170829", + "type": "arrow", + "x": 865, + "y": 224, + "width": 85, + "height": 94, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1728003799, + "versionNonce": 1950422038, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -85, + 94 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1753848408", + "type": "text", + "x": 822.5, + "y": 257.0, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 731912180, + "versionNonce": 983153961, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "XOR-combine after join", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "XOR-combine after join", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t410113131", + "type": "text", + "x": 40, + "y": 252, + "width": 586.85, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 929235386, + "versionNonce": 783543025, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one mutex serializes every push/pop; XOR is commutative, so drain order cannot change the result.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one mutex serializes every push/pop; XOR is commutative, so drain order cannot change the result.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/25-mpmc-queue.svg b/assets/diagrams/25-mpmc-queue.svg new file mode 100644 index 0000000..c91d42d --- /dev/null +++ b/assets/diagrams/25-mpmc-queue.svg @@ -0,0 +1,44 @@ + + + +producer 0 +owns i = 0 (mod P) + +producer 1 +owns i = 1 (mod P) + +... +producer P-1 + +bounded queue, cap K +mutex + not_full / not_empty +(= buffered chan in Go) +produced: atomic<long> + +consumer 0 +local (count, sum) + +consumer 1 +local (count, sum) + +... +consumer C-1 + +XOR-fold locals after join +checksum = 42b3746c6cee7465 +independent of P, C, cap + +push() +push() + + + +pop() +pop() + + + +XOR-combine after join +XOR-combine after join +one mutex serializes every push/pop; XOR is commutative, so drain order cannot change the result. + \ No newline at end of file diff --git a/assets/diagrams/26-false-sharing.excalidraw b/assets/diagrams/26-false-sharing.excalidraw new file mode 100644 index 0000000..8675fb7 --- /dev/null +++ b/assets/diagrams/26-false-sharing.excalidraw @@ -0,0 +1,1276 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r594930674", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 420, + "height": 300, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fbeeee", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 346187079, + "versionNonce": 825520233, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t833568879", + "type": "text", + "x": 32, + "y": 50, + "width": 270.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1822557321, + "versionNonce": 454311884, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pad=off \u2014 head & tail share one 64 B line", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pad=off \u2014 head & tail share one 64 B line", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r614135139", + "type": "rectangle", + "x": 460, + "y": 44, + "width": 420, + "height": 300, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef7ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 741328194, + "versionNonce": 1965785922, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t352852454", + "type": "text", + "x": 472, + "y": 50, + "width": 250.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1302986540, + "versionNonce": 1184301249, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pad=on \u2014 head & tail on separate lines", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pad=on \u2014 head & tail on separate lines", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r224978799", + "type": "rectangle", + "x": 60, + "y": 84, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1338048780, + "versionNonce": 963506597, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t168925704", + "type": "text", + "x": 70, + "y": 92, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 103074728, + "versionNonce": 534150698, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core 0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core 0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1022893250", + "type": "text", + "x": 70, + "y": 112, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1833433702, + "versionNonce": 1085473779, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "producer: writes tail", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "producer: writes tail", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1273995700", + "type": "rectangle", + "x": 250, + "y": 84, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1106918953, + "versionNonce": 1810672, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1552317452", + "type": "text", + "x": 260, + "y": 92, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1635243591, + "versionNonce": 652670524, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core 2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core 2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t733589835", + "type": "text", + "x": 260, + "y": 112, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1679100681, + "versionNonce": 302726199, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "consumer: writes head", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "consumer: writes head", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r316880055", + "type": "rectangle", + "x": 120, + "y": 200, + "width": 220, + "height": 64, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1548041799, + "versionNonce": 702999286, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t837183596", + "type": "text", + "x": 130, + "y": 208, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 716858885, + "versionNonce": 1264666096, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cache line #1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cache line #1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1467202022", + "type": "text", + "x": 130, + "y": 228, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1027247786, + "versionNonce": 408835392, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "head | tail (adjacent)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "head | tail (adjacent)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1046372782", + "type": "rectangle", + "x": 500, + "y": 84, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1848870188, + "versionNonce": 65686745, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1935302672", + "type": "text", + "x": 510, + "y": 92, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 420714038, + "versionNonce": 616993419, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core 0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core 0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t487735898", + "type": "text", + "x": 510, + "y": 112, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 253927302, + "versionNonce": 1274159466, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "producer: writes tail", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "producer: writes tail", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r495351682", + "type": "rectangle", + "x": 690, + "y": 84, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1803496525, + "versionNonce": 1227067203, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1507788098", + "type": "text", + "x": 700, + "y": 92, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 352111045, + "versionNonce": 1400206308, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core 2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core 2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1950189773", + "type": "text", + "x": 700, + "y": 112, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 502955624, + "versionNonce": 644998269, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "consumer: writes head", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "consumer: writes head", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1021185355", + "type": "rectangle", + "x": 500, + "y": 200, + "width": 150, + "height": 64, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 934157109, + "versionNonce": 1704102960, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t938202564", + "type": "text", + "x": 510, + "y": 208, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1421974969, + "versionNonce": 477076928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "line #1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "line #1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1783932463", + "type": "text", + "x": 510, + "y": 228, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 39337150, + "versionNonce": 261148965, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "head + 56 B pad", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "head + 56 B pad", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1056187343", + "type": "rectangle", + "x": 690, + "y": 200, + "width": 150, + "height": 64, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 586656859, + "versionNonce": 902250419, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t813104068", + "type": "text", + "x": 700, + "y": 208, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1397224142, + "versionNonce": 1526866910, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "line #2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "line #2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1572853626", + "type": "text", + "x": 700, + "y": 228, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1625405174, + "versionNonce": 923043555, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail + 56 B pad", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail + 56 B pad", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1224210450", + "type": "arrow", + "x": 135, + "y": 144, + "width": 65, + "height": 56, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1788024557, + "versionNonce": 787194207, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 65, + 56 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t120927213", + "type": "text", + "x": 167.5, + "y": 158.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 567899489, + "versionNonce": 333705611, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "invalidate", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "invalidate", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1074113420", + "type": "arrow", + "x": 325, + "y": 144, + "width": 65, + "height": 56, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1464759323, + "versionNonce": 788418688, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -65, + 56 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1735013052", + "type": "text", + "x": 292.5, + "y": 158.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 248542586, + "versionNonce": 1722049499, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "invalidate", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "invalidate", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a151038455", + "type": "arrow", + "x": 575, + "y": 144, + "width": 0, + "height": 56, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1101954629, + "versionNonce": 1562556579, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 56 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t518395057", + "type": "text", + "x": 575.0, + "y": 158.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 897693377, + "versionNonce": 170082285, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1724242022", + "type": "arrow", + "x": 765, + "y": 144, + "width": 0, + "height": 56, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 509745450, + "versionNonce": 1879533971, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 56 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1647902222", + "type": "text", + "x": 765.0, + "y": 158.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 277909913, + "versionNonce": 690053139, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1162552642", + "type": "text", + "x": 130, + "y": 290, + "width": 138.60000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#a12a2a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1468611539, + "versionNonce": 49638689, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "L1 load-misses ~128 M", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "L1 load-misses ~128 M", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t737285005", + "type": "text", + "x": 130, + "y": 312, + "width": 248.05, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 568525105, + "versionNonce": 1286670191, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "median 4.64 Mops (capacity 1, 10 M items)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "median 4.64 Mops (capacity 1, 10 M items)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t625518361", + "type": "text", + "x": 560, + "y": 290, + "width": 138.60000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#2a7d3a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1191594202, + "versionNonce": 1188537220, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "L1 load-misses ~108 M", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "L1 load-misses ~108 M", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t317931711", + "type": "text", + "x": 560, + "y": 312, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1266508284, + "versionNonce": 532391538, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "median 6.20 Mops \u2014 ~34% faster", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "median 6.20 Mops \u2014 ~34% faster", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/26-false-sharing.svg b/assets/diagrams/26-false-sharing.svg new file mode 100644 index 0000000..bcd98f3 --- /dev/null +++ b/assets/diagrams/26-false-sharing.svg @@ -0,0 +1,44 @@ + + + +pad=off — head & tail share one 64 B line + +pad=on — head & tail on separate lines + +core 0 +producer: writes tail + +core 2 +consumer: writes head + +cache line #1 +head | tail (adjacent) + +core 0 +producer: writes tail + +core 2 +consumer: writes head + +line #1 +head + 56 B pad + +line #2 +tail + 56 B pad + +invalidate +invalidate + +invalidate +invalidate + +local +local + +local +local +L1 load-misses ~128 M +median 4.64 Mops (capacity 1, 10 M items) +L1 load-misses ~108 M +median 6.20 Mops — ~34% faster + \ No newline at end of file diff --git a/assets/diagrams/26-memory-ordering.excalidraw b/assets/diagrams/26-memory-ordering.excalidraw new file mode 100644 index 0000000..c66d738 --- /dev/null +++ b/assets/diagrams/26-memory-ordering.excalidraw @@ -0,0 +1,1058 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r457273142", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 860, + "height": 92, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fbeeee", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 166859917, + "versionNonce": 1124358982, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1986240265", + "type": "text", + "x": 32, + "y": 50, + "width": 349.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1720445030, + "versionNonce": 172880673, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sequential consistency \u2014 strongest; Go's only setting", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sequential consistency \u2014 strongest; Go's only setting", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1227158473", + "type": "rectangle", + "x": 20, + "y": 152, + "width": 860, + "height": 92, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1324580525, + "versionNonce": 1885085640, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t629280027", + "type": "text", + "x": 32, + "y": 158, + "width": 382.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1936404631, + "versionNonce": 423071945, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "acquire / release \u2014 pairwise publish; C++ & Rust head/tail", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "acquire / release \u2014 pairwise publish; C++ & Rust head/tail", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r867369908", + "type": "rectangle", + "x": 20, + "y": 260, + "width": 860, + "height": 92, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef7ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 4991988, + "versionNonce": 1947835279, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t34915068", + "type": "text", + "x": 32, + "y": 266, + "width": 356.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 454986661, + "versionNonce": 1052975612, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "relaxed \u2014 atomicity only, no ordering; Rust ring slots", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "relaxed \u2014 atomicity only, no ordering; Rust ring slots", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1409884809", + "type": "rectangle", + "x": 40, + "y": 74, + "width": 360, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 885060305, + "versionNonce": 1768530968, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t881257529", + "type": "text", + "x": 50, + "y": 82, + "width": 277.20000000000005, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 757078363, + "versionNonce": 250648131, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one total order over all seq_cst ops", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one total order over all seq_cst ops", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t701285414", + "type": "text", + "x": 50, + "y": 102, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1254958915, + "versionNonce": 1256948829, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail.Store(t) in Go", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail.Store(t) in Go", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r498531138", + "type": "rectangle", + "x": 470, + "y": 74, + "width": 380, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 510469921, + "versionNonce": 707899993, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t466777547", + "type": "text", + "x": 480, + "y": 82, + "width": 331.1, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1561063216, + "versionNonce": 178694953, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "x86: XCHGQ (implicitly locked, full fence)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "x86: XCHGQ (implicitly locked, full fence)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t930019105", + "type": "text", + "x": 480, + "y": 102, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 833923012, + "versionNonce": 696014284, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the store pays for global order", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the store pays for global order", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r332975166", + "type": "rectangle", + "x": 40, + "y": 182, + "width": 360, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1430488073, + "versionNonce": 1966934773, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1830740033", + "type": "text", + "x": 50, + "y": 190, + "width": 261.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 290162241, + "versionNonce": 1881618210, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Release store -> Acquire load edge", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Release store -> Acquire load edge", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t431375437", + "type": "text", + "x": 50, + "y": 210, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1338401849, + "versionNonce": 1450342874, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail.store(...,Release) / .load(Acquire)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail.store(...,Release) / .load(Acquire)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r107145487", + "type": "rectangle", + "x": 470, + "y": 182, + "width": 380, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 167970272, + "versionNonce": 294596893, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1071852609", + "type": "text", + "x": 480, + "y": 190, + "width": 269.5, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 121611537, + "versionNonce": 1495415822, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "x86: plain MOV (no lock, no fence)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "x86: plain MOV (no lock, no fence)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1482765202", + "type": "text", + "x": 480, + "y": 210, + "width": 260.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 851164484, + "versionNonce": 189136887, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "TSO already orders store->store, load->load", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "TSO already orders store->store, load->load", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r986604285", + "type": "rectangle", + "x": 40, + "y": 290, + "width": 360, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 292817380, + "versionNonce": 1949953627, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t723896187", + "type": "text", + "x": 50, + "y": 298, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 8441188, + "versionNonce": 667347595, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no cross-thread ordering promise", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no cross-thread ordering promise", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t403900095", + "type": "text", + "x": 50, + "y": 318, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1491995127, + "versionNonce": 1222843586, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[t%K].store(v, Relaxed)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[t%K].store(v, Relaxed)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1022466809", + "type": "rectangle", + "x": 470, + "y": 290, + "width": 380, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1264039055, + "versionNonce": 467275599, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1612913990", + "type": "text", + "x": 480, + "y": 298, + "width": 346.5, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 279869739, + "versionNonce": 1098413255, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "x86: plain MOV (same instruction as release)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "x86: plain MOV (same instruction as release)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1608807582", + "type": "text", + "x": 480, + "y": 318, + "width": 266.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1640684673, + "versionNonce": 779136311, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "safe here only because tail's edge covers it", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "safe here only because tail's edge covers it", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1072292146", + "type": "arrow", + "x": 400, + "y": 99, + "width": 70, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1883619502, + "versionNonce": 1457451696, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 70, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t544240528", + "type": "text", + "x": 435.0, + "y": 85.0, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 820711236, + "versionNonce": 638223720, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "compiles to", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "compiles to", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1655121740", + "type": "arrow", + "x": 400, + "y": 207, + "width": 70, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 34546226, + "versionNonce": 1183372911, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 70, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t363562411", + "type": "text", + "x": 435.0, + "y": 193.0, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 317848208, + "versionNonce": 767570201, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "compiles to", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "compiles to", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a233993340", + "type": "arrow", + "x": 400, + "y": 315, + "width": 70, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 290272674, + "versionNonce": 1141495955, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 70, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t410986221", + "type": "text", + "x": 435.0, + "y": 301.0, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1449798804, + "versionNonce": 774669479, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "compiles to", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "compiles to", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t500400372", + "type": "text", + "x": 40, + "y": 362, + "width": 822.8000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 956817663, + "versionNonce": 1543954165, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "On x86 the ordering is nearly free \u2014 the difference is the promise, not the instruction. On ARM/POWER the movs would need real barriers.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "On x86 the ordering is nearly free \u2014 the difference is the promise, not the instruction. On ARM/POWER the movs would need real barriers.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/26-memory-ordering.svg b/assets/diagrams/26-memory-ordering.svg new file mode 100644 index 0000000..f46543d --- /dev/null +++ b/assets/diagrams/26-memory-ordering.svg @@ -0,0 +1,37 @@ + + + +sequential consistency — strongest; Go's only setting + +acquire / release — pairwise publish; C++ & Rust head/tail + +relaxed — atomicity only, no ordering; Rust ring slots + +one total order over all seq_cst ops +tail.Store(t) in Go + +x86: XCHGQ (implicitly locked, full fence) +the store pays for global order + +Release store -> Acquire load edge +tail.store(...,Release) / .load(Acquire) + +x86: plain MOV (no lock, no fence) +TSO already orders store->store, load->load + +no cross-thread ordering promise +buf[t%K].store(v, Relaxed) + +x86: plain MOV (same instruction as release) +safe here only because tail's edge covers it + +compiles to +compiles to + +compiles to +compiles to + +compiles to +compiles to +On x86 the ordering is nearly free — the difference is the promise, not the instruction. On ARM/POWER the movs would need real barriers. + \ No newline at end of file diff --git a/assets/diagrams/26-spsc-ring.excalidraw b/assets/diagrams/26-spsc-ring.excalidraw new file mode 100644 index 0000000..1b67ff4 --- /dev/null +++ b/assets/diagrams/26-spsc-ring.excalidraw @@ -0,0 +1,1683 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r489608926", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 860, + "height": 168, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 809979250, + "versionNonce": 387069235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1333556644", + "type": "text", + "x": 32, + "y": 50, + "width": 270.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1936680918, + "versionNonce": 1453573418, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Two threads, two owned counters \u2014 no lock", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Two threads, two owned counters \u2014 no lock", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r972144010", + "type": "rectangle", + "x": 20, + "y": 232, + "width": 860, + "height": 118, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 984095393, + "versionNonce": 915920561, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1083916532", + "type": "text", + "x": 32, + "y": 238, + "width": 409.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1932488657, + "versionNonce": 1578770717, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Shared ring buf[K] \u2014 slot for logical position p is buf[p % K]", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Shared ring buf[K] \u2014 slot for logical position p is buf[p % K]", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r258223357", + "type": "rectangle", + "x": 40, + "y": 92, + "width": 228, + "height": 96, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 468152843, + "versionNonce": 385198818, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t48189290", + "type": "text", + "x": 50, + "y": 100, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 336826789, + "versionNonce": 760097361, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Producer (core 0)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Producer (core 0)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1205751548", + "type": "text", + "x": 50, + "y": 120, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1371266299, + "versionNonce": 555337063, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "owns tail, reads head", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "owns tail, reads head", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1686300956", + "type": "text", + "x": 50, + "y": 135, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1984878549, + "versionNonce": 344433854, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[t%K] = v", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[t%K] = v", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1237689597", + "type": "text", + "x": 50, + "y": 150, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1942315210, + "versionNonce": 62006195, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail.store(t+1, Release)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail.store(t+1, Release)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1369434359", + "type": "rectangle", + "x": 632, + "y": 92, + "width": 228, + "height": 96, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 538436673, + "versionNonce": 1934906941, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1070860534", + "type": "text", + "x": 642, + "y": 100, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 412673052, + "versionNonce": 1049651310, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Consumer (core 2)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Consumer (core 2)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t793398413", + "type": "text", + "x": 642, + "y": 120, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1923168203, + "versionNonce": 502617124, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "owns head, reads tail", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "owns head, reads tail", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1719793616", + "type": "text", + "x": 642, + "y": 135, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 612698489, + "versionNonce": 667234303, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "v = buf[h%K]", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "v = buf[h%K]", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1538030580", + "type": "text", + "x": 642, + "y": 150, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 504497630, + "versionNonce": 941089902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "head.store(h+1, Release)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "head.store(h+1, Release)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r590019321", + "type": "rectangle", + "x": 352, + "y": 78, + "width": 90, + "height": 48, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1694831497, + "versionNonce": 1616124001, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t249739282", + "type": "text", + "x": 362, + "y": 86, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 833545476, + "versionNonce": 703165657, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1064177593", + "type": "text", + "x": 362, + "y": 106, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 742525426, + "versionNonce": 930628842, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one writer", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one writer", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r269417021", + "type": "rectangle", + "x": 452, + "y": 78, + "width": 90, + "height": 48, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 938703467, + "versionNonce": 72801961, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t406295938", + "type": "text", + "x": 462, + "y": 86, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1107781813, + "versionNonce": 1638641754, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "head", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "head", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t543211315", + "type": "text", + "x": 462, + "y": 106, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1274665886, + "versionNonce": 1690626763, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one writer", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one writer", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r811018369", + "type": "rectangle", + "x": 90, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1565889222, + "versionNonce": 1525624293, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t48146249", + "type": "text", + "x": 100, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 21215731, + "versionNonce": 1058834971, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[0]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[0]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1951779143", + "type": "text", + "x": 100, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1932076305, + "versionNonce": 1107180229, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "3", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "3", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r725077604", + "type": "rectangle", + "x": 214, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 696670306, + "versionNonce": 1773885749, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t251502543", + "type": "text", + "x": 224, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1035019826, + "versionNonce": 99520077, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[1]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[1]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t44092374", + "type": "text", + "x": 224, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1145273259, + "versionNonce": 1823672325, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "4", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "4", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r685126044", + "type": "rectangle", + "x": 338, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 16820783, + "versionNonce": 115848673, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1366676925", + "type": "text", + "x": 348, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 975568477, + "versionNonce": 378698240, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[2]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[2]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t141373061", + "type": "text", + "x": 348, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 344009506, + "versionNonce": 1492437022, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "free", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "free", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r728365138", + "type": "rectangle", + "x": 462, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 201436715, + "versionNonce": 1422475438, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1335179375", + "type": "text", + "x": 472, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1252759606, + "versionNonce": 602223495, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[3]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[3]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t522890668", + "type": "text", + "x": 472, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 94795486, + "versionNonce": 543075903, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "free", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "free", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1338816810", + "type": "rectangle", + "x": 586, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 352492456, + "versionNonce": 1785732816, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1289181447", + "type": "text", + "x": 596, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1963031046, + "versionNonce": 1708185663, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[4]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[4]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t901430009", + "type": "text", + "x": 596, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 9374331, + "versionNonce": 1392624331, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "free", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "free", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1682809038", + "type": "rectangle", + "x": 710, + "y": 268, + "width": 118, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1504014178, + "versionNonce": 1157302845, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1139232830", + "type": "text", + "x": 720, + "y": 276, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 570955473, + "versionNonce": 684234902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "buf[5]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "buf[5]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1983398232", + "type": "text", + "x": 720, + "y": 296, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1974148395, + "versionNonce": 1205954310, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "free", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "free", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1705193594", + "type": "arrow", + "x": 268, + "y": 108, + "width": 84, + "height": 12, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 243090385, + "versionNonce": 1295827356, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 84, + -12 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1639765169", + "type": "text", + "x": 310.0, + "y": 88.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1193195527, + "versionNonce": 349056006, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "store Rel", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "store Rel", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1283892155", + "type": "arrow", + "x": 632, + "y": 108, + "width": 90, + "height": 12, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1761047689, + "versionNonce": 631861889, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -90, + -12 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1467767818", + "type": "text", + "x": 587.0, + "y": 88.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1486050601, + "versionNonce": 520956314, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "store Rel", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "store Rel", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a115021561", + "type": "arrow", + "x": 452, + "y": 116, + "width": 184, + "height": 40, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 577178071, + "versionNonce": 1502973910, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -184, + 40 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t365575857", + "type": "text", + "x": 360.0, + "y": 122.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1011262632, + "versionNonce": 960847472, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "load Acq", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "load Acq", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a958551366", + "type": "arrow", + "x": 442, + "y": 116, + "width": 190, + "height": 40, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 107712117, + "versionNonce": 1547954682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 190, + 40 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t568943088", + "type": "text", + "x": 537.0, + "y": 122.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 676706649, + "versionNonce": 1439266625, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "load Acq", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "load Acq", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1040532190", + "type": "arrow", + "x": 397, + "y": 126, + "width": 247, + "height": 142, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1571781602, + "versionNonce": 1510100665, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -247, + 142 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1275040991", + "type": "text", + "x": 273.5, + "y": 183.0, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 892151653, + "versionNonce": 1907037755, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tail Release publishes buf[t%K]", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tail Release publishes buf[t%K]", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1136880626", + "type": "text", + "x": 40, + "y": 334, + "width": 544.5, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1578350767, + "versionNonce": 775799015, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "occupancy = tail - head = 2 | full when tail - head == K | empty when head == tail", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "occupancy = tail - head = 2 | full when tail - head == K | empty when head == tail", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/26-spsc-ring.svg b/assets/diagrams/26-spsc-ring.svg new file mode 100644 index 0000000..96d4f2a --- /dev/null +++ b/assets/diagrams/26-spsc-ring.svg @@ -0,0 +1,57 @@ + + + +Two threads, two owned counters — no lock + +Shared ring buf[K] — slot for logical position p is buf[p % K] + +Producer (core 0) +owns tail, reads head +buf[t%K] = v +tail.store(t+1, Release) + +Consumer (core 2) +owns head, reads tail +v = buf[h%K] +head.store(h+1, Release) + +tail +one writer + +head +one writer + +buf[0] +3 + +buf[1] +4 + +buf[2] +free + +buf[3] +free + +buf[4] +free + +buf[5] +free + +store Rel +store Rel + +store Rel +store Rel + +load Acq +load Acq + +load Acq +load Acq + +tail Release publishes buf[t%K] +tail Release publishes buf[t%K] +occupancy = tail - head = 2 | full when tail - head == K | empty when head == tail + \ No newline at end of file diff --git a/assets/diagrams/27-coroutine-suspension.excalidraw b/assets/diagrams/27-coroutine-suspension.excalidraw new file mode 100644 index 0000000..6a1b65e --- /dev/null +++ b/assets/diagrams/27-coroutine-suspension.excalidraw @@ -0,0 +1,1347 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1965682477", + "type": "rectangle", + "x": 20, + "y": 40, + "width": 900, + "height": 130, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 908853740, + "versionNonce": 1803679193, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1696633148", + "type": "text", + "x": 32, + "y": 46, + "width": 442.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1920647724, + "versionNonce": 1400683284, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "user \u2014 connection() coroutine (one heap-allocated frame per client)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "user \u2014 connection() coroutine (one heap-allocated frame per client)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1982507570", + "type": "rectangle", + "x": 20, + "y": 195, + "width": 900, + "height": 95, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1347432905, + "versionNonce": 1101447473, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1831202562", + "type": "text", + "x": 32, + "y": 201, + "width": 244.20000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 16743740, + "versionNonce": 704453997, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the single OS thread \u2014 Reactor::run()", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the single OS thread \u2014 Reactor::run()", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1482709554", + "type": "rectangle", + "x": 20, + "y": 315, + "width": 900, + "height": 90, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1442316002, + "versionNonce": 594381035, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t946161972", + "type": "text", + "x": 32, + "y": 321, + "width": 184.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1113758559, + "versionNonce": 905523820, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel \u2014 epoll interest list", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel \u2014 epoll interest list", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r887885626", + "type": "rectangle", + "x": 45, + "y": 78, + "width": 180, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1103361150, + "versionNonce": 1714449977, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1413430552", + "type": "text", + "x": 55, + "y": 86, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 516800779, + "versionNonce": 746501861, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "1 co_await fill()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "1 co_await fill()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t368982424", + "type": "text", + "x": 55, + "y": 106, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1622445015, + "versionNonce": 23220830, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "recv() \u2192 EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "recv() \u2192 EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1436278544", + "type": "rectangle", + "x": 258, + "y": 78, + "width": 190, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1833826714, + "versionNonce": 1651918902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1920830148", + "type": "text", + "x": 268, + "y": 86, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1115331662, + "versionNonce": 1289519070, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "2 co_await ReadAwaitable", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "2 co_await ReadAwaitable", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t949980876", + "type": "text", + "x": 268, + "y": 106, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1604432760, + "versionNonce": 986420915, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "await_suspend(h)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "await_suspend(h)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r513603019", + "type": "rectangle", + "x": 482, + "y": 78, + "width": 170, + "height": 66, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1884594889, + "versionNonce": 1104367787, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t460067314", + "type": "text", + "x": 492, + "y": 86, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1042943379, + "versionNonce": 78951808, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "3 frame suspended", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "3 frame suspended", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1553837138", + "type": "text", + "x": 492, + "y": 106, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1322000741, + "versionNonce": 1589779549, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "control returns down", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "control returns down", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1012888760", + "type": "rectangle", + "x": 700, + "y": 78, + "width": 195, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 345057696, + "versionNonce": 267947050, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t413395984", + "type": "text", + "x": 710, + "y": 86, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 973241319, + "versionNonce": 143159790, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "6 h.resume()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "6 h.resume()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t782365503", + "type": "text", + "x": 710, + "y": 106, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 939772825, + "versionNonce": 537212430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "recv() \u2192 bytes, decode", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "recv() \u2192 bytes, decode", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r655150081", + "type": "rectangle", + "x": 258, + "y": 218, + "width": 190, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1777280654, + "versionNonce": 33623453, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1648121411", + "type": "text", + "x": 268, + "y": 226, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 684459509, + "versionNonce": 98806394, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wait_readable(fd, h)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wait_readable(fd, h)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1012305181", + "type": "text", + "x": 268, + "y": 246, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1094920082, + "versionNonce": 1814596866, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "waiters_[fd]=h; arm fd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "waiters_[fd]=h; arm fd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1894893140", + "type": "rectangle", + "x": 610, + "y": 218, + "width": 190, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 718418729, + "versionNonce": 1569429187, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1165736141", + "type": "text", + "x": 620, + "y": 226, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 185515332, + "versionNonce": 698459624, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_wait(-1)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_wait(-1)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t88087487", + "type": "text", + "x": 620, + "y": 246, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 701340660, + "versionNonce": 1562935416, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "resume the parked handle", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "resume the parked handle", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1035368753", + "type": "rectangle", + "x": 340, + "y": 335, + "width": 270, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1380641405, + "versionNonce": 1787266243, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t475140058", + "type": "text", + "x": 350, + "y": 343, + "width": 215.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1679670534, + "versionNonce": 503849670, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "EPOLLIN | EPOLLONESHOT on fd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "EPOLLIN | EPOLLONESHOT on fd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1646120610", + "type": "text", + "x": 350, + "y": 363, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1902786109, + "versionNonce": 302403290, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ready \u2192 events[i].data.fd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ready \u2192 events[i].data.fd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a45724511", + "type": "arrow", + "x": 225, + "y": 111, + "width": 33, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1020838278, + "versionNonce": 1766366901, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 33, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1418832148", + "type": "text", + "x": 241.5, + "y": 97.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1975361080, + "versionNonce": 1254125253, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "EAGAIN", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "EAGAIN", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1754377630", + "type": "arrow", + "x": 353, + "y": 144, + "width": 0, + "height": 74, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1091971193, + "versionNonce": 1141474928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 74 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1706071626", + "type": "text", + "x": 353.0, + "y": 167.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 254282659, + "versionNonce": 1108085503, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "suspend", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "suspend", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a524959555", + "type": "arrow", + "x": 353, + "y": 274, + "width": 67, + "height": 61, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1410044316, + "versionNonce": 1593328695, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 67, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1401770544", + "type": "text", + "x": 386.5, + "y": 290.5, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 130210418, + "versionNonce": 1039302309, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_ctl", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_ctl", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1342249106", + "type": "arrow", + "x": 585, + "y": 361, + "width": 105, + "height": 87, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1935348647, + "versionNonce": 1672812258, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 105, + -87 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t693789376", + "type": "text", + "x": 637.5, + "y": 303.5, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 243712696, + "versionNonce": 1327531865, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ready", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ready", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1855027429", + "type": "arrow", + "x": 720, + "y": 218, + "width": 70, + "height": 74, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 865131972, + "versionNonce": 1608387140, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 70, + -74 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1163795115", + "type": "text", + "x": 755.0, + "y": 167.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 77461445, + "versionNonce": 1467533827, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "resume", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "resume", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t897374741", + "type": "text", + "x": 700, + "y": 158, + "width": 302.5, + "height": 12.5, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 409684190, + "versionNonce": 493176418, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "each decoded frame loops back to step 1 (co_await fill)", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "each decoded frame loops back to step 1 (co_await fill)", + "lineHeight": 1.25, + "baseline": 10 + }, + { + "id": "t101975871", + "type": "text", + "x": 40, + "y": 165, + "width": 434.50000000000006, + "height": 12.5, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 963254712, + "versionNonce": 1313345866, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "suspension is a return, not a blocked thread: the frame outlives the call stack", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "suspension is a return, not a blocked thread: the frame outlives the call stack", + "lineHeight": 1.25, + "baseline": 10 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/27-coroutine-suspension.svg b/assets/diagrams/27-coroutine-suspension.svg new file mode 100644 index 0000000..334f7fe --- /dev/null +++ b/assets/diagrams/27-coroutine-suspension.svg @@ -0,0 +1,47 @@ + + + +user — connection() coroutine (one heap-allocated frame per client) + +the single OS thread — Reactor::run() + +kernel — epoll interest list + +1 co_await fill() +recv() → EAGAIN + +2 co_await ReadAwaitable +await_suspend(h) + +3 frame suspended +control returns down + +6 h.resume() +recv() → bytes, decode + +wait_readable(fd, h) +waiters_[fd]=h; arm fd + +epoll_wait(-1) +resume the parked handle + +EPOLLIN | EPOLLONESHOT on fd +ready → events[i].data.fd + +EAGAIN +EAGAIN + +suspend +suspend + +epoll_ctl +epoll_ctl + +ready +ready + +resume +resume +each decoded frame loops back to step 1 (co_await fill) +suspension is a return, not a blocked thread: the frame outlives the call stack + \ No newline at end of file diff --git a/assets/diagrams/27-gmp-scheduler.excalidraw b/assets/diagrams/27-gmp-scheduler.excalidraw new file mode 100644 index 0000000..cd47b01 --- /dev/null +++ b/assets/diagrams/27-gmp-scheduler.excalidraw @@ -0,0 +1,1712 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1619956782", + "type": "rectangle", + "x": 20, + "y": 40, + "width": 900, + "height": 120, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1677228964, + "versionNonce": 1966151094, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1216722473", + "type": "text", + "x": 32, + "y": 46, + "width": 382.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 988815472, + "versionNonce": 1802496546, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "goroutines (G) \u2014 one per connection, ~2 KB growable stacks", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "goroutines (G) \u2014 one per connection, ~2 KB growable stacks", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1343562515", + "type": "rectangle", + "x": 20, + "y": 185, + "width": 900, + "height": 105, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 777929406, + "versionNonce": 1351974117, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1362343727", + "type": "text", + "x": 32, + "y": 191, + "width": 528.0, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 222285885, + "versionNonce": 1502173288, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "scheduler \u2014 GOMAXPROCS=16 logical processors (P), each bound to an OS thread (M)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "scheduler \u2014 GOMAXPROCS=16 logical processors (P), each bound to an OS thread (M)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1223256462", + "type": "rectangle", + "x": 20, + "y": 315, + "width": 900, + "height": 90, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1984282706, + "versionNonce": 558985978, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t318187704", + "type": "text", + "x": 32, + "y": 321, + "width": 270.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1459510488, + "versionNonce": 1747296878, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel \u2014 netpoller (epoll) and OS threads", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel \u2014 netpoller (epoll) and OS threads", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1083953155", + "type": "rectangle", + "x": 45, + "y": 72, + "width": 78, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 729693615, + "versionNonce": 917182718, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1178208635", + "type": "text", + "x": 55, + "y": 80, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 908048013, + "versionNonce": 1474883631, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "G", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "G", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1309535881", + "type": "text", + "x": 55, + "y": 100, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 610138945, + "versionNonce": 126610537, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "conn 1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "conn 1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r217962751", + "type": "rectangle", + "x": 133, + "y": 72, + "width": 78, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 395489245, + "versionNonce": 1518008432, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t609770599", + "type": "text", + "x": 143, + "y": 80, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 970965039, + "versionNonce": 1929057283, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "G", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "G", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t387698427", + "type": "text", + "x": 143, + "y": 100, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1331652176, + "versionNonce": 884486246, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "conn 2", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "conn 2", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r524110877", + "type": "rectangle", + "x": 221, + "y": 72, + "width": 78, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1743000975, + "versionNonce": 443833042, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t182871619", + "type": "text", + "x": 231, + "y": 80, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1312155282, + "versionNonce": 1671411979, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "G", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "G", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1856874641", + "type": "text", + "x": 231, + "y": 100, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1995680825, + "versionNonce": 1752661414, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "conn 3", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "conn 3", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r645939095", + "type": "rectangle", + "x": 309, + "y": 72, + "width": 78, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 761219929, + "versionNonce": 828322565, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t133398546", + "type": "text", + "x": 319, + "y": 80, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 283823945, + "versionNonce": 298834574, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\u2026", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\u2026", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1405714912", + "type": "text", + "x": 319, + "y": 100, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 847731734, + "versionNonce": 1901932569, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "G \u00d7 N", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "G \u00d7 N", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r742846852", + "type": "rectangle", + "x": 620, + "y": 68, + "width": 270, + "height": 64, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 165247550, + "versionNonce": 953901423, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1477803893", + "type": "text", + "x": 630, + "y": 76, + "width": 146.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1253666816, + "versionNonce": 1391628287, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "G blocked in recv()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "G blocked in recv()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t652780268", + "type": "text", + "x": 630, + "y": 96, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1703334303, + "versionNonce": 968966720, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "parked on the netpoller, not on an M", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "parked on the netpoller, not on an M", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1860451241", + "type": "rectangle", + "x": 45, + "y": 205, + "width": 150, + "height": 58, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 945024405, + "versionNonce": 1279131703, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1964293616", + "type": "text", + "x": 55, + "y": 213, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1366005417, + "versionNonce": 715816562, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "P0 \u00b7 M (thread)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "P0 \u00b7 M (thread)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t678308425", + "type": "text", + "x": 55, + "y": 233, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1187076968, + "versionNonce": 996360570, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local run queue", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local run queue", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r699551793", + "type": "rectangle", + "x": 210, + "y": 205, + "width": 150, + "height": 58, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1177944064, + "versionNonce": 1161743423, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t186763957", + "type": "text", + "x": 220, + "y": 213, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1649440166, + "versionNonce": 1910864960, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "P1 \u00b7 M (thread)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "P1 \u00b7 M (thread)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1589855913", + "type": "text", + "x": 220, + "y": 233, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 231134339, + "versionNonce": 1718947753, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "local run queue", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "local run queue", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r552355512", + "type": "rectangle", + "x": 375, + "y": 205, + "width": 120, + "height": 58, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 321572570, + "versionNonce": 1465281592, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1205790282", + "type": "text", + "x": 385, + "y": 213, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1670860499, + "versionNonce": 1148361728, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\u2026 P15", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\u2026 P15", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t364864236", + "type": "text", + "x": 385, + "y": 233, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1705749874, + "versionNonce": 1254938849, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(= 16 cores)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(= 16 cores)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1536666937", + "type": "rectangle", + "x": 620, + "y": 200, + "width": 270, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1928744150, + "versionNonce": 1945914692, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1385999043", + "type": "text", + "x": 630, + "y": 208, + "width": 231.00000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1513301325, + "versionNonce": 38092464, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "netpoller \u2014 one epoll instance", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "netpoller \u2014 one epoll instance", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t418294803", + "type": "text", + "x": 630, + "y": 228, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1561813506, + "versionNonce": 992490817, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_wait for the whole runtime", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_wait for the whole runtime", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r537358045", + "type": "rectangle", + "x": 45, + "y": 337, + "width": 330, + "height": 48, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1545427801, + "versionNonce": 523475043, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t494362885", + "type": "text", + "x": 55, + "y": 345, + "width": 354.20000000000005, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1624245122, + "versionNonce": 1885143501, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "async preemption: runtime sends SIGURG to an M", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "async preemption: runtime sends SIGURG to an M", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t893239889", + "type": "text", + "x": 55, + "y": 365, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1618648678, + "versionNonce": 990594573, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\u2192 a G running >10 ms yields", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\u2192 a G running >10 ms yields", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r381331092", + "type": "rectangle", + "x": 620, + "y": 337, + "width": 270, + "height": 48, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 729765426, + "versionNonce": 973554423, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1645286521", + "type": "text", + "x": 630, + "y": 345, + "width": 146.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1345208129, + "versionNonce": 1615638071, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll interest list", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll interest list", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1509616109", + "type": "text", + "x": 630, + "y": 365, + "width": 235.95000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 478848200, + "versionNonce": 1134526849, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "socket readable \u2192 G back to a run queue", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "socket readable \u2192 G back to a run queue", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1406855398", + "type": "arrow", + "x": 84, + "y": 128, + "width": 26, + "height": 77, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 949010988, + "versionNonce": 1130614684, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 26, + 77 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t958356772", + "type": "text", + "x": 97.0, + "y": 152.5, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1584992127, + "versionNonce": 1980568902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "run", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "run", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1764894745", + "type": "arrow", + "x": 172, + "y": 128, + "width": 22, + "height": 77, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 856112909, + "versionNonce": 1857165759, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -22, + 77 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1909143444", + "type": "arrow", + "x": 260, + "y": 128, + "width": 10, + "height": 77, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1184975442, + "versionNonce": 756380625, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 10, + 77 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1343901741", + "type": "arrow", + "x": 700, + "y": 132, + "width": 20, + "height": 68, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1679773407, + "versionNonce": 150536424, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 20, + 68 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t330117964", + "type": "text", + "x": 710.0, + "y": 152.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 6894, + "versionNonce": 1643169714, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "park", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "park", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a280576901", + "type": "arrow", + "x": 770, + "y": 266, + "width": 0, + "height": 71, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1286028897, + "versionNonce": 726308641, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 71 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t158293146", + "type": "text", + "x": 770.0, + "y": 287.5, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 344164335, + "versionNonce": 448482541, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "register fd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "register fd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1105050851", + "type": "arrow", + "x": 700, + "y": 337, + "width": 160, + "height": 205, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 493482153, + "versionNonce": 897309099, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -160, + -205 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t624899842", + "type": "text", + "x": 620.0, + "y": 220.5, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1977311202, + "versionNonce": 1063714314, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ready \u2192 runnable", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ready \u2192 runnable", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1190966231", + "type": "text", + "x": 40, + "y": 293, + "width": 627.0, + "height": 12.5, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1190123972, + "versionNonce": 1293593860, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no rewrite for async: a goroutine already suspends on I/O the way a coroutine does \u2014 the runtime hid it since 2012", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no rewrite for async: a goroutine already suspends on I/O the way a coroutine does \u2014 the runtime hid it since 2012", + "lineHeight": 1.25, + "baseline": 10 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/27-gmp-scheduler.svg b/assets/diagrams/27-gmp-scheduler.svg new file mode 100644 index 0000000..fe0cbdf --- /dev/null +++ b/assets/diagrams/27-gmp-scheduler.svg @@ -0,0 +1,57 @@ + + + +goroutines (G) — one per connection, ~2 KB growable stacks + +scheduler — GOMAXPROCS=16 logical processors (P), each bound to an OS thread (M) + +kernel — netpoller (epoll) and OS threads + +G +conn 1 + +G +conn 2 + +G +conn 3 + +… +G × N + +G blocked in recv() +parked on the netpoller, not on an M + +P0 · M (thread) +local run queue + +P1 · M (thread) +local run queue + +… P15 +(= 16 cores) + +netpoller — one epoll instance +epoll_wait for the whole runtime + +async preemption: runtime sends SIGURG to an M +→ a G running >10 ms yields + +epoll interest list +socket readable → G back to a run queue + +run +run + + + +park +park + +register fd +register fd + +ready → runnable +ready → runnable +no rewrite for async: a goroutine already suspends on I/O the way a coroutine does — the runtime hid it since 2012 + \ No newline at end of file diff --git a/assets/diagrams/27-tokio-task-lifecycle.excalidraw b/assets/diagrams/27-tokio-task-lifecycle.excalidraw new file mode 100644 index 0000000..8fe53ab --- /dev/null +++ b/assets/diagrams/27-tokio-task-lifecycle.excalidraw @@ -0,0 +1,1390 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1030511802", + "type": "rectangle", + "x": 20, + "y": 40, + "width": 900, + "height": 125, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 780743082, + "versionNonce": 214288344, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t392588556", + "type": "text", + "x": 32, + "y": 46, + "width": 534.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 635076746, + "versionNonce": 1908461310, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tokio tasks \u2014 async fn per connection (compiled to a state machine, not a thread)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tokio tasks \u2014 async fn per connection (compiled to a state machine, not a thread)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r376315815", + "type": "rectangle", + "x": 20, + "y": 190, + "width": 900, + "height": 100, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1381892063, + "versionNonce": 322782799, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1407843782", + "type": "text", + "x": 32, + "y": 196, + "width": 415.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 881475493, + "versionNonce": 1390234627, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "16 tokio-rt-worker threads (= cores) \u2014 work-stealing schedulers", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "16 tokio-rt-worker threads (= cores) \u2014 work-stealing schedulers", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1229398839", + "type": "rectangle", + "x": 20, + "y": 315, + "width": 900, + "height": 90, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1816137834, + "versionNonce": 898846618, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t763364373", + "type": "text", + "x": 32, + "y": 321, + "width": 211.20000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1545986863, + "versionNonce": 417877785, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "mio reactor \u2014 one epoll instance", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "mio reactor \u2014 one epoll instance", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1928721105", + "type": "rectangle", + "x": 45, + "y": 74, + "width": 160, + "height": 62, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 815372335, + "versionNonce": 473483358, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1742142947", + "type": "text", + "x": 55, + "y": 82, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 880001181, + "versionNonce": 1016566590, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tokio::spawn", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tokio::spawn", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t327960459", + "type": "text", + "x": 55, + "y": 102, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1458775108, + "versionNonce": 1124996109, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "handle_conn(...)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "handle_conn(...)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1975850771", + "type": "rectangle", + "x": 245, + "y": 74, + "width": 170, + "height": 62, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1756434825, + "versionNonce": 386323912, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1920286874", + "type": "text", + "x": 255, + "y": 82, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1895083546, + "versionNonce": 859606562, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": ".await \u2192 poll(cx)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": ".await \u2192 poll(cx)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1772862301", + "type": "text", + "x": 255, + "y": 102, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1573143782, + "versionNonce": 1021775651, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Poll::Pending", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Poll::Pending", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1134756326", + "type": "rectangle", + "x": 455, + "y": 74, + "width": 170, + "height": 62, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1120045939, + "versionNonce": 637830459, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1551316243", + "type": "text", + "x": 465, + "y": 82, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1544540563, + "versionNonce": 1169354842, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "task parked", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "task parked", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t807371354", + "type": "text", + "x": 465, + "y": 102, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 552543009, + "versionNonce": 616576630, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Waker held by reactor", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Waker held by reactor", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1903996326", + "type": "rectangle", + "x": 700, + "y": 74, + "width": 190, + "height": 62, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 911811627, + "versionNonce": 959920066, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1630995245", + "type": "text", + "x": 710, + "y": 82, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 149479931, + "versionNonce": 753988668, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "poll(cx) again", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "poll(cx) again", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t695862224", + "type": "text", + "x": 710, + "y": 102, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 317781958, + "versionNonce": 996288832, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Poll::Ready \u2192 frame sent", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Poll::Ready \u2192 frame sent", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r852370070", + "type": "rectangle", + "x": 245, + "y": 210, + "width": 180, + "height": 58, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 823587618, + "versionNonce": 656538984, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1678012404", + "type": "text", + "x": 255, + "y": 218, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 134088171, + "versionNonce": 602474448, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "worker thread", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "worker thread", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t502158889", + "type": "text", + "x": 255, + "y": 238, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1202503456, + "versionNonce": 202643942, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "drives poll(); cx carries Waker", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "drives poll(); cx carries Waker", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r213461101", + "type": "rectangle", + "x": 470, + "y": 210, + "width": 200, + "height": 58, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1647127808, + "versionNonce": 1335518607, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t507586580", + "type": "text", + "x": 480, + "y": 218, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1245630268, + "versionNonce": 62803535, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "run queue", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "run queue", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1522109061", + "type": "text", + "x": 480, + "y": 238, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1685836861, + "versionNonce": 1410444064, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Waker::wake() re-enqueues task", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Waker::wake() re-enqueues task", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1550047884", + "type": "rectangle", + "x": 420, + "y": 335, + "width": 300, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 726986591, + "versionNonce": 341533752, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1072228260", + "type": "text", + "x": 430, + "y": 343, + "width": 184.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1788012807, + "versionNonce": 1379017837, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "epoll_wait \u2014 fd readable", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "epoll_wait \u2014 fd readable", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1511406214", + "type": "text", + "x": 430, + "y": 363, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1869060004, + "versionNonce": 1453642213, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "calls the registered Waker", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "calls the registered Waker", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a992823122", + "type": "arrow", + "x": 125, + "y": 136, + "width": 210, + "height": 74, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 378001670, + "versionNonce": 154858574, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 210, + 74 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t840174806", + "type": "text", + "x": 230.0, + "y": 159.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 769415781, + "versionNonce": 670506665, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "schedule", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "schedule", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a855437333", + "type": "arrow", + "x": 330, + "y": 136, + "width": 0, + "height": 74, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1029585679, + "versionNonce": 855726517, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 74 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t276375994", + "type": "text", + "x": 330.0, + "y": 159.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 278612183, + "versionNonce": 992235682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "poll", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "poll", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1545355356", + "type": "arrow", + "x": 425, + "y": 105, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 64191984, + "versionNonce": 270393727, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1210227276", + "type": "text", + "x": 440.0, + "y": 91.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1977413183, + "versionNonce": 710516113, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pending", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pending", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a48788306", + "type": "arrow", + "x": 468, + "y": 136, + "width": 8, + "height": 199, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1863210705, + "versionNonce": 767874601, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -8, + 199 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1305308518", + "type": "text", + "x": 464.0, + "y": 221.5, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 499326756, + "versionNonce": 1682280780, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "register Waker+fd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "register Waker+fd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1031221627", + "type": "arrow", + "x": 700, + "y": 335, + "width": 45, + "height": 67, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1308428033, + "versionNonce": 866493950, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -45, + -67 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1744808835", + "type": "text", + "x": 677.5, + "y": 287.5, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1176341881, + "versionNonce": 342971594, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wake()", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wake()", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a826663383", + "type": "arrow", + "x": 625, + "y": 210, + "width": 135, + "height": 74, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 515084959, + "versionNonce": 684498384, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 135, + -74 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t188423681", + "type": "text", + "x": 692.5, + "y": 159.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1426954162, + "versionNonce": 338527707, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "re-poll", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "re-poll", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t700070921", + "type": "text", + "x": 40, + "y": 290, + "width": 693.0, + "height": 12.5, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 829132514, + "versionNonce": 647194604, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "a task is a heap object the size of its locals \u2014 20k tasks share 16 threads; a Waker is the callback that says 'poll me again'", + "fontSize": 10, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "a task is a heap object the size of its locals \u2014 20k tasks share 16 threads; a Waker is the callback that says 'poll me again'", + "lineHeight": 1.25, + "baseline": 10 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/27-tokio-task-lifecycle.svg b/assets/diagrams/27-tokio-task-lifecycle.svg new file mode 100644 index 0000000..3ecc1af --- /dev/null +++ b/assets/diagrams/27-tokio-task-lifecycle.svg @@ -0,0 +1,49 @@ + + + +tokio tasks — async fn per connection (compiled to a state machine, not a thread) + +16 tokio-rt-worker threads (= cores) — work-stealing schedulers + +mio reactor — one epoll instance + +tokio::spawn +handle_conn(...) + +.await → poll(cx) +Poll::Pending + +task parked +Waker held by reactor + +poll(cx) again +Poll::Ready → frame sent + +worker thread +drives poll(); cx carries Waker + +run queue +Waker::wake() re-enqueues task + +epoll_wait — fd readable +calls the registered Waker + +schedule +schedule + +poll +poll + +pending +pending + +register Waker+fd +register Waker+fd + +wake() +wake() + +re-poll +re-poll +a task is a heap object the size of its locals — 20k tasks share 16 threads; a Waker is the callback that says 'poll me again' + \ No newline at end of file diff --git a/assets/diagrams/README.md b/assets/diagrams/README.md index 1a245c9..6e18c6b 100644 --- a/assets/diagrams/README.md +++ b/assets/diagrams/README.md @@ -38,3 +38,19 @@ here as you add them: | `13-pid-reuse-race-pidfd` | 13 | The five-step pid-reuse race timeline — spawn, crash, reap, recycle, kill(2) hitting a stranger — above the four-step pidfd lane that pins identity to an fd, annotated with the real fdinfo Pid: and strace evidence from pmon v2 | | `18-pipe-internals` | 18 | One live pipe as the kernel sees it — write end fd 1 and read end fd 5 around the 16-slot ring of pages, with the measured 65536 B capacity, the F_SETPIPE_SZ page rounding, and what PIPE_BUF atomicity does and does not promise | | `18-log-topology` | 18 | pmon v4's full log pipeline — child stdio through two O_CLOEXEC pipes into the supervisor's poll loop and the framed log, then tail splicing file→FIFO to a detachable cat reader, annotated with the run's real fds, inodes, flags, and splice return values | +| `25-futex-fast-slow-path` | 25 | pthread mutex as a futex — the uncontended fast path (userspace CAS 0→1, no syscall) beside the contended slow path (FUTEX_WAIT parks the loser on a kernel wait queue keyed by (mm, &word), FUTEX_WAKE releases it), annotated with the measured 5,186→488,818 futex-call and 455→123,986 voluntary-context-switch scaling | +| `25-mpmc-queue` | 25 | workq's bounded MPMC job queue shared three ways — P producers push, one mutex + not_full/not_empty condvars (a buffered channel in Go) bound capacity K, C consumers pop into per-consumer accumulators XOR-folded after join into the P/C/cap-independent checksum 42b3746c6cee7465 | +| `26-spsc-ring` | 26 | spscring's Lamport SPSC ring — producer-owned tail and consumer-owned head, each Release store publishing the slot the other thread Acquire-loads, over a buf[K] slot row with occupancy = tail - head | +| `26-memory-ordering` | 26 | The same hand-off under three orderings — seq-cst (Go, XCHGQ), acquire/release (C++/Rust head/tail, plain MOV), relaxed (Rust slots, plain MOV) — and what each compiles to on x86-64 TSO | +| `26-false-sharing` | 26 | head and tail sharing one 64 B cache line ping-ponging between two cores versus padded onto separate lines, annotated with the real ~128 M vs ~108 M L1-dcache-load-misses and 4.64 vs 6.20 Mops | +| `21-server-lifecycle` | 21 | chatterd serve's listener path — socket → SO_REUSEADDR → bind → listen(128) → the accept4 loop spawning one thread per accepted fd (4, 5), with the SIGINT shutdown path that flips g_stop and closes the listener | +| `21-frame-format` | 21 | the length-prefixed wire frame byte for byte — a little-endian u32 length (0f 00 00 00 = 15) then name·NUL·text for alice sending "hello bob", plus the JOIN vs CHAT distinction and the len bounds | +| `23-two-vm-topology` | 23 | The two-guest lab topology on one virbr0 L2 segment — systems-target (192.168.124.7) and systems-peer (192.168.124.95) both joined to group 239.23.7.1:51824 with TTL 1, the two beacons (35 B and 34 B) crossing the bridge, and the TCP :9241 chat-frame bridge between them | +| `23-discovery-sequence` | 23 | One peer's discovery path — the connectionless UDP band (sendto beacon, recv one datagram, parse + dedup by name) bridging down into the connection-oriented TCP band (connect with retry, the reused ch21 length-prefixed frame each way, the printed discovered/says lines) | +| `24-clock-taxonomy` | 24 | The four POSIX clocks clock_gettime can read — REALTIME (steppable wall time), MONOTONIC (steady), BOOTTIME (monotonic + suspend), PROCESS_CPUTIME_ID — over the monotonic-vs-wall bug: an NTP step rewinds a wall-clock deadline while a CLOCK_MONOTONIC deadline fires on real silence | +| `24-heartbeat-timeout-seq` | 24 | chatterd v3's heartbeat/deadline/reconnect sequence — HELLO link, PING every 300 ms resetting the monotonic deadline, the listener killed, the deadline firing "peer A timed out", then the jittered exponential backoff (107, 245, 712 ms from the live run) | +| `27-coroutine-suspension` | 27 | chatterd v4's C++20 coroutine engine — one heap-allocated `connection()` frame per client suspending on `co_await ReadAwaitable` when `recv` hits EAGAIN, `wait_readable` parking the `std::coroutine_handle` in `waiters_[fd]` with `EPOLLIN|EPOLLONESHOT`, and the reactor's single `epoll_wait(-1)` resuming the exact handle on readiness — proving suspension is a return, not a blocked thread | +| `27-gmp-scheduler` | 27 | Go's G-M-P runtime for the same daemon — goroutines (one per connection) multiplexed by GOMAXPROCS=16 logical processors (P) bound to OS threads (M), the netpoller (one runtime epoll instance) parking a G blocked in `recv` and handing it back runnable on readiness, and SIGURG async preemption retiring a G that runs past 10 ms — why Go needed no async rewrite | +| `27-tokio-task-lifecycle` | 27 | tokio's task/Waker cycle for the Rust async engine — `tokio::spawn`ed `async fn` tasks polled by 16 `tokio-rt-worker` threads (= cores), `.await` returning `Poll::Pending` and registering a Waker with the mio reactor, `epoll_wait` calling that Waker on fd readiness to re-enqueue the task, and the re-poll to `Poll::Ready` — 20k tasks over 16 threads | +| `22-connection-state-machine` | 22 | chatterd v1's epoll engine per-connection state machine — recv() drained to EAGAIN into inbuf, take_frame reassembling whole frames, broadcast appending to every conn's outbuf, and the send()/EAGAIN backpressure loop that arms EPOLLOUT and resumes flush() on the next epoll_wait wakeup | +| `22-netpoller-vs-epoll` | 22 | the chapter's key comparison — C++/Rust one thread over a hand-written epoll interest list (4 epoll_wait calls, explicit outbuf+EPOLLOUT backpressure) beside Go's goroutine-per-conn with the runtime netpoller hidden underneath (165 edge-triggered epoll_pwait calls, ~7 OS threads, blocking-channel backpressure) | diff --git a/examples/20-shared-memory-coordination/verify.lua b/examples/20-shared-memory-coordination/verify.lua index c64d587..807bb2a 100644 --- a/examples/20-shared-memory-coordination/verify.lua +++ b/examples/20-shared-memory-coordination/verify.lua @@ -85,18 +85,23 @@ done ]] .. demo .. [[ watch @WD@/kv.shm --events ]] .. updates .. [[ \ > @WD@/watch.out 2>&1 & wpid=$! -sleep 0.25 -# Bracket trick so this shell's own command line never matches the pattern. -sprocs=$(pgrep -c -f "/ap[p] serve" || true) -wprocs=$(pgrep -c -f "/ap[p] watch" || true) +# Confirm both processes we spawned are simultaneously alive mid-run. Test the +# exact PIDs with kill -0 rather than pgrep -f on a command-line pattern: -f +# matches substrings of every process's argv, so a parent/wrapper shell whose +# script text contains "app serve" would inflate the count. kill -0 is exact. +both_alive=0 +for i in $(seq 1 60); do + if kill -0 "$spid" 2>/dev/null && kill -0 "$wpid" 2>/dev/null; then both_alive=1; break; fi + sleep 0.02 +done s_rc=0; wait $spid || s_rc=$? w_rc=0; wait $wpid || w_rc=$? -echo "sprocs=$sprocs wprocs=$wprocs s_rc=$s_rc w_rc=$w_rc" +echo "both_alive=$both_alive s_rc=$s_rc w_rc=$w_rc" echo "--- serve ---"; cat @WD@/serve.out echo "--- watch ---"; cat @WD@/watch.out ]]) checks.expect_exit(sw, 0, lang .. ": serve+watch harness exits 0") -checks.expect_match(sw.out, "sprocs=1 wprocs=1", lang .. ": one serve and one watch process alive mid-run") +checks.expect_match(sw.out, "both_alive=1", lang .. ": serve and watch processes both alive mid-run") checks.expect_match(sw.out, "s_rc=0 w_rc=0", lang .. ": both processes exit 0") checks.expect_match(sw.out, "serve: file=" .. wd .. "/kv%.shm updates=6 interval_ms=80", lang .. ": serve announces its parameters") diff --git a/examples/21-tcp-sockets/README.md b/examples/21-tcp-sockets/README.md new file mode 100644 index 0000000..fcc1ee3 --- /dev/null +++ b/examples/21-tcp-sockets/README.md @@ -0,0 +1,179 @@ +# 21-tcp-sockets — chatterd v0 + +`chatterd` is the book's peer-to-peer chat program. This first cut is a +thread-per-connection TCP **chat room**: one binary is both the daemon and the +client. + +``` +# terminal 1 — the room +$ ./demo.sh cpp run serve --port 9000 +chatterd: listening on 127.0.0.1:9000 +chatterd: bob joined +chatterd: alice joined + +# terminal 2 — bob +$ ./demo.sh cpp run chatctl --port 9000 --name bob +server: bob joined +server: alice joined +alice: hello bob + +# terminal 3 — alice +$ ./demo.sh cpp run chatctl --port 9000 --name alice +server: alice joined +hello bob # typed on stdin; sent to everyone else +``` + +`serve` accepts connections and runs each client on its own thread (C++ +`std::jthread`, a Go goroutine, a Rust `std::thread`). Every message a client +sends is broadcast to all **other** clients; a join notice is broadcast to +**all** members, the newcomer included. `chatctl` connects, announces its name, +then sends one frame per line of stdin while a reader thread prints everything +that arrives as `: `. + +## The wire protocol (fixed for the whole book) + +This chapter introduces **the** chatterd chat frame: chapters 22, 24, and 27 +reuse this exact header and only ever *add* frame types, never change it. A +frame that a `chatctl` from this chapter reads or writes is byte-identical to +the one a v1 (ch22), v3 (ch24), or v4 (ch27) build would produce. + +``` + byte: 0 1 2 3 4 5 6 6+L-1 + +----+----+------+------+----+----+---------------------------+ + | 'C'| 'H'| ver | type | length | L bytes of UTF-8 | + | | | 0x01 | | (u16BE) | payload | + +----+----+------+------+----+----+---------------------------+ +``` + +- **magic** — the two bytes `0x43 0x48` (ASCII `"CH"`). Any other value means + "not a chatterd frame"; the reader closes the connection. +- **version** — one byte, `0x01`. This chapter is version 0 of the *program* + but the wire's version byte has always been `1`; only the frame *header* + would ever bump it, and it never has. +- **type** — one byte selecting the payload's shape. This chapter uses three + of the five book-wide types: + + | Type | Value | Payload | Who sends it | Meaning | + |---|---|---|---|---| + | **JOIN** | `1` | `name` | client, once, right after connecting | "I am `name`" | + | **MSG** | `2` | `text` | client, per stdin line | "I say `text`" | + | **DELIVER** | `3` | `name` `00` `text` | server, to every client | a broadcast | + + `WELCOME` (`4`) and `PING` (`5`) are reserved for chapters 22 and 24; this + program never sends either, and a `chatctl` reader here skips any frame type + it does not recognize instead of erroring out — which is exactly what lets a + ch21 client stay connected to a newer server that does send them. +- **length** — a 2-byte unsigned payload length in **big-endian** order, + `0..65535`. It counts the payload only, never the 6 header bytes. +- **payload** — `length` bytes of UTF-8, shaped per the `type` above. A + `DELIVER` payload's `name` and `text` are separated by a single `0x00` NUL + byte; neither field may itself contain a NUL. + +The server itself never invents new frame *types*. A client's `JOIN` sets its +name and a `MSG` carries one line of chat; the server relays every `MSG` as a +`DELIVER` and synthesises **notices** as `DELIVER` frames whose sender name is +the reserved string `server`: + +- on a join → `DELIVER` `server` `00` ` joined`, sent to every member. +- on a disconnect → `DELIVER` `server` `00` ` left`, sent to the rest. + +So a client that reads a `DELIVER` payload `server\0alice joined` prints +`server: alice joined`, and one that reads `alice\0hello bob` prints +`alice: hello bob`. That is the whole protocol for this chapter; chapters +22–24 and 27 add `WELCOME`/`PING` and new engines around it, never a new +header. + +### Worked bytes + +`alice`'s JOIN frame (payload `alice`, 5 bytes): + +``` +43 48 01 01 00 05 61 6c 69 63 65 +\___/ ^^ ^^ \___/ \____________/ +magic ver type len=5 "alice" + "CH" JOIN +``` + +`alice` sending `hello bob` as a MSG frame (payload `hello bob`, 9 bytes): + +``` +43 48 01 02 00 09 68 65 6c 6c 6f 20 62 6f 62 +\___/ ^^ ^^ \___/ \______________________/ +magic ver type len=9 "hello bob" + "CH" MSG +``` + +The server's broadcast of that message as a DELIVER frame (payload +`alice\0hello bob`, 15 bytes): + +``` +43 48 01 03 00 0f 61 6c 69 63 65 00 68 65 6c 6c 6f 20 62 6f 62 +\___/ ^^ ^^ \___/ \____________/ ^^ \______________________/ +magic ver type len=15 "alice" NUL "hello bob" + "CH" DELIVER +``` + +## The CLI + +``` +chatterd serve --port PORT [--host HOST] # default host 127.0.0.1 +chatterd chatctl --port PORT --name NAME [--host HOST] +``` + +Exit codes are the same across all three languages: `0` clean, `1` a runtime +error (bind refused, connect refused, …), `2` a usage error. `serve` runs until +**SIGINT**, then closes the listener, lets the connection threads wind down, and +exits `0`. `chatctl` runs until stdin reaches EOF. Empty stdin lines are +dropped so a client never emits an empty-text frame (that would read as a +second JOIN). + +## The three implementations + +Identical CLI, identical stdout/stderr shapes, identical exit codes — one +`verify.lua` covers all three. + +| | connection model | listener shutdown | fd ownership | +|---|---|---|---| +| **C++23** | one `std::jthread` per client; a `std::stop_callback` shuts the socket down on stop so the blocking read unwinds | `std::atomic` set by the SIGINT handler, polled by a `poll(2)` that SIGINT interrupts | RAII `Socket` owns every fd; no naked `close` | +| **Go 1.26** | one goroutine per client + a per-client writer goroutine draining a channel; a `select` over send/`done`/`ctx` never wedges a broadcaster | a `context` cancelled on SIGINT; `ln.Close()` unblocks `Accept` | `net.Conn` / `net.Listener` | +| **Rust 2024** | one `std::thread` per client; broadcasts write to `try_clone`d `TcpStream`s under a `Mutex` | `AtomicBool` set by the SIGINT handler, polled by a non-blocking `accept` loop | listener built via `nix` and taken as an `OwnedFd`; `TcpStream` owns each connection | + +No async runtime here on purpose — this chapter is about blocking sockets and +threads. Chapter 27 is where `chatterd` meets tokio. + +## Gotchas worth remembering + +- **Length prefix, not newline.** TCP is a byte stream with no message + boundaries; a single `read` may return half a frame or two frames. Every + reader loops until it has all 6 header bytes, then exactly `length` payload + bytes. `\0` inside a `DELIVER` payload is why a newline delimiter would not + do. +- **Broadcast excludes the sender; join notices do not.** A speaker already + sees what they typed, so their own message is not echoed back. A join, + though, goes to every member — which is why both clients in a two-party room + each see a join line. +- **SIGINT and a blocking accept.** A flag set in the handler is invisible to a + thread parked in `accept`. C++ interrupts `poll` with the signal; Go closes + the listener out from under `Accept`; Rust makes `accept` non-blocking and + polls the flag. All three reach the same observable end: the listener closes + and the process exits `0`. +- **`SO_REUSEADDR`.** Set on the listener so a fresh `serve` can rebind the port + while a previous run's connections linger in `TIME_WAIT`. + +## Files + +``` +demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +verify.lua # behavioral checks (delivered lines, joins, clean SIGINT exit) +cpp/ go/ rust/ # one chatterd each; binary always ./…/app; demo.sh contract identical +``` + +## Verification + +`python3 scripts/test-all-examples.py --only 21-tcp-sockets` builds all three +and runs `verify.lua` per language. The checks are behavioral: two real +`chatctl` clients are driven over loopback through scripted stdin fifos, and the +test asserts the **actually delivered lines** — bob receives `alice: hello`, +alice sees no echo of her own message, both clients see a join notice from the +`server` sender, the daemon logs each join, and SIGINT leaves the listener +closed with exit `0`. diff --git a/examples/21-tcp-sockets/cpp/.clang-format b/examples/21-tcp-sockets/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/21-tcp-sockets/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/21-tcp-sockets/cpp/.clang-tidy b/examples/21-tcp-sockets/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/21-tcp-sockets/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/21-tcp-sockets/cpp/CMakeLists.txt b/examples/21-tcp-sockets/cpp/CMakeLists.txt new file mode 100644 index 0000000..1304d26 --- /dev/null +++ b/examples/21-tcp-sockets/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/21-tcp-sockets/cpp/CMakePresets.json b/examples/21-tcp-sockets/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/21-tcp-sockets/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/21-tcp-sockets/cpp/demo.sh b/examples/21-tcp-sockets/cpp/demo.sh new file mode 100755 index 0000000..81d3103 --- /dev/null +++ b/examples/21-tcp-sockets/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/21-tcp-sockets/cpp/src/main.cpp b/examples/21-tcp-sockets/cpp/src/main.cpp new file mode 100644 index 0000000..4ffab88 --- /dev/null +++ b/examples/21-tcp-sockets/cpp/src/main.cpp @@ -0,0 +1,514 @@ +// chatterd v0 — a thread-per-connection TCP chat room. +// +// One binary, two subcommands: +// +// chatterd serve --port P [--host H] +// Accept TCP connections; each client runs on its own std::jthread. +// Every chat frame received is broadcast to all OTHER clients; a join +// notice is broadcast to ALL clients (including the newcomer). SIGINT +// closes the listener and exits 0. +// +// chatterd chatctl --port P --name N [--host H] +// Connect, announce N with a join frame, then send one frame per line +// of stdin. A reader jthread prints every received frame as +// ": ". +// +// THE WIRE PROTOCOL (fixed for the whole book, ch21-ch27). Every message is +// the canonical chatterd CHAT FRAME: +// +// +-------+---------+------+----------------+-------------------+ +// | magic | version | type | length (u16BE) | payload (UTF-8) | +// | 2B | 1B | 1B | 2B | `length` bytes | +// +-------+---------+------+----------------+-------------------+ +// +// magic is the two bytes 0x43 0x48 ("CH"); version is 0x01. `length` counts +// the payload only (0..65535), never the 6-byte header. This chapter (v0) +// uses three of the five frame types: +// +// JOIN (1) payload = name client -> server, once, at connect +// MSG (2) payload = text client -> server, per stdin line +// DELIVER (3) payload = name 0x00 text server -> all clients, a broadcast +// +// WELCOME (4) and PING (5) are reserved for later chapters (ch22, ch24); this +// program never sends them, and a chatctl reader here ignores any frame type +// it doesn't recognize so it stays interoperable with newer servers. The +// server relays a client's MSG as a DELIVER and synthesises join/leave +// DELIVER notices from the reserved sender name "server". +// +// C++23: a RAII Socket owns every fd (no naked close); std::expected carries +// each fallible step; std::jthread + std::stop_token drive the connection +// threads; a std::atomic flag plus a SIGINT-interruptible poll(2) shut the +// listener down. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace { + +// The canonical chatterd chat frame header: magic "CH", version 1. +constexpr std::uint8_t kMagic0 = 0x43; +constexpr std::uint8_t kMagic1 = 0x48; +constexpr std::uint8_t kWireVersion = 0x01; +constexpr std::size_t kHeaderSize = 6; // magic(2) + version(1) + type(1) + length(2) + +enum FrameType : std::uint8_t { + kJoin = 1, + kMsg = 2, + kDeliver = 3, + kWelcome = 4, + kPing = 5, +}; + +[[nodiscard]] std::string errno_text() { + return std::error_code{errno, std::system_category()}.message(); +} + +template +using Result = std::expected; + +// --------------------------------------------------------------------------- +// RAII Socket — the sole owner of every descriptor in this program. +// --------------------------------------------------------------------------- + +class Socket { +public: + Socket() = default; + explicit Socket(int fd) : fd_{fd} {} + Socket(const Socket&) = delete; + Socket& operator=(const Socket&) = delete; + Socket(Socket&& o) noexcept : fd_{std::exchange(o.fd_, -1)} {} + Socket& operator=(Socket&& o) noexcept { + if (this != &o) { + reset(); + fd_ = std::exchange(o.fd_, -1); + } + return *this; + } + ~Socket() { reset(); } + + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + } + fd_ = -1; + } + [[nodiscard]] int get() const noexcept { return fd_; } + [[nodiscard]] bool valid() const noexcept { return fd_ >= 0; } + +private: + int fd_ = -1; +}; + +// --------------------------------------------------------------------------- +// The canonical chat frame: magic + version + type + u16BE length + payload. +// --------------------------------------------------------------------------- + +struct RawFrame { + std::uint8_t type; + std::string payload; +}; + +// Read exactly n bytes; false on EOF or hard error (EINTR is retried). +bool read_full(int fd, void* buf, std::size_t n) { + auto* p = static_cast(buf); + std::size_t off = 0; + while (off < n) { + ssize_t r = ::read(fd, p + off, n - off); + if (r < 0) { + if (errno == EINTR) { + continue; + } + return false; + } + if (r == 0) { + return false; // peer closed + } + off += static_cast(r); + } + return true; +} + +void write_all(int fd, std::string_view text) { + std::size_t off = 0; + while (off < text.size()) { + ssize_t n = ::write(fd, text.data() + off, text.size() - off); + if (n < 0) { + if (errno == EINTR) { + continue; + } + return; // peer gone; nothing sensible to do here + } + off += static_cast(n); + } +} + +std::string build_frame(std::uint8_t type, std::string_view payload) { + // payload.size() is always <= 65535 for the frames this program builds + // (names/messages typed on a terminal); the length field is u16BE. + auto len = static_cast(payload.size()); + std::string frame; + frame.reserve(kHeaderSize + payload.size()); + frame.push_back(static_cast(kMagic0)); + frame.push_back(static_cast(kMagic1)); + frame.push_back(static_cast(kWireVersion)); + frame.push_back(static_cast(type)); + frame.push_back(static_cast((len >> 8) & 0xff)); + frame.push_back(static_cast(len & 0xff)); + frame.append(payload); + return frame; +} + +std::string build_join(std::string_view name) { return build_frame(kJoin, name); } +std::string build_msg(std::string_view text) { return build_frame(kMsg, text); } +std::string build_deliver(std::string_view name, std::string_view text) { + std::string payload; + payload.reserve(name.size() + 1 + text.size()); + payload.append(name); + payload.push_back('\0'); + payload.append(text); + return build_frame(kDeliver, payload); +} + +std::optional read_frame(int fd) { + std::uint8_t hdr[kHeaderSize]; + if (!read_full(fd, hdr, kHeaderSize)) { + return std::nullopt; + } + if (hdr[0] != kMagic0 || hdr[1] != kMagic1 || hdr[2] != kWireVersion) { + return std::nullopt; // wrong magic/version: not a chatterd frame + } + std::uint8_t type = hdr[3]; + std::uint16_t len = (static_cast(hdr[4]) << 8) | hdr[5]; + std::string payload(len, '\0'); + if (len > 0 && !read_full(fd, payload.data(), len)) { + return std::nullopt; + } + return RawFrame{type, std::move(payload)}; +} + +// --------------------------------------------------------------------------- +// The hub: every connected client, keyed by id. +// --------------------------------------------------------------------------- + +constexpr int kBroadcastAll = -1; + +class Hub { +public: + void add(int id, int fd) { + std::lock_guard lk(m_); + clients_.push_back({id, std::string{}, fd}); + } + void set_name(int id, std::string_view name) { + std::lock_guard lk(m_); + if (auto* c = find(id)) { + c->name = name; + } + } + // Remove a client and return the name it last carried (empty if unnamed). + std::string remove(int id) { + std::lock_guard lk(m_); + for (auto it = clients_.begin(); it != clients_.end(); ++it) { + if (it->id == id) { + std::string name = it->name; + clients_.erase(it); + return name; + } + } + return {}; + } + // Write frame to every client; when except >= 0 that id is skipped. + void broadcast(std::string_view frame, int except) { + std::lock_guard lk(m_); + for (const auto& c : clients_) { + if (c.id != except) { + write_all(c.fd, frame); + } + } + } + +private: + struct Client { + int id; + std::string name; + int fd; + }; + Client* find(int id) { + auto it = std::ranges::find(clients_, id, &Client::id); + return it == clients_.end() ? nullptr : &*it; + } + + std::mutex m_; + std::vector clients_; +}; + +// One connection's lifetime. The stop_callback shuts the socket down when the +// jthread is asked to stop, which unblocks the read below during shutdown. +void serve_client(std::stop_token st, Socket conn, int id, Hub& hub) { + const int fd = conn.get(); + std::stop_callback on_stop(st, [fd] { ::shutdown(fd, SHUT_RDWR); }); + + hub.add(id, fd); + std::string my_name; + for (;;) { + auto raw = read_frame(fd); + if (!raw) { + break; + } + if (raw->type == kJoin) { + if (raw->payload.empty()) { + break; // malformed: JOIN carries no name + } + my_name = raw->payload; + hub.set_name(id, my_name); + std::println(stderr, "chatterd: {} joined", my_name); + hub.broadcast(build_deliver("server", std::format("{} joined", my_name)), + kBroadcastAll); + } else if (raw->type == kMsg) { + if (my_name.empty()) { + continue; // MSG before JOIN: nothing sensible to attribute it to + } + hub.broadcast(build_deliver(my_name, raw->payload), id); + } + // Any other type (DELIVER/WELCOME/PING) is not sent by a client in this + // chapter; ignore it rather than tearing the connection down. + } + + std::string name = hub.remove(id); + if (!name.empty()) { + std::println(stderr, "chatterd: {} left", name); + hub.broadcast(build_deliver("server", std::format("{} left", name)), kBroadcastAll); + } +} + +// --------------------------------------------------------------------------- +// serve +// --------------------------------------------------------------------------- + +std::atomic g_stop{false}; + +extern "C" void on_sigint(int) { g_stop.store(true, std::memory_order_relaxed); } + +Result listen_tcp(const std::string& host, std::uint16_t port) { + Socket sock{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!sock.valid()) { + return std::unexpected(std::format("socket: {}", errno_text())); + } + int one = 1; + ::setsockopt(sock.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(std::format("{}: not an IPv4 address", host)); + } + if (::bind(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected(std::format("bind {}:{}: {}", host, port, errno_text())); + } + if (::listen(sock.get(), 128) < 0) { + return std::unexpected(std::format("listen: {}", errno_text())); + } + return sock; +} + +int cmd_serve(const std::string& host, std::uint16_t port) { + auto lsock = listen_tcp(host, port); + if (!lsock) { + std::println(stderr, "chatterd: error: {}", lsock.error()); + return 1; + } + std::signal(SIGINT, on_sigint); + std::println(stderr, "chatterd: listening on {}:{}", host, port); + + Hub hub; + std::vector threads; + int next_id = 0; + + while (!g_stop.load(std::memory_order_relaxed)) { + pollfd pfd{lsock->get(), POLLIN, 0}; + int pr = ::poll(&pfd, 1, 200); // wake at least every 200 ms to re-check g_stop + if (pr < 0) { + if (errno == EINTR) { + continue; // SIGINT lands here; the loop re-checks g_stop + } + std::println(stderr, "chatterd: error: poll: {}", errno_text()); + return 1; + } + if (pr == 0 || !(pfd.revents & POLLIN)) { + continue; + } + Socket conn{::accept4(lsock->get(), nullptr, nullptr, SOCK_CLOEXEC)}; + if (!conn.valid()) { + if (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK) { + continue; + } + std::println(stderr, "chatterd: error: accept: {}", errno_text()); + return 1; + } + int id = ++next_id; + threads.emplace_back( + [&hub, id, c = std::move(conn)](std::stop_token st) mutable { + serve_client(st, std::move(c), id, hub); + }); + } + + std::println(stderr, "chatterd: shutting down"); + // Returning destroys `threads`: each jthread's destructor requests stop + // (firing the stop_callback that shuts the socket) and joins. The listener + // Socket closes last, when lsock goes out of scope. + for (auto& t : threads) { + t.request_stop(); + } + return 0; +} + +// --------------------------------------------------------------------------- +// chatctl +// --------------------------------------------------------------------------- + +Result connect_tcp(const std::string& host, std::uint16_t port) { + Socket sock{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!sock.valid()) { + return std::unexpected(std::format("socket: {}", errno_text())); + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(std::format("{}: not an IPv4 address", host)); + } + if (::connect(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected(std::format("connect {}:{}: {}", host, port, errno_text())); + } + return sock; +} + +int cmd_chatctl(const std::string& host, std::uint16_t port, const std::string& name) { + auto conn = connect_tcp(host, port); + if (!conn) { + std::println(stderr, "chatctl: error: {}", conn.error()); + return 1; + } + // Announce ourselves: a JOIN frame's payload is just our name. + write_all(conn->get(), build_join(name)); + + std::jthread reader([fd = conn->get()](std::stop_token) { + for (;;) { + auto raw = read_frame(fd); + if (!raw) { + break; + } + if (raw->type != kDeliver) { + continue; // WELCOME/PING/etc. are for later chapters; ignore them + } + auto nul = raw->payload.find('\0'); + if (nul == std::string::npos) { + continue; // malformed DELIVER; skip rather than crash + } + std::println("{}: {}", raw->payload.substr(0, nul), raw->payload.substr(nul + 1)); + std::fflush(stdout); // stdout is fully buffered to a pipe/file + } + }); + + std::string line; + while (std::getline(std::cin, line)) { + if (line.empty()) { + continue; // never emit an empty-text frame; that would read as a join + } + write_all(conn->get(), build_msg(line)); + } + // stdin EOF: tear the socket down so the reader sees EOF and joins. + ::shutdown(conn->get(), SHUT_RDWR); + return 0; +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +int usage() { + std::println(stderr, "usage: chatterd serve --port PORT [--host HOST]"); + std::println(stderr, " chatterd chatctl --port PORT --name NAME [--host HOST]"); + return 2; +} + +std::optional parse_port(const std::string& s) { + try { + std::size_t pos = 0; + int v = std::stoi(s, &pos); + if (pos != s.size() || v < 1 || v > 65535) { + return std::nullopt; + } + return static_cast(v); + } catch (...) { + return std::nullopt; + } +} + +} // namespace + +int main(int argc, char** argv) { + std::vector args(argv + 1, argv + argc); + if (args.empty()) { + return usage(); + } + + const std::string sub = args[0]; + if (sub != "serve" && sub != "chatctl") { + return usage(); + } + + std::string host = "127.0.0.1"; + std::string port_str; + std::string name; + for (std::size_t i = 1; i < args.size(); ++i) { + if (args[i] == "--port" && i + 1 < args.size()) { + port_str = args[++i]; + } else if (args[i] == "--host" && i + 1 < args.size()) { + host = args[++i]; + } else if (args[i] == "--name" && i + 1 < args.size()) { + name = args[++i]; + } else { + return usage(); + } + } + if (port_str.empty()) { + return usage(); + } + auto port = parse_port(port_str); + if (!port) { + return usage(); + } + + if (sub == "serve") { + return cmd_serve(host, *port); + } + if (name.empty()) { + return usage(); + } + return cmd_chatctl(host, *port, name); +} diff --git a/examples/21-tcp-sockets/demo.sh b/examples/21-tcp-sockets/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/21-tcp-sockets/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/21-tcp-sockets/go/demo.sh b/examples/21-tcp-sockets/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/21-tcp-sockets/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/21-tcp-sockets/go/go.mod b/examples/21-tcp-sockets/go/go.mod new file mode 100644 index 0000000..b9d5bae --- /dev/null +++ b/examples/21-tcp-sockets/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/21-tcp-sockets/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/21-tcp-sockets/go/go.sum b/examples/21-tcp-sockets/go/go.sum new file mode 100644 index 0000000..e69de29 diff --git a/examples/21-tcp-sockets/go/main.go b/examples/21-tcp-sockets/go/main.go new file mode 100644 index 0000000..f40b636 --- /dev/null +++ b/examples/21-tcp-sockets/go/main.go @@ -0,0 +1,415 @@ +// chatterd v0 — a thread-per-connection TCP chat room. +// +// One binary, two subcommands: +// +// chatterd serve --port P [--host H] +// Accept TCP connections; each client runs on its own goroutine. Every +// chat frame received is broadcast to all OTHER clients; a join notice +// is broadcast to ALL clients (including the newcomer). SIGINT closes +// the listener and exits 0. +// +// chatterd chatctl --port P --name N [--host H] +// Connect, announce N with a join frame, then send one frame per line +// of stdin. A reader goroutine prints every received frame as +// ": ". +// +// THE WIRE PROTOCOL (fixed for the whole book, ch21-ch27). Every message is +// the canonical chatterd CHAT FRAME: +// +// +-------+---------+------+----------------+------------------+ +// | magic | version | type | length (u16BE) | payload (UTF-8) | +// | 2B | 1B | 1B | 2B | `length` bytes | +// +-------+---------+------+----------------+------------------+ +// +// magic is the two bytes 0x43 0x48 ("CH"); version is 0x01. length counts the +// payload only (0..65535), never the 6-byte header. This chapter (v0) uses +// three of the five frame types: +// +// JOIN (1) payload = name client -> server, once, at connect +// MSG (2) payload = text client -> server, per stdin line +// DELIVER (3) payload = name 0x00 text server -> all clients, a broadcast +// +// WELCOME (4) and PING (5) are reserved for later chapters (ch22, ch24); this +// program never sends them, and a chatctl reader here ignores any frame type +// it doesn't recognize so it stays interoperable with newer servers. The +// server relays a client's MSG as a DELIVER and synthesises join/leave +// DELIVER notices from the reserved sender name "server". +// +// Go: net.Conn per connection, goroutines for readers and per-client writers, +// a context cancelled on SIGINT, and a select over the send/done/ctx channels +// so a slow or departed client never wedges a broadcaster. +package main + +import ( + "bufio" + "context" + "encoding/binary" + "fmt" + "io" + "net" + "os" + "os/signal" + "strconv" + "strings" + "sync" + "syscall" +) + +// The canonical chatterd chat frame header: magic "CH", version 1. +const ( + magic0 = 0x43 + magic1 = 0x48 + version = 0x01 + + frameJoin = 1 + frameMsg = 2 + frameDeliver = 3 + frameWelcome = 4 + framePing = 5 + + headerSize = 6 // magic(2) + version(1) + type(1) + length(2) +) + +func usage() int { + fmt.Fprintln(os.Stderr, "usage: chatterd serve --port PORT [--host HOST]") + fmt.Fprintln(os.Stderr, " chatterd chatctl --port PORT --name NAME [--host HOST]") + return 2 +} + +// --------------------------------------------------------------------------- +// The canonical chat frame: magic + version + type + u16BE length + payload. +// --------------------------------------------------------------------------- + +func buildFrame(typ byte, payload []byte) []byte { + // len(payload) is always <= 65535 for the frames this program builds + // (names/messages typed on a terminal); the length field is u16BE. + out := make([]byte, headerSize+len(payload)) + out[0] = magic0 + out[1] = magic1 + out[2] = version + out[3] = typ + binary.BigEndian.PutUint16(out[4:6], uint16(len(payload))) + copy(out[headerSize:], payload) + return out +} + +func buildJoin(name string) []byte { return buildFrame(frameJoin, []byte(name)) } +func buildMsg(text string) []byte { return buildFrame(frameMsg, []byte(text)) } + +func buildDeliver(name, text string) []byte { + payload := make([]byte, 0, len(name)+1+len(text)) + payload = append(payload, name...) + payload = append(payload, 0) + payload = append(payload, text...) + return buildFrame(frameDeliver, payload) +} + +// readFrame returns the frame's type and payload, or an error; io.EOF marks a +// clean close. +func readFrame(r *bufio.Reader) (byte, []byte, error) { + var hdr [headerSize]byte + if _, err := io.ReadFull(r, hdr[:]); err != nil { + return 0, nil, err + } + if hdr[0] != magic0 || hdr[1] != magic1 || hdr[2] != version { + return 0, nil, fmt.Errorf("not a chatterd frame (bad magic/version)") + } + typ := hdr[3] + n := binary.BigEndian.Uint16(hdr[4:6]) + payload := make([]byte, n) + if n > 0 { + if _, err := io.ReadFull(r, payload); err != nil { + return 0, nil, err + } + } + return typ, payload, nil +} + +// splitDeliver separates a DELIVER payload's "name\0text" into its two parts. +func splitDeliver(payload []byte) (string, string, bool) { + sep := -1 + for i, b := range payload { + if b == 0 { + sep = i + break + } + } + if sep < 0 { + return "", "", false + } + return string(payload[:sep]), string(payload[sep+1:]), true +} + +// --------------------------------------------------------------------------- +// The hub: every connected client, keyed by id. +// --------------------------------------------------------------------------- + +type client struct { + id int + name string + out chan []byte + done chan struct{} +} + +type hub struct { + mu sync.Mutex + clients map[int]*client +} + +func newHub() *hub { return &hub{clients: make(map[int]*client)} } + +func (h *hub) add(c *client) { + h.mu.Lock() + h.clients[c.id] = c + h.mu.Unlock() +} + +func (h *hub) setName(id int, name string) { + h.mu.Lock() + if c, ok := h.clients[id]; ok { + c.name = name + } + h.mu.Unlock() +} + +// remove drops a client and returns the name it last carried. +func (h *hub) remove(id int) string { + h.mu.Lock() + defer h.mu.Unlock() + c, ok := h.clients[id] + if !ok { + return "" + } + delete(h.clients, id) + close(c.done) // stop the writer goroutine + return c.name +} + +// broadcast queues frame on every client's outbound channel; when except >= 0 +// that id is skipped. A departed client (done closed) is passed over rather +// than blocking the broadcaster. +func (h *hub) broadcast(frame []byte, except int) { + h.mu.Lock() + targets := make([]*client, 0, len(h.clients)) + for _, c := range h.clients { + if c.id != except { + targets = append(targets, c) + } + } + h.mu.Unlock() + for _, c := range targets { + select { + case c.out <- frame: + case <-c.done: + } + } +} + +const broadcastAll = -1 + +// serveClient owns one connection: a writer goroutine drains c.out to the +// socket, while this goroutine reads frames and drives the hub. +func (h *hub) serveClient(ctx context.Context, conn net.Conn, id int) { + c := &client{id: id, out: make(chan []byte, 64), done: make(chan struct{})} + h.add(c) + + go func() { + for { + select { + case frame := <-c.out: + if _, err := conn.Write(frame); err != nil { + return + } + case <-c.done: + return + case <-ctx.Done(): + return + } + } + }() + + defer func() { + name := h.remove(id) + conn.Close() + if name != "" { + fmt.Fprintf(os.Stderr, "chatterd: %s left\n", name) + h.broadcast(buildDeliver("server", name+" left"), broadcastAll) + } + }() + + r := bufio.NewReader(conn) + var myName string + for { + typ, payload, err := readFrame(r) + if err != nil { + return + } + switch typ { + case frameJoin: + if len(payload) == 0 { + return // malformed: JOIN carries no name + } + myName = string(payload) + h.setName(id, myName) + fmt.Fprintf(os.Stderr, "chatterd: %s joined\n", myName) + h.broadcast(buildDeliver("server", myName+" joined"), broadcastAll) + case frameMsg: + if myName == "" { + continue // MSG before JOIN: nothing sensible to attribute it to + } + h.broadcast(buildDeliver(myName, string(payload)), id) + default: + // DELIVER/WELCOME/PING are not sent by a client in this chapter; + // ignore rather than tearing the connection down. + } + } +} + +// --------------------------------------------------------------------------- +// serve +// --------------------------------------------------------------------------- + +func cmdServe(host string, port int) error { + addr := fmt.Sprintf("%s:%d", host, port) + ln, err := net.Listen("tcp", addr) // Go sets SO_REUSEADDR on TCP listeners + if err != nil { + return fmt.Errorf("listen %s: %w", addr, err) + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT) + go func() { + <-sigCh + fmt.Fprintln(os.Stderr, "chatterd: shutting down") + cancel() + ln.Close() // unblock Accept + }() + + fmt.Fprintf(os.Stderr, "chatterd: listening on %s\n", addr) + + h := newHub() + id := 0 + for { + conn, err := ln.Accept() + if err != nil { + if ctx.Err() != nil { + return nil // shutdown requested; a clean exit + } + return fmt.Errorf("accept: %w", err) + } + id++ + go h.serveClient(ctx, conn, id) + } +} + +// --------------------------------------------------------------------------- +// chatctl +// --------------------------------------------------------------------------- + +func cmdChatctl(host string, port int, name string) error { + addr := fmt.Sprintf("%s:%d", host, port) + conn, err := net.Dial("tcp", addr) + if err != nil { + return fmt.Errorf("connect %s: %w", addr, err) + } + defer conn.Close() + + // Announce ourselves: a JOIN frame's payload is just our name. + if _, err := conn.Write(buildJoin(name)); err != nil { + return fmt.Errorf("write %s: %w", addr, err) + } + + readerDone := make(chan struct{}) + go func() { + defer close(readerDone) + r := bufio.NewReader(conn) + for { + typ, payload, err := readFrame(r) + if err != nil { + return + } + if typ != frameDeliver { + continue // WELCOME/PING/etc. are for later chapters; ignore them + } + n, t, ok := splitDeliver(payload) + if !ok { + continue // malformed DELIVER; skip rather than crash + } + fmt.Printf("%s: %s\n", n, t) + } + }() + + sc := bufio.NewScanner(os.Stdin) + for sc.Scan() { + line := sc.Text() + if line == "" { + continue // never emit an empty-text frame; that reads as a join + } + if _, err := conn.Write(buildMsg(line)); err != nil { + break + } + } + // stdin EOF: close the socket so the reader sees EOF, then wait for it to + // finish so every received line is flushed before we exit. + conn.Close() + <-readerDone + return nil +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +func main() { + args := os.Args[1:] + if len(args) == 0 { + os.Exit(usage()) + } + sub := args[0] + if sub != "serve" && sub != "chatctl" { + os.Exit(usage()) + } + + host := "127.0.0.1" + portStr := "" + name := "" + for i := 1; i < len(args); i++ { + switch { + case args[i] == "--port" && i+1 < len(args): + portStr = args[i+1] + i++ + case args[i] == "--host" && i+1 < len(args): + host = args[i+1] + i++ + case args[i] == "--name" && i+1 < len(args): + name = args[i+1] + i++ + default: + os.Exit(usage()) + } + } + if portStr == "" { + os.Exit(usage()) + } + port, err := strconv.Atoi(portStr) + if err != nil || port < 1 || port > 65535 { + os.Exit(usage()) + } + + if sub == "serve" { + if err := cmdServe(host, port); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: %v\n", err) + os.Exit(1) + } + return + } + if strings.TrimSpace(name) == "" { + os.Exit(usage()) + } + if err := cmdChatctl(host, port, name); err != nil { + fmt.Fprintf(os.Stderr, "chatctl: error: %v\n", err) + os.Exit(1) + } +} diff --git a/examples/21-tcp-sockets/rust/Cargo.lock b/examples/21-tcp-sockets/rust/Cargo.lock new file mode 100644 index 0000000..febc9bd --- /dev/null +++ b/examples/21-tcp-sockets/rust/Cargo.lock @@ -0,0 +1,69 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chatterd" +version = "0.1.0" +dependencies = [ + "anyhow", + "nix", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "memoffset" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset", +] diff --git a/examples/21-tcp-sockets/rust/Cargo.toml b/examples/21-tcp-sockets/rust/Cargo.toml new file mode 100644 index 0000000..4f45a07 --- /dev/null +++ b/examples/21-tcp-sockets/rust/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "chatterd" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +anyhow = "1" +nix = { version = "0.31", features = ["socket", "signal", "net"] } diff --git a/examples/21-tcp-sockets/rust/demo.sh b/examples/21-tcp-sockets/rust/demo.sh new file mode 100755 index 0000000..c2ef7e4 --- /dev/null +++ b/examples/21-tcp-sockets/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + exec "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/21-tcp-sockets/rust/rust-toolchain.toml b/examples/21-tcp-sockets/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/21-tcp-sockets/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/21-tcp-sockets/rust/src/main.rs b/examples/21-tcp-sockets/rust/src/main.rs new file mode 100644 index 0000000..749443c --- /dev/null +++ b/examples/21-tcp-sockets/rust/src/main.rs @@ -0,0 +1,415 @@ +// chatterd v0 — a thread-per-connection TCP chat room. +// +// One binary, two subcommands: +// +// chatterd serve --port P [--host H] +// Accept TCP connections; each client runs on its own std::thread. +// Every chat frame received is broadcast to all OTHER clients; a join +// notice is broadcast to ALL clients (including the newcomer). SIGINT +// closes the listener and exits 0. +// +// chatterd chatctl --port P --name N [--host H] +// Connect, announce N with a join frame, then send one frame per line +// of stdin. A reader thread prints every received frame as +// ": ". +// +// THE WIRE PROTOCOL (fixed for the whole book, ch21-ch27). Every message is +// the canonical chatterd CHAT FRAME: +// +// +-------+---------+------+----------------+------------------+ +// | magic | version | type | length (u16BE) | payload (UTF-8) | +// | 2B | 1B | 1B | 2B | `length` bytes | +// +-------+---------+------+----------------+------------------+ +// +// magic is the two bytes 0x43 0x48 ("CH"); version is 0x01. `length` counts +// the payload only (0..65535), never the 6-byte header. This chapter (v0) +// uses three of the five frame types: +// +// JOIN (1) payload = name client -> server, once, at connect +// MSG (2) payload = text client -> server, per stdin line +// DELIVER (3) payload = name 0x00 text server -> all clients, a broadcast +// +// WELCOME (4) and PING (5) are reserved for later chapters (ch22, ch24); this +// program never sends them, and a chatctl reader here ignores any frame type +// it doesn't recognize so it stays interoperable with newer servers. The +// server relays a client's MSG as a DELIVER and synthesises join/leave +// DELIVER notices from the reserved sender name "server". +// +// Rust 2024: no async runtime (this chapter is about blocking sockets and +// threads). The listener is built through nix so it can carry SO_REUSEADDR and +// arrive as an OwnedFd; a SIGINT handler flips an AtomicBool the non-blocking +// accept loop polls; every connection owns a std::thread and a TcpStream. + +use std::collections::HashMap; +use std::io::{BufReader, Read, Write}; +use std::net::{Shutdown, SocketAddrV4, TcpListener, TcpStream}; +use std::os::fd::{AsRawFd, OwnedFd}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread::{self, JoinHandle}; +use std::time::Duration; + +use anyhow::{Context, Result, anyhow, bail}; +use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction}; +use nix::sys::socket::sockopt::ReuseAddr; +use nix::sys::socket::{ + AddressFamily, Backlog, SockFlag, SockType, SockaddrIn, bind, listen, setsockopt, socket, +}; + +// The canonical chatterd chat frame header: magic "CH", version 1. +const MAGIC: [u8; 2] = [0x43, 0x48]; +const WIRE_VERSION: u8 = 0x01; +const HEADER_SIZE: usize = 6; // magic(2) + version(1) + type(1) + length(2) + +const FRAME_JOIN: u8 = 1; +const FRAME_MSG: u8 = 2; +const FRAME_DELIVER: u8 = 3; +#[allow(dead_code)] // WELCOME is reserved for ch22; not sent or read here +const FRAME_WELCOME: u8 = 4; +#[allow(dead_code)] // PING is reserved for ch24; not sent or read here +const FRAME_PING: u8 = 5; + +const BROADCAST_ALL: usize = 0; // sentinel id: no real client ever holds it + +fn usage() -> i32 { + eprintln!("usage: chatterd serve --port PORT [--host HOST]"); + eprintln!(" chatterd chatctl --port PORT --name NAME [--host HOST]"); + 2 +} + +// --------------------------------------------------------------------------- +// The canonical chat frame: magic + version + type + u16BE length + payload. +// --------------------------------------------------------------------------- + +fn build_frame(frame_type: u8, payload: &[u8]) -> Vec { + // payload.len() is always <= 65535 for the frames this program builds + // (names/messages typed on a terminal); the length field is u16BE. + let mut out = Vec::with_capacity(HEADER_SIZE + payload.len()); + out.extend_from_slice(&MAGIC); + out.push(WIRE_VERSION); + out.push(frame_type); + out.extend_from_slice(&(payload.len() as u16).to_be_bytes()); + out.extend_from_slice(payload); + out +} + +fn build_join(name: &str) -> Vec { + build_frame(FRAME_JOIN, name.as_bytes()) +} + +fn build_msg(text: &str) -> Vec { + build_frame(FRAME_MSG, text.as_bytes()) +} + +fn build_deliver(name: &str, text: &str) -> Vec { + let mut payload = Vec::with_capacity(name.len() + 1 + text.len()); + payload.extend_from_slice(name.as_bytes()); + payload.push(0); + payload.extend_from_slice(text.as_bytes()); + build_frame(FRAME_DELIVER, &payload) +} + +/// Read one frame; `Ok(None)` marks a clean EOF, `Err` a protocol fault. +fn read_frame(r: &mut R) -> Result)>> { + let mut hdr = [0u8; HEADER_SIZE]; + match r.read_exact(&mut hdr) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None), + Err(e) => return Err(e.into()), + } + if hdr[0..2] != MAGIC || hdr[2] != WIRE_VERSION { + bail!("not a chatterd frame (bad magic/version)"); + } + let frame_type = hdr[3]; + let len = u16::from_be_bytes([hdr[4], hdr[5]]); + let mut payload = vec![0u8; len as usize]; + if len > 0 { + match r.read_exact(&mut payload) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None), + Err(e) => return Err(e.into()), + } + } + Ok(Some((frame_type, payload))) +} + +/// Split a DELIVER payload's `name\0text` into its two parts. +fn split_deliver(payload: &[u8]) -> Option<(String, String)> { + let sep = payload.iter().position(|&b| b == 0)?; + let name = String::from_utf8_lossy(&payload[..sep]).into_owned(); + let text = String::from_utf8_lossy(&payload[sep + 1..]).into_owned(); + Some((name, text)) +} + +// --------------------------------------------------------------------------- +// The hub: every connected client, keyed by id. +// --------------------------------------------------------------------------- + +struct Client { + name: String, + writer: TcpStream, // a clone of the accepted stream, for broadcasts +} + +#[derive(Default)] +struct Hub { + clients: Mutex>, +} + +impl Hub { + fn add(&self, id: usize, writer: TcpStream) { + self.clients + .lock() + .unwrap() + .insert(id, Client { name: String::new(), writer }); + } + + fn set_name(&self, id: usize, name: &str) { + if let Some(c) = self.clients.lock().unwrap().get_mut(&id) { + c.name = name.to_owned(); + } + } + + /// Remove a client and return the name it last carried. + fn remove(&self, id: usize) -> Option { + self.clients.lock().unwrap().remove(&id).map(|c| c.name) + } + + /// Write `frame` to every client; `except == BROADCAST_ALL` skips nobody. + fn broadcast(&self, frame: &[u8], except: usize) { + let mut guard = self.clients.lock().unwrap(); + for (&id, c) in guard.iter_mut() { + if id != except { + let _ = c.writer.write_all(frame); + } + } + } + + /// Unblock every connection's blocking read so its thread can wind down. + fn shutdown_all(&self) { + for c in self.clients.lock().unwrap().values() { + let _ = c.writer.shutdown(Shutdown::Both); + } + } +} + +/// One connection's lifetime: read frames, drive the hub, announce the leave. +fn serve_client(hub: Arc, stream: TcpStream, id: usize) { + let writer = match stream.try_clone() { + Ok(w) => w, + Err(_) => return, + }; + hub.add(id, writer); + + let mut r = BufReader::new(stream); + let mut my_name = String::new(); + loop { + match read_frame(&mut r) { + Ok(Some((FRAME_JOIN, payload))) => { + if payload.is_empty() { + break; // malformed: JOIN carries no name + } + my_name = String::from_utf8_lossy(&payload).into_owned(); + hub.set_name(id, &my_name); + eprintln!("chatterd: {my_name} joined"); + hub.broadcast(&build_deliver("server", &format!("{my_name} joined")), BROADCAST_ALL); + } + Ok(Some((FRAME_MSG, payload))) => { + if my_name.is_empty() { + continue; // MSG before JOIN: nothing sensible to attribute it to + } + let text = String::from_utf8_lossy(&payload).into_owned(); + hub.broadcast(&build_deliver(&my_name, &text), id); + } + Ok(Some(_)) => continue, // DELIVER/WELCOME/PING not sent by a client; ignore + _ => break, // EOF or a protocol fault ends the connection + } + } + + if let Some(name) = hub.remove(id) + && !name.is_empty() + { + eprintln!("chatterd: {name} left"); + hub.broadcast(&build_deliver("server", &format!("{name} left")), BROADCAST_ALL); + } +} + +// --------------------------------------------------------------------------- +// serve +// --------------------------------------------------------------------------- + +static STOP: AtomicBool = AtomicBool::new(false); + +extern "C" fn on_sigint(_: i32) { + STOP.store(true, Ordering::Relaxed); +} + +/// Build a listening socket through nix so it carries SO_REUSEADDR and arrives +/// as an OwnedFd we hand straight to the std TcpListener. +fn listen_tcp(host: &str, port: u16) -> Result { + let addr: SocketAddrV4 = format!("{host}:{port}") + .parse() + .with_context(|| format!("{host}: not an IPv4 address"))?; + let fd: OwnedFd = socket( + AddressFamily::Inet, + SockType::Stream, + SockFlag::SOCK_CLOEXEC, + None, + ) + .context("socket")?; + setsockopt(&fd, ReuseAddr, &true).context("SO_REUSEADDR")?; + bind(fd.as_raw_fd(), &SockaddrIn::from(addr)).with_context(|| format!("bind {host}:{port}"))?; + listen(&fd, Backlog::new(128).unwrap()).context("listen")?; + Ok(TcpListener::from(fd)) +} + +fn install_sigint() -> Result<()> { + let action = SigAction::new( + SigHandler::Handler(on_sigint), + SaFlags::empty(), + SigSet::empty(), + ); + // SAFETY: on_sigint only stores into a static AtomicBool, which is + // async-signal-safe. + unsafe { sigaction(Signal::SIGINT, &action) }.context("sigaction")?; + Ok(()) +} + +fn cmd_serve(host: &str, port: u16) -> Result<()> { + let listener = listen_tcp(host, port)?; + listener.set_nonblocking(true).context("set_nonblocking")?; + install_sigint()?; + eprintln!("chatterd: listening on {host}:{port}"); + + let hub = Arc::new(Hub::default()); + let mut handles: Vec> = Vec::new(); + let mut next_id: usize = BROADCAST_ALL; // first real id is 1 + + while !STOP.load(Ordering::Relaxed) { + match listener.accept() { + Ok((conn, _)) => { + conn.set_nonblocking(false).context("client set_blocking")?; + next_id += 1; + let id = next_id; + let hub = Arc::clone(&hub); + handles.push(thread::spawn(move || serve_client(hub, conn, id))); + } + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { + thread::sleep(Duration::from_millis(20)); + } + Err(e) => return Err(anyhow!("accept: {e}")), + } + } + + eprintln!("chatterd: shutting down"); + hub.shutdown_all(); // unblock every reader + for h in handles { + let _ = h.join(); + } + Ok(()) +} + +// --------------------------------------------------------------------------- +// chatctl +// --------------------------------------------------------------------------- + +fn cmd_chatctl(host: &str, port: u16, name: &str) -> Result<()> { + let mut stream = + TcpStream::connect((host, port)).with_context(|| format!("connect {host}:{port}"))?; + + // Announce ourselves: a JOIN frame's payload is just our name. + stream + .write_all(&build_join(name)) + .with_context(|| format!("write {host}:{port}"))?; + + let reader_stream = stream.try_clone().context("clone stream")?; + let reader = thread::spawn(move || { + let mut r = BufReader::new(reader_stream); + while let Ok(Some((frame_type, payload))) = read_frame(&mut r) { + if frame_type != FRAME_DELIVER { + continue; // WELCOME/PING/etc. are for later chapters; ignore them + } + if let Some((n, t)) = split_deliver(&payload) { + println!("{n}: {t}"); + } + } + }); + + let stdin = std::io::stdin(); + let mut line = String::new(); + loop { + line.clear(); + match stdin.read_line(&mut line) { + Ok(0) => break, // EOF + Ok(_) => { + let msg = line.trim_end_matches('\n'); + if msg.is_empty() { + continue; // never emit an empty-text frame; that reads as a join + } + if stream.write_all(&build_msg(msg)).is_err() { + break; + } + } + Err(_) => break, + } + } + // stdin EOF: close the socket so the reader sees EOF, then join it so every + // received line is flushed before we exit. + let _ = stream.shutdown(Shutdown::Both); + let _ = reader.join(); + Ok(()) +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +fn main() -> std::process::ExitCode { + let args: Vec = std::env::args().skip(1).collect(); + let Some(sub) = args.first().cloned() else { + return std::process::ExitCode::from(usage() as u8); + }; + if sub != "serve" && sub != "chatctl" { + return std::process::ExitCode::from(usage() as u8); + } + + let mut host = String::from("127.0.0.1"); + let mut port_str = String::new(); + let mut name = String::new(); + let mut i = 1; + while i < args.len() { + if args[i] == "--port" && i + 1 < args.len() { + port_str = args[i + 1].clone(); + i += 2; + } else if args[i] == "--host" && i + 1 < args.len() { + host = args[i + 1].clone(); + i += 2; + } else if args[i] == "--name" && i + 1 < args.len() { + name = args[i + 1].clone(); + i += 2; + } else { + return std::process::ExitCode::from(usage() as u8); + } + } + let Ok(port) = port_str.parse::() else { + return std::process::ExitCode::from(usage() as u8); + }; + if port == 0 { + return std::process::ExitCode::from(usage() as u8); + } + + if sub == "serve" { + if let Err(err) = cmd_serve(&host, port) { + eprintln!("chatterd: error: {err:#}"); + return std::process::ExitCode::from(1); + } + } else { + if name.trim().is_empty() { + return std::process::ExitCode::from(usage() as u8); + } + if let Err(err) = cmd_chatctl(&host, port, &name) { + eprintln!("chatctl: error: {err:#}"); + return std::process::ExitCode::from(1); + } + } + std::process::ExitCode::SUCCESS +} diff --git a/examples/21-tcp-sockets/verify.lua b/examples/21-tcp-sockets/verify.lua new file mode 100644 index 0000000..94a0083 --- /dev/null +++ b/examples/21-tcp-sockets/verify.lua @@ -0,0 +1,161 @@ +-- Verify chatterd v0 (21-tcp-sockets) for LSP_LANG. +-- +-- Asserts observable behavior, identical across cpp/go/rust: +-- 1. usage errors exit 2; chatctl against a closed port exits 1 and reports +-- the failure on stderr. +-- 2. A real two-client session over loopback TCP: start `serve` in the +-- background, connect two chatctl clients (bob, then alice) through +-- scripted stdin fifos, and assert the actually delivered lines — +-- * alice's message reaches bob as "alice: hello" +-- * alice never sees her own message echoed back (messages go to +-- OTHER clients only) +-- * both clients see a join notice from the reserved "server" sender +-- * the server logs each join on stderr +-- 3. SIGINT tears the listener down and the server exits 0. +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set; the interpreter may be lua or luajit. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- Pick a free loopback port (bound and released; the server sets SO_REUSEADDR +-- so the immediate rebind succeeds even against a lingering TIME_WAIT). +local port = checks.run( + "python3 -c 'import socket; s=socket.socket(); s.bind((\"127.0.0.1\",0)); " .. + "print(s.getsockname()[1]); s.close()'").out:gsub("%s+$", "") +if not port:match("^%d+$") then + checks.skip("could not allocate a TCP port") +end + +local wd = checks.run("mktemp -d").out:gsub("%s+$", "") +if wd == "" or not wd:match("^/") then + checks.skip("mktemp -d did not yield a directory") +end + +local function sh(script) + return checks.run((script:gsub("@WD@", wd):gsub("@PORT@", port):gsub("@DEMO@", demo))) +end + +-- --------------------------------------------------------------------------- +-- 1. error shapes +-- --------------------------------------------------------------------------- + +local noargs = checks.run(demo) +checks.expect_exit(noargs, 2, lang .. ": no arguments exits 2") +checks.expect_match(noargs.out, "usage: chatterd", lang .. ": no arguments prints usage") + +local badsub = checks.run(demo .. " frobnicate") +checks.expect_exit(badsub, 2, lang .. ": unknown subcommand exits 2") + +local noport = checks.run(demo .. " serve") +checks.expect_exit(noport, 2, lang .. ": serve without --port exits 2") + +local noname = sh(demo .. " chatctl --port @PORT@") +checks.expect_exit(noname, 2, lang .. ": chatctl without --name exits 2") + +-- Nothing is listening yet: connect must fail with a runtime error, exit 1. +local refused = sh(demo .. " chatctl --port @PORT@ --name x < /dev/null") +checks.expect_exit(refused, 1, lang .. ": chatctl against a closed port exits 1") +checks.expect_match(refused.out, "chatctl: error:", lang .. ": connect failure is reported") + +-- --------------------------------------------------------------------------- +-- 2 + 3. two-client session, then a clean SIGINT shutdown +-- --------------------------------------------------------------------------- + +local session = sh([[ +set -u +wait_for() { # wait_for FILE PATTERN + for _ in $(seq 1 200); do grep -q "$2" "$1" 2>/dev/null && return 0; sleep 0.05; done + return 1 +} + +@DEMO@ serve --port @PORT@ --host 127.0.0.1 > @WD@/srv.out 2>&1 & +SRV=$! +for _ in $(seq 1 200); do ss -ltn 2>/dev/null | grep -q ":@PORT@ " && break; sleep 0.05; done + +mkfifo @WD@/afifo @WD@/bfifo + +# bob joins first, then alice — so bob is present to witness alice's join. +@DEMO@ chatctl --port @PORT@ --name bob < @WD@/bfifo > @WD@/b.out 2>&1 & +B=$! +exec 4> @WD@/bfifo +wait_for @WD@/srv.out "bob joined" + +@DEMO@ chatctl --port @PORT@ --name alice < @WD@/afifo > @WD@/a.out 2>&1 & +A=$! +exec 3> @WD@/afifo +wait_for @WD@/a.out "server: alice joined" +wait_for @WD@/b.out "server: alice joined" + +# alice speaks; bob must receive it. +printf 'hello\n' >&3 +wait_for @WD@/b.out "alice: hello" + +# Close alice first and let her exit fully, so a.out is frozen before we look. +exec 3>&- +wait "$A" +exec 4>&- +wait "$B" + +# Per-file observations (avoids cross-section ambiguity in the transcript). +grep -q "^alice: hello$" @WD@/b.out && echo "BOB_GOT_HELLO=yes" || echo "BOB_GOT_HELLO=no" +grep -q "^server: alice joined$" @WD@/b.out && echo "BOB_SAW_JOIN=yes" || echo "BOB_SAW_JOIN=no" +grep -q "^server: bob joined$" @WD@/b.out && echo "BOB_SAW_OWN_JOIN=yes" || echo "BOB_SAW_OWN_JOIN=no" +grep -q "joined$" @WD@/a.out && echo "ALICE_SAW_JOIN=yes" || echo "ALICE_SAW_JOIN=no" +grep -q "^alice: hello$" @WD@/a.out && echo "ALICE_SELF_ECHO=yes" || echo "ALICE_SELF_ECHO=no" + +echo "===ALICE==="; cat @WD@/a.out +echo "===BOB==="; cat @WD@/b.out +echo "===END===" + +# SIGINT the listener; it must close and exit 0. +kill -INT "$SRV" +for _ in $(seq 1 200); do kill -0 "$SRV" 2>/dev/null || break; sleep 0.05; done +if kill -0 "$SRV" 2>/dev/null; then kill -9 "$SRV"; echo "server_exit=timeout"; else + wait "$SRV"; echo "server_exit=$?" +fi + +# The server log now includes the shutdown line printed on SIGINT. +echo "===SERVER==="; cat @WD@/srv.out +]]) + +-- Message delivery to the OTHER client. +checks.expect_match(session.out, "BOB_GOT_HELLO=yes", + lang .. ": alice's message is delivered to bob as 'alice: hello'") +checks.expect_match(session.out, "ALICE_SELF_ECHO=no", + lang .. ": a message is not echoed back to its own sender") + +-- Both clients see a join. +checks.expect_match(session.out, "ALICE_SAW_JOIN=yes", lang .. ": alice sees a join notice") +checks.expect_match(session.out, "BOB_SAW_JOIN=yes", lang .. ": bob sees alice's join notice") +checks.expect_match(session.out, "BOB_SAW_OWN_JOIN=yes", + lang .. ": a joiner is told of its own join (join goes to all members)") + +-- The delivered lines really appear in the transcript. +checks.expect_match(session.out, "\nalice: hello\n", + lang .. ": 'alice: hello' is present in bob's transcript") +checks.expect_match(session.out, "server: alice joined", + lang .. ": the join notice carries the reserved 'server' sender") + +-- Server-side observations. +checks.expect_match(session.out, "chatterd: listening on 127%.0%.0%.1:" .. port, + lang .. ": server announces the listen address") +checks.expect_match(session.out, "chatterd: bob joined", lang .. ": server logs bob's join") +checks.expect_match(session.out, "chatterd: alice joined", lang .. ": server logs alice's join") +checks.expect_match(session.out, "chatterd: shutting down", lang .. ": server logs the shutdown") + +-- Clean SIGINT teardown. +checks.expect_match(session.out, "server_exit=0", lang .. ": SIGINT shuts the listener and exits 0") + +checks.run("rm -rf " .. wd) +checks.finish() diff --git a/examples/22-scaling-the-server/README.md b/examples/22-scaling-the-server/README.md new file mode 100644 index 0000000..f67c380 --- /dev/null +++ b/examples/22-scaling-the-server/README.md @@ -0,0 +1,153 @@ +# 22-scaling-the-server — chatterd v1 + +chatterd is a peer-to-peer chat daemon we grow across chapters 21–24 and 27. +ch21 introduced the length-prefixed frame protocol and a thread-per-connection +server. ch22 is about **how a server holds many connections at once**: it keeps +the threaded engine and adds a single-threaded **epoll** engine — one thread, +nonblocking accept, and a per-connection read/write state machine with an +outbound queue and real backpressure. + +The daemon and its client are the same binary (`app`): `serve` is the daemon, +`send` / `listen` / `flood` are the client (`chatctl`) modes. + +``` +$ ./demo.sh cpp run serve --engine epoll --port 0 & +chatterd: serving engine=epoll on 127.0.0.1:41453 + +$ ./demo.sh cpp run listen --port 41453 bob --count 1 & +$ ./demo.sh cpp run send --port 41453 alice "hello, chatterd" +alice: hello, chatterd +# bob's terminal prints the same line: +alice: hello, chatterd + +$ ./demo.sh cpp run flood --port 41453 50 +flood: connected 50 +flood: delivered 50 + +# SIGINT/SIGTERM the daemon: +chatterd: stopped engine=epoll messages=2 peak_conns=50 +``` + +Every MSG is delivered to **every** connected client, including the sender, so +a client confirms its message was accepted by seeing its own line echoed. + +## What the chapter is really about + +Two ways to service N connections from one process: + +- **`--engine threaded`** — one OS thread per connection blocks in `recv()`; a + shared registry (mutex + per-socket write mutex) fans each message out. Simple + to reason about, one kernel thread per client. The accept loop `poll(2)`s the + listening socket alongside a `signalfd`, so a signal breaks it out cleanly. +- **`--engine epoll`** — one thread owns an `epoll` set over the listener, a + `signalfd`, and every client socket. Accept is nonblocking (`accept4` in a + loop until `EAGAIN`); each connection is a state machine with an **inbound + buffer** (frames are reassembled across reads) and an **outbound queue**. + Broadcasting appends the encoded frame to every connection's queue and tries + to drain it; if the socket buffer is full the remainder stays queued and the + connection re-arms `EPOLLOUT`, which drives the rest of the flush later. That + is **backpressure**: a slow reader can't corrupt or drop another client's + frames, and the writer never blocks the whole server. + +`verify.lua` proves the epoll engine handles **50 concurrent connections** and +delivers one broadcast to **all 50** (`flood: delivered 50`, no dropped frames), +that the threaded engine does the same, and that both shut down cleanly with a +`messages`/`peak_conns` summary that matches what was driven through them. + +### A note on Go + +Go exposes the same `--engine threaded|epoll` flag, but **both** engines are +the same idiomatic code: one goroutine per connection reading, one writing, and +a hub goroutine fanning out over channels. There is no hand-rolled epoll — the +Go runtime's **network poller** already multiplexes every socket through +`epoll_wait(2)` and parks a *goroutine* (not an OS thread) on each blocked +Read/Write. The epoll the other two languages write by hand, Go's netpoller runs +underneath; the engine flag only changes the label. Observable behavior is +identical across all three. + +## Wire format + +One frame protocol, introduced in ch21 and reused unchanged by every later +chatterd version and every language. All multi-byte fields are **big-endian**. + +``` + byte: 0 1 2 3 4 5 6 ............. 6+LEN-1 + +-----+-----+-------+------+-----+-----+---------------------------+ + | 'C' | 'H' | 0x01 | TYPE | LENGTH | PAYLOAD | + +-----+-----+-------+------+-----+-----+---------------------------+ + magic version u16 (LEN) LEN bytes + +magic = 0x43 0x48 ('C','H') +version = 0x01 +TYPE = 1 byte (see below) +LENGTH = uint16 big-endian, number of PAYLOAD bytes (0..65535) +PAYLOAD = LENGTH bytes +``` + +Header is always 6 bytes; total frame length is `6 + LENGTH`. + +| TYPE | name | dir | payload | +|------|---------|-----|---------| +| 0x01 | JOIN | client → server | nickname, UTF-8 | +| 0x02 | MSG | client → server | message text, UTF-8 | +| 0x03 | DELIVER | server → client | `nick + 0x00 (NUL) + text` | +| 0x04 | WELCOME | server → client | empty (sent once, on accept) | +| 0x05 | PING | — | reserved (introduced in ch24; unused here) | + +This is the same frame every chatterd chapter and language speaks: a v0 +(ch21) client can chat through a v1 (this chapter's) or v3/v4 server and back +because JOIN/MSG/DELIVER never change shape — later chapters only add types +(WELCOME here, PING in ch24), never touch the 6-byte header or repurpose an +existing type. + +`DELIVER` carries the originating nickname, then a single NUL (`0x00`) byte, +then the message text — split the payload on its first NUL to recover the two +fields (a nickname is never allowed to contain a NUL). WELCOME is sent the +instant the server accepts a connection — receiving it is proof the connection +is registered in the server's fan-out set, which is how `flood` knows all N +peers are present before it triggers the broadcast. + +Example — `alice` sends `hi` (`msg`), and the server delivers it: + +``` +client -> server 43 48 01 01 00 05 61 6c 69 63 65 JOIN "alice" +client -> server 43 48 01 02 00 02 68 69 MSG "hi" +server -> client 43 48 01 03 00 08 61 6c 69 63 65 00 68 69 DELIVER nick="alice" text="hi" +``` + +## CLI + +Identical across the three languages; exit codes 0 ok / 1 runtime / 2 usage. + +``` +serve --engine threaded|epoll --port P run the daemon (P=0 picks an ephemeral port, + printed as "on 127.0.0.1:") +send --port P NICK TEXT join as NICK, broadcast TEXT, print each + delivered line as "nick: text", stop after + seeing your own echo +listen --port P NICK --count N join as NICK, print N delivered lines, exit +flood --port P N [--text TEXT] open N concurrent connections, wait for all to + be welcomed, have one broadcast TEXT, and report + "flood: connected " / "flood: delivered " +``` + +The daemon logs two lines to stderr: `chatterd: serving engine= on +127.0.0.1:` at startup and `chatterd: stopped engine= messages= +peak_conns=

` on SIGINT/SIGTERM, where `messages` is the number of MSG frames +broadcast and `peak_conns` the high-water mark of simultaneous connections. + +## The three implementations + +Same CLI, stdout/stderr shapes, exit codes, and wire protocol, so one +`verify.lua` covers all three. + +| | epoll engine | threaded engine | shutdown | +|---|---|---|---| +| **C++23** | hand-rolled `epoll_create1`/`epoll_ctl`/`epoll_wait`; per-conn `EConn` with `inbuf`/`outbuf`, `EPOLLOUT` toggled on backpressure; RAII `Fd` owns every socket, `std::expected` for setup | `std::jthread` per connection, shared `Registry` (mutex + per-socket write mutex), `std::atomic` counters | `signalfd` in the loop / poll set; `SHUT_RDWR` wakes reader threads, jthreads join | +| **Go 1.26** | netpoller (see note) — goroutine per conn + hub goroutine, channel fan-out with blocking sends for backpressure | *same code* as epoll; the flag only changes the printed label | `signal.Notify`, close listener, `quit`/`done` channels, atomics read after the hub drains | +| **Rust 2024** | `mio` `Poll`/`Token`/`Interest`; per-conn `EConn` with `inbuf`/`outbuf`, reregister `READABLE\|WRITABLE` on backpressure; `signalfd` registered via `SourceFd`; `OwnedFd`-backed `mio::net` sockets | `std::thread` per connection, `Arc` (mutex map + per-socket `Mutex` write clone), atomics | `nix::poll` over listener + `signalfd`; `TcpStream::shutdown` wakes readers, threads join | + +Build/run: `./demo.sh [cpp|go|rust] build` and `./demo.sh run `. +`./demo.sh ` with no args runs a short end-to-end demo (epoll daemon, a +two-party exchange, a 50-client flood, clean shutdown). Everything runs on +loopback; no VM is involved. diff --git a/examples/22-scaling-the-server/cpp/.clang-format b/examples/22-scaling-the-server/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/22-scaling-the-server/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/22-scaling-the-server/cpp/.clang-tidy b/examples/22-scaling-the-server/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/22-scaling-the-server/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/22-scaling-the-server/cpp/CMakeLists.txt b/examples/22-scaling-the-server/cpp/CMakeLists.txt new file mode 100644 index 0000000..05ec36d --- /dev/null +++ b/examples/22-scaling-the-server/cpp/CMakeLists.txt @@ -0,0 +1,11 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app src/main.cpp) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/22-scaling-the-server/cpp/CMakePresets.json b/examples/22-scaling-the-server/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/22-scaling-the-server/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/22-scaling-the-server/cpp/demo.sh b/examples/22-scaling-the-server/cpp/demo.sh new file mode 100755 index 0000000..5fe9aec --- /dev/null +++ b/examples/22-scaling-the-server/cpp/demo.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + # exec so a backgrounded `run serve ...` can be signalled directly + exec "./$BIN" "$@" + fi +} + +demo() { + build + local tmp + tmp=$(mktemp -d) + "./$BIN" serve --engine epoll --port 0 2>"$tmp/serve.err" & + local sp=$! port="" + for _ in $(seq 1 100); do + port=$(sed -n 's/.*on 127\.0\.0\.1:\([0-9]*\).*/\1/p' "$tmp/serve.err") + [[ -n "$port" ]] && break + sleep 0.05 + done + echo "chatterd (epoll) listening on 127.0.0.1:$port" + "./$BIN" listen --port "$port" bob --count 1 >"$tmp/bob.out" & + local lp=$! + sleep 0.2 + "./$BIN" send --port "$port" alice "hello from alice" + wait "$lp" + echo "bob received: $(cat "$tmp/bob.out")" + "./$BIN" flood --port "$port" 50 + kill -TERM "$sp" + wait "$sp" 2>/dev/null || true + cat "$tmp/serve.err" + rm -rf "$tmp" +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + demo + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/22-scaling-the-server/cpp/src/main.cpp b/examples/22-scaling-the-server/cpp/src/main.cpp new file mode 100644 index 0000000..e6aac8b --- /dev/null +++ b/examples/22-scaling-the-server/cpp/src/main.cpp @@ -0,0 +1,829 @@ +// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// +// ch21 introduced the length-prefixed frame protocol and a threaded server +// (one thread per connection). ch22 — "scaling the server" — keeps that +// threaded engine working and adds a single-threaded epoll engine: one thread, +// many connections, nonblocking accept plus a per-connection read/write state +// machine with an outbound queue and real backpressure (partial writes are +// carried across epoll_wait iterations, EPOLLOUT drives the flush). +// +// Wire format (identical across cpp/go/rust and every version), big-endian: +// +// offset 0 2 magic 'C' 'H' (0x43 0x48) +// offset 2 1 version 0x01 +// offset 3 1 type +// offset 4 2 length u16 payload byte count +// offset 6 N payload +// +// type 0x01 JOIN c->s payload = nickname (UTF-8) +// type 0x02 MSG c->s payload = message text (UTF-8) +// type 0x03 DELIVER s->c payload = nick + 0x00 (NUL) + text +// type 0x04 WELCOME s->c payload = empty (sent on accept) +// +// The server delivers every MSG to every connected client, including the +// originator, so a client confirms acceptance by seeing its own line echoed. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +[[nodiscard]] std::error_code last_error() { + return {errno, std::system_category()}; +} + +// --------------------------------------------------------------------------- +// RAII file descriptor — the only owner of every fd/socket in this program. +// --------------------------------------------------------------------------- + +class Fd { +public: + Fd() = default; + explicit Fd(int fd) : fd_{fd} {} + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + Fd(Fd&& other) noexcept : fd_{std::exchange(other.fd_, -1)} {} + Fd& operator=(Fd&& other) noexcept { + if (this != &other) { + reset(); + fd_ = std::exchange(other.fd_, -1); + } + return *this; + } + ~Fd() { reset(); } + + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + } + fd_ = -1; + } + [[nodiscard]] int get() const noexcept { return fd_; } + +private: + int fd_ = -1; +}; + +[[nodiscard]] std::expected checked(int fd) { + if (fd < 0) { + return std::unexpected(last_error()); + } + return Fd{fd}; +} + +// --------------------------------------------------------------------------- +// Frame protocol +// --------------------------------------------------------------------------- + +constexpr std::uint8_t kVersion = 0x01; +enum class Type : std::uint8_t { join = 0x01, msg = 0x02, deliver = 0x03, welcome = 0x04 }; + +struct Frame { + std::uint8_t type; + std::string payload; +}; + +[[nodiscard]] std::string encode(Type t, std::string_view payload) { + std::string f; + f.reserve(6 + payload.size()); + f.push_back('C'); + f.push_back('H'); + f.push_back(static_cast(kVersion)); + f.push_back(static_cast(std::to_underlying(t))); + f.push_back(static_cast((payload.size() >> 8) & 0xff)); + f.push_back(static_cast(payload.size() & 0xff)); + f.append(payload); + return f; +} + +[[nodiscard]] std::string deliver_payload(std::string_view nick, std::string_view text) { + std::string p; + p.reserve(nick.size() + 1 + text.size()); + p.append(nick); + p.push_back('\0'); + p.append(text); + return p; +} + +// Pull one complete frame off the front of buf, consuming its bytes. Returns +// nullopt when buf holds only a partial frame (need more bytes). +[[nodiscard]] std::optional take_frame(std::string& buf) { + if (buf.size() < 6) { + return std::nullopt; + } + const auto* b = reinterpret_cast(buf.data()); + if (b[0] != 'C' || b[1] != 'H' || b[2] != kVersion) { + buf.clear(); // desync on a foreign speaker; drop the stream + return std::nullopt; + } + const std::size_t len = (static_cast(b[4]) << 8) | b[5]; + if (buf.size() < 6 + len) { + return std::nullopt; + } + Frame fr{b[3], buf.substr(6, len)}; + buf.erase(0, 6 + len); + return fr; +} + +// --------------------------------------------------------------------------- +// Sockets +// --------------------------------------------------------------------------- + +[[nodiscard]] std::expected, std::error_code> +make_listener(std::uint16_t port, bool nonblock) { + const int type = SOCK_STREAM | SOCK_CLOEXEC | (nonblock ? SOCK_NONBLOCK : 0); + auto s = checked(::socket(AF_INET, type, 0)); + if (!s) { + return std::unexpected(s.error()); + } + int one = 1; + ::setsockopt(s->get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof one); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = htons(port); + if (::bind(s->get(), reinterpret_cast(&addr), sizeof addr) != 0) { + return std::unexpected(last_error()); + } + if (::listen(s->get(), 128) != 0) { + return std::unexpected(last_error()); + } + sockaddr_in bound{}; + socklen_t blen = sizeof bound; + if (::getsockname(s->get(), reinterpret_cast(&bound), &blen) != 0) { + return std::unexpected(last_error()); + } + return std::pair{std::move(*s), ntohs(bound.sin_port)}; +} + +[[nodiscard]] std::expected make_signalfd() { + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + if (::sigprocmask(SIG_BLOCK, &mask, nullptr) != 0) { + return std::unexpected(last_error()); + } + return checked(::signalfd(-1, &mask, SFD_NONBLOCK | SFD_CLOEXEC)); +} + +// Blocking write of every byte; returns false if the peer went away. +[[nodiscard]] bool write_all(int fd, std::string_view data) { + while (!data.empty()) { + const ssize_t n = ::send(fd, data.data(), data.size(), MSG_NOSIGNAL); + if (n > 0) { + data.remove_prefix(static_cast(n)); + } else if (n < 0 && errno == EINTR) { + continue; + } else { + return false; + } + } + return true; +} + +// --------------------------------------------------------------------------- +// Threaded engine — one thread per connection, a shared broadcast registry. +// --------------------------------------------------------------------------- + +struct Conn { + Fd fd; + std::mutex wmu; // serialize writes to this socket + std::string nick = "?"; // set by the JOIN frame +}; + +class Registry { +public: + void add(std::uint64_t id, std::shared_ptr c) { + std::scoped_lock lk{mu_}; + conns_.emplace(id, std::move(c)); + const auto n = static_cast(conns_.size()); + if (n > peak_.load(std::memory_order_relaxed)) { + peak_.store(n, std::memory_order_relaxed); + } + } + void remove(std::uint64_t id) { + std::scoped_lock lk{mu_}; + conns_.erase(id); + } + void broadcast(std::string_view frame) { + std::vector> snap; + { + std::scoped_lock lk{mu_}; + snap.reserve(conns_.size()); + for (auto& [_, c] : conns_) { + snap.push_back(c); + } + } + for (auto& c : snap) { + std::scoped_lock wl{c->wmu}; + (void)write_all(c->fd.get(), frame); // a dead peer is reaped by its reader + } + } + void shutdown_all() { + std::scoped_lock lk{mu_}; + for (auto& [_, c] : conns_) { + ::shutdown(c->fd.get(), SHUT_RDWR); // wake the reader blocked in recv() + } + } + [[nodiscard]] int peak() const { return peak_.load(std::memory_order_relaxed); } + +private: + std::mutex mu_; + std::unordered_map> conns_; + std::atomic peak_{0}; +}; + +void serve_conn(std::stop_token stop, std::shared_ptr conn, std::uint64_t id, + Registry& reg, std::atomic& messages) { + (void)write_all(conn->fd.get(), encode(Type::welcome, "")); + std::string inbuf; + std::array buf{}; + while (!stop.stop_requested()) { + const ssize_t n = ::recv(conn->fd.get(), buf.data(), buf.size(), 0); + if (n <= 0) { + break; // EOF, error, or shutdown() at teardown + } + inbuf.append(buf.data(), static_cast(n)); + while (auto fr = take_frame(inbuf)) { + if (fr->type == std::to_underlying(Type::join)) { + std::scoped_lock wl{conn->wmu}; + conn->nick = fr->payload; + } else if (fr->type == std::to_underlying(Type::msg)) { + std::string nick; + { + std::scoped_lock wl{conn->wmu}; + nick = conn->nick; + } + messages.fetch_add(1, std::memory_order_relaxed); + reg.broadcast(encode(Type::deliver, deliver_payload(nick, fr->payload))); + } + } + } + reg.remove(id); +} + +int serve_threaded(std::uint16_t port) { + auto listener = make_listener(port, /*nonblock=*/false); + if (!listener) { + std::println(stderr, "chatterd: listen: {}", listener.error().message()); + return 1; + } + auto& [lfd, bound] = *listener; + auto sigfd = make_signalfd(); + if (!sigfd) { + std::println(stderr, "chatterd: signalfd: {}", sigfd.error().message()); + return 1; + } + std::println(stderr, "chatterd: serving engine=threaded on 127.0.0.1:{}", bound); + + Registry reg; + std::atomic messages{0}; + std::vector workers; + std::uint64_t next_id = 0; + + for (;;) { + std::array pfds{ + pollfd{.fd = lfd.get(), .events = POLLIN, .revents = 0}, + pollfd{.fd = sigfd->get(), .events = POLLIN, .revents = 0}, + }; + if (::poll(pfds.data(), pfds.size(), -1) < 0) { + if (errno == EINTR) { + continue; + } + std::println(stderr, "chatterd: poll: {}", last_error().message()); + return 1; + } + if ((pfds[1].revents & POLLIN) != 0) { + break; // SIGINT/SIGTERM + } + if ((pfds[0].revents & POLLIN) != 0) { + auto client = checked(::accept4(lfd.get(), nullptr, nullptr, SOCK_CLOEXEC)); + if (!client) { + continue; + } + auto conn = std::make_shared(); + conn->fd = std::move(*client); + const auto id = next_id++; + reg.add(id, conn); + workers.emplace_back([conn, id, ®, &messages](std::stop_token st) { + serve_conn(std::move(st), conn, id, reg, messages); + }); + } + } + + lfd.reset(); + reg.shutdown_all(); + workers.clear(); // jthread destructors request stop and join every worker + std::println(stderr, "chatterd: stopped engine=threaded messages={} peak_conns={}", + messages.load(), reg.peak()); + return 0; +} + +// --------------------------------------------------------------------------- +// Epoll engine — one thread, nonblocking, per-connection state machines. +// --------------------------------------------------------------------------- + +struct EConn { + Fd fd; + std::string nick = "?"; + std::string inbuf; + std::string outbuf; // pending outbound bytes (the backpressure queue) + bool want_write = false; + bool dead = false; +}; + +class EpollServer { +public: + static constexpr std::uint64_t kListen = 0; + static constexpr std::uint64_t kSignal = 1; + + EpollServer(Fd epoll, Fd listener) : epoll_{std::move(epoll)}, listener_{std::move(listener)} {} + + [[nodiscard]] std::expected register_fd(int fd, std::uint64_t tok, + std::uint32_t ev) { + epoll_event e{.events = ev, .data = {.u64 = tok}}; + if (::epoll_ctl(epoll_.get(), EPOLL_CTL_ADD, fd, &e) != 0) { + return std::unexpected(last_error()); + } + return {}; + } + void set_sigfd(Fd sig) { sigfd_ = std::move(sig); } + + int run() { + std::array events{}; + for (;;) { + const int n = ::epoll_wait(epoll_.get(), events.data(), events.size(), -1); + if (n < 0) { + if (errno == EINTR) { + continue; + } + std::println(stderr, "chatterd: epoll_wait: {}", last_error().message()); + return 1; + } + for (int i = 0; i < n; ++i) { + const auto tok = events[i].data.u64; + if (tok == kListen) { + accept_ready(); + } else if (tok == kSignal) { + reap_closed(); + std::println(stderr, + "chatterd: stopped engine=epoll messages={} peak_conns={}", + messages_, peak_); + return 0; + } else { + if (!conns_.contains(tok)) { + continue; // reaped earlier in this batch + } + if ((events[i].events & (EPOLLHUP | EPOLLERR)) != 0) { + conns_.at(tok).dead = true; + } + if ((events[i].events & EPOLLIN) != 0) { + on_readable(tok); + } + if (conns_.contains(tok) && (events[i].events & EPOLLOUT) != 0) { + flush(tok); + } + } + } + reap_closed(); + } + } + +private: + void accept_ready() { + for (;;) { + const int cfd = ::accept4(listener_.get(), nullptr, nullptr, + SOCK_CLOEXEC | SOCK_NONBLOCK); + if (cfd < 0) { + return; // EAGAIN: backlog drained + } + const auto tok = next_tok_++; + EConn conn; + conn.fd = Fd{cfd}; + epoll_event e{.events = EPOLLIN, .data = {.u64 = tok}}; + if (::epoll_ctl(epoll_.get(), EPOLL_CTL_ADD, cfd, &e) != 0) { + continue; // conn.fd closes on scope exit + } + conns_.emplace(tok, std::move(conn)); + if (static_cast(conns_.size()) > peak_) { + peak_ = static_cast(conns_.size()); + } + queue(tok, encode(Type::welcome, "")); + } + } + + void on_readable(std::uint64_t tok) { + auto& conn = conns_.at(tok); + std::array buf{}; + for (;;) { + const ssize_t n = ::recv(conn.fd.get(), buf.data(), buf.size(), 0); + if (n > 0) { + conn.inbuf.append(buf.data(), static_cast(n)); + } else if (n == 0) { + conn.dead = true; + return; + } else if (errno == EAGAIN || errno == EWOULDBLOCK) { + break; + } else if (errno == EINTR) { + continue; + } else { + conn.dead = true; + return; + } + } + while (auto fr = take_frame(conn.inbuf)) { + if (fr->type == std::to_underlying(Type::join)) { + conn.nick = fr->payload; + } else if (fr->type == std::to_underlying(Type::msg)) { + ++messages_; + broadcast(encode(Type::deliver, deliver_payload(conn.nick, fr->payload))); + } + } + } + + void broadcast(const std::string& frame) { + for (auto& [tok, _] : conns_) { + queue(tok, frame); + } + } + + // Append to the connection's outbound queue and try to drain it now. + void queue(std::uint64_t tok, const std::string& frame) { + conns_.at(tok).outbuf.append(frame); + flush(tok); + } + + void flush(std::uint64_t tok) { + auto& conn = conns_.at(tok); + while (!conn.outbuf.empty()) { + const ssize_t n = ::send(conn.fd.get(), conn.outbuf.data(), conn.outbuf.size(), + MSG_NOSIGNAL); + if (n > 0) { + conn.outbuf.erase(0, static_cast(n)); + } else if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) { + break; // socket buffer full: keep the rest, wait for EPOLLOUT + } else if (n < 0 && errno == EINTR) { + continue; + } else { + conn.dead = true; + return; + } + } + // Register/clear EPOLLOUT interest to match the backlog state. + const bool need = !conn.outbuf.empty(); + if (need != conn.want_write) { + conn.want_write = need; + epoll_event e{.events = EPOLLIN | (need ? EPOLLOUT : 0u), .data = {.u64 = tok}}; + ::epoll_ctl(epoll_.get(), EPOLL_CTL_MOD, conn.fd.get(), &e); + } + } + + void reap_closed() { + for (auto it = conns_.begin(); it != conns_.end();) { + if (it->second.dead) { + ::epoll_ctl(epoll_.get(), EPOLL_CTL_DEL, it->second.fd.get(), nullptr); + it = conns_.erase(it); + } else { + ++it; + } + } + } + + Fd epoll_; + Fd listener_; + Fd sigfd_; + std::unordered_map conns_; + std::uint64_t next_tok_ = 2; // 0/1 reserved for listen/signal + std::uint64_t messages_ = 0; + int peak_ = 0; +}; + +int serve_epoll(std::uint16_t port) { + auto listener = make_listener(port, /*nonblock=*/true); + if (!listener) { + std::println(stderr, "chatterd: listen: {}", listener.error().message()); + return 1; + } + auto& [lfd, bound] = *listener; + auto sigfd = make_signalfd(); + if (!sigfd) { + std::println(stderr, "chatterd: signalfd: {}", sigfd.error().message()); + return 1; + } + auto epoll = checked(::epoll_create1(EPOLL_CLOEXEC)); + if (!epoll) { + std::println(stderr, "chatterd: epoll_create1: {}", epoll.error().message()); + return 1; + } + + const int lfd_raw = lfd.get(); + const int sfd_raw = sigfd->get(); + EpollServer server{std::move(*epoll), std::move(lfd)}; + if (auto r = server.register_fd(lfd_raw, EpollServer::kListen, EPOLLIN); !r) { + std::println(stderr, "chatterd: epoll_ctl: {}", r.error().message()); + return 1; + } + if (auto r = server.register_fd(sfd_raw, EpollServer::kSignal, EPOLLIN); !r) { + std::println(stderr, "chatterd: epoll_ctl: {}", r.error().message()); + return 1; + } + server.set_sigfd(std::move(*sigfd)); + + std::println(stderr, "chatterd: serving engine=epoll on 127.0.0.1:{}", bound); + return server.run(); +} + +// --------------------------------------------------------------------------- +// Client +// --------------------------------------------------------------------------- + +[[nodiscard]] std::expected connect_to(std::uint16_t port) { + auto s = checked(::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)); + if (!s) { + return std::unexpected(s.error()); + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = htons(port); + if (::connect(s->get(), reinterpret_cast(&addr), sizeof addr) != 0) { + return std::unexpected(last_error()); + } + return std::move(*s); +} + +// Blocking read of one frame; nullopt on EOF or SO_RCVTIMEO timeout. +[[nodiscard]] std::optional recv_frame(int fd, std::string& inbuf) { + for (;;) { + if (auto fr = take_frame(inbuf)) { + return fr; + } + std::array buf{}; + const ssize_t n = ::recv(fd, buf.data(), buf.size(), 0); + if (n <= 0) { + return std::nullopt; + } + inbuf.append(buf.data(), static_cast(n)); + } +} + +[[nodiscard]] std::pair split_deliver(const std::string& payload) { + const auto nul = payload.find('\0'); + if (nul == std::string::npos) { + return {"?", payload}; + } + return {payload.substr(0, nul), payload.substr(nul + 1)}; +} + +int cmd_send(std::uint16_t port, const std::string& nick, const std::string& text) { + auto fd = connect_to(port); + if (!fd) { + std::println(stderr, "chatctl: cannot connect to 127.0.0.1:{}: {}", port, + fd.error().message()); + return 1; + } + if (!write_all(fd->get(), encode(Type::join, nick)) || + !write_all(fd->get(), encode(Type::msg, text))) { + std::println(stderr, "chatctl: send failed"); + return 1; + } + std::string inbuf; + while (auto fr = recv_frame(fd->get(), inbuf)) { + if (fr->type != std::to_underlying(Type::deliver)) { + continue; // skip WELCOME + } + auto [dn, dt] = split_deliver(fr->payload); + std::println("{}: {}", dn, dt); + std::fflush(stdout); + if (dn == nick && dt == text) { + return 0; // saw our own message echoed back + } + } + return 0; +} + +int cmd_listen(std::uint16_t port, const std::string& nick, int count) { + auto fd = connect_to(port); + if (!fd) { + std::println(stderr, "chatctl: cannot connect to 127.0.0.1:{}: {}", port, + fd.error().message()); + return 1; + } + timeval tv{.tv_sec = 10, .tv_usec = 0}; + ::setsockopt(fd->get(), SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + if (!write_all(fd->get(), encode(Type::join, nick))) { + std::println(stderr, "chatctl: send failed"); + return 1; + } + std::string inbuf; + int got = 0; + while (got < count) { + auto fr = recv_frame(fd->get(), inbuf); + if (!fr) { + std::println(stderr, "chatctl: timed out after {} of {} messages", got, count); + return 1; + } + if (fr->type != std::to_underlying(Type::deliver)) { + continue; + } + auto [dn, dt] = split_deliver(fr->payload); + std::println("{}: {}", dn, dt); + std::fflush(stdout); + ++got; + } + return 0; +} + +int cmd_flood(std::uint16_t port, int n, const std::string& text) { + std::atomic joined{0}; + std::atomic delivered{0}; + std::atomic go{false}; + std::vector conns; + conns.reserve(static_cast(n)); + + for (int k = 0; k < n; ++k) { + conns.emplace_back([&, k] { + auto fd = connect_to(port); + if (!fd) { + return; + } + timeval tv{.tv_sec = 10, .tv_usec = 0}; + ::setsockopt(fd->get(), SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + if (!write_all(fd->get(), encode(Type::join, "flooder" + std::to_string(k)))) { + return; + } + std::string inbuf; + // Wait for the WELCOME: proof the server has us in its fan-out set. + for (;;) { + auto fr = recv_frame(fd->get(), inbuf); + if (!fr) { + return; + } + if (fr->type == std::to_underlying(Type::welcome)) { + break; + } + } + joined.fetch_add(1, std::memory_order_relaxed); + while (!go.load(std::memory_order_acquire)) { + std::this_thread::sleep_for(std::chrono::milliseconds{1}); + } + if (k == 0) { + (void)write_all(fd->get(), encode(Type::msg, text)); + } + // Exactly one broadcast is expected — conn0's message reaches all. + for (;;) { + auto fr = recv_frame(fd->get(), inbuf); + if (!fr) { + return; + } + if (fr->type == std::to_underlying(Type::deliver)) { + delivered.fetch_add(1, std::memory_order_relaxed); + return; + } + } + }); + } + + // Wait for every connection to be welcomed before releasing the sender. + for (int i = 0; i < 5000 && joined.load() < n; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds{1}); + } + go.store(true, std::memory_order_release); + conns.clear(); // join all + + const int j = joined.load(); + const int d = delivered.load(); + std::println("flood: connected {}", j); + std::println("flood: delivered {}", d); + return (j == n && d == n) ? 0 : 1; +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +int usage() { + std::println(stderr, "usage: chatterd "); + std::println(stderr, " serve --engine threaded|epoll --port P run the chat daemon"); + std::println(stderr, " send --port P NICK TEXT join, broadcast, print echoes"); + std::println(stderr, " listen --port P NICK --count N join, print N delivered lines"); + std::println(stderr, " flood --port P N [--text TEXT] open N conns, broadcast, count"); + return 2; +} + +[[nodiscard]] std::optional flag(const std::vector& a, + std::string_view name) { + for (std::size_t i = 0; i + 1 < a.size(); ++i) { + if (a[i] == name) { + return std::string{a[i + 1]}; + } + } + return std::nullopt; +} + +[[nodiscard]] std::optional parse_port(const std::optional& s) { + if (!s) { + return std::nullopt; + } + try { + const int p = std::stoi(*s); + if (p < 0 || p > 65535) { + return std::nullopt; + } + return static_cast(p); + } catch (const std::exception&) { + return std::nullopt; + } +} + +} // namespace + +int main(int argc, char** argv) { + const std::vector args{argv + 1, argv + argc}; + if (args.empty()) { + return usage(); + } + const auto cmd = args[0]; + + if (cmd == "serve") { + const auto engine = flag(args, "--engine"); + const auto port = parse_port(flag(args, "--port")); + if (!engine || !port) { + return usage(); + } + if (*engine == "threaded") { + return serve_threaded(*port); + } + if (*engine == "epoll") { + return serve_epoll(*port); + } + return usage(); + } + if (cmd == "send" && args.size() == 5 && args[1] == "--port") { + const auto port = parse_port(std::string{args[2]}); + if (!port) { + return usage(); + } + return cmd_send(*port, std::string{args[3]}, std::string{args[4]}); + } + if (cmd == "listen" && args.size() == 6 && args[1] == "--port" && args[4] == "--count") { + const auto port = parse_port(std::string{args[2]}); + if (!port) { + return usage(); + } + try { + const int c = std::stoi(std::string{args[5]}); + if (c <= 0) { + return usage(); + } + return cmd_listen(*port, std::string{args[3]}, c); + } catch (const std::exception&) { + return usage(); + } + } + if (cmd == "flood" && args.size() >= 4 && args[1] == "--port") { + const auto port = parse_port(std::string{args[2]}); + if (!port) { + return usage(); + } + const std::string text = flag(args, "--text").value_or("flood"); + try { + const int n = std::stoi(std::string{args[3]}); + if (n <= 0) { + return usage(); + } + return cmd_flood(*port, n, text); + } catch (const std::exception&) { + return usage(); + } + } + return usage(); +} diff --git a/examples/22-scaling-the-server/demo.sh b/examples/22-scaling-the-server/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/22-scaling-the-server/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/22-scaling-the-server/go/demo.sh b/examples/22-scaling-the-server/go/demo.sh new file mode 100755 index 0000000..2849fc1 --- /dev/null +++ b/examples/22-scaling-the-server/go/demo.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + # exec so a backgrounded `run serve ...` can be signalled directly + exec "./$BIN" "$@" +} + +demo() { + build + local tmp + tmp=$(mktemp -d) + "./$BIN" serve --engine epoll --port 0 2>"$tmp/serve.err" & + local sp=$! port="" + for _ in $(seq 1 100); do + port=$(sed -n 's/.*on 127\.0\.0\.1:\([0-9]*\).*/\1/p' "$tmp/serve.err") + [[ -n "$port" ]] && break + sleep 0.05 + done + echo "chatterd (epoll) listening on 127.0.0.1:$port" + "./$BIN" listen --port "$port" bob --count 1 >"$tmp/bob.out" & + local lp=$! + sleep 0.2 + "./$BIN" send --port "$port" alice "hello from alice" + wait "$lp" + echo "bob received: $(cat "$tmp/bob.out")" + "./$BIN" flood --port "$port" 50 + kill -TERM "$sp" + wait "$sp" 2>/dev/null || true + cat "$tmp/serve.err" + rm -rf "$tmp" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") demo ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/22-scaling-the-server/go/go.mod b/examples/22-scaling-the-server/go/go.mod new file mode 100644 index 0000000..81a5043 --- /dev/null +++ b/examples/22-scaling-the-server/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/22-scaling-the-server/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/22-scaling-the-server/go/main.go b/examples/22-scaling-the-server/go/main.go new file mode 100644 index 0000000..c19b2ac --- /dev/null +++ b/examples/22-scaling-the-server/go/main.go @@ -0,0 +1,503 @@ +// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// +// ch21 introduced the length-prefixed frame protocol and a threaded server; +// ch22 — "scaling the server" — is about how a server holds thousands of +// connections. The C++ and Rust versions offer two engines: a thread-per- +// connection "threaded" engine and a single-thread "epoll" engine with +// explicit nonblocking read/write state machines. +// +// Go exposes the SAME "--engine threaded|epoll" flag, but BOTH engines are the +// same code: one goroutine per connection reading, one writing, and a central +// hub goroutine that fans messages out over channels. There is no hand-rolled +// epoll here on purpose — the Go runtime's network poller already multiplexes +// every socket through epoll_wait(2) under the hood and parks a goroutine +// (not an OS thread) on each blocked Read/Write. The "epoll" the other +// languages write by hand, Go's netpoller runs for us; the engine flag only +// changes the label we print. Observable behavior is identical across all three. +// +// Wire format (identical across cpp/go/rust and every version), big-endian: +// +// offset 0 2 magic 'C' 'H' (0x43 0x48) +// offset 2 1 version 0x01 +// offset 3 1 type +// offset 4 2 length u16 payload byte count +// offset 6 N payload +// +// type 0x01 JOIN c->s payload = nickname (UTF-8) +// type 0x02 MSG c->s payload = message text (UTF-8) +// type 0x03 DELIVER s->c payload = nick + 0x00 (NUL) + text +// type 0x04 WELCOME s->c payload = empty (sent on accept) +package main + +import ( + "bytes" + "errors" + "fmt" + "net" + "os" + "os/signal" + "strconv" + "sync" + "sync/atomic" + "syscall" + "time" +) + +const version = 0x01 + +const ( + typeJoin = 0x01 + typeMsg = 0x02 + typeDeliver = 0x03 + typeWelcome = 0x04 +) + +// --------------------------------------------------------------------------- +// Frame protocol +// --------------------------------------------------------------------------- + +func encode(typ byte, payload []byte) []byte { + f := make([]byte, 6+len(payload)) + f[0], f[1], f[2], f[3] = 'C', 'H', version, typ + f[4] = byte(len(payload) >> 8) + f[5] = byte(len(payload)) + copy(f[6:], payload) + return f +} + +func deliverPayload(nick string, text []byte) []byte { + p := make([]byte, 0, len(nick)+1+len(text)) + p = append(p, nick...) + p = append(p, 0) + p = append(p, text...) + return p +} + +func splitDeliver(payload []byte) (nick, text string) { + i := bytes.IndexByte(payload, 0) + if i < 0 { + return "?", string(payload) + } + return string(payload[:i]), string(payload[i+1:]) +} + +// frameReader pulls whole frames off a stream, buffering partial reads. +type frameReader struct { + conn net.Conn + buf []byte + tmp []byte +} + +func newFrameReader(conn net.Conn) *frameReader { + return &frameReader{conn: conn, tmp: make([]byte, 4096)} +} + +// next blocks until one whole frame is available; returns the read error +// (io.EOF or a timeout) once the stream can yield no more. +func (fr *frameReader) next() (typ byte, payload []byte, err error) { + for { + if len(fr.buf) >= 6 { + if fr.buf[0] != 'C' || fr.buf[1] != 'H' || fr.buf[2] != version { + return 0, nil, errors.New("protocol desync") + } + ln := int(fr.buf[4])<<8 | int(fr.buf[5]) + if len(fr.buf) >= 6+ln { + typ = fr.buf[3] + payload = append([]byte(nil), fr.buf[6:6+ln]...) + fr.buf = append([]byte(nil), fr.buf[6+ln:]...) + return typ, payload, nil + } + } + n, rerr := fr.conn.Read(fr.tmp) + if n > 0 { + fr.buf = append(fr.buf, fr.tmp[:n]...) + } + if rerr != nil { + return 0, nil, rerr + } + } +} + +func writeAll(conn net.Conn, data []byte) error { + for len(data) > 0 { + n, err := conn.Write(data) + if err != nil { + return err + } + data = data[n:] + } + return nil +} + +// --------------------------------------------------------------------------- +// Server — one hub goroutine fans out to one goroutine-pair per connection. +// --------------------------------------------------------------------------- + +type client struct { + conn net.Conn + send chan []byte + nick string // owned by this client's readPump goroutine +} + +type hub struct { + register chan *client + unregister chan *client + broadcast chan []byte + quit chan struct{} + done chan struct{} + clients map[*client]bool + messages atomic.Uint64 + peak atomic.Int64 +} + +func newHub() *hub { + return &hub{ + register: make(chan *client), + unregister: make(chan *client), + broadcast: make(chan []byte), + quit: make(chan struct{}), + done: make(chan struct{}), + clients: make(map[*client]bool), + } +} + +func (h *hub) run() { + defer close(h.done) + for { + select { + case c := <-h.register: + h.clients[c] = true + if int64(len(h.clients)) > h.peak.Load() { + h.peak.Store(int64(len(h.clients))) + } + c.send <- encode(typeWelcome, nil) + case c := <-h.unregister: + if h.clients[c] { + delete(h.clients, c) + close(c.send) + } + case frame := <-h.broadcast: + h.messages.Add(1) + // Blocking send is the backpressure: a slow client slows the hub + // rather than dropping a frame. writePump keeps draining even on + // write errors, so this never deadlocks on a dead peer. + for c := range h.clients { + c.send <- frame + } + case <-h.quit: + for c := range h.clients { + c.conn.Close() + } + return + } + } +} + +func (h *hub) readPump(c *client) { + defer func() { + select { + case h.unregister <- c: + case <-h.done: + } + }() + fr := newFrameReader(c.conn) + for { + typ, payload, err := fr.next() + if err != nil { + return + } + switch typ { + case typeJoin: + c.nick = string(payload) + case typeMsg: + frame := encode(typeDeliver, deliverPayload(c.nick, payload)) + select { + case h.broadcast <- frame: + case <-h.done: + return + } + } + } +} + +func (c *client) writePump() { + failed := false + for frame := range c.send { + if failed { + continue // keep draining so the hub never blocks on us + } + if err := writeAll(c.conn, frame); err != nil { + failed = true + c.conn.Close() + } + } + c.conn.Close() +} + +func serve(engine string, port int) error { + ln, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", port)) + if err != nil { + return fmt.Errorf("listen: %w", err) + } + bound := ln.Addr().(*net.TCPAddr).Port + fmt.Fprintf(os.Stderr, "chatterd: serving engine=%s on 127.0.0.1:%d\n", engine, bound) + + h := newHub() + go h.run() + + go func() { + for { + conn, err := ln.Accept() + if err != nil { + return // listener closed at shutdown + } + c := &client{conn: conn, send: make(chan []byte, 64), nick: "?"} + select { + case h.register <- c: + go c.writePump() + go h.readPump(c) + case <-h.done: + conn.Close() + return + } + } + }() + + sigs := make(chan os.Signal, 1) + signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM) + <-sigs + + ln.Close() + close(h.quit) + <-h.done + fmt.Fprintf(os.Stderr, "chatterd: stopped engine=%s messages=%d peak_conns=%d\n", + engine, h.messages.Load(), h.peak.Load()) + return nil +} + +// --------------------------------------------------------------------------- +// Client +// --------------------------------------------------------------------------- + +func dial(port int) (net.Conn, error) { + conn, err := net.Dial("tcp", fmt.Sprintf("127.0.0.1:%d", port)) + if err != nil { + return nil, fmt.Errorf("cannot connect to 127.0.0.1:%d: %w", port, err) + } + return conn, nil +} + +func cmdSend(port int, nick, text string) error { + conn, err := dial(port) + if err != nil { + return err + } + defer conn.Close() + if err := writeAll(conn, encode(typeJoin, []byte(nick))); err != nil { + return fmt.Errorf("send: %w", err) + } + if err := writeAll(conn, encode(typeMsg, []byte(text))); err != nil { + return fmt.Errorf("send: %w", err) + } + fr := newFrameReader(conn) + for { + typ, payload, err := fr.next() + if err != nil { + return nil + } + if typ != typeDeliver { + continue // skip WELCOME + } + dn, dt := splitDeliver(payload) + fmt.Printf("%s: %s\n", dn, dt) + if dn == nick && dt == text { + return nil // saw our own message echoed back + } + } +} + +func cmdListen(port int, nick string, count int) error { + conn, err := dial(port) + if err != nil { + return err + } + defer conn.Close() + if err := writeAll(conn, encode(typeJoin, []byte(nick))); err != nil { + return fmt.Errorf("send: %w", err) + } + conn.SetReadDeadline(time.Now().Add(10 * time.Second)) + fr := newFrameReader(conn) + got := 0 + for got < count { + typ, payload, err := fr.next() + if err != nil { + return fmt.Errorf("timed out after %d of %d messages", got, count) + } + if typ != typeDeliver { + continue + } + dn, dt := splitDeliver(payload) + fmt.Printf("%s: %s\n", dn, dt) + got++ + } + return nil +} + +// errQuiet flags a nonzero exit without printing a "chatctl: ..." line — the +// flood counts are already on stdout for the caller to inspect. +var errQuiet = errors.New("flood incomplete") + +func cmdFlood(port, n int, text string) error { + var joined, delivered atomic.Int64 + release := make(chan struct{}) + var wg sync.WaitGroup + + for k := 0; k < n; k++ { + wg.Add(1) + go func(k int) { + defer wg.Done() + conn, err := dial(port) + if err != nil { + return + } + defer conn.Close() + conn.SetReadDeadline(time.Now().Add(10 * time.Second)) + if err := writeAll(conn, encode(typeJoin, []byte(fmt.Sprintf("flooder%d", k)))); err != nil { + return + } + fr := newFrameReader(conn) + // Wait for the WELCOME: proof the server has us in its fan-out set. + for { + typ, _, err := fr.next() + if err != nil { + return + } + if typ == typeWelcome { + break + } + } + joined.Add(1) + <-release + if k == 0 { + if err := writeAll(conn, encode(typeMsg, []byte(text))); err != nil { + return + } + } + // Exactly one broadcast is expected — conn0's message reaches all. + for { + typ, _, err := fr.next() + if err != nil { + return + } + if typ == typeDeliver { + delivered.Add(1) + return + } + } + }(k) + } + + // Wait for every connection to be welcomed before releasing the sender. + for i := 0; i < 5000 && joined.Load() < int64(n); i++ { + time.Sleep(time.Millisecond) + } + close(release) + wg.Wait() + + j, d := int(joined.Load()), int(delivered.Load()) + fmt.Printf("flood: connected %d\n", j) + fmt.Printf("flood: delivered %d\n", d) + if j != n || d != n { + return errQuiet + } + return nil +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +func usage() { + fmt.Fprintln(os.Stderr, "usage: chatterd ") + fmt.Fprintln(os.Stderr, " serve --engine threaded|epoll --port P run the chat daemon") + fmt.Fprintln(os.Stderr, " send --port P NICK TEXT join, broadcast, print echoes") + fmt.Fprintln(os.Stderr, " listen --port P NICK --count N join, print N delivered lines") + fmt.Fprintln(os.Stderr, " flood --port P N [--text TEXT] open N conns, broadcast, count") + os.Exit(2) +} + +func flagVal(args []string, name string) (string, bool) { + for i := 0; i+1 < len(args); i++ { + if args[i] == name { + return args[i+1], true + } + } + return "", false +} + +func parsePort(s string) (int, bool) { + p, err := strconv.Atoi(s) + if err != nil || p < 0 || p > 65535 { + return 0, false + } + return p, true +} + +func main() { + args := os.Args[1:] + if len(args) == 0 { + usage() + } + + var err error + switch { + case args[0] == "serve": + engine, ok1 := flagVal(args, "--engine") + portStr, ok2 := flagVal(args, "--port") + port, ok3 := 0, false + if ok2 { + port, ok3 = parsePort(portStr) + } + if !ok1 || !ok3 || (engine != "threaded" && engine != "epoll") { + usage() + } + err = serve(engine, port) + + case args[0] == "send" && len(args) == 5 && args[1] == "--port": + port, ok := parsePort(args[2]) + if !ok { + usage() + } + err = cmdSend(port, args[3], args[4]) + + case args[0] == "listen" && len(args) == 6 && args[1] == "--port" && args[4] == "--count": + port, ok := parsePort(args[2]) + count, err2 := strconv.Atoi(args[5]) + if !ok || err2 != nil || count <= 0 { + usage() + } + err = cmdListen(port, args[3], count) + + case args[0] == "flood" && len(args) >= 4 && args[1] == "--port": + port, ok := parsePort(args[2]) + n, err2 := strconv.Atoi(args[3]) + if !ok || err2 != nil || n <= 0 { + usage() + } + text := "flood" + if t, ok := flagVal(args, "--text"); ok { + text = t + } + err = cmdFlood(port, n, text) + + default: + usage() + } + + if err != nil { + if !errors.Is(err, errQuiet) { + fmt.Fprintf(os.Stderr, "chatctl: %v\n", err) + } + os.Exit(1) + } +} diff --git a/examples/22-scaling-the-server/rust/Cargo.lock b/examples/22-scaling-the-server/rust/Cargo.lock new file mode 100644 index 0000000..f2cddb4 --- /dev/null +++ b/examples/22-scaling-the-server/rust/Cargo.lock @@ -0,0 +1,93 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chatterd" +version = "0.1.0" +dependencies = [ + "anyhow", + "mio", + "nix", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "mio" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "log", + "wasi", + "windows-sys", +] + +[[package]] +name = "nix" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] diff --git a/examples/22-scaling-the-server/rust/Cargo.toml b/examples/22-scaling-the-server/rust/Cargo.toml new file mode 100644 index 0000000..5624850 --- /dev/null +++ b/examples/22-scaling-the-server/rust/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "chatterd" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +anyhow = "1" +mio = { version = "1", features = ["os-poll", "os-ext", "net"] } +nix = { version = "0.30", features = ["signal", "poll"] } diff --git a/examples/22-scaling-the-server/rust/demo.sh b/examples/22-scaling-the-server/rust/demo.sh new file mode 100755 index 0000000..c199ade --- /dev/null +++ b/examples/22-scaling-the-server/rust/demo.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + # exec so a backgrounded `run serve ...` can be signalled directly + exec "./$BIN" "$@" + fi +} + +demo() { + build + local tmp + tmp=$(mktemp -d) + "./$BIN" serve --engine epoll --port 0 2>"$tmp/serve.err" & + local sp=$! port="" + for _ in $(seq 1 100); do + port=$(sed -n 's/.*on 127\.0\.0\.1:\([0-9]*\).*/\1/p' "$tmp/serve.err") + [[ -n "$port" ]] && break + sleep 0.05 + done + echo "chatterd (epoll) listening on 127.0.0.1:$port" + "./$BIN" listen --port "$port" bob --count 1 >"$tmp/bob.out" & + local lp=$! + sleep 0.2 + "./$BIN" send --port "$port" alice "hello from alice" + wait "$lp" + echo "bob received: $(cat "$tmp/bob.out")" + "./$BIN" flood --port "$port" 50 + kill -TERM "$sp" + wait "$sp" 2>/dev/null || true + cat "$tmp/serve.err" + rm -rf "$tmp" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") demo ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/22-scaling-the-server/rust/rust-toolchain.toml b/examples/22-scaling-the-server/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/22-scaling-the-server/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/22-scaling-the-server/rust/src/main.rs b/examples/22-scaling-the-server/rust/src/main.rs new file mode 100644 index 0000000..0490ad0 --- /dev/null +++ b/examples/22-scaling-the-server/rust/src/main.rs @@ -0,0 +1,717 @@ +// chatterd v1 — a peer-to-peer chat daemon growing chapter by chapter. +// +// ch21 introduced the length-prefixed frame protocol and a threaded server +// (one thread per connection). ch22 — "scaling the server" — keeps that +// threaded engine working and adds a single-threaded epoll engine, built with +// mio: one thread, many connections, nonblocking accept plus a per-connection +// read/write state machine with an outbound queue and real backpressure +// (partial writes are carried across poll iterations, WRITABLE drives the flush). +// +// Wire format (identical across cpp/go/rust and every version), big-endian: +// +// offset 0 2 magic 'C' 'H' (0x43 0x48) +// offset 2 1 version 0x01 +// offset 3 1 type +// offset 4 2 length u16 payload byte count +// offset 6 N payload +// +// type 0x01 JOIN c->s payload = nickname (UTF-8) +// type 0x02 MSG c->s payload = message text (UTF-8) +// type 0x03 DELIVER s->c payload = nick + 0x00 (NUL) + text +// type 0x04 WELCOME s->c payload = empty (sent on accept) +// +// The server delivers every MSG to every connected client, including the +// originator, so a client confirms acceptance by seeing its own line echoed. + +use std::collections::HashMap; +use std::io::{ErrorKind, Read, Write}; +use std::net::{Shutdown, TcpListener as StdTcpListener, TcpStream as StdTcpStream}; +use std::os::fd::{AsFd, AsRawFd, BorrowedFd}; +use std::process::ExitCode; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread::{self, JoinHandle}; +use std::time::Duration; + +use anyhow::{Context, Result}; +use mio::net::{TcpListener as MioTcpListener, TcpStream as MioTcpStream}; +use mio::unix::SourceFd; +use mio::{Events, Interest, Poll, Token}; +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sys::signal::{SigSet, Signal}; +use nix::sys::signalfd::{SfdFlags, SignalFd}; + +const VERSION: u8 = 0x01; +const TYPE_JOIN: u8 = 0x01; +const TYPE_MSG: u8 = 0x02; +const TYPE_DELIVER: u8 = 0x03; +const TYPE_WELCOME: u8 = 0x04; + +// --------------------------------------------------------------------------- +// Frame protocol +// --------------------------------------------------------------------------- + +fn encode(typ: u8, payload: &[u8]) -> Vec { + let mut f = Vec::with_capacity(6 + payload.len()); + f.extend_from_slice(&[ + b'C', + b'H', + VERSION, + typ, + (payload.len() >> 8) as u8, + payload.len() as u8, + ]); + f.extend_from_slice(payload); + f +} + +fn deliver_payload(nick: &str, text: &[u8]) -> Vec { + let nb = nick.as_bytes(); + let mut p = Vec::with_capacity(nb.len() + 1 + text.len()); + p.extend_from_slice(nb); + p.push(0); + p.extend_from_slice(text); + p +} + +fn split_deliver(payload: &[u8]) -> (String, String) { + match payload.iter().position(|&b| b == 0) { + Some(nul) => ( + String::from_utf8_lossy(&payload[..nul]).into_owned(), + String::from_utf8_lossy(&payload[nul + 1..]).into_owned(), + ), + None => ("?".into(), String::from_utf8_lossy(payload).into_owned()), + } +} + +// Pull one complete frame off the front of buf, consuming its bytes. Returns +// None when buf holds only a partial frame (need more bytes). +fn take_frame(buf: &mut Vec) -> Option<(u8, Vec)> { + if buf.len() < 6 { + return None; + } + if buf[0] != b'C' || buf[1] != b'H' || buf[2] != VERSION { + buf.clear(); // desync on a foreign speaker; drop the stream + return None; + } + let len = ((buf[4] as usize) << 8) | buf[5] as usize; + if buf.len() < 6 + len { + return None; + } + let typ = buf[3]; + let payload = buf[6..6 + len].to_vec(); + buf.drain(0..6 + len); + Some((typ, payload)) +} + +// --------------------------------------------------------------------------- +// Signals — SIGINT/SIGTERM blocked process-wide, delivered as readable data. +// --------------------------------------------------------------------------- + +fn block_signals() -> Result { + let mut set = SigSet::empty(); + set.add(Signal::SIGINT); + set.add(Signal::SIGTERM); + set.thread_block().context("sigprocmask")?; + SignalFd::with_flags(&set, SfdFlags::SFD_NONBLOCK | SfdFlags::SFD_CLOEXEC) + .context("signalfd") +} + +// --------------------------------------------------------------------------- +// Threaded engine — one std thread per connection, a shared broadcast registry. +// --------------------------------------------------------------------------- + +struct ConnHandle { + id: u64, + w: Mutex, // a clone used for writes + shutdown + nick: Mutex, +} + +struct ConnRegistry { + conns: Mutex>>, + messages: AtomicU64, + peak: AtomicUsize, +} + +impl ConnRegistry { + fn new() -> Self { + Self { + conns: Mutex::new(HashMap::new()), + messages: AtomicU64::new(0), + peak: AtomicUsize::new(0), + } + } + fn add(&self, h: Arc) { + let mut m = self.conns.lock().unwrap(); + m.insert(h.id, h); + let n = m.len(); + if n > self.peak.load(Ordering::Relaxed) { + self.peak.store(n, Ordering::Relaxed); + } + } + fn remove(&self, id: u64) { + self.conns.lock().unwrap().remove(&id); + } + fn broadcast(&self, frame: &[u8]) { + let snap: Vec> = self.conns.lock().unwrap().values().cloned().collect(); + for h in snap { + let mut s = h.w.lock().unwrap(); + let _ = s.write_all(frame); // a dead peer is reaped by its reader + } + } + fn shutdown_all(&self) { + for h in self.conns.lock().unwrap().values() { + let s = h.w.lock().unwrap(); + let _ = s.shutdown(Shutdown::Both); // wake the reader blocked in read() + } + } +} + +fn serve_conn(mut stream: StdTcpStream, handle: Arc, reg: Arc) { + { + let mut w = handle.w.lock().unwrap(); + let _ = w.write_all(&encode(TYPE_WELCOME, &[])); + } + let mut inbuf: Vec = Vec::new(); + let mut buf = [0u8; 4096]; + while let Ok(n) = stream.read(&mut buf) { + if n == 0 { + break; // EOF or shutdown() at teardown + } + inbuf.extend_from_slice(&buf[..n]); + while let Some((typ, payload)) = take_frame(&mut inbuf) { + if typ == TYPE_JOIN { + *handle.nick.lock().unwrap() = String::from_utf8_lossy(&payload).into_owned(); + } else if typ == TYPE_MSG { + reg.messages.fetch_add(1, Ordering::Relaxed); + let nick = handle.nick.lock().unwrap().clone(); + reg.broadcast(&encode(TYPE_DELIVER, &deliver_payload(&nick, &payload))); + } + } + } + reg.remove(handle.id); +} + +fn serve_threaded(port: u16) -> Result<()> { + let listener = StdTcpListener::bind(("127.0.0.1", port)).context("listen")?; + let bound = listener.local_addr().context("getsockname")?.port(); + let sigfd = block_signals()?; + let sig_raw = sigfd.as_raw_fd(); + eprintln!("chatterd: serving engine=threaded on 127.0.0.1:{bound}"); + + let reg = Arc::new(ConnRegistry::new()); + let mut workers: Vec> = Vec::new(); + let mut next_id: u64 = 0; + + 'accept: loop { + let sig_borrow = unsafe { BorrowedFd::borrow_raw(sig_raw) }; + let mut fds = [ + PollFd::new(listener.as_fd(), PollFlags::POLLIN), + PollFd::new(sig_borrow, PollFlags::POLLIN), + ]; + match poll(&mut fds, PollTimeout::NONE) { + Ok(_) => {} + Err(nix::errno::Errno::EINTR) => continue, + Err(e) => return Err(e).context("poll"), + } + let sig_ready = fds[1].revents().is_some_and(|r| r.contains(PollFlags::POLLIN)); + let listen_ready = fds[0].revents().is_some_and(|r| r.contains(PollFlags::POLLIN)); + if sig_ready { + break 'accept; // SIGINT/SIGTERM + } + if listen_ready { + let (stream, _) = match listener.accept() { + Ok(pair) => pair, + Err(_) => continue, + }; + let wclone = stream.try_clone().context("try_clone")?; + let id = next_id; + next_id += 1; + let handle = Arc::new(ConnHandle { + id, + w: Mutex::new(wclone), + nick: Mutex::new("?".into()), + }); + reg.add(handle.clone()); + let reg2 = reg.clone(); + workers.push(thread::spawn(move || serve_conn(stream, handle, reg2))); + } + } + + reg.shutdown_all(); + for w in workers { + let _ = w.join(); + } + eprintln!( + "chatterd: stopped engine=threaded messages={} peak_conns={}", + reg.messages.load(Ordering::Relaxed), + reg.peak.load(Ordering::Relaxed) + ); + Ok(()) +} + +// --------------------------------------------------------------------------- +// Epoll engine — one thread, mio-driven, per-connection state machines. +// --------------------------------------------------------------------------- + +struct EConn { + stream: MioTcpStream, + nick: String, + inbuf: Vec, + outbuf: Vec, // pending outbound bytes (the backpressure queue) + want_write: bool, + dead: bool, +} + +// Append to a connection's outbound queue and try to drain it now. +fn queue(reg: &mio::Registry, conns: &mut HashMap, tok: usize, frame: &[u8]) { + if let Some(c) = conns.get_mut(&tok) { + c.outbuf.extend_from_slice(frame); + } + flush(reg, conns, tok); +} + +fn flush(reg: &mio::Registry, conns: &mut HashMap, tok: usize) { + let c = match conns.get_mut(&tok) { + Some(c) => c, + None => return, + }; + while !c.outbuf.is_empty() { + match c.stream.write(&c.outbuf) { + Ok(0) => { + c.dead = true; + return; + } + Ok(n) => { + c.outbuf.drain(0..n); + } + Err(e) if e.kind() == ErrorKind::WouldBlock => break, // buffer full: wait for WRITABLE + Err(e) if e.kind() == ErrorKind::Interrupted => continue, + Err(_) => { + c.dead = true; + return; + } + } + } + let need = !c.outbuf.is_empty(); + if need != c.want_write { + c.want_write = need; + let interest = if need { + Interest::READABLE | Interest::WRITABLE + } else { + Interest::READABLE + }; + let _ = reg.reregister(&mut c.stream, Token(tok), interest); + } +} + +fn broadcast(reg: &mio::Registry, conns: &mut HashMap, frame: &[u8]) { + let toks: Vec = conns.keys().copied().collect(); + for tok in toks { + queue(reg, conns, tok, frame); + } +} + +fn on_readable( + reg: &mio::Registry, + conns: &mut HashMap, + tok: usize, + messages: &mut u64, +) { + let mut tmp = [0u8; 4096]; + loop { + let res = match conns.get_mut(&tok) { + Some(c) => c.stream.read(&mut tmp), + None => return, + }; + match res { + Ok(0) => { + conns.get_mut(&tok).unwrap().dead = true; + break; + } + Ok(n) => conns.get_mut(&tok).unwrap().inbuf.extend_from_slice(&tmp[..n]), + Err(e) if e.kind() == ErrorKind::WouldBlock => break, + Err(e) if e.kind() == ErrorKind::Interrupted => continue, + Err(_) => { + conns.get_mut(&tok).unwrap().dead = true; + break; + } + } + } + loop { + let frame = conns.get_mut(&tok).and_then(|c| take_frame(&mut c.inbuf)); + match frame { + None => break, + Some((typ, payload)) => { + if typ == TYPE_JOIN { + if let Some(c) = conns.get_mut(&tok) { + c.nick = String::from_utf8_lossy(&payload).into_owned(); + } + } else if typ == TYPE_MSG { + *messages += 1; + let nick = conns.get(&tok).map(|c| c.nick.clone()).unwrap_or_else(|| "?".into()); + let frame = encode(TYPE_DELIVER, &deliver_payload(&nick, &payload)); + broadcast(reg, conns, &frame); + } + } + } + } +} + +fn accept_ready( + reg: &mio::Registry, + conns: &mut HashMap, + listener: &mut MioTcpListener, + next_tok: &mut usize, + peak: &mut usize, +) { + loop { + match listener.accept() { + Ok((mut stream, _)) => { + let tok = *next_tok; + *next_tok += 1; + if reg.register(&mut stream, Token(tok), Interest::READABLE).is_err() { + continue; + } + conns.insert( + tok, + EConn { + stream, + nick: "?".into(), + inbuf: Vec::new(), + outbuf: Vec::new(), + want_write: false, + dead: false, + }, + ); + if conns.len() > *peak { + *peak = conns.len(); + } + queue(reg, conns, tok, &encode(TYPE_WELCOME, &[])); + } + Err(e) if e.kind() == ErrorKind::WouldBlock => break, + Err(_) => break, + } + } +} + +fn reap(reg: &mio::Registry, conns: &mut HashMap) { + let dead: Vec = conns.iter().filter(|(_, c)| c.dead).map(|(t, _)| *t).collect(); + for tok in dead { + if let Some(mut c) = conns.remove(&tok) { + let _ = reg.deregister(&mut c.stream); + } + } +} + +fn serve_epoll(port: u16) -> Result<()> { + let mut poll_inst = Poll::new().context("epoll_create1")?; + let reg = poll_inst.registry().try_clone().context("dup epoll")?; + + let addr = format!("127.0.0.1:{port}").parse().expect("valid loopback addr"); + let mut listener = MioTcpListener::bind(addr).context("listen")?; + let bound = listener.local_addr().context("getsockname")?.port(); + reg.register(&mut listener, Token(0), Interest::READABLE).context("epoll_ctl")?; + + let sigfd = block_signals()?; + let sig_raw = sigfd.as_raw_fd(); + reg.register(&mut SourceFd(&sig_raw), Token(1), Interest::READABLE).context("epoll_ctl")?; + + eprintln!("chatterd: serving engine=epoll on 127.0.0.1:{bound}"); + + let mut conns: HashMap = HashMap::new(); + let mut next_tok: usize = 2; + let mut peak: usize = 0; + let mut messages: u64 = 0; + let mut events = Events::with_capacity(1024); + + loop { + if let Err(e) = poll_inst.poll(&mut events, None) { + if e.kind() == ErrorKind::Interrupted { + continue; + } + return Err(e).context("epoll_wait"); + } + for event in events.iter() { + match event.token().0 { + 0 => accept_ready(®, &mut conns, &mut listener, &mut next_tok, &mut peak), + 1 => { + reap(®, &mut conns); + eprintln!( + "chatterd: stopped engine=epoll messages={messages} peak_conns={peak}" + ); + return Ok(()); + } + tok => { + if !conns.contains_key(&tok) { + continue; // reaped earlier in this batch + } + if (event.is_read_closed() || event.is_error()) + && let Some(c) = conns.get_mut(&tok) + { + c.dead = true; + } + if event.is_readable() { + on_readable(®, &mut conns, tok, &mut messages); + } + if conns.contains_key(&tok) && event.is_writable() { + flush(®, &mut conns, tok); + } + } + } + } + reap(®, &mut conns); + } +} + +// --------------------------------------------------------------------------- +// Client +// --------------------------------------------------------------------------- + +fn connect_to(port: u16) -> Result { + StdTcpStream::connect(("127.0.0.1", port)) + .with_context(|| format!("cannot connect to 127.0.0.1:{port}")) +} + +// Blocking frame reader over a std stream, buffering partial reads. +struct FrameStream { + stream: StdTcpStream, + buf: Vec, + tmp: [u8; 4096], +} + +impl FrameStream { + fn new(stream: StdTcpStream) -> Self { + Self { stream, buf: Vec::new(), tmp: [0u8; 4096] } + } + // Ok(Some(frame)); Ok(None) on EOF or SO_RCVTIMEO timeout. + fn next(&mut self) -> std::io::Result)>> { + loop { + if let Some(f) = take_frame(&mut self.buf) { + return Ok(Some(f)); + } + match self.stream.read(&mut self.tmp) { + Ok(0) => return Ok(None), + Ok(n) => self.buf.extend_from_slice(&self.tmp[..n]), + Err(e) if e.kind() == ErrorKind::WouldBlock || e.kind() == ErrorKind::TimedOut => { + return Ok(None); + } + Err(e) if e.kind() == ErrorKind::Interrupted => continue, + Err(e) => return Err(e), + } + } + } +} + +fn cmd_send(port: u16, nick: &str, text: &str) -> Result { + let stream = connect_to(port)?; + stream.set_nodelay(true).ok(); + let wstream = stream.try_clone().context("try_clone")?; + let mut w = wstream; + w.write_all(&encode(TYPE_JOIN, nick.as_bytes())).context("send")?; + w.write_all(&encode(TYPE_MSG, text.as_bytes())).context("send")?; + let mut fs = FrameStream::new(stream); + while let Some((typ, payload)) = fs.next().context("recv")? { + if typ != TYPE_DELIVER { + continue; // skip WELCOME + } + let (dn, dt) = split_deliver(&payload); + println!("{dn}: {dt}"); + if dn == nick && dt == text { + return Ok(ExitCode::SUCCESS); // saw our own message echoed back + } + } + Ok(ExitCode::SUCCESS) +} + +fn cmd_listen(port: u16, nick: &str, count: usize) -> Result { + let stream = connect_to(port)?; + stream.set_read_timeout(Some(Duration::from_secs(10))).ok(); + let mut w = stream.try_clone().context("try_clone")?; + w.write_all(&encode(TYPE_JOIN, nick.as_bytes())).context("send")?; + let mut fs = FrameStream::new(stream); + let mut got = 0usize; + while got < count { + match fs.next().context("recv")? { + None => { + eprintln!("chatctl: timed out after {got} of {count} messages"); + return Ok(ExitCode::from(1)); + } + Some((typ, payload)) => { + if typ != TYPE_DELIVER { + continue; + } + let (dn, dt) = split_deliver(&payload); + println!("{dn}: {dt}"); + got += 1; + } + } + } + Ok(ExitCode::SUCCESS) +} + +fn cmd_flood(port: u16, n: usize, text: &str) -> Result { + let joined = Arc::new(AtomicUsize::new(0)); + let delivered = Arc::new(AtomicUsize::new(0)); + let go = Arc::new(AtomicBool::new(false)); + let mut handles: Vec> = Vec::with_capacity(n); + + for k in 0..n { + let joined = joined.clone(); + let delivered = delivered.clone(); + let go = go.clone(); + let text = text.to_string(); + handles.push(thread::spawn(move || { + let stream = match connect_to(port) { + Ok(s) => s, + Err(_) => return, + }; + stream.set_read_timeout(Some(Duration::from_secs(10))).ok(); + let mut w = match stream.try_clone() { + Ok(s) => s, + Err(_) => return, + }; + if w.write_all(&encode(TYPE_JOIN, format!("flooder{k}").as_bytes())).is_err() { + return; + } + let mut fs = FrameStream::new(stream); + // Wait for the WELCOME: proof the server has us in its fan-out set. + loop { + match fs.next() { + Ok(Some((typ, _))) if typ == TYPE_WELCOME => break, + Ok(Some(_)) => continue, + _ => return, + } + } + joined.fetch_add(1, Ordering::Relaxed); + while !go.load(Ordering::Acquire) { + thread::sleep(Duration::from_millis(1)); + } + if k == 0 { + let _ = w.write_all(&encode(TYPE_MSG, text.as_bytes())); + } + // Exactly one broadcast is expected — conn0's message reaches all. + loop { + match fs.next() { + Ok(Some((typ, _))) if typ == TYPE_DELIVER => { + delivered.fetch_add(1, Ordering::Relaxed); + return; + } + Ok(Some(_)) => continue, + _ => return, + } + } + })); + } + + // Wait for every connection to be welcomed before releasing the sender. + for _ in 0..5000 { + if joined.load(Ordering::Relaxed) >= n { + break; + } + thread::sleep(Duration::from_millis(1)); + } + go.store(true, Ordering::Release); + for h in handles { + let _ = h.join(); + } + + let j = joined.load(Ordering::Relaxed); + let d = delivered.load(Ordering::Relaxed); + println!("flood: connected {j}"); + println!("flood: delivered {d}"); + if j == n && d == n { + Ok(ExitCode::SUCCESS) + } else { + Ok(ExitCode::from(1)) + } +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +fn usage() -> ExitCode { + eprintln!("usage: chatterd "); + eprintln!(" serve --engine threaded|epoll --port P run the chat daemon"); + eprintln!(" send --port P NICK TEXT join, broadcast, print echoes"); + eprintln!(" listen --port P NICK --count N join, print N delivered lines"); + eprintln!(" flood --port P N [--text TEXT] open N conns, broadcast, count"); + ExitCode::from(2) +} + +fn flag(args: &[String], name: &str) -> Option { + args.windows(2).find(|w| w[0] == name).map(|w| w[1].clone()) +} + +fn parse_port(s: &str) -> Option { + s.parse::().ok().filter(|p| *p <= 65535).map(|p| p as u16) +} + +fn run() -> Result { + let args: Vec = std::env::args().skip(1).collect(); + if args.is_empty() { + return Ok(usage()); + } + match args[0].as_str() { + "serve" => { + let engine = flag(&args, "--engine"); + let port = flag(&args, "--port").as_deref().and_then(parse_port); + let (engine, port) = match (engine, port) { + (Some(e), Some(p)) => (e, p), + _ => return Ok(usage()), + }; + let result = match engine.as_str() { + "threaded" => serve_threaded(port), + "epoll" => serve_epoll(port), + _ => return Ok(usage()), + }; + match result { + Ok(()) => Ok(ExitCode::SUCCESS), + Err(e) => { + eprintln!("chatterd: {e:#}"); + Ok(ExitCode::from(1)) + } + } + } + "send" if args.len() == 5 && args[1] == "--port" => { + let port = match parse_port(&args[2]) { + Some(p) => p, + None => return Ok(usage()), + }; + cmd_send(port, &args[3], &args[4]) + } + "listen" if args.len() == 6 && args[1] == "--port" && args[4] == "--count" => { + let port = match parse_port(&args[2]) { + Some(p) => p, + None => return Ok(usage()), + }; + match args[5].parse::() { + Ok(c) if c > 0 => cmd_listen(port, &args[3], c), + _ => Ok(usage()), + } + } + "flood" if args.len() >= 4 && args[1] == "--port" => { + let port = match parse_port(&args[2]) { + Some(p) => p, + None => return Ok(usage()), + }; + let text = flag(&args, "--text").unwrap_or_else(|| "flood".into()); + match args[3].parse::() { + Ok(n) if n > 0 => cmd_flood(port, n, &text), + _ => Ok(usage()), + } + } + _ => Ok(usage()), + } +} + +fn main() -> ExitCode { + match run() { + Ok(code) => code, + Err(e) => { + eprintln!("chatctl: {e:#}"); + ExitCode::from(1) + } + } +} diff --git a/examples/22-scaling-the-server/verify.lua b/examples/22-scaling-the-server/verify.lua new file mode 100644 index 0000000..1bb0f97 --- /dev/null +++ b/examples/22-scaling-the-server/verify.lua @@ -0,0 +1,152 @@ +-- Verify chatterd v1 (22-scaling-the-server) for LSP_LANG. +-- +-- Asserts observable behavior, identical across cpp/go/rust: +-- 1. epoll engine, two-party transcript — a listener receives the exact +-- "alice: hello world" line the sender broadcast; server stop summary +-- reports messages=1 peak_conns=2; clean exit +-- 2. epoll engine, load — `flood 50` opens 50 concurrent connections and one +-- broadcast is DELIVERED to all 50 (delivered==connected==50, no dropped +-- frames), exit 0; server stop summary reports messages=1 peak_conns=50 +-- 3. threaded engine — the same two-party transcript and 50-client flood +-- still work and shut down cleanly +-- 4. error shapes — no subcommand and bad flags exit 2; a client against a +-- dead port exits 1 +-- +-- Each scenario runs its own fresh daemon so the message and peak counts are +-- deterministic. Invoked from the example directory with LSP_LANG (cpp|go|rust) +-- and EXAMPLE_DIR set; the interpreter may be lua or luajit. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +local wd = checks.run("mktemp -d").out:gsub("%s+$", "") +if wd == "" or not wd:match("^/") then + checks.skip("mktemp -d did not yield a directory") +end + +-- Run a fresh daemon of ENGINE, discover its ephemeral port, execute BODY +-- (which may use $port and @WD@/@DEMO@), then SIGTERM the daemon and dump its +-- stderr. Returns the combined transcript + exit code. +local function scenario(engine, body) + local template = [[ +set -e +@DEMO@ serve --engine @ENGINE@ --port 0 2>@WD@/serve.err & +spid=$! +port="" +for _ in $(seq 1 200); do + port=$(sed -n 's/.*on 127\.0\.0\.1:\([0-9]*\).*/\1/p' @WD@/serve.err) + [ -n "$port" ] && break + sleep 0.05 +done +[ -n "$port" ] || { echo NO_PORT; kill $spid 2>/dev/null; exit 3; } +@BODY@ +kill -TERM $spid 2>/dev/null || true +for _ in $(seq 1 200); do kill -0 $spid 2>/dev/null || break; sleep 0.05; done +echo "=== serve.err ===" +cat @WD@/serve.err +]] + local script = template + :gsub("@ENGINE@", engine) + :gsub("@BODY@", body) + :gsub("@DEMO@", demo) + :gsub("@WD@", wd) + return checks.run(script) +end + +local two_party = [[ +@DEMO@ listen --port $port bob --count 1 > @WD@/bob.out 2>/dev/null & +lp=$! +sleep 0.3 +if @DEMO@ send --port $port alice "hello world" > @WD@/alice.out 2>/dev/null; then se=0; else se=$?; fi +echo "SEND_EXIT=$se" +if wait $lp; then le=0; else le=$?; fi +echo "LISTEN_EXIT=$le" +echo "ALICE=$(cat @WD@/alice.out)" +echo "BOB=$(cat @WD@/bob.out)" +]] + +local flood = [[ +if @DEMO@ flood --port $port 50 > @WD@/flood.out 2>/dev/null; then fe=0; else fe=$?; fi +echo "FLOOD_EXIT=$fe" +cat @WD@/flood.out +]] + +-- --------------------------------------------------------------------------- +-- 1. epoll engine — two-party transcript +-- --------------------------------------------------------------------------- + +local e1 = scenario("epoll", two_party) +checks.expect_exit(e1, 0, lang .. ": epoll two-party scenario exits 0") +checks.expect_match(e1.out, "chatterd: serving engine=epoll on 127%.0%.0%.1:%d+", + lang .. ": epoll daemon announces its bound loopback port") +checks.expect_match(e1.out, "SEND_EXIT=0", lang .. ": epoll send exits 0") +checks.expect_match(e1.out, "LISTEN_EXIT=0", lang .. ": epoll listen exits 0") +checks.expect_match(e1.out, "ALICE=alice: hello world", + lang .. ": epoll sender sees its own message echoed back") +checks.expect_match(e1.out, "BOB=alice: hello world", + lang .. ": epoll listener receives the broadcast verbatim") +checks.expect_match(e1.out, "chatterd: stopped engine=epoll messages=1 peak_conns=2", + lang .. ": epoll stop summary reports 1 message across 2 peak connections") + +-- --------------------------------------------------------------------------- +-- 2. epoll engine — 50-connection flood, broadcast delivered to all +-- --------------------------------------------------------------------------- + +local e2 = scenario("epoll", flood) +checks.expect_exit(e2, 0, lang .. ": epoll flood scenario exits 0") +checks.expect_match(e2.out, "flood: connected 50", + lang .. ": epoll accepts all 50 concurrent flood connections") +checks.expect_match(e2.out, "flood: delivered 50", + lang .. ": epoll broadcast is delivered to all 50 connections (no dropped frames)") +checks.expect_match(e2.out, "FLOOD_EXIT=0", lang .. ": epoll flood exits 0") +checks.expect_match(e2.out, "chatterd: stopped engine=epoll messages=1 peak_conns=50", + lang .. ": epoll stop summary reports 1 message across 50 peak connections") + +-- --------------------------------------------------------------------------- +-- 3. threaded engine — same transcript and flood still work +-- --------------------------------------------------------------------------- + +local t1 = scenario("threaded", two_party) +checks.expect_exit(t1, 0, lang .. ": threaded two-party scenario exits 0") +checks.expect_match(t1.out, "BOB=alice: hello world", + lang .. ": threaded listener receives the broadcast verbatim") +checks.expect_match(t1.out, "chatterd: stopped engine=threaded messages=1 peak_conns=2", + lang .. ": threaded stop summary reports 1 message across 2 peak connections") + +local t2 = scenario("threaded", flood) +checks.expect_exit(t2, 0, lang .. ": threaded flood scenario exits 0") +checks.expect_match(t2.out, "flood: delivered 50", + lang .. ": threaded broadcast is delivered to all 50 connections") +checks.expect_match(t2.out, "chatterd: stopped engine=threaded messages=1 peak_conns=50", + lang .. ": threaded stop summary reports 1 message across 50 peak connections") + +-- --------------------------------------------------------------------------- +-- 4. error shapes +-- --------------------------------------------------------------------------- + +local noargs = checks.run(demo) +checks.expect_exit(noargs, 2, lang .. ": no subcommand exits 2") +checks.expect_match(noargs.out, "usage: chatterd", lang .. ": no subcommand prints usage") + +local badeng = checks.run(demo .. " serve --engine bogus --port 0") +checks.expect_exit(badeng, 2, lang .. ": unknown engine exits 2") + +local badargs = checks.run(demo .. " flood --port 0") +checks.expect_exit(badargs, 2, lang .. ": flood without a count exits 2") + +local dead = checks.run(demo .. " send --port 1 alice hi") +checks.expect_exit(dead, 1, lang .. ": a client against a dead port exits 1") +checks.expect_match(dead.out, "chatctl: cannot connect", + lang .. ": the connect failure is reported") + +checks.run("rm -rf " .. wd) +checks.finish() diff --git a/examples/23-udp-and-peer-discovery/README.md b/examples/23-udp-and-peer-discovery/README.md new file mode 100644 index 0000000..f9e35b4 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/README.md @@ -0,0 +1,182 @@ +# 23-udp-and-peer-discovery — chatterd v2 + +chatterd grows peer discovery. Where chapters 21–22 built the TCP side (the +canonical chatterd chat frame between two known endpoints), `chatterd +discover` adds the missing half: finding peers you were never told about. It +sends periodic **UDP multicast beacons**, listens for everyone else's, and the +first time it hears a new peer it opens a TCP connection to that peer's +advertised port and exchanges one chat frame each way — the same canonical +wire frame every chatterd version speaks, reused here for JOIN/DELIVER. + +``` +$ ./demo.sh go run discover --group 239.23.7.1 --port 51823 \ + --name alice --tcp-port 9231 --iface 127.0.0.1 & +$ ./demo.sh go run discover --group 239.23.7.1 --port 51823 \ + --name bob --tcp-port 9232 --iface 127.0.0.1 & + +# alice's stdout: +discovered peer bob at 127.0.0.1:9232 +peer bob says: hello from bob + +# bob's stdout: +discovered peer alice at 127.0.0.1:9231 +peer alice says: hello from alice +``` + +Each process announces on stderr (`chatterd: announcing as on +: (tcp :)`), runs for `--rounds` beacons spaced +`--announce-ms` apart, then exits 0 after an 800 ms grace window so in-flight +exchanges finish. + +## What the chapter is really about + +UDP multicast is the one-to-many primitive. A beacon sent once to a group +address (`239.0.0.0/8`, the administratively-scoped block) reaches every +member on the interface without the sender knowing who or how many they are. +Discovery is: **join the group, announce yourself, and learn from what you +hear.** The three implementations exercise the same setsockopt dance — + +- **`IP_ADD_MEMBERSHIP`** (`ip_mreq { imr_multiaddr = group, imr_interface = + iface }`) — join the group on a specific interface. Without this the kernel + drops group traffic before it reaches the socket. +- **`IP_MULTICAST_IF`** — pin *outgoing* multicast to the same interface, so a + multi-homed host sends the beacon where its peers are listening. +- **`IP_MULTICAST_LOOP`** — deliver our own multicast to other sockets **on + this host**. This is what makes the local two-process verify work over the + loopback interface; across two machines it is irrelevant (the packet is on + the wire either way). +- **`IP_MULTICAST_TTL = 1`** — keep beacons on the local link; do not let a + router forward them. + +The receive side dedupes by peer name, so "discovered peer …" prints exactly +once per peer no matter how many beacons arrive. + +### Loopback multicast note + +The loopback device `lo` is not flagged `MULTICAST`, but the IPv4 stack still +loops group traffic **when the multicast interface is set to `127.0.0.1`**. +That is why the local verify passes `--iface 127.0.0.1`: both processes join +and send on the loopback address and the kernel loops each beacon to the +other. On the VMs the interface is the guest's `enp1s0` address and the +beacons cross the `virbr0` bridge for real. + +## Wire formats (byte-exact, identical across all three languages) + +### 1. Discovery beacon — UDP datagram to `:` + +An ASCII text datagram, no framing (UDP preserves message boundaries): + +``` +CHATTERD1 +``` + +- `CHATTERD1` — literal magic / version tag (9 bytes). +- single ASCII spaces separate the four fields; `` contains no space. +- `` — decimal, the TCP port this peer accepts chat frames on. +- `` — dotted-quad the peer wants to be dialed at (its `--iface`). + +A receiver splits on whitespace, requires exactly 4 fields with `tok[0] == +"CHATTERD1"`, and ignores a beacon whose `` equals its own. + +Captured on the peer guest's `enp1s0` during the two-host run (`tcpdump -X`), +a beacon from systems-target — 35 payload bytes: + +``` +IP 192.168.124.7.40516 > 239.23.7.1.51824: UDP, length 35 + 0x001a: 4348 4154 ....CHAT + 0x0020: 5445 5244 3120 7461 7267 6574 2039 3234 TERD1.target.924 + 0x0030: 3120 3139 322e 3136 382e 3132 342e 37 1.192.168.124.7 += "CHATTERD1 target 9241 192.168.124.7" +``` + +### 2. Chat frame — TCP, the canonical chatterd frame (reused, not this chapter's invention) + +This is **not** a ch23-specific format — it is the one wire frame every +chatterd version (ch21's thread-per-connection server, ch22's epoll engine, +ch24's heartbeat client, ch27's async transport) reads and writes. ch23 just +dials it peer-to-peer instead of client-to-server. The UDP beacon above is a +completely separate object; this frame only ever travels over the TCP +connection opened *after* discovery. + +``` ++-------+-------+------+-----------------+------------------------+ +| magic | ver | type | length | payload | +| 2B | 1B | 1B | 2B, big-endian | bytes, UTF-8 | ++-------+-------+------+-----------------+------------------------+ + 0x43 0x48 0x01 JOIN=1 0..65535 type-specific: + "CH" MSG=2 JOIN = name + DELIVER=3 MSG = text + WELCOME=4 DELIVER = name + 0x00 (NUL) + text + PING=5 WELCOME = text + PING = empty +``` + +The header (magic/version/type/length) is byte-identical across every +chatterd version; a version only ever *adds* a type, never changes the +header. ch23's post-discovery exchange uses two of the five types, one frame +each way: + +1. the discoverer dials `:` and sends a **JOIN** frame + whose payload is its own name (``) — the same "announce on connect" + JOIN chapters 21/22/24/27 use; +2. the listener reads that frame, replies with a **DELIVER** frame whose + payload is `\0hello from ` (name, then a NUL byte, then the + greeting text), and closes; +3. the discoverer reads the DELIVER frame, splits the payload on the NUL + byte, and prints `peer says: `. + +Both peers discover each other independently, so each dials the other once and +each prints one `peer … says:` line — the message crossed the network. Because +the header is shared, a ch23 peer's JOIN/DELIVER exchange is wire-compatible +with a ch21/ch22/ch24/ch27 chatterd speaking the same frame; the newer-only +types (WELCOME, PING) simply never appear in this chapter's exchange. + +## The three implementations + +Identical CLI, stdout/stderr shapes, and exit codes (**0** ok / **1** runtime +error / **2** usage). Malformed `--group`/`--iface` or an unknown flag is a +usage error (2); a syntactically valid but unbindable `--iface` fails at bind +time (1). + +| | multicast socket | concurrency | +|---|---|---| +| **C++23** | raw `setsockopt` `IP_ADD_MEMBERSHIP` / `IP_MULTICAST_IF` / `IP_MULTICAST_LOOP` / `IP_MULTICAST_TTL`; every fd in an RAII `Fd`, no naked `close`; `std::expected` on each fallible step; output via `std::println` | accept + receive loops on `std::jthread`s, stopped through their `stop_token`; dedup set under a `std::mutex` | +| **Go 1.26** | `net.ListenMulticastUDP` for the joined receiver; the beacon sender is a `golang.org/x/sys/unix` socket with `IP_MULTICAST_IF`/`_LOOP`/`_TTL` set via `Setsockopt*` | accept + receive goroutines coordinated by a `context` + `sync.WaitGroup`; errors wrapped with `%w` | +| **Rust (2024)** | `socket2::Socket` (`join_multicast_v4` / `set_multicast_if_v4` / `set_multicast_loop_v4` / `set_multicast_ttl_v4`), converted `into()` a std `UdpSocket` (an `OwnedFd`) shared via `Arc` | accept + receive `std::thread`s stopped through an `AtomicBool`; `Result`/`?` throughout, no `tokio` (this chapter is sockets, not async) | + +## Running it + +``` +./demo.sh build # build all three +./demo.sh go run discover --group 239.23.7.1 --port 51823 \ + --name alice --tcp-port 9231 --iface 127.0.0.1 +``` + +Flags: `--group ` and `--port ` and `--name ` are required; +`--tcp-port` (default 9101), `--iface` (default 127.0.0.1), `--announce-ms` +(default 200), `--rounds` (default 10) are optional. + +## Verification + +`verify.lua` (mode `vm-peer`) runs the **local** two-process assertion on +loopback multicast — it clears `TARGET` so `demo.sh run` executes on the host, +starts `alice` and `bob` with distinct names and TCP ports, and asserts each +discovers the other exactly once, exchanges the chat frame, and exits 0, plus +the usage/exit-code shapes. Distinct ports per language keep back-to-back +cpp/go/rust runs off each other's `TIME_WAIT` sockets. + +### Two-host demo (systems-target + systems-peer) + +The real observable is two daemons on two machines. Deploy the binary to both +guests and run `discover` on each with its own `--iface`; the beacons cross +`virbr0` and each guest discovers and greets the other: + +``` +# on systems-target (192.168.124.7): +discovered peer peer at 192.168.124.95:9231 +peer peer says: hello from peer + +# on systems-peer (192.168.124.95): +discovered peer target at 192.168.124.7:9231 +peer target says: hello from target +``` diff --git a/examples/23-udp-and-peer-discovery/cpp/.clang-format b/examples/23-udp-and-peer-discovery/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/23-udp-and-peer-discovery/cpp/.clang-tidy b/examples/23-udp-and-peer-discovery/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/23-udp-and-peer-discovery/cpp/CMakeLists.txt b/examples/23-udp-and-peer-discovery/cpp/CMakeLists.txt new file mode 100644 index 0000000..1304d26 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/23-udp-and-peer-discovery/cpp/CMakePresets.json b/examples/23-udp-and-peer-discovery/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/23-udp-and-peer-discovery/cpp/demo.sh b/examples/23-udp-and-peer-discovery/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/23-udp-and-peer-discovery/cpp/src/main.cpp b/examples/23-udp-and-peer-discovery/cpp/src/main.cpp new file mode 100644 index 0000000..04731c3 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/cpp/src/main.cpp @@ -0,0 +1,452 @@ +// chatterd v2 (C++23) — UDP multicast peer discovery + a TCP chat exchange. +// Grows the same chatterd program introduced in chapter 21: the TCP frame is +// the canonical chatterd chat frame (2-byte magic "CH" + 1-byte version + +// 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by every +// chatterd version (ch21-ch27). The UDP discovery beacon below is a +// completely separate wire object — a plain ASCII datagram, no relation to +// the chat frame's magic/version/type/length header. +// +// chatterd discover --group 239.7.7.7 --port 51888 --name alice \ +// --tcp-port 9101 --iface 127.0.0.1 [--announce-ms 200] [--rounds 10] +// +// Every socket lives inside an RAII Fd, so there is no naked close(); fallible +// operations return std::expected; the listen loops run on jthreads that stop +// through their stop_token; output goes through std::println. + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr std::string_view kBeaconMagic = "CHATTERD1"; // UDP beacon tag only + +// --- canonical chatterd chat frame (TCP) ----------------------------------- +// magic 'C' 'H', version 0x01, type (1 byte), length (2 bytes big-endian, +// 0..65535), payload (UTF-8, type-specific). This header is byte-identical +// across every chatterd version; ch23's post-discovery greeting uses only +// JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the +// listener's reply) — the same types chapters 21/22/24/27 share. +constexpr std::uint8_t kFrameMagic0 = 0x43; // 'C' +constexpr std::uint8_t kFrameMagic1 = 0x48; // 'H' +constexpr std::uint8_t kFrameVersion = 0x01; +constexpr std::uint8_t kFrameJoin = 1; +constexpr std::uint8_t kFrameMsg = 2; +constexpr std::uint8_t kFrameDeliver = 3; +constexpr std::uint8_t kFrameWelcome = 4; +constexpr std::uint8_t kFramePing = 5; +constexpr std::size_t kMaxFramePayload = 0xFFFF; + +// --- RAII around every descriptor ----------------------------------------- +class Fd { + public: + Fd() = default; + explicit Fd(int fd) : fd_(fd) {} + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + Fd(Fd&& o) noexcept : fd_(o.fd_) { o.fd_ = -1; } + Fd& operator=(Fd&& o) noexcept { + if (this != &o) { + reset(); + fd_ = o.fd_; + o.fd_ = -1; + } + return *this; + } + ~Fd() { reset(); } + [[nodiscard]] int get() const { return fd_; } + [[nodiscard]] explicit operator bool() const { return fd_ >= 0; } + void reset() { + if (fd_ >= 0) ::close(fd_); + fd_ = -1; + } + + private: + int fd_ = -1; +}; + +std::string errno_msg(std::string_view what) { + return std::string(what) + ": " + std::strerror(errno); +} + +// --- canonical chatterd chat frame I/O -------------------------------------- +std::expected write_all(int fd, const void* buf, size_t n) { + const auto* p = static_cast(buf); + while (n > 0) { + ssize_t w = ::send(fd, p, n, MSG_NOSIGNAL); + if (w < 0) { + if (errno == EINTR) continue; + return std::unexpected(errno_msg("send")); + } + p += w; + n -= static_cast(w); + } + return {}; +} + +std::expected read_all(int fd, void* buf, size_t n) { + auto* p = static_cast(buf); + while (n > 0) { + ssize_t r = ::recv(fd, p, n, 0); + if (r < 0) { + if (errno == EINTR) continue; + return std::unexpected(errno_msg("recv")); + } + if (r == 0) return std::unexpected(std::string("recv: peer closed early")); + p += r; + n -= static_cast(r); + } + return {}; +} + +std::expected write_frame(int fd, std::uint8_t type, + std::string_view payload) { + if (payload.size() > kMaxFramePayload) + return std::unexpected(std::string("frame payload too large: ") + + std::to_string(payload.size())); + std::uint8_t hdr[6] = { + kFrameMagic0, kFrameMagic1, kFrameVersion, type, + static_cast((payload.size() >> 8) & 0xFF), + static_cast(payload.size() & 0xFF), + }; + if (auto r = write_all(fd, hdr, sizeof(hdr)); !r) return r; + return write_all(fd, payload.data(), payload.size()); +} + +std::expected, std::string> read_frame(int fd) { + std::uint8_t hdr[6]; + if (auto r = read_all(fd, hdr, sizeof(hdr)); !r) return std::unexpected(r.error()); + if (hdr[0] != kFrameMagic0 || hdr[1] != kFrameMagic1) + return std::unexpected(std::string("bad frame magic")); + if (hdr[2] != kFrameVersion) + return std::unexpected(std::string("unsupported frame version: ") + + std::to_string(hdr[2])); + std::uint8_t type = hdr[3]; + std::uint16_t n = static_cast((hdr[4] << 8) | hdr[5]); + std::string payload(n, '\0'); + if (n > 0) { + if (auto r = read_all(fd, payload.data(), n); !r) + return std::unexpected(r.error()); + } + return std::make_pair(type, std::move(payload)); +} + +// --- socket construction --------------------------------------------------- +std::expected parse_ipv4(std::string_view s) { + in_addr a{}; + std::string tmp(s); + if (::inet_pton(AF_INET, tmp.c_str(), &a) != 1) + return std::unexpected(std::string("bad IPv4 address: ") + tmp); + return a; +} + +void set_rcv_timeout(int fd, int ms) { + timeval tv{.tv_sec = ms / 1000, .tv_usec = (ms % 1000) * 1000}; + ::setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); +} + +std::expected make_multicast_socket(in_addr group, in_addr iface, + std::uint16_t port) { + Fd fd(::socket(AF_INET, SOCK_DGRAM, 0)); + if (!fd) return std::unexpected(errno_msg("socket(udp)")); + int one = 1; + if (::setsockopt(fd.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) < 0) + return std::unexpected(errno_msg("SO_REUSEADDR")); + + sockaddr_in bind_addr{}; + bind_addr.sin_family = AF_INET; + bind_addr.sin_addr.s_addr = htonl(INADDR_ANY); + bind_addr.sin_port = htons(port); + if (::bind(fd.get(), reinterpret_cast(&bind_addr), sizeof(bind_addr)) < 0) + return std::unexpected(errno_msg("bind(udp)")); + + ip_mreq mreq{}; + mreq.imr_multiaddr = group; + mreq.imr_interface = iface; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) < 0) + return std::unexpected(errno_msg("IP_ADD_MEMBERSHIP")); + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_IF, &iface, sizeof(iface)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_IF")); + unsigned char loop = 1; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_LOOP, &loop, sizeof(loop)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_LOOP")); + unsigned char ttl = 1; + if (::setsockopt(fd.get(), IPPROTO_IP, IP_MULTICAST_TTL, &ttl, sizeof(ttl)) < 0) + return std::unexpected(errno_msg("IP_MULTICAST_TTL")); + set_rcv_timeout(fd.get(), 200); + return fd; +} + +std::expected make_tcp_listener(in_addr iface, std::uint16_t port) { + Fd fd(::socket(AF_INET, SOCK_STREAM, 0)); + if (!fd) return std::unexpected(errno_msg("socket(tcp)")); + int one = 1; + ::setsockopt(fd.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr = iface; + addr.sin_port = htons(port); + if (::bind(fd.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) + return std::unexpected(errno_msg("bind(tcp)")); + if (::listen(fd.get(), 16) < 0) return std::unexpected(errno_msg("listen")); + set_rcv_timeout(fd.get(), 200); + return fd; +} + +std::expected dial_tcp(in_addr ip, std::uint16_t port) { + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr = ip; + addr.sin_port = htons(port); + // Retry: the peer's beacon may arrive a hair before its listener is ready. + for (int attempt = 0; attempt < 40; ++attempt) { + Fd fd(::socket(AF_INET, SOCK_STREAM, 0)); + if (!fd) return std::unexpected(errno_msg("socket(dial)")); + if (::connect(fd.get(), reinterpret_cast(&addr), sizeof(addr)) == 0) + return fd; + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + return std::unexpected(std::string("connect: giving up")); +} + +// --- config ---------------------------------------------------------------- +struct Config { + std::string group; + std::uint16_t port = 0; + std::string name; + std::uint16_t tcp_port = 9101; + std::string iface = "127.0.0.1"; + int announce_ms = 200; + int rounds = 10; +}; + +std::optional parse_u16(std::string_view s) { + int v = 0; + auto [p, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || p != s.data() + s.size() || v < 1 || v > 65535) + return std::nullopt; + return static_cast(v); +} + +std::optional parse_int(std::string_view s) { + int v = 0; + auto [p, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || p != s.data() + s.size() || v < 0) return std::nullopt; + return v; +} + +void usage() { + std::println(stderr, + "usage: chatterd discover --group --port --name " + "[--tcp-port ] [--iface ] [--announce-ms ] [--rounds ]"); +} + +std::expected parse_args(int argc, char** argv) { + if (argc < 2 || std::string_view(argv[1]) != "discover") { + usage(); + return std::unexpected(2); + } + Config c; + for (int i = 2; i < argc; ++i) { + std::string_view a = argv[i]; + auto need = [&](std::string_view name) -> const char* { + if (i + 1 >= argc) { + std::println(stderr, "chatterd: {} needs a value", name); + return nullptr; + } + return argv[++i]; + }; + if (a == "--group") { + const char* v = need(a); + if (!v) return std::unexpected(2); + if (!parse_ipv4(v)) { usage(); return std::unexpected(2); } + c.group = v; + } else if (a == "--port") { + const char* v = need(a); + auto p = v ? parse_u16(v) : std::nullopt; + if (!p) { usage(); return std::unexpected(2); } + c.port = *p; + } else if (a == "--name") { + const char* v = need(a); + if (!v) return std::unexpected(2); + c.name = v; + } else if (a == "--tcp-port") { + const char* v = need(a); + auto p = v ? parse_u16(v) : std::nullopt; + if (!p) { usage(); return std::unexpected(2); } + c.tcp_port = *p; + } else if (a == "--iface") { + const char* v = need(a); + if (!v) return std::unexpected(2); + if (!parse_ipv4(v)) { usage(); return std::unexpected(2); } + c.iface = v; + } else if (a == "--announce-ms") { + const char* v = need(a); + auto p = v ? parse_int(v) : std::nullopt; + if (!p || *p == 0) { usage(); return std::unexpected(2); } + c.announce_ms = *p; + } else if (a == "--rounds") { + const char* v = need(a); + auto p = v ? parse_int(v) : std::nullopt; + if (!p || *p == 0) { usage(); return std::unexpected(2); } + c.rounds = *p; + } else { + std::println(stderr, "chatterd: unknown argument: {}", a); + usage(); + return std::unexpected(2); + } + } + if (c.group.empty() || c.port == 0 || c.name.empty()) { + usage(); + return std::unexpected(2); + } + return c; +} + +// Shared dedup set for discovered peer names. +struct Seen { + std::mutex m; + std::unordered_set names; + bool first(const std::string& n) { + std::scoped_lock lk(m); + return names.insert(n).second; + } +}; + +// Split a DELIVER payload "\0". +std::pair split_nul(const std::string& s) { + auto pos = s.find('\0'); + if (pos == std::string::npos) return {s, ""}; + return {s.substr(0, pos), s.substr(pos + 1)}; +} + +} // namespace + +int main(int argc, char** argv) { + auto cfg = parse_args(argc, argv); + if (!cfg) return cfg.error(); + const Config& c = *cfg; + + auto group = parse_ipv4(c.group); + if (!group) { + std::println(stderr, "chatterd: error: {}", group.error()); + return 1; + } + auto iface = parse_ipv4(c.iface); + if (!iface) { + std::println(stderr, "chatterd: error: {}", iface.error()); + return 1; + } + + auto tcp = make_tcp_listener(*iface, c.tcp_port); + if (!tcp) { + std::println(stderr, "chatterd: error: {}", tcp.error()); + return 1; + } + auto udp = make_multicast_socket(*group, *iface, c.port); + if (!udp) { + std::println(stderr, "chatterd: error: {}", udp.error()); + return 1; + } + + std::println(stderr, "chatterd: announcing as {} on {}:{} (tcp {}:{})", c.name, + c.group, c.port, c.iface, c.tcp_port); + + Seen seen; + // DELIVER payload sent back to whoever dials us: "\0hello from ". + const std::string deliver_payload = c.name + '\0' + "hello from " + c.name; + + // TCP accept loop: read the dialer's JOIN frame, reply with our DELIVER + // frame (our name + NUL + greeting text), close. + std::jthread accepter([&](std::stop_token stop) { + while (!stop.stop_requested()) { + int cfd = ::accept(tcp->get(), nullptr, nullptr); + if (cfd < 0) continue; // timeout or interrupt -> re-check stop + Fd conn(cfd); + if (auto f = read_frame(conn.get()); f) { + (void)write_frame(conn.get(), kFrameDeliver, deliver_payload); + } + } + }); + + // UDP receive loop: on each new peer, print + dial + exchange one message. + std::jthread receiver([&](std::stop_token stop) { + std::vector buf(2048); + while (!stop.stop_requested()) { + ssize_t n = ::recv(udp->get(), buf.data(), buf.size() - 1, 0); + if (n <= 0) continue; + std::string msg(buf.data(), static_cast(n)); + // Parse "CHATTERD1 ". + std::vector tok; + size_t start = 0; + while (start < msg.size() && tok.size() < 4) { + size_t sp = msg.find(' ', start); + if (sp == std::string::npos) sp = msg.size(); + tok.emplace_back(msg.substr(start, sp - start)); + start = sp + 1; + } + if (tok.size() != 4 || tok[0] != kBeaconMagic) continue; + const std::string& pname = tok[1]; + if (pname == c.name) continue; // our own beacon + auto pport = parse_u16(tok[2]); + auto pip = parse_ipv4(tok[3]); + if (!pport || !pip) continue; + if (!seen.first(pname)) continue; // already discovered + + std::println("discovered peer {} at {}:{}", pname, tok[3], *pport); + auto conn = dial_tcp(*pip, *pport); + if (!conn) { + std::println(stderr, "chatterd: error: dial {}: {}", pname, conn.error()); + continue; + } + // JOIN (announce ourselves), then read the peer's DELIVER reply. + if (auto s = write_frame(conn->get(), kFrameJoin, c.name); !s) { + std::println(stderr, "chatterd: error: {}", s.error()); + continue; + } + auto reply = read_frame(conn->get()); + if (!reply) { + std::println(stderr, "chatterd: error: {}", reply.error()); + continue; + } + auto [rname, rtext] = split_nul(reply->second); + std::println("peer {} says: {}", rname, rtext); + } + }); + + // Announce loop: periodic multicast beacons, then a grace window for the + // final exchanges to complete before we stop the listen threads. + sockaddr_in dst{}; + dst.sin_family = AF_INET; + dst.sin_addr = *group; + dst.sin_port = htons(c.port); + std::string beacon = std::string(kBeaconMagic) + " " + c.name + " " + + std::to_string(c.tcp_port) + " " + c.iface; + for (int r = 0; r < c.rounds; ++r) { + ::sendto(udp->get(), beacon.data(), beacon.size(), 0, + reinterpret_cast(&dst), sizeof(dst)); + std::this_thread::sleep_for(std::chrono::milliseconds(c.announce_ms)); + } + std::this_thread::sleep_for(std::chrono::milliseconds(800)); + + receiver.request_stop(); + accepter.request_stop(); + return 0; +} diff --git a/examples/23-udp-and-peer-discovery/demo.sh b/examples/23-udp-and-peer-discovery/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/23-udp-and-peer-discovery/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/23-udp-and-peer-discovery/go/demo.sh b/examples/23-udp-and-peer-discovery/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/23-udp-and-peer-discovery/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/23-udp-and-peer-discovery/go/go.mod b/examples/23-udp-and-peer-discovery/go/go.mod new file mode 100644 index 0000000..bac243a --- /dev/null +++ b/examples/23-udp-and-peer-discovery/go/go.mod @@ -0,0 +1,7 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/23-udp-and-peer-discovery/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 diff --git a/examples/23-udp-and-peer-discovery/go/go.sum b/examples/23-udp-and-peer-discovery/go/go.sum new file mode 100644 index 0000000..37ee2d4 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/go/go.sum @@ -0,0 +1,2 @@ +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= diff --git a/examples/23-udp-and-peer-discovery/go/main.go b/examples/23-udp-and-peer-discovery/go/main.go new file mode 100644 index 0000000..ae27e27 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/go/main.go @@ -0,0 +1,428 @@ +// chatterd v2 (Go) — UDP multicast peer discovery + a TCP chat exchange. +// Grows the same chatterd program introduced in chapter 21: the TCP frame is +// the canonical chatterd chat frame (2-byte magic "CH" + 1-byte version + +// 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by every +// chatterd version (ch21-ch27). The UDP discovery beacon below is a +// completely separate wire object — a plain ASCII datagram, no relation to +// the chat frame's magic/version/type/length header. +// +// chatterd discover --group 239.7.7.7 --port 51888 --name alice \ +// --tcp-port 9101 --iface 127.0.0.1 [--announce-ms 200] [--rounds 10] +// +// The multicast listener uses net.ListenMulticastUDP; the beacon sender is a +// raw socket whose IP_MULTICAST_IF/LOOP/TTL are set through golang.org/x/sys/ +// unix. The accept and receive loops run as goroutines coordinated by a context +// and a WaitGroup; errors are wrapped with %w. +package main + +import ( + "context" + "fmt" + "io" + "net" + "os" + "strconv" + "strings" + "sync" + "time" + + "golang.org/x/sys/unix" +) + +const beaconMagic = "CHATTERD1" // UDP beacon tag only + +// --- canonical chatterd chat frame (TCP) ------------------------------------ +// magic 'C' 'H', version 0x01, type (1 byte), length (2 bytes big-endian, +// 0..65535), payload (UTF-8, type-specific). This header is byte-identical +// across every chatterd version; ch23's post-discovery greeting uses only +// JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the +// listener's reply) — the same types chapters 21/22/24/27 share. +const ( + frameMagic0 = 0x43 // 'C' + frameMagic1 = 0x48 // 'H' + frameVersion = 0x01 +) + +const ( + frameJoin = 1 + frameMsg = 2 + frameDeliver = 3 + frameWelcome = 4 + framePing = 5 +) + +const maxFramePayload = 0xFFFF + +type config struct { + group string + port int + name string + tcpPort int + iface string + announceMs int + rounds int +} + +func usage() { + fmt.Fprintln(os.Stderr, + "usage: chatterd discover --group --port --name "+ + "[--tcp-port ] [--iface ] [--announce-ms ] [--rounds ]") +} + +// parseArgs returns (cfg, exitCode, ok). On any parse error it prints usage and +// returns ok=false with exit code 2. +func parseArgs(argv []string) (config, int, bool) { + c := config{tcpPort: 9101, iface: "127.0.0.1", announceMs: 200, rounds: 10} + if len(argv) < 2 || argv[1] != "discover" { + usage() + return c, 2, false + } + args := argv[2:] + next := func(i *int, name string) (string, bool) { + if *i+1 >= len(args) { + fmt.Fprintf(os.Stderr, "chatterd: %s needs a value\n", name) + return "", false + } + *i++ + return args[*i], true + } + posInt := func(s string, allowZero bool) (int, bool) { + v, err := strconv.Atoi(s) + if err != nil || v < 0 || (!allowZero && v == 0) { + return 0, false + } + return v, true + } + port16 := func(s string) (int, bool) { + v, err := strconv.Atoi(s) + if err != nil || v < 1 || v > 65535 { + return 0, false + } + return v, true + } + for i := 0; i < len(args); i++ { + a := args[i] + switch a { + case "--group": + v, ok := next(&i, a) + if !ok { + return c, 2, false + } + if ip := net.ParseIP(v); ip == nil || ip.To4() == nil { + usage() + return c, 2, false + } + c.group = v + case "--name": + v, ok := next(&i, a) + if !ok { + return c, 2, false + } + c.name = v + case "--iface": + v, ok := next(&i, a) + if !ok { + return c, 2, false + } + if ip := net.ParseIP(v); ip == nil || ip.To4() == nil { + usage() + return c, 2, false + } + c.iface = v + case "--port": + v, ok := next(&i, a) + p, pok := port16(v) + if !ok || !pok { + usage() + return c, 2, false + } + c.port = p + case "--tcp-port": + v, ok := next(&i, a) + p, pok := port16(v) + if !ok || !pok { + usage() + return c, 2, false + } + c.tcpPort = p + case "--announce-ms": + v, ok := next(&i, a) + p, pok := posInt(v, false) + if !ok || !pok { + usage() + return c, 2, false + } + c.announceMs = p + case "--rounds": + v, ok := next(&i, a) + p, pok := posInt(v, false) + if !ok || !pok { + usage() + return c, 2, false + } + c.rounds = p + default: + fmt.Fprintf(os.Stderr, "chatterd: unknown argument: %s\n", a) + usage() + return c, 2, false + } + } + if c.group == "" || c.port == 0 || c.name == "" { + usage() + return c, 2, false + } + return c, 0, true +} + +// --- canonical chatterd chat frame I/O --------------------------------------- + +func writeFrame(w io.Writer, typ byte, payload []byte) error { + if len(payload) > maxFramePayload { + return fmt.Errorf("frame payload too large: %d", len(payload)) + } + hdr := [6]byte{ + frameMagic0, frameMagic1, frameVersion, typ, + byte(len(payload) >> 8), byte(len(payload)), + } + if _, err := w.Write(hdr[:]); err != nil { + return fmt.Errorf("write frame header: %w", err) + } + if _, err := w.Write(payload); err != nil { + return fmt.Errorf("write frame body: %w", err) + } + return nil +} + +func readFrame(r io.Reader) (byte, []byte, error) { + var hdr [6]byte + if _, err := io.ReadFull(r, hdr[:]); err != nil { + return 0, nil, fmt.Errorf("read frame header: %w", err) + } + if hdr[0] != frameMagic0 || hdr[1] != frameMagic1 { + return 0, nil, fmt.Errorf("bad frame magic") + } + if hdr[2] != frameVersion { + return 0, nil, fmt.Errorf("unsupported frame version: %d", hdr[2]) + } + typ := hdr[3] + n := int(hdr[4])<<8 | int(hdr[5]) + body := make([]byte, n) + if _, err := io.ReadFull(r, body); err != nil { + return 0, nil, fmt.Errorf("read frame body: %w", err) + } + return typ, body, nil +} + +func interfaceForIP(ip net.IP) (*net.Interface, error) { + ifaces, err := net.Interfaces() + if err != nil { + return nil, fmt.Errorf("list interfaces: %w", err) + } + for i := range ifaces { + addrs, err := ifaces[i].Addrs() + if err != nil { + continue + } + for _, a := range addrs { + var aip net.IP + switch v := a.(type) { + case *net.IPNet: + aip = v.IP + case *net.IPAddr: + aip = v.IP + } + if aip.Equal(ip) { + return &ifaces[i], nil + } + } + } + return nil, fmt.Errorf("no interface has address %s", ip) +} + +// makeSender builds the raw UDP socket used to emit beacons, pinning the +// outgoing multicast interface and enabling local loopback so a second peer on +// the same host still receives our beacons. +func makeSender(ip4 [4]byte) (int, error) { + fd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM, 0) + if err != nil { + return -1, fmt.Errorf("socket: %w", err) + } + if err := unix.SetsockoptInet4Addr(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_IF, ip4); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_IF: %w", err) + } + if err := unix.SetsockoptByte(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_LOOP, 1); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_LOOP: %w", err) + } + if err := unix.SetsockoptByte(fd, unix.IPPROTO_IP, unix.IP_MULTICAST_TTL, 1); err != nil { + unix.Close(fd) + return -1, fmt.Errorf("IP_MULTICAST_TTL: %w", err) + } + return fd, nil +} + +func dial(ip string, port int) (net.Conn, error) { + addr := net.JoinHostPort(ip, strconv.Itoa(port)) + // Retry: the peer's beacon may arrive a hair before its listener is ready. + for i := 0; i < 40; i++ { + c, err := net.DialTimeout("tcp", addr, 500*time.Millisecond) + if err == nil { + return c, nil + } + time.Sleep(50 * time.Millisecond) + } + return nil, fmt.Errorf("connect %s: giving up", addr) +} + +func acceptLoop(ctx context.Context, wg *sync.WaitGroup, ln *net.TCPListener, deliverPayload []byte) { + defer wg.Done() + for { + _ = ln.SetDeadline(time.Now().Add(200 * time.Millisecond)) + conn, err := ln.Accept() + if err != nil { + if ctx.Err() != nil { + return + } + continue + } + // Read the dialer's JOIN frame, reply with our DELIVER frame (our + // name + NUL + greeting text), close. + if _, _, rerr := readFrame(conn); rerr == nil { + _ = writeFrame(conn, frameDeliver, deliverPayload) + } + conn.Close() + } +} + +func recvLoop(ctx context.Context, wg *sync.WaitGroup, udp *net.UDPConn, cfg config) { + defer wg.Done() + seen := make(map[string]bool) + buf := make([]byte, 2048) + for { + _ = udp.SetReadDeadline(time.Now().Add(200 * time.Millisecond)) + n, _, err := udp.ReadFromUDP(buf) + if err != nil { + if ctx.Err() != nil { + return + } + continue + } + tok := strings.Fields(string(buf[:n])) + if len(tok) != 4 || tok[0] != beaconMagic { + continue + } + pname := tok[1] + if pname == cfg.name || seen[pname] { // our own beacon or already known + continue + } + pport, err := strconv.Atoi(tok[2]) + if err != nil || pport < 1 || pport > 65535 { + continue + } + pip := tok[3] + seen[pname] = true + + fmt.Printf("discovered peer %s at %s:%d\n", pname, pip, pport) + conn, err := dial(pip, pport) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: dial %s: %v\n", pname, err) + continue + } + // JOIN (announce ourselves), then read the peer's DELIVER reply. + if err := writeFrame(conn, frameJoin, []byte(cfg.name)); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: %v\n", err) + conn.Close() + continue + } + _, payload, err := readFrame(conn) + conn.Close() + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: %v\n", err) + continue + } + parts := strings.SplitN(string(payload), "\x00", 2) + rname, rtext := parts[0], "" + if len(parts) == 2 { + rtext = parts[1] + } + fmt.Printf("peer %s says: %s\n", rname, rtext) + } +} + +func discover(cfg config) error { + ip := net.ParseIP(cfg.iface) + if ip == nil || ip.To4() == nil { + return fmt.Errorf("bad --iface address: %s", cfg.iface) + } + gip := net.ParseIP(cfg.group) + if gip == nil || gip.To4() == nil { + return fmt.Errorf("bad --group address: %s", cfg.group) + } + var ip4, group4 [4]byte + copy(ip4[:], ip.To4()) + copy(group4[:], gip.To4()) + + ifi, err := interfaceForIP(ip) + if err != nil { + return err + } + + lnAny, err := net.Listen("tcp", net.JoinHostPort(cfg.iface, strconv.Itoa(cfg.tcpPort))) + if err != nil { + return fmt.Errorf("listen tcp: %w", err) + } + ln := lnAny.(*net.TCPListener) + defer ln.Close() + + udp, err := net.ListenMulticastUDP("udp4", ifi, &net.UDPAddr{IP: gip, Port: cfg.port}) + if err != nil { + return fmt.Errorf("listen multicast: %w", err) + } + defer udp.Close() + + sfd, err := makeSender(ip4) + if err != nil { + return err + } + defer unix.Close(sfd) + + fmt.Fprintf(os.Stderr, "chatterd: announcing as %s on %s:%d (tcp %s:%d)\n", + cfg.name, cfg.group, cfg.port, cfg.iface, cfg.tcpPort) + + deliverPayload := []byte(cfg.name + "\x00hello from " + cfg.name) + ctx, cancel := context.WithCancel(context.Background()) + var wg sync.WaitGroup + wg.Add(2) + go acceptLoop(ctx, &wg, ln, deliverPayload) + go recvLoop(ctx, &wg, udp, cfg) + + beacon := []byte(fmt.Sprintf("%s %s %d %s", beaconMagic, cfg.name, cfg.tcpPort, cfg.iface)) + dst := &unix.SockaddrInet4{Port: cfg.port, Addr: group4} + for r := 0; r < cfg.rounds; r++ { + if err := unix.Sendto(sfd, beacon, 0, dst); err != nil { + cancel() + wg.Wait() + return fmt.Errorf("sendto beacon: %w", err) + } + time.Sleep(time.Duration(cfg.announceMs) * time.Millisecond) + } + // Grace window so final exchanges complete, then unblock the loops. + time.Sleep(800 * time.Millisecond) + cancel() + ln.Close() + udp.Close() + wg.Wait() + return nil +} + +func main() { + cfg, code, ok := parseArgs(os.Args) + if !ok { + os.Exit(code) + } + if err := discover(cfg); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: %v\n", err) + os.Exit(1) + } +} diff --git a/examples/23-udp-and-peer-discovery/rust/Cargo.lock b/examples/23-udp-and-peer-discovery/rust/Cargo.lock new file mode 100644 index 0000000..8987ac6 --- /dev/null +++ b/examples/23-udp-and-peer-discovery/rust/Cargo.lock @@ -0,0 +1,41 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "chatterd" +version = "0.2.0" +dependencies = [ + "socket2", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] diff --git a/examples/23-udp-and-peer-discovery/rust/Cargo.toml b/examples/23-udp-and-peer-discovery/rust/Cargo.toml new file mode 100644 index 0000000..59b604e --- /dev/null +++ b/examples/23-udp-and-peer-discovery/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "chatterd" +version = "0.2.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +socket2 = "0.6" diff --git a/examples/23-udp-and-peer-discovery/rust/demo.sh b/examples/23-udp-and-peer-discovery/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/23-udp-and-peer-discovery/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/23-udp-and-peer-discovery/rust/rust-toolchain.toml b/examples/23-udp-and-peer-discovery/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/23-udp-and-peer-discovery/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/23-udp-and-peer-discovery/rust/src/main.rs b/examples/23-udp-and-peer-discovery/rust/src/main.rs new file mode 100644 index 0000000..239bade --- /dev/null +++ b/examples/23-udp-and-peer-discovery/rust/src/main.rs @@ -0,0 +1,354 @@ +// chatterd v2 (Rust 2024) — UDP multicast peer discovery + a TCP chat +// exchange. Grows the same chatterd program introduced in chapter 21: the TCP +// frame is the canonical chatterd chat frame (2-byte magic "CH" + 1-byte +// version + 1-byte type + 2-byte big-endian length + UTF-8 payload) shared by +// every chatterd version (ch21-ch27). The UDP discovery beacon below is a +// completely separate wire object — a plain ASCII datagram, no relation to +// the chat frame's magic/version/type/length header. +// +// chatterd discover --group 239.7.7.7 --port 51888 --name alice \ +// --tcp-port 9101 --iface 127.0.0.1 [--announce-ms 200] [--rounds 10] +// +// The multicast socket is built with the socket2 crate (IP_ADD_MEMBERSHIP / +// IP_MULTICAST_IF / IP_MULTICAST_LOOP), then converted into a std UdpSocket +// (an OwnedFd) shared across threads. Fallible paths use Result/?; the accept +// and receive loops run on std threads coordinated by an AtomicBool. + +use std::collections::HashSet; +use std::error::Error; +use std::io::{self, Read, Write}; +use std::net::{Ipv4Addr, SocketAddr, SocketAddrV4, TcpListener, TcpStream, UdpSocket}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::thread; +use std::time::Duration; + +use socket2::{Domain, Protocol, SockAddr, Socket, Type}; + +const BEACON_MAGIC: &str = "CHATTERD1"; // UDP beacon tag only + +// --- canonical chatterd chat frame (TCP) ----------------------------------- +// magic 'C' 'H', version 0x01, type (1 byte), length (2 bytes big-endian, +// 0..65535), payload (UTF-8, type-specific). This header is byte-identical +// across every chatterd version; ch23's post-discovery greeting uses only +// JOIN (announce the dialer's name) and DELIVER (name + NUL + text, the +// listener's reply) — the same types chapters 21/22/24/27 share. +const FRAME_MAGIC: [u8; 2] = [0x43, 0x48]; // "CH" +const FRAME_VERSION: u8 = 0x01; +const FRAME_JOIN: u8 = 1; +#[allow(dead_code)] +const FRAME_MSG: u8 = 2; +const FRAME_DELIVER: u8 = 3; +#[allow(dead_code)] +const FRAME_WELCOME: u8 = 4; +#[allow(dead_code)] +const FRAME_PING: u8 = 5; +const MAX_FRAME_PAYLOAD: usize = 0xFFFF; + +type Fallible = Result>; + +#[derive(Clone)] +struct Config { + group: Ipv4Addr, + port: u16, + name: String, + tcp_port: u16, + iface: Ipv4Addr, + announce_ms: u64, + rounds: u32, +} + +fn usage() { + eprintln!( + "usage: chatterd discover --group --port --name \ +[--tcp-port ] [--iface ] [--announce-ms ] [--rounds ]" + ); +} + +fn parse_port(s: &str) -> Result { + s.parse::().ok().filter(|&n| n >= 1).ok_or_else(|| { + usage(); + 2 + }) +} + +/// Parse argv. On any error prints usage and returns Err(exit_code). +fn parse_args() -> Result { + let argv: Vec = std::env::args().collect(); + if argv.len() < 2 || argv[1] != "discover" { + usage(); + return Err(2); + } + let mut group: Option = None; + let mut port: Option = None; + let mut name: Option = None; + let mut tcp_port: u16 = 9101; + let mut iface: Ipv4Addr = Ipv4Addr::LOCALHOST; + let mut announce_ms: u64 = 200; + let mut rounds: u32 = 10; + + let args = &argv[2..]; + let take = |i: &mut usize, flag: &str| -> Result { + if *i + 1 >= args.len() { + eprintln!("chatterd: {flag} needs a value"); + return Err(2); + } + *i += 1; + Ok(args[*i].clone()) + }; + let mut i = 0; + while i < args.len() { + let a = args[i].as_str(); + match a { + "--group" => { + let v = take(&mut i, a)?; + group = Some(v.parse().map_err(|_| { + usage(); + 2 + })?); + } + "--name" => name = Some(take(&mut i, a)?), + "--iface" => { + let v = take(&mut i, a)?; + iface = v.parse().map_err(|_| { + usage(); + 2 + })?; + } + "--port" => { + let v = take(&mut i, a)?; + port = Some(parse_port(&v)?); + } + "--tcp-port" => { + let v = take(&mut i, a)?; + tcp_port = parse_port(&v)?; + } + "--announce-ms" => { + let v = take(&mut i, a)?; + announce_ms = v.parse::().ok().filter(|&n| n > 0).ok_or_else(|| { + usage(); + 2 + })?; + } + "--rounds" => { + let v = take(&mut i, a)?; + rounds = v.parse::().ok().filter(|&n| n > 0).ok_or_else(|| { + usage(); + 2 + })?; + } + _ => { + eprintln!("chatterd: unknown argument: {a}"); + usage(); + return Err(2); + } + } + i += 1; + } + + match (group, port, name) { + (Some(group), Some(port), Some(name)) => Ok(Config { + group, + port, + name, + tcp_port, + iface, + announce_ms, + rounds, + }), + _ => { + usage(); + Err(2) + } + } +} + +// --- canonical chatterd chat frame I/O -------------------------------------- + +fn write_frame(w: &mut impl Write, typ: u8, payload: &[u8]) -> io::Result<()> { + if payload.len() > MAX_FRAME_PAYLOAD { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("frame payload too large: {}", payload.len()), + )); + } + let mut hdr = [0u8; 6]; + hdr[0] = FRAME_MAGIC[0]; + hdr[1] = FRAME_MAGIC[1]; + hdr[2] = FRAME_VERSION; + hdr[3] = typ; + hdr[4] = (payload.len() >> 8) as u8; + hdr[5] = (payload.len() & 0xFF) as u8; + w.write_all(&hdr)?; + w.write_all(payload)?; + Ok(()) +} + +fn read_frame(r: &mut impl Read) -> io::Result<(u8, Vec)> { + let mut hdr = [0u8; 6]; + r.read_exact(&mut hdr)?; + if hdr[0] != FRAME_MAGIC[0] || hdr[1] != FRAME_MAGIC[1] { + return Err(io::Error::new(io::ErrorKind::InvalidData, "bad frame magic")); + } + if hdr[2] != FRAME_VERSION { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("unsupported frame version: {}", hdr[2]), + )); + } + let typ = hdr[3]; + let n = ((hdr[4] as usize) << 8) | hdr[5] as usize; + let mut body = vec![0u8; n]; + r.read_exact(&mut body)?; + Ok((typ, body)) +} + +/// Build the multicast UDP socket and hand back a std UdpSocket (an OwnedFd). +fn make_multicast(cfg: &Config) -> Fallible { + let sock = Socket::new(Domain::IPV4, Type::DGRAM, Some(Protocol::UDP))?; + sock.set_reuse_address(true)?; + let bind = SocketAddrV4::new(Ipv4Addr::UNSPECIFIED, cfg.port); + sock.bind(&SockAddr::from(bind))?; + sock.join_multicast_v4(&cfg.group, &cfg.iface)?; + sock.set_multicast_if_v4(&cfg.iface)?; + sock.set_multicast_loop_v4(true)?; + sock.set_multicast_ttl_v4(1)?; + sock.set_read_timeout(Some(Duration::from_millis(200)))?; + Ok(sock.into()) +} + +fn dial(ip: Ipv4Addr, port: u16) -> Fallible { + let addr = SocketAddr::from((ip, port)); + // Retry: the peer's beacon may arrive a hair before its listener is ready. + for _ in 0..40 { + if let Ok(s) = TcpStream::connect_timeout(&addr, Duration::from_millis(500)) { + return Ok(s); + } + thread::sleep(Duration::from_millis(50)); + } + Err(format!("connect {addr}: giving up").into()) +} + +fn accept_loop(listener: TcpListener, deliver_payload: Vec, stop: Arc) { + listener.set_nonblocking(true).ok(); + while !stop.load(Ordering::Relaxed) { + match listener.accept() { + Ok((mut conn, _)) => { + // Read the dialer's JOIN frame, reply with our DELIVER frame + // (our name + NUL + greeting text), close. + if read_frame(&mut conn).is_ok() { + let _ = write_frame(&mut conn, FRAME_DELIVER, &deliver_payload); + } + } + Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => { + thread::sleep(Duration::from_millis(50)); + } + Err(_) => thread::sleep(Duration::from_millis(50)), + } + } +} + +fn recv_loop(udp: Arc, cfg: Config, stop: Arc) { + let mut seen: HashSet = HashSet::new(); + let mut buf = [0u8; 2048]; + while !stop.load(Ordering::Relaxed) { + let n = match udp.recv_from(&mut buf) { + Ok((n, _)) => n, + Err(ref e) + if e.kind() == io::ErrorKind::WouldBlock + || e.kind() == io::ErrorKind::TimedOut => + { + continue; + } + Err(_) => continue, + }; + let msg = String::from_utf8_lossy(&buf[..n]); + let tok: Vec<&str> = msg.split_whitespace().collect(); + if tok.len() != 4 || tok[0] != BEACON_MAGIC { + continue; + } + let pname = tok[1]; + if pname == cfg.name || seen.contains(pname) { + continue; // our own beacon or already known + } + let (pport, pip) = match (tok[2].parse::(), tok[3].parse::()) { + (Ok(p), Ok(ip)) if p >= 1 => (p, ip), + _ => continue, + }; + seen.insert(pname.to_string()); + + println!("discovered peer {pname} at {pip}:{pport}"); + let mut conn = match dial(pip, pport) { + Ok(c) => c, + Err(e) => { + eprintln!("chatterd: error: dial {pname}: {e}"); + continue; + } + }; + // JOIN (announce ourselves), then read the peer's DELIVER reply. + if let Err(e) = write_frame(&mut conn, FRAME_JOIN, cfg.name.as_bytes()) { + eprintln!("chatterd: error: {e}"); + continue; + } + let (_typ, payload) = match read_frame(&mut conn) { + Ok(r) => r, + Err(e) => { + eprintln!("chatterd: error: {e}"); + continue; + } + }; + let reply = String::from_utf8_lossy(&payload); + let (rname, rtext) = reply.split_once('\0').unwrap_or((reply.as_ref(), "")); + println!("peer {rname} says: {rtext}"); + } +} + +fn discover(cfg: &Config) -> Fallible<()> { + let listener = TcpListener::bind((cfg.iface, cfg.tcp_port)) + .map_err(|e| format!("listen tcp {}:{}: {e}", cfg.iface, cfg.tcp_port))?; + let udp = Arc::new(make_multicast(cfg)?); + + eprintln!( + "chatterd: announcing as {} on {}:{} (tcp {}:{})", + cfg.name, cfg.group, cfg.port, cfg.iface, cfg.tcp_port + ); + + let deliver_payload = format!("{}\0hello from {}", cfg.name, cfg.name).into_bytes(); + let stop = Arc::new(AtomicBool::new(false)); + + let acc = { + let (p, s) = (deliver_payload.clone(), stop.clone()); + thread::spawn(move || accept_loop(listener, p, s)) + }; + let rcv = { + let (u, c, s) = (udp.clone(), cfg.clone(), stop.clone()); + thread::spawn(move || recv_loop(u, c, s)) + }; + + let beacon = format!("{} {} {} {}", BEACON_MAGIC, cfg.name, cfg.tcp_port, cfg.iface); + let dst = SocketAddr::from((cfg.group, cfg.port)); + for _ in 0..cfg.rounds { + udp.send_to(beacon.as_bytes(), dst)?; + thread::sleep(Duration::from_millis(cfg.announce_ms)); + } + // Grace window so final exchanges complete, then stop the loops. + thread::sleep(Duration::from_millis(800)); + stop.store(true, Ordering::Relaxed); + let _ = acc.join(); + let _ = rcv.join(); + Ok(()) +} + +fn main() { + let code = match parse_args() { + Ok(cfg) => match discover(&cfg) { + Ok(()) => 0, + Err(e) => { + eprintln!("chatterd: error: {e}"); + 1 + } + }, + Err(code) => code, + }; + std::process::exit(code); +} diff --git a/examples/23-udp-and-peer-discovery/verify.lua b/examples/23-udp-and-peer-discovery/verify.lua new file mode 100644 index 0000000..a2ceefc --- /dev/null +++ b/examples/23-udp-and-peer-discovery/verify.lua @@ -0,0 +1,127 @@ +-- Verify chatterd v2 (23-udp-and-peer-discovery) for LSP_LANG. +-- +-- Asserts observable behavior, identical across cpp/go/rust. This lane is +-- mode:vm-peer, but the verify itself is the LOCAL two-process assertion: two +-- chatterd instances with distinct names run on loopback multicast, discover +-- each other, and exchange one canonical chatterd TCP chat frame (JOIN then +-- DELIVER — the same frame ch21/22/24/27 share). (The real two-host demo +-- across systems-target/systems-peer is driven separately by the chapter's +-- demo; see README.) demo.sh run is forced local here by clearing TARGET, so +-- the assertion never touches a VM. +-- +-- Checks: +-- 1. usage errors exit 2 (no subcommand, unknown flag, malformed --group); +-- an unassignable --iface fails at bind time with exit 1 +-- 2. two instances (alice, bob) each print exactly one +-- "discovered peer at 127.0.0.1:" line +-- 3. after discovery each opens a TCP connection to the peer's advertised +-- port and prints "peer says: hello from " +-- 4. both processes exit 0 and announce themselves on stderr +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set; the interpreter may be lua or luajit. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +-- Force demo.sh run to execute locally even when the runner set TARGET for the +-- vm-peer lane: an empty TARGET makes demo.sh take its loopback branch. +local demo = "TARGET= ./demo.sh " .. lang .. " run discover" + +-- Distinct ports per language so back-to-back cpp/go/rust runs never collide on +-- a lingering TIME_WAIT socket. +local base = ({ cpp = 0, go = 100, rust = 200 })[lang] +local group = "239.23.7.1" +local uport = 51823 + base +local aport = 9231 + base +local bport = 9232 + base + +local wd = checks.run("mktemp -d").out:gsub("%s+$", "") +if wd == "" or not wd:match("^/") then + checks.skip("mktemp -d did not yield a directory") +end + +local function sh(script) + return checks.run((script:gsub("@WD@", wd):gsub("@DEMO@", demo))) +end + +-- --------------------------------------------------------------------------- +-- 1. error shapes +-- --------------------------------------------------------------------------- + +local noargs = checks.run("TARGET= ./demo.sh " .. lang .. " run") +checks.expect_exit(noargs, 2, lang .. ": no subcommand exits 2") +checks.expect_match(noargs.out, "usage: chatterd discover", lang .. ": prints usage") + +local unknown = sh("@DEMO@ --group " .. group .. " --port " .. uport .. + " --name x --frobnicate") +checks.expect_exit(unknown, 2, lang .. ": unknown flag exits 2") + +local badgroup = sh("@DEMO@ --group 999.1.1.1 --port " .. uport .. " --name x") +checks.expect_exit(badgroup, 2, lang .. ": malformed --group exits 2") + +-- 192.0.2.0/24 is TEST-NET-1: valid syntax, never assigned to this host, so the +-- TCP bind fails at runtime -> exit 1 (not a usage error). +local badiface = sh("@DEMO@ --group " .. group .. " --port " .. uport .. + " --name x --iface 192.0.2.1 --tcp-port " .. aport .. " --rounds 1") +checks.expect_exit(badiface, 1, lang .. ": unassignable --iface exits 1") +checks.expect_match(badiface.out, "chatterd: error:", lang .. ": bind failure is reported") + +-- --------------------------------------------------------------------------- +-- 2 + 3. two instances discover each other and exchange a message +-- --------------------------------------------------------------------------- + +local params = "--group " .. group .. " --port " .. uport .. + " --iface 127.0.0.1 --announce-ms 150 --rounds 12" + +local run = sh([[ +@DEMO@ ]] .. params .. [[ --name alice --tcp-port ]] .. aport .. [[ \ + > @WD@/alice.out 2> @WD@/alice.err & apid=$! +@DEMO@ ]] .. params .. [[ --name bob --tcp-port ]] .. bport .. [[ \ + > @WD@/bob.out 2> @WD@/bob.err & bpid=$! +wait $apid; echo "alice-exit=$?" >> @WD@/exits +wait $bpid; echo "bob-exit=$?" >> @WD@/exits +]]) +checks.expect_exit(run, 0, lang .. ": both instances run to completion") + +local exits = sh("cat @WD@/exits").out +checks.expect_match(exits, "alice%-exit=0", lang .. ": alice exits 0") +checks.expect_match(exits, "bob%-exit=0", lang .. ": bob exits 0") + +local aout = sh("cat @WD@/alice.out").out +local bout = sh("cat @WD@/bob.out").out +local aerr = sh("cat @WD@/alice.err").out +local berr = sh("cat @WD@/bob.err").out + +checks.expect_match(aout, "discovered peer bob at 127%.0%.0%.1:" .. bport, + lang .. ": alice discovers bob at his advertised tcp port") +checks.expect_match(bout, "discovered peer alice at 127%.0%.0%.1:" .. aport, + lang .. ": bob discovers alice at her advertised tcp port") + +checks.expect_match(aout, "peer bob says: hello from bob", + lang .. ": alice receives bob's chat frame over TCP") +checks.expect_match(bout, "peer alice says: hello from alice", + lang .. ": bob receives alice's chat frame over TCP") + +-- discovery is deduped: exactly one "discovered peer" line per process. +local _, an = aout:gsub("discovered peer ", "") +local _, bn = bout:gsub("discovered peer ", "") +checks.expect_match("count=" .. an, "^count=1$", + lang .. ": alice discovers bob exactly once (dedup)") +checks.expect_match("count=" .. bn, "^count=1$", + lang .. ": bob discovers alice exactly once (dedup)") + +checks.expect_match(aerr, "chatterd: announcing as alice on " .. group, + lang .. ": alice announces on stderr") +checks.expect_match(berr, "chatterd: announcing as bob on " .. group, + lang .. ": bob announces on stderr") + +checks.run("rm -rf " .. wd) +checks.finish() diff --git a/examples/24-time-timers-deadlines/README.md b/examples/24-time-timers-deadlines/README.md new file mode 100644 index 0000000..221566f --- /dev/null +++ b/examples/24-time-timers-deadlines/README.md @@ -0,0 +1,168 @@ +# 24-time-timers-deadlines — chatterd v3, heartbeats + deadlines + +Chapter 24 grows `chatterd` — the peer-to-peer chat daemon introduced in +chapter 21 — into a version that survives a peer going away. The lesson is +*time*: how to keep a connection alive with a periodic heartbeat, how to +declare a peer dead by measuring **silence on a monotonic clock**, and how to +come back with a **jittered exponential backoff** instead of a hot reconnect +loop. A standalone `clockprobe` subcommand measures the machine's actual timer +resolution so the numbers stop being abstract. + +``` +chatterd clockprobe # measure clock/timer resolution +chatterd listen --name N --addr HOST:PORT [opts] # passive peer +chatterd connect --name N --peer NAME@HOST:PORT [opts] # active peer (dials + reconnects) + +opts: --heartbeat-ms H (default 1000) PING interval + --timeout-ms T (default 3000) drop a silent peer after this + --backoff-ms B (default 200) base reconnect backoff + --max-backoff-ms M (default 5000) backoff ceiling + --message MSG send one chat message after linking + --seed S seed the jitter RNG (deterministic tests) +``` + +## Behavior (identical across C++, Go, and Rust) + +- **`clockprobe`** prints exactly one line and exits 0: + + ``` + clockprobe: CLOCK_MONOTONIC res=ns nanosleep(1ms) actual=us timerfd(1ms) actual=us + ``` + + `res` is `clock_getres(CLOCK_MONOTONIC)`; the two `actual` fields are the + measured wall-time of a 1 ms `nanosleep(2)` and a 1 ms one-shot `timerfd`, + each timed on a **monotonic/steady clock** (never `CLOCK_REALTIME`). On a + typical host both land near 1050–1100 µs — the overshoot is the scheduler's + wake-up latency, which is exactly the point. + +- **`listen`** binds, prints `chatterd: listening on :` (a + `:0` port is reported as the actual ephemeral port), and accepts a peer. + +- **`connect`** prints `chatterd: connecting to at ` and + dials. + +- On a completed JOIN handshake **both** sides print + `chatterd: linked with `; if `--message` was given, the message + is sent and the receiver prints `chatterd: message from : `. + +- While linked, each side sends a `PING` every `--heartbeat-ms`. **Any** + received frame (JOIN/MSG/PING) is "traffic" and resets a + `--timeout-ms` deadline. When the deadline fires with no traffic the peer + is dropped: `chatterd: peer timed out`. A closed socket (peer + killed) is handled the same way — reads stop, but it is the *deadline*, on + the monotonic clock, that declares death. + +- After a drop the **connector** reconnects with equal-jitter exponential + backoff — `delay = B/2 + rand(0..B/2)`, with `B` doubling each failed + attempt up to `--max-backoff-ms` and reset to the base on a fresh + connection — printing `chatterd: reconnecting to in ms` before + each wait. The **listener** simply returns to `accept()` to welcome a + reconnecting peer. + +- **SIGINT / SIGTERM** at any point (linked, mid-backoff, or waiting to + accept) prints `chatterd: shutting down` and exits 0. + +- No/bad arguments print a usage block on stderr and exit 2. + +## Wire format (identical across languages and every chatterd version) + +This is **the** chatterd chat frame: introduced in chapter 21, and extended +in later chapters only by adding new `TYPE` values — the header itself never +changes, so every chatterd version (thread-per-connection in ch21, epoll in +ch22, the UDP-discovered peers of ch23, this chapter's heartbeats, the async +runtimes of ch27) reads and writes exactly these bytes. All multi-byte +integers are big-endian. The header is a fixed 6 bytes: + +``` + offset size field value / meaning + ------ ---- -------- --------------------------------------------------- + 0 1 MAGIC0 0x43 ('C') + 1 1 MAGIC1 0x48 ('H') + 2 1 VERSION 0x01 + 3 1 TYPE 0x01 JOIN payload = peer name (UTF-8) + 0x02 MSG payload = chat text (UTF-8) + 0x03 DELIVER payload = name + 0x00 (NUL) + text + 0x04 WELCOME payload = motd/roster text (UTF-8) + 0x05 PING payload = empty (v3 addition) + 0x06..0xFF reserved for future versions + 4 2 LENGTH uint16 payload length N (0..65535) + 6 N PAYLOAD N bytes +``` + +Framing rules, byte-exact and version-stable: + +- A reader accumulates bytes and consumes a frame only once all `6 + N` bytes + are present; partial reads are buffered. +- `MAGIC`/`VERSION` mismatch is a protocol error and drops the connection. +- `PING` is the frame type added in **v3** (this chapter) for heartbeating. + `DELIVER` and `WELCOME` belong to the server-broadcast chatterd versions + (ch22, ch27) — this chapter's peer-to-peer `listen`/`connect` model never + emits them, but since the header is shared, a peer that ever receives one + (or any `0x06..0xFF` value) treats it as generic liveness traffic and skips + its `LENGTH` bytes rather than erroring. `JOIN` (the peer-name handshake) + and `MSG` (chat text) are unchanged since ch21, so a v3 peer here still + links with older or newer chatterd peers for everything except + heartbeating — dropping a silent peer is the only thing PING adds. + +Example: a JOIN announcing the name `A` is the 7 bytes +`43 48 01 01 00 01 41`; a PING is the 6 bytes `43 48 01 05 00 00`. + +## One subject, three timer stacks + +The protocol and the observable lines are identical; how each language builds +the heartbeat/deadline loop is the lesson. + +- **C++ — `timerfd_create(CLOCK_MONOTONIC)` + `poll(2)`.** A periodic timerfd + is the heartbeat; a `std::chrono::steady_clock` deadline is peer liveness + (the poll timeout is the time left until it). The peer socket, the heartbeat + timerfd, and a `signalfd(2)` shutdown doorbell share one `poll` loop. Every + fd is owned by a RAII `UniqueFd`; sends use `MSG_NOSIGNAL`, and errors flow + through `std::expected`. +- **Go — `time.Ticker` + `time.Timer` + `context`.** A `Ticker` fires the + heartbeat, a `Timer` is the deadline (`Reset` on every frame), and the + context from `signal.NotifyContext` is shutdown. A per-connection reader + goroutine turns the socket into a channel of frames, so the session is one + `select` over ticker / deadline / frames / `ctx.Done()`. `time.Now` carries + a monotonic reading, so `time.Since` never sees a wall-clock step. +- **Rust — `nix` timerfd + `std::time::Instant` + `poll`.** A periodic + `nix::sys::timerfd::TimerFd` is the heartbeat, an `Instant` deadline is peer + liveness, and a `nix` `SignalFd` is shutdown — all three in one + `nix::poll::poll` loop over `std` `TcpStream`/`TcpListener` fds (`OwnedFd` + ownership throughout). Errors flow through `Result`/`?`. No tokio here — + async is chapter 27. + +## Layout + +``` +24-time-timers-deadlines/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated behavioral check (clockprobe + two-peer scenario) +├── cpp/ go/ rust/ # one implementation each, same demo.sh contract +``` + +## The demo contract + +Each language directory has a `demo.sh` with the identical interface used +book-wide: `./demo.sh build`, `./demo.sh run [args]`, and a bare `./demo.sh` +that builds then runs. With `TARGET` set, `run` deploys to that lab VM — but +this chapter runs purely on the host (loopback), so `TARGET` is unused here. + +Try it by hand: + +``` +# terminal 1 — passive peer on an ephemeral port +./demo.sh cpp run listen --name A --addr 127.0.0.1:9000 --heartbeat-ms 500 --timeout-ms 1500 --message "hi from A" +# terminal 2 — active peer; ^C peer A and watch B time out then back off +./demo.sh cpp run connect --name B --peer A@127.0.0.1:9000 --heartbeat-ms 500 --timeout-ms 1500 --message "hi from B" +``` + +## Verification + +`verify.lua` (run by `scripts/test-all-examples.py` with `LSP_LANG` set) +asserts observable behavior, not exit 0: that `clockprobe` prints all three +resolution fields with parseable ns/µs on a `CLOCK_MONOTONIC` line (never +`CLOCK_REALTIME`), that two loopback peers link and deliver a message each +way, that killing the listener makes the connector print `peer A timed out` +followed by two-or-more growing, jittered `reconnecting to A in ms` lines, +and that a `SIGTERM` shuts the connector down cleanly (exit 0). Exit 0 = pass, +1 = fail, 77 = skip. diff --git a/examples/24-time-timers-deadlines/cpp/.clang-format b/examples/24-time-timers-deadlines/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/24-time-timers-deadlines/cpp/.clang-tidy b/examples/24-time-timers-deadlines/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/24-time-timers-deadlines/cpp/CMakeLists.txt b/examples/24-time-timers-deadlines/cpp/CMakeLists.txt new file mode 100644 index 0000000..7cbca73 --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd_timers CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/24-time-timers-deadlines/cpp/CMakePresets.json b/examples/24-time-timers-deadlines/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/24-time-timers-deadlines/cpp/demo.sh b/examples/24-time-timers-deadlines/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/24-time-timers-deadlines/cpp/src/main.cpp b/examples/24-time-timers-deadlines/cpp/src/main.cpp new file mode 100644 index 0000000..209a5a0 --- /dev/null +++ b/examples/24-time-timers-deadlines/cpp/src/main.cpp @@ -0,0 +1,652 @@ +// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// +// chatterd clockprobe +// chatterd listen --name N --addr HOST:PORT [opts] +// chatterd connect --name N --peer NAME@HOST:PORT [opts] +// +// The C++ take on deadlines: a periodic timerfd(CLOCK_MONOTONIC) drives the +// keepalive heartbeat, and a std::chrono::steady_clock deadline drives peer +// liveness. Both are fed to a single poll(2) loop alongside the peer socket +// and a signalfd(2) doorbell for clean shutdown. Nothing here reads the wall +// clock: a peer is declared dead by *silence* measured on a monotonic clock, +// never by a wall-clock timestamp that a settimeofday(2) could rewind. +// +// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// was introduced in chapter 21 and extended only by adding TYPE values in +// later chapters; the header never changes. All integers big-endian: +// +// byte 0 MAGIC0 = 0x43 ('C') +// byte 1 MAGIC1 = 0x48 ('H') +// byte 2 VERSION = 0x01 +// byte 3 TYPE 1=JOIN 2=MSG 3=DELIVER 4=WELCOME 5=PING +// byte 4-5 LENGTH uint16 payload length N +// byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING +// empty); DELIVER is name + 0x00 (NUL) + text. +// +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the +// header is shared, are simply treated as liveness traffic here if ever seen. +// PING is the v3 addition over v0/v1. Any received frame is "traffic" and +// resets the deadline; a peer with no traffic for --timeout-ms is dropped. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +using namespace std::chrono; + +// --------------------------------------------------------------------------- +// RAII around a file descriptor — never a naked close(2). +// --------------------------------------------------------------------------- +class UniqueFd { + public: + UniqueFd() = default; + explicit UniqueFd(int fd) : fd_(fd) {} + UniqueFd(const UniqueFd&) = delete; + UniqueFd& operator=(const UniqueFd&) = delete; + UniqueFd(UniqueFd&& other) noexcept : fd_(std::exchange(other.fd_, -1)) {} + UniqueFd& operator=(UniqueFd&& other) noexcept { + if (this != &other) { + reset(); + fd_ = std::exchange(other.fd_, -1); + } + return *this; + } + ~UniqueFd() { reset(); } + [[nodiscard]] int get() const { return fd_; } + [[nodiscard]] bool valid() const { return fd_ >= 0; } + + private: + void reset() { + if (fd_ >= 0) { + ::close(fd_); + fd_ = -1; + } + } + int fd_ = -1; +}; + +[[nodiscard]] std::error_code errno_ec() { + return {errno, std::generic_category()}; +} + +[[noreturn]] void die(std::string_view what, std::error_code ec) { + std::println(stderr, "chatterd: error: {}: {}", what, ec.message()); + std::exit(1); +} + +[[noreturn]] void usage() { + std::println(stderr, "usage: chatterd clockprobe"); + std::println(stderr, " chatterd listen --name N --addr HOST:PORT [opts]"); + std::println(stderr, " chatterd connect --name N --peer NAME@HOST:PORT [opts]"); + std::println(stderr, "opts: --heartbeat-ms H --timeout-ms T --backoff-ms B " + "--max-backoff-ms M --message MSG --seed S"); + std::exit(2); +} + +// --------------------------------------------------------------------------- +// Protocol +// --------------------------------------------------------------------------- +constexpr std::uint8_t kMagic0 = 'C'; +constexpr std::uint8_t kMagic1 = 'H'; +constexpr std::uint8_t kVersion = 1; +constexpr std::size_t kHeader = 6; +constexpr std::size_t kMaxPayload = 65535; + +// DELIVER (3) and WELCOME (4) belong to other chatterd versions; this v3 +// peer never emits them and treats them as generic traffic if received. +enum class Type : std::uint8_t { Join = 1, Msg = 2, Deliver = 3, Welcome = 4, Ping = 5 }; + +struct Frame { + Type type; + std::string payload; +}; + +[[nodiscard]] std::string encode(Type type, std::string_view payload) { + std::string out; + out.resize(kHeader + payload.size()); + out[0] = static_cast(kMagic0); + out[1] = static_cast(kMagic1); + out[2] = static_cast(kVersion); + out[3] = static_cast(type); + out[4] = static_cast((payload.size() >> 8) & 0xff); + out[5] = static_cast(payload.size() & 0xff); + std::memcpy(out.data() + kHeader, payload.data(), payload.size()); + return out; +} + +// Pop one complete frame off the front of buf. nullopt = need more bytes; a +// bad magic/version sets `bad` so the caller can drop the connection. +[[nodiscard]] std::optional pop_frame(std::string& buf, bool& bad) { + if (buf.size() < kHeader) { + return std::nullopt; + } + const auto b = reinterpret_cast(buf.data()); + if (b[0] != kMagic0 || b[1] != kMagic1 || b[2] != kVersion) { + bad = true; + return std::nullopt; + } + const std::size_t len = (static_cast(b[4]) << 8) | b[5]; + if (buf.size() < kHeader + len) { + return std::nullopt; + } + Frame f{static_cast(b[3]), buf.substr(kHeader, len)}; + buf.erase(0, kHeader + len); + return f; +} + +[[nodiscard]] std::expected send_all(int fd, std::string_view data) { + std::size_t off = 0; + while (off < data.size()) { + const ssize_t n = ::send(fd, data.data() + off, data.size() - off, MSG_NOSIGNAL); + if (n < 0) { + if (errno == EINTR) { + continue; + } + return std::unexpected(errno_ec()); + } + off += static_cast(n); + } + return {}; +} + +[[nodiscard]] std::expected +send_frame(int fd, Type type, std::string_view payload = {}) { + return send_all(fd, encode(type, payload)); +} + +// --------------------------------------------------------------------------- +// Timerfd helper (CLOCK_MONOTONIC), periodic. +// --------------------------------------------------------------------------- +[[nodiscard]] UniqueFd periodic_timerfd(long period_ms) { + UniqueFd tfd(::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC)); + if (!tfd.valid()) { + die("timerfd_create", errno_ec()); + } + itimerspec its{}; + its.it_interval.tv_sec = period_ms / 1000; + its.it_interval.tv_nsec = (period_ms % 1000) * 1'000'000; + its.it_value = its.it_interval; + if (::timerfd_settime(tfd.get(), 0, &its, nullptr) < 0) { + die("timerfd_settime", errno_ec()); + } + return tfd; +} + +// --------------------------------------------------------------------------- +// Options +// --------------------------------------------------------------------------- +struct Opts { + std::string name; + std::string listen_addr; + std::string peer_name; + std::string peer_addr; + long heartbeat_ms = 1000; + long timeout_ms = 3000; + long backoff_ms = 200; + long max_backoff_ms = 5000; + std::string message; + bool has_message = false; + unsigned seed = 0; + bool has_seed = false; +}; + +[[nodiscard]] std::expected parse_long(std::string_view s, long min) { + const std::string tmp(s); + errno = 0; + char* end = nullptr; + const long v = std::strtol(tmp.c_str(), &end, 10); + if (errno != 0 || end == tmp.c_str() || *end != '\0' || v < min) { + return std::unexpected("bad numeric value: " + tmp); + } + return v; +} + +// Split "HOST:PORT" -> {host, port}; the port keeps its string form so ":0" +// (an ephemeral bind) round-trips unchanged. +[[nodiscard]] std::expected, std::string> +split_host_port(std::string_view hp) { + const auto colon = hp.rfind(':'); + if (colon == std::string_view::npos || colon == 0 || colon + 1 == hp.size()) { + return std::unexpected("expected HOST:PORT, got " + std::string(hp)); + } + return std::pair{std::string(hp.substr(0, colon)), std::string(hp.substr(colon + 1))}; +} + +// --------------------------------------------------------------------------- +// Sockets +// --------------------------------------------------------------------------- +[[nodiscard]] std::expected +make_listener(const std::string& host, const std::string& port, std::string& bound_out) { + addrinfo hints{}; + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = AI_PASSIVE; + addrinfo* res = nullptr; + if (::getaddrinfo(host.c_str(), port.c_str(), &hints, &res) != 0) { + return std::unexpected(errno_ec()); + } + UniqueFd fd(::socket(res->ai_family, res->ai_socktype | SOCK_CLOEXEC, res->ai_protocol)); + if (!fd.valid()) { + const auto ec = errno_ec(); + ::freeaddrinfo(res); + return std::unexpected(ec); + } + int one = 1; + ::setsockopt(fd.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof one); + if (::bind(fd.get(), res->ai_addr, res->ai_addrlen) < 0 || ::listen(fd.get(), 8) < 0) { + const auto ec = errno_ec(); + ::freeaddrinfo(res); + return std::unexpected(ec); + } + ::freeaddrinfo(res); + + // Read back the actually-bound port (matters for an ephemeral ":0"). + sockaddr_in sin{}; + socklen_t sl = sizeof sin; + ::getsockname(fd.get(), reinterpret_cast(&sin), &sl); + char ip[INET_ADDRSTRLEN] = {}; + ::inet_ntop(AF_INET, &sin.sin_addr, ip, sizeof ip); + bound_out = std::format("{}:{}", ip, ntohs(sin.sin_port)); + return fd; +} + +[[nodiscard]] std::expected +dial(const std::string& host, const std::string& port) { + addrinfo hints{}; + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_STREAM; + addrinfo* res = nullptr; + if (::getaddrinfo(host.c_str(), port.c_str(), &hints, &res) != 0) { + return std::unexpected(errno_ec()); + } + UniqueFd fd(::socket(res->ai_family, res->ai_socktype | SOCK_CLOEXEC, res->ai_protocol)); + if (!fd.valid()) { + const auto ec = errno_ec(); + ::freeaddrinfo(res); + return std::unexpected(ec); + } + const int rc = ::connect(fd.get(), res->ai_addr, res->ai_addrlen); + const auto ec = errno_ec(); + ::freeaddrinfo(res); + if (rc < 0) { + return std::unexpected(ec); + } + return fd; +} + +// --------------------------------------------------------------------------- +// Shutdown doorbell: SIGINT/SIGTERM turned into a pollable fd. +// --------------------------------------------------------------------------- +[[nodiscard]] UniqueFd make_shutdown_fd() { + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + if (::sigprocmask(SIG_BLOCK, &mask, nullptr) < 0) { + die("sigprocmask", errno_ec()); + } + UniqueFd fd(::signalfd(-1, &mask, SFD_CLOEXEC)); + if (!fd.valid()) { + die("signalfd", errno_ec()); + } + return fd; +} + +// --------------------------------------------------------------------------- +// A single peer session over an established socket. +// --------------------------------------------------------------------------- +enum class Outcome { TimedOut, Shutdown, PeerError }; + +struct SessionResult { + Outcome outcome; + bool linked = false; // did we complete a JOIN handshake at least once? + std::string peer; // name learned from the peer's JOIN, if any +}; + +// Drives one connection: sends JOIN, exchanges heartbeats, prints the linked +// line and any received messages, and returns when the peer goes silent +// (TimedOut), we are asked to quit (Shutdown), or the socket errors. +SessionResult run_session(const Opts& opts, UniqueFd sock, int shutdown_fd, + std::string peer_display) { + UniqueFd hb = periodic_timerfd(opts.heartbeat_ms); + + if (auto r = send_frame(sock.get(), Type::Join, opts.name); !r) { + return {Outcome::PeerError, false, peer_display}; + } + + std::string buf; + bool linked = false; + bool sock_dead = false; // peer sent EOF; keep the deadline running + const auto timeout = milliseconds(opts.timeout_ms); + auto deadline = steady_clock::now() + timeout; + + for (;;) { + int poll_ms = 0; + { + const auto left = deadline - steady_clock::now(); + poll_ms = left <= 0ns + ? 0 + : static_cast(duration_cast(left).count()) + 1; + } + std::vector pfds; + pfds.push_back({shutdown_fd, POLLIN, 0}); + pfds.push_back({hb.get(), POLLIN, 0}); + const bool watch_sock = !sock_dead; + if (watch_sock) { + pfds.push_back({sock.get(), POLLIN, 0}); + } + + const int n = ::poll(pfds.data(), pfds.size(), poll_ms); + if (n < 0) { + if (errno == EINTR) { + continue; + } + die("poll", errno_ec()); + } + if (n == 0) { + return {Outcome::TimedOut, linked, peer_display}; // deadline: peer went silent + } + if (pfds[0].revents & POLLIN) { + return {Outcome::Shutdown, linked, peer_display}; + } + if (pfds[1].revents & POLLIN) { + std::uint64_t ticks = 0; + [[maybe_unused]] auto rd = ::read(hb.get(), &ticks, sizeof ticks); + if (!sock_dead) { + (void)send_frame(sock.get(), Type::Ping); // a broken pipe is left to the deadline + } + } + if (watch_sock && (pfds[2].revents & (POLLIN | POLLHUP))) { + char tmp[4096]; + const ssize_t rn = ::read(sock.get(), tmp, sizeof tmp); + if (rn > 0) { + buf.append(tmp, static_cast(rn)); + deadline = steady_clock::now() + timeout; // traffic: extend life + bool bad = false; + while (auto f = pop_frame(buf, bad)) { + if (f->type == Type::Join) { + peer_display = f->payload; + if (!linked) { + linked = true; + std::println("chatterd: {} linked with {}", opts.name, peer_display); + if (opts.has_message) { + (void)send_frame(sock.get(), Type::Msg, opts.message); + } + } + } else if (f->type == Type::Msg) { + std::println("chatterd: {} message from {}: {}", opts.name, peer_display, + f->payload); + } + // PING / DELIVER / WELCOME / reserved: liveness only, already counted. + } + if (bad || buf.size() > kHeader + kMaxPayload) { + return {Outcome::PeerError, linked, peer_display}; + } + } else { + // EOF (0) or error: stop reading, let the deadline declare death. + sock_dead = true; + } + } + } +} + +// --------------------------------------------------------------------------- +// Wait out a backoff, but wake early for shutdown. true = shut down now. +// --------------------------------------------------------------------------- +[[nodiscard]] bool sleep_or_shutdown(long ms, int shutdown_fd) { + pollfd pfd{shutdown_fd, POLLIN, 0}; + for (;;) { + const int n = ::poll(&pfd, 1, static_cast(ms)); + if (n < 0) { + if (errno == EINTR) { + continue; + } + die("poll", errno_ec()); + } + return n > 0; // readable = a shutdown signal arrived + } +} + +// --------------------------------------------------------------------------- +// Roles +// --------------------------------------------------------------------------- +int run_listen(const Opts& opts) { + UniqueFd shutdown_fd = make_shutdown_fd(); + const auto hp = split_host_port(opts.listen_addr); + if (!hp) { + std::println(stderr, "chatterd: error: {}", hp.error()); + return 2; + } + std::string bound; + auto listener = make_listener(hp->first, hp->second, bound); + if (!listener) { + die("listen", listener.error()); + } + std::println("chatterd: {} listening on {}", opts.name, bound); + + std::string linked_peer = "peer"; + for (;;) { + pollfd pfds[2] = {{shutdown_fd.get(), POLLIN, 0}, {listener->get(), POLLIN, 0}}; + const int n = ::poll(pfds, 2, -1); + if (n < 0) { + if (errno == EINTR) { + continue; + } + die("poll", errno_ec()); + } + if (pfds[0].revents & POLLIN) { + std::println("chatterd: {} shutting down", opts.name); + return 0; + } + if (pfds[1].revents & POLLIN) { + UniqueFd conn(::accept4(listener->get(), nullptr, nullptr, SOCK_CLOEXEC)); + if (!conn.valid()) { + continue; + } + const auto res = run_session(opts, std::move(conn), shutdown_fd.get(), linked_peer); + if (res.outcome == Outcome::Shutdown) { + std::println("chatterd: {} shutting down", opts.name); + return 0; + } + if (res.outcome == Outcome::TimedOut && res.linked) { + std::println("chatterd: peer {} timed out", res.peer); + } + // Loop back to accept: a reconnecting peer is welcomed again. + } + } +} + +int run_connect(const Opts& opts) { + UniqueFd shutdown_fd = make_shutdown_fd(); + const auto hp = split_host_port(opts.peer_addr); + if (!hp) { + std::println(stderr, "chatterd: error: {}", hp.error()); + return 2; + } + std::println("chatterd: {} connecting to {} at {}", opts.name, opts.peer_name, opts.peer_addr); + + std::mt19937 rng(opts.has_seed ? opts.seed : std::random_device{}()); + long backoff = opts.backoff_ms; + + for (;;) { + auto sock = dial(hp->first, hp->second); + if (sock) { + backoff = opts.backoff_ms; // a fresh TCP connection resets backoff + const auto res = run_session(opts, std::move(*sock), shutdown_fd.get(), opts.peer_name); + if (res.outcome == Outcome::Shutdown) { + std::println("chatterd: {} shutting down", opts.name); + return 0; + } + if (res.outcome == Outcome::TimedOut) { + std::println("chatterd: peer {} timed out", opts.peer_name); + } + } + // Jittered exponential backoff (equal jitter: half fixed, half random). + const long half = backoff / 2; + std::uniform_int_distribution jit(0, half > 0 ? half : 0); + const long delay = half + jit(rng); + std::println("chatterd: reconnecting to {} in {}ms", opts.peer_name, delay); + if (sleep_or_shutdown(delay, shutdown_fd.get())) { + std::println("chatterd: {} shutting down", opts.name); + return 0; + } + backoff = std::min(backoff * 2, opts.max_backoff_ms); + } +} + +int run_clockprobe() { + timespec res{}; + ::clock_getres(CLOCK_MONOTONIC, &res); + const long res_ns = res.tv_sec * 1'000'000'000L + res.tv_nsec; + + const auto t0 = steady_clock::now(); + timespec req{0, 1'000'000}; // 1 ms + timespec rem{}; + while (::nanosleep(&req, &rem) != 0 && errno == EINTR) { + req = rem; + } + const auto sleep_us = duration_cast(steady_clock::now() - t0).count(); + + UniqueFd tfd(::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC)); + if (!tfd.valid()) { + die("timerfd_create", errno_ec()); + } + itimerspec its{}; + its.it_value.tv_nsec = 1'000'000; // one-shot 1 ms + if (::timerfd_settime(tfd.get(), 0, &its, nullptr) < 0) { + die("timerfd_settime", errno_ec()); + } + const auto t1 = steady_clock::now(); + std::uint64_t ticks = 0; + [[maybe_unused]] auto rd = ::read(tfd.get(), &ticks, sizeof ticks); + const auto tfd_us = duration_cast(steady_clock::now() - t1).count(); + + std::println("clockprobe: CLOCK_MONOTONIC res={}ns nanosleep(1ms) actual={}us " + "timerfd(1ms) actual={}us", + res_ns, sleep_us, tfd_us); + return 0; +} + +// --------------------------------------------------------------------------- +// Argument parsing +// --------------------------------------------------------------------------- +[[nodiscard]] std::expected parse_common(const std::vector& args) { + Opts o; + for (std::size_t i = 0; i < args.size(); ++i) { + auto need = [&](const std::string& flag) -> std::expected { + if (i + 1 >= args.size()) { + return std::unexpected(flag + " needs a value"); + } + return args[++i]; + }; + const std::string& a = args[i]; + if (a == "--name") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + o.name = *v; + } else if (a == "--addr") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + o.listen_addr = *v; + } else if (a == "--peer") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + const auto at = v->find('@'); + if (at == std::string::npos) { + return std::unexpected("--peer wants NAME@HOST:PORT"); + } + o.peer_name = v->substr(0, at); + o.peer_addr = v->substr(at + 1); + } else if (a == "--heartbeat-ms" || a == "--timeout-ms" || a == "--backoff-ms" || + a == "--max-backoff-ms") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + auto num = parse_long(*v, 1); + if (!num) return std::unexpected(num.error()); + if (a == "--heartbeat-ms") o.heartbeat_ms = *num; + else if (a == "--timeout-ms") o.timeout_ms = *num; + else if (a == "--backoff-ms") o.backoff_ms = *num; + else o.max_backoff_ms = *num; + } else if (a == "--message") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + o.message = *v; + o.has_message = true; + } else if (a == "--seed") { + auto v = need(a); + if (!v) return std::unexpected(v.error()); + auto num = parse_long(*v, 0); + if (!num) return std::unexpected(num.error()); + o.seed = static_cast(*num); + o.has_seed = true; + } else { + return std::unexpected("unknown flag " + a); + } + } + if (o.name.empty()) { + return std::unexpected("--name is required"); + } + return o; +} + +} // namespace + +int main(int argc, char** argv) { + std::setvbuf(stdout, nullptr, _IOLBF, 0); + + const std::vector args(argv + 1, argv + argc); + if (args.empty()) { + usage(); + } + const std::string& sub = args[0]; + const std::vector rest(args.begin() + 1, args.end()); + + if (sub == "clockprobe") { + return run_clockprobe(); + } + if (sub == "listen" || sub == "connect") { + auto opts = parse_common(rest); + if (!opts) { + std::println(stderr, "chatterd: error: {}", opts.error()); + usage(); + } + if (sub == "listen") { + if (opts->listen_addr.empty()) { + std::println(stderr, "chatterd: error: listen needs --addr"); + usage(); + } + return run_listen(*opts); + } + if (opts->peer_addr.empty()) { + std::println(stderr, "chatterd: error: connect needs --peer"); + usage(); + } + return run_connect(*opts); + } + usage(); +} diff --git a/examples/24-time-timers-deadlines/demo.sh b/examples/24-time-timers-deadlines/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/24-time-timers-deadlines/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/24-time-timers-deadlines/go/demo.sh b/examples/24-time-timers-deadlines/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/24-time-timers-deadlines/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/24-time-timers-deadlines/go/go.mod b/examples/24-time-timers-deadlines/go/go.mod new file mode 100644 index 0000000..fa59b48 --- /dev/null +++ b/examples/24-time-timers-deadlines/go/go.mod @@ -0,0 +1,7 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/24-time-timers-deadlines/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 diff --git a/examples/24-time-timers-deadlines/go/go.sum b/examples/24-time-timers-deadlines/go/go.sum new file mode 100644 index 0000000..37ee2d4 --- /dev/null +++ b/examples/24-time-timers-deadlines/go/go.sum @@ -0,0 +1,2 @@ +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= diff --git a/examples/24-time-timers-deadlines/go/main.go b/examples/24-time-timers-deadlines/go/main.go new file mode 100644 index 0000000..f0cbb98 --- /dev/null +++ b/examples/24-time-timers-deadlines/go/main.go @@ -0,0 +1,467 @@ +// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// +// chatterd clockprobe +// chatterd listen --name N --addr HOST:PORT [opts] +// chatterd connect --name N --peer NAME@HOST:PORT [opts] +// +// The Go take on deadlines: a time.Ticker fires the keepalive heartbeat, a +// time.Timer is the peer-liveness deadline (Reset on every received frame), +// and a context from signal.NotifyContext is the clean-shutdown signal. A +// per-connection reader goroutine turns the socket into a channel of frames, +// so the whole session is one select over ticker / deadline / frames / +// ctx.Done(). time.Now carries a monotonic reading, so "silence" is measured +// on the monotonic clock, never the wall clock. +// +// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// was introduced in chapter 21 and extended only by adding TYPE values in +// later chapters; the header never changes. All integers big-endian: +// +// byte 0 MAGIC0 = 0x43 ('C') +// byte 1 MAGIC1 = 0x48 ('H') +// byte 2 VERSION = 0x01 +// byte 3 TYPE 1=JOIN 2=MSG 3=DELIVER 4=WELCOME 5=PING +// byte 4-5 LENGTH uint16 payload length N +// byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING +// empty); DELIVER is name + 0x00 (NUL) + text. +// +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the +// header is shared, are simply treated as liveness traffic here if ever seen. +// PING is the v3 addition over v0/v1. Any received frame is "traffic" and +// resets the deadline; a peer with no traffic for --timeout-ms is dropped. +package main + +import ( + "context" + "encoding/binary" + "fmt" + "io" + "math/rand" + "net" + "os" + "os/signal" + "strconv" + "strings" + "time" + "unsafe" + + "golang.org/x/sys/unix" +) + +// --------------------------------------------------------------------------- +// Protocol +// --------------------------------------------------------------------------- + +const ( + magic0 = 'C' + magic1 = 'H' + version = 1 + header = 6 + + typeJoin = 1 + typeMsg = 2 + typePing = 5 + // typeDeliver = 3 and typeWelcome = 4 belong to other chatterd versions + // (server broadcast / connect motd); v3 never emits them and treats them + // as generic liveness traffic if ever received. +) + +type frame struct { + typ byte + payload []byte +} + +func encode(typ byte, payload []byte) []byte { + out := make([]byte, header+len(payload)) + out[0], out[1], out[2], out[3] = magic0, magic1, version, typ + binary.BigEndian.PutUint16(out[4:6], uint16(len(payload))) + copy(out[header:], payload) + return out +} + +func sendFrame(conn net.Conn, typ byte, payload []byte) error { + if _, err := conn.Write(encode(typ, payload)); err != nil { + return fmt.Errorf("send frame: %w", err) + } + return nil +} + +// readFrames pumps decoded frames onto ch until the socket closes; a final +// zero-value frame (typ==0) marks EOF so the session can keep its deadline +// running rather than react to the FIN directly. +func readFrames(conn net.Conn, ch chan<- frame) { + defer close(ch) + r := io.Reader(conn) + head := make([]byte, header) + for { + if _, err := io.ReadFull(r, head); err != nil { + ch <- frame{} // EOF sentinel + return + } + if head[0] != magic0 || head[1] != magic1 || head[2] != version { + ch <- frame{} // protocol error: treat as gone + return + } + n := binary.BigEndian.Uint16(head[4:6]) + payload := make([]byte, n) + if _, err := io.ReadFull(r, payload); err != nil { + ch <- frame{} + return + } + ch <- frame{typ: head[3], payload: payload} + } +} + +// --------------------------------------------------------------------------- +// Options +// --------------------------------------------------------------------------- + +type opts struct { + name string + listenAddr string + peerName string + peerAddr string + heartbeatMs int64 + timeoutMs int64 + backoffMs int64 + maxBackoffMs int64 + message string + hasMessage bool + seed int64 + hasSeed bool +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: chatterd clockprobe") + fmt.Fprintln(os.Stderr, " chatterd listen --name N --addr HOST:PORT [opts]") + fmt.Fprintln(os.Stderr, " chatterd connect --name N --peer NAME@HOST:PORT [opts]") + fmt.Fprintln(os.Stderr, "opts: --heartbeat-ms H --timeout-ms T --backoff-ms B "+ + "--max-backoff-ms M --message MSG --seed S") + os.Exit(2) +} + +func parseCommon(args []string) (opts, error) { + o := opts{heartbeatMs: 1000, timeoutMs: 3000, backoffMs: 200, maxBackoffMs: 5000} + need := func(i *int, flag string) (string, error) { + if *i+1 >= len(args) { + return "", fmt.Errorf("%s needs a value", flag) + } + *i++ + return args[*i], nil + } + num := func(s, flag string, min int64) (int64, error) { + v, err := strconv.ParseInt(s, 10, 64) + if err != nil || v < min { + return 0, fmt.Errorf("bad value for %s: %s", flag, s) + } + return v, nil + } + for i := 0; i < len(args); i++ { + a := args[i] + switch a { + case "--name": + v, err := need(&i, a) + if err != nil { + return o, err + } + o.name = v + case "--addr": + v, err := need(&i, a) + if err != nil { + return o, err + } + o.listenAddr = v + case "--peer": + v, err := need(&i, a) + if err != nil { + return o, err + } + at := strings.IndexByte(v, '@') + if at < 0 { + return o, fmt.Errorf("--peer wants NAME@HOST:PORT") + } + o.peerName, o.peerAddr = v[:at], v[at+1:] + case "--heartbeat-ms", "--timeout-ms", "--backoff-ms", "--max-backoff-ms": + v, err := need(&i, a) + if err != nil { + return o, err + } + n, err := num(v, a, 1) + if err != nil { + return o, err + } + switch a { + case "--heartbeat-ms": + o.heartbeatMs = n + case "--timeout-ms": + o.timeoutMs = n + case "--backoff-ms": + o.backoffMs = n + default: + o.maxBackoffMs = n + } + case "--message": + v, err := need(&i, a) + if err != nil { + return o, err + } + o.message, o.hasMessage = v, true + case "--seed": + v, err := need(&i, a) + if err != nil { + return o, err + } + n, err := num(v, a, 0) + if err != nil { + return o, err + } + o.seed, o.hasSeed = n, true + default: + return o, fmt.Errorf("unknown flag %s", a) + } + } + if o.name == "" { + return o, fmt.Errorf("--name is required") + } + return o, nil +} + +// --------------------------------------------------------------------------- +// Session +// --------------------------------------------------------------------------- + +type outcome int + +const ( + timedOut outcome = iota + shutdown + peerError +) + +type sessionResult struct { + outcome outcome + linked bool + peer string // name learned from the peer's JOIN, if any +} + +// runSession drives one connection until the peer goes silent (timedOut), the +// context is cancelled (shutdown), or the socket errors. +func runSession(ctx context.Context, o opts, conn net.Conn, peerDisplay string) sessionResult { + defer conn.Close() + + frames := make(chan frame, 8) + go readFrames(conn, frames) + + ticker := time.NewTicker(time.Duration(o.heartbeatMs) * time.Millisecond) + defer ticker.Stop() + deadline := time.NewTimer(time.Duration(o.timeoutMs) * time.Millisecond) + defer deadline.Stop() + + _ = sendFrame(conn, typeJoin, []byte(o.name)) + + linked := false + sockDead := false + for { + var frameCh <-chan frame + if !sockDead { + frameCh = frames + } + select { + case <-ctx.Done(): + return sessionResult{shutdown, linked, peerDisplay} + case <-deadline.C: + return sessionResult{timedOut, linked, peerDisplay} // peer went silent + case <-ticker.C: + if !sockDead { + _ = sendFrame(conn, typePing, nil) + } + case f, ok := <-frameCh: + if !ok || f.typ == 0 { + // EOF / protocol error: stop reading, let the deadline fire. + sockDead = true + continue + } + // Traffic: extend the deadline. + if !deadline.Stop() { + select { + case <-deadline.C: + default: + } + } + deadline.Reset(time.Duration(o.timeoutMs) * time.Millisecond) + switch f.typ { + case typeJoin: + peerDisplay = string(f.payload) + if !linked { + linked = true + fmt.Printf("chatterd: %s linked with %s\n", o.name, peerDisplay) + if o.hasMessage { + _ = sendFrame(conn, typeMsg, []byte(o.message)) + } + } + case typeMsg: + fmt.Printf("chatterd: %s message from %s: %s\n", o.name, peerDisplay, f.payload) + } + // PING / DELIVER / WELCOME / reserved: liveness only, already counted. + } + } +} + +// --------------------------------------------------------------------------- +// Roles +// --------------------------------------------------------------------------- + +func runListen(ctx context.Context, o opts) int { + lc := net.ListenConfig{} + ln, err := lc.Listen(ctx, "tcp", o.listenAddr) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: listen: %v\n", err) + return 1 + } + defer ln.Close() + fmt.Printf("chatterd: %s listening on %s\n", o.name, ln.Addr().String()) + + go func() { <-ctx.Done(); ln.Close() }() // unblock Accept on shutdown + + for { + conn, err := ln.Accept() + if err != nil { + if ctx.Err() != nil { + fmt.Printf("chatterd: %s shutting down\n", o.name) + return 0 + } + continue + } + res := runSession(ctx, o, conn, "peer") + switch res.outcome { + case shutdown: + fmt.Printf("chatterd: %s shutting down\n", o.name) + return 0 + case timedOut: + if res.linked { + fmt.Printf("chatterd: peer %s timed out\n", res.peer) + } + } + // Loop back to accept: a reconnecting peer is welcomed again. + } +} + +func runConnect(ctx context.Context, o opts) int { + fmt.Printf("chatterd: %s connecting to %s at %s\n", o.name, o.peerName, o.peerAddr) + + var rng *rand.Rand + if o.hasSeed { + rng = rand.New(rand.NewSource(o.seed)) + } else { + rng = rand.New(rand.NewSource(time.Now().UnixNano())) + } + backoff := o.backoffMs + dialer := net.Dialer{} + + for { + conn, err := dialer.DialContext(ctx, "tcp", o.peerAddr) + if err == nil { + backoff = o.backoffMs // a fresh TCP connection resets backoff + res := runSession(ctx, o, conn, o.peerName) + switch res.outcome { + case shutdown: + fmt.Printf("chatterd: %s shutting down\n", o.name) + return 0 + case timedOut: + fmt.Printf("chatterd: peer %s timed out\n", o.peerName) + } + } else if ctx.Err() != nil { + fmt.Printf("chatterd: %s shutting down\n", o.name) + return 0 + } + // Jittered exponential backoff (equal jitter: half fixed, half random). + half := backoff / 2 + delay := half + if half > 0 { + delay += rng.Int63n(half + 1) + } + fmt.Printf("chatterd: reconnecting to %s in %dms\n", o.peerName, delay) + select { + case <-ctx.Done(): + fmt.Printf("chatterd: %s shutting down\n", o.name) + return 0 + case <-time.After(time.Duration(delay) * time.Millisecond): + } + backoff = min(backoff*2, o.maxBackoffMs) + } +} + +func runClockprobe() int { + var res unix.Timespec + if err := unix.ClockGetres(unix.CLOCK_MONOTONIC, &res); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: clock_getres: %v\n", err) + return 1 + } + resNs := res.Sec*1_000_000_000 + int64(res.Nsec) + + t0 := time.Now() + req := unix.NsecToTimespec(1_000_000) // 1 ms + for unix.Nanosleep(&req, &req) == unix.EINTR { + } + sleepUs := time.Since(t0).Microseconds() + + tfd, err := unix.TimerfdCreate(unix.CLOCK_MONOTONIC, unix.TFD_CLOEXEC) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: timerfd_create: %v\n", err) + return 1 + } + defer unix.Close(tfd) + its := unix.ItimerSpec{Value: unix.Timespec{Nsec: 1_000_000}} // one-shot 1 ms + if err := unix.TimerfdSettime(tfd, 0, &its, nil); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: timerfd_settime: %v\n", err) + return 1 + } + t1 := time.Now() + var ticks uint64 + buf := (*[8]byte)(unsafe.Pointer(&ticks))[:] + _, _ = unix.Read(tfd, buf) + tfdUs := time.Since(t1).Microseconds() + + fmt.Printf("clockprobe: CLOCK_MONOTONIC res=%dns nanosleep(1ms) actual=%dus "+ + "timerfd(1ms) actual=%dus\n", resNs, sleepUs, tfdUs) + return 0 +} + +func main() { + args := os.Args[1:] + if len(args) == 0 { + usage() + } + sub, rest := args[0], args[1:] + + if sub == "clockprobe" { + os.Exit(runClockprobe()) + } + if sub != "listen" && sub != "connect" { + usage() + } + + o, err := parseCommon(rest) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: error: %v\n", err) + usage() + } + + ctx, stop := signal.NotifyContext(context.Background(), unix.SIGINT, unix.SIGTERM) + defer stop() + + switch sub { + case "listen": + if o.listenAddr == "" { + fmt.Fprintln(os.Stderr, "chatterd: error: listen needs --addr") + usage() + } + os.Exit(runListen(ctx, o)) + case "connect": + if o.peerAddr == "" { + fmt.Fprintln(os.Stderr, "chatterd: error: connect needs --peer") + usage() + } + os.Exit(runConnect(ctx, o)) + } +} diff --git a/examples/24-time-timers-deadlines/rust/Cargo.lock b/examples/24-time-timers-deadlines/rust/Cargo.lock new file mode 100644 index 0000000..2beef0b --- /dev/null +++ b/examples/24-time-timers-deadlines/rust/Cargo.lock @@ -0,0 +1,46 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chatterd-timers" +version = "0.3.0" +dependencies = [ + "nix", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] diff --git a/examples/24-time-timers-deadlines/rust/Cargo.toml b/examples/24-time-timers-deadlines/rust/Cargo.toml new file mode 100644 index 0000000..46932d1 --- /dev/null +++ b/examples/24-time-timers-deadlines/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "chatterd-timers" +version = "0.3.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +nix = { version = "0.31", features = ["time", "poll", "signal"] } diff --git a/examples/24-time-timers-deadlines/rust/demo.sh b/examples/24-time-timers-deadlines/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/24-time-timers-deadlines/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/24-time-timers-deadlines/rust/rust-toolchain.toml b/examples/24-time-timers-deadlines/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/24-time-timers-deadlines/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/24-time-timers-deadlines/rust/src/main.rs b/examples/24-time-timers-deadlines/rust/src/main.rs new file mode 100644 index 0000000..3893c12 --- /dev/null +++ b/examples/24-time-timers-deadlines/rust/src/main.rs @@ -0,0 +1,492 @@ +// chatterd v3 — heartbeats + deadlines (chapter 24: time, timers, deadlines). +// +// chatterd clockprobe +// chatterd listen --name N --addr HOST:PORT [opts] +// chatterd connect --name N --peer NAME@HOST:PORT [opts] +// +// The Rust take on deadlines: a periodic timerfd(CLOCK_MONOTONIC) from nix +// drives the keepalive heartbeat, a std::time::Instant deadline drives peer +// liveness, and a signalfd doorbell drives clean shutdown. All three are fed +// to one poll(2) loop alongside the peer socket. Instant is a monotonic +// clock, so a peer is declared dead by silence, never by a wall-clock reading +// that a clock step could rewind. No tokio here — that is chapter 27. +// +// Wire format — this is THE canonical chatterd chat frame, unchanged since it +// was introduced in chapter 21 and extended only by adding TYPE values in +// later chapters; the header never changes. All integers big-endian: +// +// byte 0 MAGIC0 = 0x43 ('C') +// byte 1 MAGIC1 = 0x48 ('H') +// byte 2 VERSION = 0x01 +// byte 3 TYPE 1=JOIN 2=MSG 3=DELIVER 4=WELCOME 5=PING +// byte 4-5 LENGTH uint16 payload length N +// byte 6.. PAYLOAD N bytes: JOIN/MSG/WELCOME/PING are UTF-8 text (PING +// empty); DELIVER is name + 0x00 (NUL) + text. +// +// This v3 (chapter 24) peer only ever emits JOIN, MSG, and PING — DELIVER and +// WELCOME belong to the server-broadcast versions (ch22/ch27) and, since the +// header is shared, are simply treated as liveness traffic here if ever seen. +// PING is the v3 addition over v0/v1. Any received frame is "traffic" and +// resets the deadline; a peer with no traffic for --timeout-ms is dropped. + +use std::io::{Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::os::fd::AsFd; +use std::time::{Duration, Instant}; + +use nix::errno::Errno; +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sys::signal::{SigSet, Signal}; +use nix::sys::signalfd::{SfdFlags, SignalFd}; +use nix::sys::time::{TimeSpec, TimeValLike}; +use nix::sys::timerfd::{ + ClockId as TfdClockId, Expiration, TimerFd, TimerFlags, TimerSetTimeFlags, +}; +use nix::time::{ClockId, ClockNanosleepFlags, clock_getres, clock_gettime, clock_nanosleep}; + +// --------------------------------------------------------------------------- +// Protocol +// --------------------------------------------------------------------------- +const MAGIC0: u8 = b'C'; +const MAGIC1: u8 = b'H'; +const VERSION: u8 = 1; +const HEADER: usize = 6; +const MAX_PAYLOAD: usize = 65535; + +const TYPE_JOIN: u8 = 1; +const TYPE_MSG: u8 = 2; +const TYPE_PING: u8 = 5; +// TYPE_DELIVER = 3 and TYPE_WELCOME = 4 belong to other chatterd versions +// (server broadcast / connect motd); v3 never emits them and treats them as +// generic liveness traffic if ever received. + +fn encode(typ: u8, payload: &[u8]) -> Vec { + let mut out = Vec::with_capacity(HEADER + payload.len()); + let len = payload.len() as u16; + out.extend_from_slice(&[MAGIC0, MAGIC1, VERSION, typ, (len >> 8) as u8, (len & 0xff) as u8]); + out.extend_from_slice(payload); + out +} + +fn send_frame(mut w: impl Write, typ: u8, payload: &[u8]) -> std::io::Result<()> { + w.write_all(&encode(typ, payload)) +} + +// --------------------------------------------------------------------------- +// Options +// --------------------------------------------------------------------------- +#[derive(Clone, Default)] +struct Opts { + name: String, + listen_addr: String, + peer_name: String, + peer_addr: String, + heartbeat_ms: i64, + timeout_ms: i64, + backoff_ms: i64, + max_backoff_ms: i64, + message: Option, + seed: Option, +} + +fn usage() -> ! { + eprintln!("usage: chatterd clockprobe"); + eprintln!(" chatterd listen --name N --addr HOST:PORT [opts]"); + eprintln!(" chatterd connect --name N --peer NAME@HOST:PORT [opts]"); + eprintln!( + "opts: --heartbeat-ms H --timeout-ms T --backoff-ms B \ + --max-backoff-ms M --message MSG --seed S" + ); + std::process::exit(2); +} + +fn parse_common(args: &[String]) -> Result { + let mut o = Opts { + heartbeat_ms: 1000, + timeout_ms: 3000, + backoff_ms: 200, + max_backoff_ms: 5000, + ..Default::default() + }; + let mut i = 0; + let need = |i: &mut usize, flag: &str| -> Result { + if *i + 1 >= args.len() { + return Err(format!("{flag} needs a value")); + } + *i += 1; + Ok(args[*i].clone()) + }; + let num = |s: &str, flag: &str, min: i64| -> Result { + s.parse::().ok().filter(|&v| v >= min).ok_or_else(|| format!("bad value for {flag}: {s}")) + }; + while i < args.len() { + let a = args[i].clone(); + match a.as_str() { + "--name" => o.name = need(&mut i, &a)?, + "--addr" => o.listen_addr = need(&mut i, &a)?, + "--peer" => { + let v = need(&mut i, &a)?; + let at = v.find('@').ok_or("--peer wants NAME@HOST:PORT")?; + o.peer_name = v[..at].to_string(); + o.peer_addr = v[at + 1..].to_string(); + } + "--heartbeat-ms" => o.heartbeat_ms = num(&need(&mut i, &a)?, &a, 1)?, + "--timeout-ms" => o.timeout_ms = num(&need(&mut i, &a)?, &a, 1)?, + "--backoff-ms" => o.backoff_ms = num(&need(&mut i, &a)?, &a, 1)?, + "--max-backoff-ms" => o.max_backoff_ms = num(&need(&mut i, &a)?, &a, 1)?, + "--message" => o.message = Some(need(&mut i, &a)?), + "--seed" => o.seed = Some(num(&need(&mut i, &a)?, &a, 0)? as u64), + other => return Err(format!("unknown flag {other}")), + } + i += 1; + } + if o.name.is_empty() { + return Err("--name is required".to_string()); + } + Ok(o) +} + +// --------------------------------------------------------------------------- +// A tiny seeded PRNG (xorshift64*) so jitter is deterministic under --seed +// without pulling in an external crate. +// --------------------------------------------------------------------------- +struct Rng(u64); + +impl Rng { + fn new(seed: u64) -> Self { + Rng(if seed == 0 { 0x9E37_79B9_7F4A_7C15 } else { seed }) + } + fn next_u64(&mut self) -> u64 { + let mut x = self.0; + x ^= x >> 12; + x ^= x << 25; + x ^= x >> 27; + self.0 = x; + x.wrapping_mul(0x2545_F491_4F6C_DD1D) + } + fn below(&mut self, n: i64) -> i64 { + if n <= 0 { 0 } else { (self.next_u64() % n as u64) as i64 } + } +} + +// --------------------------------------------------------------------------- +// Shutdown doorbell + timerfd helpers +// --------------------------------------------------------------------------- +fn install_shutdown() -> SignalFd { + let mut mask = SigSet::empty(); + mask.add(Signal::SIGINT); + mask.add(Signal::SIGTERM); + mask.thread_block().expect("block signals"); + SignalFd::with_flags(&mask, SfdFlags::SFD_CLOEXEC).expect("signalfd") +} + +fn periodic_timerfd(period_ms: i64) -> TimerFd { + let tfd = TimerFd::new(TfdClockId::CLOCK_MONOTONIC, TimerFlags::empty()).expect("timerfd"); + tfd.set(Expiration::Interval(TimeSpec::milliseconds(period_ms)), TimerSetTimeFlags::empty()) + .expect("timerfd_settime"); + tfd +} + +fn poll_timeout_until(deadline: Instant) -> PollTimeout { + let left = deadline.saturating_duration_since(Instant::now()); + if left.is_zero() { + PollTimeout::ZERO + } else { + let ms = left.as_millis().saturating_add(1).min(u16::MAX as u128) as u16; + PollTimeout::from(ms) + } +} + +// --------------------------------------------------------------------------- +// Session +// --------------------------------------------------------------------------- +#[derive(PartialEq)] +enum Outcome { + TimedOut, + Shutdown, + PeerError, +} + +struct SessionResult { + outcome: Outcome, + linked: bool, + peer: String, // name learned from the peer's JOIN, if any +} + +/// Drives one connection until the peer goes silent (TimedOut), we are asked +/// to quit (Shutdown), or the socket errors (PeerError). +fn run_session(o: &Opts, mut stream: TcpStream, sfd: &SignalFd, mut peer: String) -> SessionResult { + let hb = periodic_timerfd(o.heartbeat_ms); + let timeout = Duration::from_millis(o.timeout_ms as u64); + + let _ = send_frame(&stream, TYPE_JOIN, o.name.as_bytes()); + + let mut buf: Vec = Vec::new(); + let mut linked = false; + let mut sock_dead = false; // peer sent EOF; keep the deadline running + let mut deadline = Instant::now() + timeout; + + loop { + let (sd_ready, hb_ready, sock_ready) = { + let mut pfds = vec![ + PollFd::new(sfd.as_fd(), PollFlags::POLLIN), + PollFd::new(hb.as_fd(), PollFlags::POLLIN), + ]; + if !sock_dead { + pfds.push(PollFd::new(stream.as_fd(), PollFlags::POLLIN)); + } + match poll(&mut pfds, poll_timeout_until(deadline)) { + Ok(0) => return SessionResult { outcome: Outcome::TimedOut, linked, peer: peer.clone() }, + Ok(_) => {} + Err(Errno::EINTR) => continue, + Err(e) => panic!("poll: {e}"), + } + let ev = |idx: usize, flags: PollFlags| { + pfds[idx].revents().map(|r| r.intersects(flags)).unwrap_or(false) + }; + let sk = if sock_dead { + false + } else { + ev(2, PollFlags::POLLIN | PollFlags::POLLHUP) + }; + (ev(0, PollFlags::POLLIN), ev(1, PollFlags::POLLIN), sk) + }; + + if sd_ready { + return SessionResult { outcome: Outcome::Shutdown, linked, peer: peer.clone() }; + } + if hb_ready { + let _ = hb.wait(); + if !sock_dead { + let _ = send_frame(&stream, TYPE_PING, &[]); // a broken pipe is left to the deadline + } + } + if sock_ready { + let mut tmp = [0u8; 4096]; + match stream.read(&mut tmp) { + Ok(0) => sock_dead = true, // EOF: let the deadline declare death + Ok(n) => { + buf.extend_from_slice(&tmp[..n]); + deadline = Instant::now() + timeout; // traffic: extend life + loop { + // Capture the type byte before pop_frame drains the header. + let typ = if buf.len() >= HEADER { buf[3] } else { 0 }; + match parse_next(&mut buf) { + Ok(Some(payload)) => { + if typ == TYPE_JOIN { + peer = String::from_utf8_lossy(&payload).into_owned(); + if !linked { + linked = true; + println!("chatterd: {} linked with {}", o.name, peer); + if let Some(m) = &o.message { + let _ = send_frame(&stream, TYPE_MSG, m.as_bytes()); + } + } + } else if typ == TYPE_MSG { + println!( + "chatterd: {} message from {}: {}", + o.name, + peer, + String::from_utf8_lossy(&payload) + ); + } + // PING / DELIVER / WELCOME / reserved: liveness only, already counted. + } + Ok(None) => break, + Err(()) => return SessionResult { outcome: Outcome::PeerError, linked, peer: peer.clone() }, + } + } + if buf.len() > HEADER + MAX_PAYLOAD { + return SessionResult { outcome: Outcome::PeerError, linked, peer: peer.clone() }; + } + } + Err(_) => sock_dead = true, + } + } + } +} + +/// Pop one frame's payload from `buf` (header already validated by the caller +/// reading `buf[3]`). Returns the payload bytes so the type stays with the +/// caller. `Err(())` on bad magic/version. +fn parse_next(buf: &mut Vec) -> Result>, ()> { + if buf.len() < HEADER { + return Ok(None); + } + if buf[0] != MAGIC0 || buf[1] != MAGIC1 || buf[2] != VERSION { + return Err(()); + } + let len = ((buf[4] as usize) << 8) | buf[5] as usize; + if buf.len() < HEADER + len { + return Ok(None); + } + let payload = buf[HEADER..HEADER + len].to_vec(); + buf.drain(..HEADER + len); + Ok(Some(payload)) +} + +/// Wait out a backoff, but wake early for shutdown. `true` = shut down now. +fn sleep_or_shutdown(ms: i64, sfd: &SignalFd) -> bool { + let clamp = ms.clamp(0, u16::MAX as i64) as u16; + loop { + let mut pfds = [PollFd::new(sfd.as_fd(), PollFlags::POLLIN)]; + match poll(&mut pfds, PollTimeout::from(clamp)) { + Ok(0) => return false, // backoff elapsed + Ok(_) => return true, // a shutdown signal arrived + Err(Errno::EINTR) => continue, + Err(e) => panic!("poll: {e}"), + } + } +} + +// --------------------------------------------------------------------------- +// Roles +// --------------------------------------------------------------------------- +fn run_listen(o: &Opts, sfd: &SignalFd) -> i32 { + let listener = match TcpListener::bind(&o.listen_addr) { + Ok(l) => l, + Err(e) => { + eprintln!("chatterd: error: listen: {e}"); + return 1; + } + }; + let bound = listener.local_addr().expect("local_addr"); + println!("chatterd: {} listening on {}", o.name, bound); + + loop { + let ready = { + let mut pfds = [ + PollFd::new(sfd.as_fd(), PollFlags::POLLIN), + PollFd::new(listener.as_fd(), PollFlags::POLLIN), + ]; + match poll(&mut pfds, PollTimeout::NONE) { + Ok(_) => {} + Err(Errno::EINTR) => continue, + Err(e) => panic!("poll: {e}"), + } + let ev = |idx: usize| pfds[idx].revents().map(|r| r.intersects(PollFlags::POLLIN)).unwrap_or(false); + (ev(0), ev(1)) + }; + if ready.0 { + println!("chatterd: {} shutting down", o.name); + return 0; + } + if ready.1 { + let stream = match listener.accept() { + Ok((s, _)) => s, + Err(_) => continue, + }; + let res = run_session(o, stream, sfd, "peer".to_string()); + match res.outcome { + Outcome::Shutdown => { + println!("chatterd: {} shutting down", o.name); + return 0; + } + Outcome::TimedOut if res.linked => { + println!("chatterd: peer {} timed out", res.peer); + } + _ => {} + } + // Loop back to accept: a reconnecting peer is welcomed again. + } + } +} + +fn run_connect(o: &Opts, sfd: &SignalFd) -> i32 { + println!("chatterd: {} connecting to {} at {}", o.name, o.peer_name, o.peer_addr); + let seed = o.seed.unwrap_or_else(|| { + clock_gettime(ClockId::CLOCK_MONOTONIC).map(|t| t.tv_nsec() as u64).unwrap_or(1) + }); + let mut rng = Rng::new(seed); + let mut backoff = o.backoff_ms; + + loop { + if let Ok(stream) = TcpStream::connect(&o.peer_addr) { + backoff = o.backoff_ms; // a fresh TCP connection resets backoff + let res = run_session(o, stream, sfd, o.peer_name.clone()); + match res.outcome { + Outcome::Shutdown => { + println!("chatterd: {} shutting down", o.name); + return 0; + } + Outcome::TimedOut => { + println!("chatterd: peer {} timed out", o.peer_name); + } + Outcome::PeerError => {} + } + } + // Jittered exponential backoff (equal jitter: half fixed, half random). + let half = backoff / 2; + let delay = half + rng.below(half + 1); + println!("chatterd: reconnecting to {} in {}ms", o.peer_name, delay); + if sleep_or_shutdown(delay, sfd) { + println!("chatterd: {} shutting down", o.name); + return 0; + } + backoff = (backoff * 2).min(o.max_backoff_ms); + } +} + +fn run_clockprobe() -> i32 { + let res = clock_getres(ClockId::CLOCK_MONOTONIC).expect("clock_getres"); + let res_ns = res.tv_sec() as i64 * 1_000_000_000 + res.tv_nsec() as i64; + + let t0 = Instant::now(); + let req = TimeSpec::nanoseconds(1_000_000); // 1 ms + while clock_nanosleep(ClockId::CLOCK_MONOTONIC, ClockNanosleepFlags::empty(), &req) + == Err(Errno::EINTR) + {} + let sleep_us = t0.elapsed().as_micros(); + + let tfd = TimerFd::new(TfdClockId::CLOCK_MONOTONIC, TimerFlags::empty()).expect("timerfd"); + tfd.set(Expiration::OneShot(TimeSpec::nanoseconds(1_000_000)), TimerSetTimeFlags::empty()) + .expect("timerfd_settime"); + let t1 = Instant::now(); + tfd.wait().expect("timerfd wait"); + let tfd_us = t1.elapsed().as_micros(); + + println!( + "clockprobe: CLOCK_MONOTONIC res={res_ns}ns nanosleep(1ms) actual={sleep_us}us \ + timerfd(1ms) actual={tfd_us}us" + ); + 0 +} + +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + if args.is_empty() { + usage(); + } + let sub = args[0].as_str(); + let rest = &args[1..]; + + if sub == "clockprobe" { + std::process::exit(run_clockprobe()); + } + if sub != "listen" && sub != "connect" { + usage(); + } + + let o = parse_common(rest).unwrap_or_else(|e| { + eprintln!("chatterd: error: {e}"); + usage(); + }); + + // Block SIGINT/SIGTERM and consume them via a signalfd doorbell. + let sfd = install_shutdown(); + + let code = if sub == "listen" { + if o.listen_addr.is_empty() { + eprintln!("chatterd: error: listen needs --addr"); + usage(); + } + run_listen(&o, &sfd) + } else { + if o.peer_addr.is_empty() { + eprintln!("chatterd: error: connect needs --peer"); + usage(); + } + run_connect(&o, &sfd) + }; + std::process::exit(code); +} diff --git a/examples/24-time-timers-deadlines/verify.lua b/examples/24-time-timers-deadlines/verify.lua new file mode 100644 index 0000000..54688dd --- /dev/null +++ b/examples/24-time-timers-deadlines/verify.lua @@ -0,0 +1,147 @@ +-- Verify chatterd v3 (chapter 24: time, timers, deadlines). Asserts OBSERVABLE +-- behavior, not exit-0: +-- * clockprobe prints a monotonic-clock line with three parseable fields +-- (res ns, nanosleep 1ms us, timerfd 1ms us) and names CLOCK_MONOTONIC, +-- never CLOCK_REALTIME (deadlines must be measured on a steady clock). +-- * two peers link over loopback and deliver a chat message each way. +-- * killing the listener makes the connector declare "peer A timed out" +-- (the deadline firing on silence) then reconnect with a jittered, +-- growing backoff ("reconnecting to A in ms"). +-- * a SIGTERM shuts the connector down cleanly (exit 0). +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set. Plain Lua 5.1+ (works under luajit). + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local bins = { + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +} +local bin = bins[lang] + +local function expect_true(cond, label) + checks.expect_match(cond and "yes" or "no", "yes", label) +end + +-- The orchestrator builds before verifying; fail loudly here if it did not. +local r = checks.run("test -x " .. bin) +checks.expect_exit(r, 0, lang .. ": built binary " .. bin .. " exists") + +-- --------------------------------------------------------------------------- +-- 1. Usage / arg handling. +-- --------------------------------------------------------------------------- +r = checks.run("./demo.sh " .. lang .. " run") +checks.expect_exit(r, 2, lang .. ": bare invocation exits 2") +checks.expect_match(r.out, "usage: chatterd", lang .. ": prints usage") + +r = checks.run("./" .. bin .. " bogus-subcommand") +checks.expect_exit(r, 2, lang .. ": unknown subcommand exits 2") + +-- --------------------------------------------------------------------------- +-- 2. clockprobe: three parseable fields on a monotonic-clock line. +-- --------------------------------------------------------------------------- +r = checks.run("./demo.sh " .. lang .. " run clockprobe") +checks.expect_exit(r, 0, lang .. ": clockprobe exits 0") +checks.expect_match(r.out, "clockprobe: CLOCK_MONOTONIC res=%d+ns", + lang .. ": clockprobe reports CLOCK_MONOTONIC resolution in ns") +checks.expect_match(r.out, "nanosleep%(1ms%) actual=%d+us", + lang .. ": clockprobe reports nanosleep(1ms) actual in us") +checks.expect_match(r.out, "timerfd%(1ms%) actual=%d+us", + lang .. ": clockprobe reports timerfd(1ms) actual in us") +expect_true(not r.out:find("CLOCK_REALTIME"), + lang .. ": clockprobe uses a monotonic clock, not the wall clock") +-- The two 1ms sleepers should each land in a sane [1ms, 100ms] window. +local ns_us = tonumber(r.out:match("nanosleep%(1ms%) actual=(%d+)us")) +local tfd_us = tonumber(r.out:match("timerfd%(1ms%) actual=(%d+)us")) +expect_true(ns_us ~= nil and ns_us >= 1000 and ns_us <= 100000, + lang .. ": nanosleep(1ms) actually slept ~1ms (got " .. tostring(ns_us) .. "us)") +expect_true(tfd_us ~= nil and tfd_us >= 1000 and tfd_us <= 100000, + lang .. ": timerfd(1ms) actually waited ~1ms (got " .. tostring(tfd_us) .. "us)") + +-- --------------------------------------------------------------------------- +-- 3. Heartbeat + deadline + reconnect: two peers, kill one, drive the other. +-- --------------------------------------------------------------------------- +r = checks.run(string.format([[ + set -u + bin=%s + work=$(mktemp -d) + alog=$work/a.log; blog=$work/b.log + "./$bin" listen --name A --addr 127.0.0.1:0 \ + --heartbeat-ms 80 --timeout-ms 320 --message "hi-from-A" >"$alog" 2>&1 & + apid=$! + port="" + for i in $(seq 1 100); do + port=$(sed -n 's/.*listening on 127.0.0.1:\([0-9]*\).*/\1/p' "$alog") + [ -n "$port" ] && break + sleep 0.05 + done + echo "port=$port" + "./$bin" connect --name B --peer "A@127.0.0.1:$port" \ + --heartbeat-ms 80 --timeout-ms 320 --backoff-ms 120 --max-backoff-ms 2000 \ + --message "hi-from-B" --seed 42 >"$blog" 2>&1 & + bpid=$! + trap 'kill -9 $apid $bpid 2>/dev/null; rm -rf "$work"' EXIT + # wait for both directions of the chat to land + for i in $(seq 1 100); do + grep -q "message from A" "$blog" 2>/dev/null \ + && grep -q "message from B" "$alog" 2>/dev/null && break + sleep 0.05 + done + echo "=== A LOG (link phase) ==="; cat "$alog" + echo "=== B LOG (link phase) ==="; cat "$blog" + # kill the listener; the connector must notice the silence and reconnect + kill -9 $apid 2>/dev/null + for i in $(seq 1 160); do + n=$(grep -c "reconnecting to A in" "$blog" 2>/dev/null || echo 0) + [ "$n" -ge 3 ] && break + sleep 0.05 + done + # clean shutdown of the connector + kill -TERM $bpid 2>/dev/null + wait $bpid; echo "b_exit=$?" + echo "=== B LOG (final) ==="; cat "$blog" +]], bin)) + +checks.expect_match(r.out, "port=%d+", lang .. ": listener bound an ephemeral port") +checks.expect_match(r.out, "chatterd: A linked with B", lang .. ": listener links with connector") +checks.expect_match(r.out, "chatterd: B linked with A", lang .. ": connector links with listener") +checks.expect_match(r.out, "chatterd: A message from B: hi%-from%-B", + lang .. ": message delivered connector -> listener") +checks.expect_match(r.out, "chatterd: B message from A: hi%-from%-A", + lang .. ": message delivered listener -> connector") +checks.expect_match(r.out, "chatterd: peer A timed out", + lang .. ": connector declares the listener timed out after silence") + +-- Count and inspect the jittered reconnect lines. +local delays = {} +for ms in r.out:gmatch("chatterd: reconnecting to A in (%d+)ms") do + delays[#delays + 1] = tonumber(ms) +end +expect_true(#delays >= 2, + lang .. ": at least two reconnect attempts (got " .. #delays .. ")") +-- Equal-jitter backoff from base 120ms: first delay in [60,120], and the +-- window grows (later attempts can exceed the first base ceiling). +expect_true(#delays >= 1 and delays[1] >= 60 and delays[1] <= 120, + lang .. ": first backoff is jittered within [60,120]ms (got " .. + tostring(delays[1]) .. ")") +local grew = false +for _, d in ipairs(delays) do + if d > 120 then grew = true end +end +expect_true(grew, + lang .. ": backoff grows past the base ceiling across attempts") + +checks.expect_match(r.out, "b_exit=0", lang .. ": SIGTERM shuts the connector down with exit 0") +checks.expect_match(r.out, "chatterd: B shutting down", + lang .. ": connector prints its clean-shutdown line") + +checks.finish() diff --git a/examples/25-shared-state-and-the-futex/README.md b/examples/25-shared-state-and-the-futex/README.md new file mode 100644 index 0000000..a369b67 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/README.md @@ -0,0 +1,152 @@ +# 25-shared-state-and-the-futex + +Chapter 25's example: `workq`, a bounded **MPMC job queue + worker pool** +implemented three times — C++23, Go, and Rust — with each language's native +shared-state toolkit, but one observable contract so a single `verify.lua` +covers all three. The queue's blocking is exactly the mutex/condvar dance the +kernel implements with **futexes**; `strace`-ing any of these binaries shows +`futex(2)` calls under load. + +## The program + +``` +workq --producers P --consumers C --items N [--buggy] [--seed S] [--cap K] +``` + +`P` producer threads push `N` items **total** into a bounded blocking queue of +capacity `K` (default 256); `C` consumer threads pop items and fold each item's +payload into a running checksum. When every item has been produced and drained, +the program prints one line: + +``` +workq: produced= consumed= checksum= ms= +``` + +- `produced` / `consumed` are exact counts (both equal `N` on the correct path). +- `checksum` is a 16-digit zero-padded lowercase hex `u64`. +- `ms` is wall-clock milliseconds (timing — the only non-deterministic field). + +Defaults: `--seed 0x0123456789ABCDEF`, `--cap 256`. `--producers`, +`--consumers`, and `--items` are required. + +**Errors** (all to stderr, exit 2, always followed by the usage line): +missing a required flag → usage only; `--foo` → `workq: unknown flag: --foo`; +a flag with no value → `workq: --producers needs a value`; a non-integer value +→ `workq: not an integer: x`; `P<1`, `C<1`, `N<0`, or `K<1` → a bounds message. + +## The value contract (identical across languages and runs) + +Each item index `i ∈ [0, N)` has a payload that is a **pure function** of +`(seed, i)` — a splitmix64 finalizer over `seed + (i+1)·GOLDEN`, all in wrapping +`u64` arithmetic: + +``` +GOLDEN = 0x9E3779B97F4A7C15 MIX1 = 0xBF58476D1CE4E5B9 MIX2 = 0x94D049BB133111EB + +payload(seed, i): + x = seed + (i+1)*GOLDEN # wrapping u64 + x = (x ^ (x >> 30)) * MIX1 # wrapping + x = (x ^ (x >> 27)) * MIX2 # wrapping + x = x ^ (x >> 31) + return x +``` + +Producer `p` owns the indices `{ i : i mod P == p }`, so the **set** of items is +`{0..N-1}` for any `P`. The checksum folds payloads with **XOR**, which is +commutative and associative, so the result is independent of how items are +split among producers or the order consumers drain them. For a fixed +`(seed, N)` the checksum is therefore one fixed constant — byte-identical in +C++, Go, and Rust: + +| seed | N | checksum | +|---|---|---| +| `0x0123456789ABCDEF` (default) | 8 | `697cd06b54533772` | +| `0x0123456789ABCDEF` (default) | 100000 | `42b3746c6cee7465` | +| `42` | 50000 | `fceb5608a4b73dc0` | +| any | 0 | `0000000000000000` | + +Real output (default seed, all three binaries print the same line): + +``` +$ ./demo.sh cpp run --producers 4 --consumers 4 --items 100000 +workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms=49 +$ ./demo.sh go run --producers 1 --consumers 8 --items 100000 --cap 4 +workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms=102 +$ ./demo.sh rust run --producers 8 --consumers 3 --items 100000 --cap 1024 +workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms=62 +``` + +## Three shared-state shapes, one contract + +| | the bounded queue | the checksum accumulator | +|---|---|---| +| **C++23** | `std::mutex` + two `std::condition_variable`s (`not_full` / `not_empty`) guarding a `std::deque`; `std::jthread` producers/consumers, `std::atomic` produced counter | per-consumer local fold, XOR-combined after join | +| **Go** | a buffered `chan uint64` (capacity `K`) as the blocking queue; a closer goroutine `sync.WaitGroup.Wait()`s the producers then `close()`s the channel, ending every consumer's `range` | per-consumer local fold, XOR-combined after the consumer `WaitGroup` | +| **Rust** | `Mutex>` + two `Condvar`s inside `std::thread::scope`; no `Arc` — the scope lets the threads borrow the queue directly | per-consumer `(count, sum)` returned from `ScopedJoinHandle::join`, XOR-combined | + +None of these needs an atomic on the hot path except the produced counter — the +mutex/condvar (or channel) already establishes the happens-before edges. + +## The `--buggy` demo (a real data race — chapter material, not verified) + +`--buggy` keeps the same correct, synchronized queue but routes **every** +consumer into a single **unsynchronized** shared counter + checksum. That is a +genuine data race; `verify.lua` never runs it, but the chapter does: + +- **Go** builds with `-race` (see `go/demo.sh`), so `--buggy` prints + `WARNING: DATA RACE` and ends with `Found N data race(s)` and a nonzero exit. + The corrupted `consumed` comes out below `N` (lost updates), e.g. `94730`. +- **C++** has a ThreadSanitizer preset: `cmake --preset tsan && cmake --build + --preset tsan`, then `./build/tsan/app ... --buggy` reports + `WARNING: ThreadSanitizer: data race` at the two shared-write lines. +- **Rust** won't let you write this in safe code at all — the buggy path needs + an `unsafe` `*mut` shared across threads. Run it and the borrow checker's + point is made empirically: `consumed` and `checksum` come out different every + time (`consumed=97282`, `95773`, …), never the correct `100000` / + `42b3746c6cee7465`. + +## Layout + +``` +25-shared-state-and-the-futex/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check (correct path only) +├── cpp/ # CMake presets (release / tsan / asan); std::mutex+condvar, jthread +├── go/ # go build -race (toolchain go1.26.5); buffered channel queue +└── rust/ # cargo build (rustup pin 1.97.1); Mutex+Condvar, thread::scope +``` + +## The demo contract + +Every language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of + running locally. + +The top-level `demo.sh` dispatches: `./demo.sh cpp run …` execs `cpp/demo.sh +run …`; bare `./demo.sh` (or `all`) builds and runs all three. + +## Verification + +`verify.lua` runs from this directory with `LSP_LANG` set to one language at a +time and asserts, per language (25 checks each): + +1. the error contract — no args / unknown flag / missing required flag / bad + integer / value-less flag all print the usage line and exit 2; +2. a fixed run (`P=4 C=4 N=100000`, default seed) reports + `produced=100000 consumed=100000` and the shared checksum constant + `42b3746c6cee7465`; +3. determinism — three more shapes with different `P`, `C`, and `--cap` over the + same `(N, seed)` all yield that same constant with `produced==consumed==N`; +4. seed `42` (`N=50000`) yields its own shared constant `fceb5608a4b73dc0`, and + `N=0` drains to `produced=0 consumed=0 checksum=0000000000000000`. + +The `--buggy` race is a sanitizer/`-race` chapter demo and is deliberately +outside the passing verify. + +Verified on the Fedora 44 reference host (kernel 7.1.3-200.fc44, GCC 16.1.1, +go1.26.5, rustc 1.97.1): all three languages PASS 25/25, Go under `-race`. diff --git a/examples/25-shared-state-and-the-futex/cpp/.clang-format b/examples/25-shared-state-and-the-futex/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/25-shared-state-and-the-futex/cpp/.clang-tidy b/examples/25-shared-state-and-the-futex/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/25-shared-state-and-the-futex/cpp/CMakeLists.txt b/examples/25-shared-state-and-the-futex/cpp/CMakeLists.txt new file mode 100644 index 0000000..86f35c7 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/CMakeLists.txt @@ -0,0 +1,11 @@ +cmake_minimum_required(VERSION 3.25) +project(workq CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app src/main.cpp) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/25-shared-state-and-the-futex/cpp/CMakePresets.json b/examples/25-shared-state-and-the-futex/cpp/CMakePresets.json new file mode 100644 index 0000000..e398922 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/CMakePresets.json @@ -0,0 +1,73 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "tsan", + "inherits": "_base", + "displayName": "ThreadSanitizer instrumented build (for --buggy demo)", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=thread", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=thread" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "tsan", "configurePreset": "tsan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/25-shared-state-and-the-futex/cpp/demo.sh b/examples/25-shared-state-and-the-futex/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/25-shared-state-and-the-futex/cpp/src/main.cpp b/examples/25-shared-state-and-the-futex/cpp/src/main.cpp new file mode 100644 index 0000000..8d1ee77 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/cpp/src/main.cpp @@ -0,0 +1,255 @@ +// workq — a bounded MPMC job queue and worker pool (C++23). +// +// P producers push N items total into a bounded blocking queue; C consumers +// pop items and fold each item's payload into a running checksum. The correct +// path synchronizes with std::mutex + std::condition_variable (the futex-backed +// primitives this chapter is about); --buggy swaps the per-consumer local +// accumulators for a single UNSYNCHRONIZED shared counter + checksum so +// ThreadSanitizer (build/tsan) has a real data race to report. +// +// The item set {0..N-1} is fixed for a given (seed, N), each item's payload is +// a pure function of its index, and the checksum folds them with XOR — which is +// commutative — so the correct run's checksum is deterministic regardless of P, +// C, or the order consumers happen to drain the queue. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr std::uint64_t kGolden = 0x9E3779B97F4A7C15ull; +constexpr std::uint64_t kMix1 = 0xBF58476D1CE4E5B9ull; +constexpr std::uint64_t kMix2 = 0x94D049BB133111EBull; +constexpr std::uint64_t kSeedDefault = 0x0123456789ABCDEFull; + +// Pure per-index payload: a splitmix64 finalizer over seed + (i+1)*golden. +// uint64_t arithmetic wraps by definition, so this is identical in Go and Rust. +std::uint64_t Payload(std::uint64_t seed, std::uint64_t i) { + std::uint64_t x = seed + (i + 1) * kGolden; + x = (x ^ (x >> 30)) * kMix1; + x = (x ^ (x >> 27)) * kMix2; + x = x ^ (x >> 31); + return x; +} + +struct Config { + long producers = 0; + long consumers = 0; + long items = 0; // N, the total across all producers + std::uint64_t seed = kSeedDefault; + long cap = 256; + bool buggy = false; +}; + +struct ParseError { + std::string message; // printed after the "workq: " prefix; empty => usage only +}; + +constexpr std::string_view kUsage = + "usage: workq --producers P --consumers C --items N [--buggy] " + "[--seed S] [--cap K]"; + +std::expected ParseLong(std::string_view s) { + long v = 0; + auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || ptr != s.data() + s.size()) { + return std::unexpected(std::string("not an integer: ") + std::string(s)); + } + return v; +} + +std::expected ParseSeed(std::string_view s) { + int base = 10; + std::string_view body = s; + if (s.size() > 2 && (s.substr(0, 2) == "0x" || s.substr(0, 2) == "0X")) { + base = 16; + body = s.substr(2); + } + std::uint64_t v = 0; + auto [ptr, ec] = std::from_chars(body.data(), body.data() + body.size(), v, base); + if (ec != std::errc{} || ptr != body.data() + body.size()) { + return std::unexpected(std::string("not an integer: ") + std::string(s)); + } + return v; +} + +std::expected Parse(int argc, char** argv) { + Config c; + bool has_p = false, has_c = false, has_n = false; + for (int i = 1; i < argc; ++i) { + std::string_view a = argv[i]; + auto value = [&](std::string_view* out) -> bool { + if (i + 1 >= argc) return false; + *out = argv[++i]; + return true; + }; + if (a == "--buggy") { + c.buggy = true; + } else if (a == "--producers" || a == "--consumers" || a == "--items" || + a == "--cap") { + std::string_view v; + if (!value(&v)) return std::unexpected(ParseError{std::string(a) + " needs a value"}); + auto n = ParseLong(v); + if (!n) return std::unexpected(ParseError{n.error()}); + if (a == "--producers") { c.producers = *n; has_p = true; } + else if (a == "--consumers") { c.consumers = *n; has_c = true; } + else if (a == "--items") { c.items = *n; has_n = true; } + else { c.cap = *n; } + } else if (a == "--seed") { + std::string_view v; + if (!value(&v)) return std::unexpected(ParseError{std::string(a) + " needs a value"}); + auto s = ParseSeed(v); + if (!s) return std::unexpected(ParseError{s.error()}); + c.seed = *s; + } else { + return std::unexpected(ParseError{"unknown flag: " + std::string(a)}); + } + } + if (!has_p || !has_c || !has_n) { + return std::unexpected(ParseError{""}); // usage only + } + if (c.producers < 1 || c.consumers < 1) { + return std::unexpected(ParseError{"--producers and --consumers must be >= 1"}); + } + if (c.items < 0) { + return std::unexpected(ParseError{"--items must be >= 0"}); + } + if (c.cap < 1) { + return std::unexpected(ParseError{"--cap must be >= 1"}); + } + return c; +} + +// A bounded blocking MPMC queue: one mutex, two condition variables. Producers +// block on not_full_, consumers block on not_empty_; Close() wakes drained +// consumers. This is the shared state whose access the futex serializes. +class BoundedQueue { + public: + explicit BoundedQueue(std::size_t cap) : cap_(cap) {} + + void Push(std::uint64_t v) { + std::unique_lock lock(mu_); + not_full_.wait(lock, [&] { return q_.size() < cap_; }); + q_.push_back(v); + not_empty_.notify_one(); + } + + // Pop one item; returns false once the queue is closed and drained. + bool Pop(std::uint64_t& out) { + std::unique_lock lock(mu_); + not_empty_.wait(lock, [&] { return !q_.empty() || closed_; }); + if (q_.empty()) return false; // closed_ is true here + out = q_.front(); + q_.pop_front(); + not_full_.notify_one(); + return true; + } + + void Close() { + { + std::unique_lock lock(mu_); + closed_ = true; + } + not_empty_.notify_all(); + } + + private: + std::mutex mu_; + std::condition_variable not_full_; + std::condition_variable not_empty_; + std::deque q_; + std::size_t cap_; + bool closed_ = false; +}; + +int Run(const Config& cfg) { + BoundedQueue queue(static_cast(cfg.cap)); + std::atomic produced{0}; + + // Correct path: each consumer folds into its own slot; no shared writes. + std::vector local_counts(static_cast(cfg.consumers), 0); + std::vector local_sums(static_cast(cfg.consumers), 0); + + // Buggy path: one shared counter + checksum, mutated with no lock at all. + long shared_consumed = 0; + std::uint64_t shared_checksum = 0; + + const auto t0 = std::chrono::steady_clock::now(); + + std::vector producers; + producers.reserve(static_cast(cfg.producers)); + for (long p = 0; p < cfg.producers; ++p) { + producers.emplace_back([&, p] { + for (long i = p; i < cfg.items; i += cfg.producers) { + queue.Push(Payload(cfg.seed, static_cast(i))); + produced.fetch_add(1, std::memory_order_relaxed); + } + }); + } + + std::vector consumers; + consumers.reserve(static_cast(cfg.consumers)); + for (long idx = 0; idx < cfg.consumers; ++idx) { + consumers.emplace_back([&, idx] { + std::uint64_t v; + while (queue.Pop(v)) { + if (cfg.buggy) { + shared_consumed += 1; // DATA RACE: unsynchronized shared counter + shared_checksum ^= v; // DATA RACE: unsynchronized shared checksum + } else { + local_counts[static_cast(idx)] += 1; + local_sums[static_cast(idx)] ^= v; + } + } + }); + } + + for (auto& t : producers) t.join(); + queue.Close(); + for (auto& t : consumers) t.join(); + + const auto t1 = std::chrono::steady_clock::now(); + const long ms = + std::chrono::duration_cast(t1 - t0).count(); + + long consumed = shared_consumed; + std::uint64_t checksum = shared_checksum; + if (!cfg.buggy) { + consumed = 0; + checksum = 0; + for (long i = 0; i < cfg.consumers; ++i) { + consumed += local_counts[static_cast(i)]; + checksum ^= local_sums[static_cast(i)]; + } + } + + std::println("workq: produced={} consumed={} checksum={:016x} ms={}", + produced.load(), consumed, checksum, ms); + return 0; +} + +} // namespace + +int main(int argc, char** argv) { + auto cfg = Parse(argc, argv); + if (!cfg) { + if (!cfg.error().message.empty()) { + std::println(stderr, "workq: {}", cfg.error().message); + } + std::println(stderr, "{}", kUsage); + return 2; + } + return Run(*cfg); +} diff --git a/examples/25-shared-state-and-the-futex/demo.sh b/examples/25-shared-state-and-the-futex/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/25-shared-state-and-the-futex/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/25-shared-state-and-the-futex/go/demo.sh b/examples/25-shared-state-and-the-futex/go/demo.sh new file mode 100755 index 0000000..46d4d9f --- /dev/null +++ b/examples/25-shared-state-and-the-futex/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -race -o "$BIN" . # race detector on: this is the shared-state chapter +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/25-shared-state-and-the-futex/go/go.mod b/examples/25-shared-state-and-the-futex/go/go.mod new file mode 100644 index 0000000..13a8925 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/25-shared-state-and-the-futex/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/25-shared-state-and-the-futex/go/main.go b/examples/25-shared-state-and-the-futex/go/main.go new file mode 100644 index 0000000..60396ca --- /dev/null +++ b/examples/25-shared-state-and-the-futex/go/main.go @@ -0,0 +1,198 @@ +// workq — a bounded MPMC job queue and worker pool (Go 1.26). +// +// P producers push N items total into a bounded blocking queue (a buffered +// channel — the CSP-flavored blocking queue), C consumers pop and fold each +// item's payload into a checksum. A closer goroutine waits on the producers' +// WaitGroup and closes the channel, which ends every consumer's range loop. +// +// The correct path gives each consumer its own accumulator (no shared writes); +// --buggy funnels every consumer into ONE unsynchronized shared counter + +// checksum, which `go build -race` (see demo.sh) reports as a data race. +// +// Item payloads are a pure function of index and folded with XOR, so the +// correct checksum is deterministic for a given (seed, N) regardless of P/C. +package main + +import ( + "fmt" + "os" + "strconv" + "sync" + "sync/atomic" + "time" +) + +const ( + golden = 0x9E3779B97F4A7C15 + mix1 = 0xBF58476D1CE4E5B9 + mix2 = 0x94D049BB133111EB + seedDefault = 0x0123456789ABCDEF + usage = "usage: workq --producers P --consumers C --items N [--buggy] [--seed S] [--cap K]" +) + +// payload is the splitmix64 finalizer over seed + (i+1)*golden. uint64 wraps, +// matching the C++ and Rust implementations byte for byte. +func payload(seed, i uint64) uint64 { + x := seed + (i+1)*golden + x = (x ^ (x >> 30)) * mix1 + x = (x ^ (x >> 27)) * mix2 + x = x ^ (x >> 31) + return x +} + +type config struct { + producers int + consumers int + items int + cap int + seed uint64 + buggy bool +} + +func parseSeed(s string) (uint64, error) { + return strconv.ParseUint(s, 0, 64) // base 0: accepts 0x-prefixed hex +} + +// parseArgs returns (cfg, msg, ok). On failure ok is false; a non-empty msg is +// printed after the "workq: " prefix, and the usage line always follows. +func parseArgs(args []string) (config, string, bool) { + c := config{seed: seedDefault, cap: 256} + var hasP, hasC, hasN bool + i := 0 + need := func() (string, bool) { + if i+1 >= len(args) { + return "", false + } + i++ + return args[i], true + } + for ; i < len(args); i++ { + a := args[i] + switch a { + case "--buggy": + c.buggy = true + case "--producers", "--consumers", "--items", "--cap": + v, ok := need() + if !ok { + return c, a + " needs a value", false + } + n, err := strconv.Atoi(v) + if err != nil { + return c, "not an integer: " + v, false + } + switch a { + case "--producers": + c.producers, hasP = n, true + case "--consumers": + c.consumers, hasC = n, true + case "--items": + c.items, hasN = n, true + case "--cap": + c.cap = n + } + case "--seed": + v, ok := need() + if !ok { + return c, a + " needs a value", false + } + n, err := parseSeed(v) + if err != nil { + return c, "not an integer: " + v, false + } + c.seed = n + default: + return c, "unknown flag: " + a, false + } + } + if !hasP || !hasC || !hasN { + return c, "", false + } + if c.producers < 1 || c.consumers < 1 { + return c, "--producers and --consumers must be >= 1", false + } + if c.items < 0 { + return c, "--items must be >= 0", false + } + if c.cap < 1 { + return c, "--cap must be >= 1", false + } + return c, "", true +} + +func run(c config) { + queue := make(chan uint64, c.cap) // buffered channel == bounded blocking queue + var produced int64 + + // Correct path: one accumulator slot per consumer, no shared writes. + localCounts := make([]int, c.consumers) + localSums := make([]uint64, c.consumers) + + // Buggy path: a single shared counter + checksum, mutated with no lock. + var sharedConsumed int + var sharedChecksum uint64 + + start := time.Now() + + var producers sync.WaitGroup + for p := 0; p < c.producers; p++ { + producers.Add(1) + go func(p int) { + defer producers.Done() + for i := p; i < c.items; i += c.producers { + queue <- payload(c.seed, uint64(i)) + atomic.AddInt64(&produced, 1) + } + }(p) + } + // Closer: end the consumers' range once every item has been pushed. + go func() { + producers.Wait() + close(queue) + }() + + var consumers sync.WaitGroup + for idx := 0; idx < c.consumers; idx++ { + consumers.Add(1) + go func(idx int) { + defer consumers.Done() + for v := range queue { + if c.buggy { + sharedConsumed++ // DATA RACE: unsynchronized shared counter + sharedChecksum ^= v // DATA RACE: unsynchronized shared checksum + } else { + localCounts[idx]++ + localSums[idx] ^= v + } + } + }(idx) + } + consumers.Wait() + + ms := time.Since(start).Milliseconds() + + consumed := sharedConsumed + checksum := sharedChecksum + if !c.buggy { + consumed = 0 + checksum = 0 + for i := 0; i < c.consumers; i++ { + consumed += localCounts[i] + checksum ^= localSums[i] + } + } + + fmt.Printf("workq: produced=%d consumed=%d checksum=%016x ms=%d\n", + atomic.LoadInt64(&produced), consumed, checksum, ms) +} + +func main() { + c, msg, ok := parseArgs(os.Args[1:]) + if !ok { + if msg != "" { + fmt.Fprintf(os.Stderr, "workq: %s\n", msg) + } + fmt.Fprintln(os.Stderr, usage) + os.Exit(2) + } + run(c) +} diff --git a/examples/25-shared-state-and-the-futex/rust/Cargo.lock b/examples/25-shared-state-and-the-futex/rust/Cargo.lock new file mode 100644 index 0000000..1978958 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "workq" +version = "0.1.0" diff --git a/examples/25-shared-state-and-the-futex/rust/Cargo.toml b/examples/25-shared-state-and-the-futex/rust/Cargo.toml new file mode 100644 index 0000000..6e9dae7 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/rust/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "workq" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] diff --git a/examples/25-shared-state-and-the-futex/rust/demo.sh b/examples/25-shared-state-and-the-futex/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/25-shared-state-and-the-futex/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/25-shared-state-and-the-futex/rust/rust-toolchain.toml b/examples/25-shared-state-and-the-futex/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/25-shared-state-and-the-futex/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/25-shared-state-and-the-futex/rust/src/main.rs b/examples/25-shared-state-and-the-futex/rust/src/main.rs new file mode 100644 index 0000000..1063b58 --- /dev/null +++ b/examples/25-shared-state-and-the-futex/rust/src/main.rs @@ -0,0 +1,287 @@ +// workq — a bounded MPMC job queue and worker pool (Rust 2024). +// +// P producers push N items total into a bounded blocking queue built from a +// Mutex plus two Condvars — the futex-backed pair this chapter is +// about. C consumers pop items and fold each payload into a checksum. +// +// The correct path gives every consumer its own accumulator and combines them +// after the scoped threads join; nothing shared is written without the lock. +// --buggy is the deliberate counterexample: safe Rust will NOT let two threads +// write one plain counter, so the buggy path reaches for `unsafe` and a raw +// pointer to force the data race — and then prints the corrupted result that +// the borrow checker exists to prevent. (No sanitizer needed to see it fail; +// run it a few times and consumed/checksum wander.) +// +// Payloads are a pure function of index folded with XOR, so the correct +// checksum is deterministic for a given (seed, N) regardless of P and C — and +// byte-identical to the C++ and Go builds. + +use std::collections::VecDeque; +use std::sync::atomic::{AtomicI64, Ordering}; +use std::sync::{Condvar, Mutex}; +use std::thread; +use std::time::Instant; + +const GOLDEN: u64 = 0x9E3779B97F4A7C15; +const MIX1: u64 = 0xBF58476D1CE4E5B9; +const MIX2: u64 = 0x94D049BB133111EB; +const SEED_DEFAULT: u64 = 0x0123456789ABCDEF; +const USAGE: &str = + "usage: workq --producers P --consumers C --items N [--buggy] [--seed S] [--cap K]"; + +/// splitmix64 finalizer over seed + (i+1)*golden, wrapping u64 throughout. +fn payload(seed: u64, i: u64) -> u64 { + let mut x = seed.wrapping_add(i.wrapping_add(1).wrapping_mul(GOLDEN)); + x = (x ^ (x >> 30)).wrapping_mul(MIX1); + x = (x ^ (x >> 27)).wrapping_mul(MIX2); + x ^ (x >> 31) +} + +struct Config { + producers: i64, + consumers: i64, + items: i64, + cap: usize, + seed: u64, + buggy: bool, +} + +fn parse_seed(s: &str) -> Option { + if let Some(hex) = s.strip_prefix("0x").or_else(|| s.strip_prefix("0X")) { + u64::from_str_radix(hex, 16).ok() + } else { + s.parse::().ok() + } +} + +/// Ok(cfg) or Err(msg): Err(None) => usage only, Err(Some(m)) => "workq: m" + usage. +fn parse(args: &[String]) -> Result> { + let (mut producers, mut consumers, mut items) = (None, None, None); + let mut cap: i64 = 256; + let mut seed = SEED_DEFAULT; + let mut buggy = false; + + let mut it = args.iter(); + while let Some(a) = it.next() { + match a.as_str() { + "--buggy" => buggy = true, + "--producers" | "--consumers" | "--items" | "--cap" => { + let v = it.next().ok_or_else(|| Some(format!("{a} needs a value")))?; + let n: i64 = v + .parse() + .map_err(|_| Some(format!("not an integer: {v}")))?; + match a.as_str() { + "--producers" => producers = Some(n), + "--consumers" => consumers = Some(n), + "--items" => items = Some(n), + _ => cap = n, + } + } + "--seed" => { + let v = it.next().ok_or_else(|| Some(format!("{a} needs a value")))?; + seed = parse_seed(v).ok_or_else(|| Some(format!("not an integer: {v}")))?; + } + other => return Err(Some(format!("unknown flag: {other}"))), + } + } + + let (producers, consumers, items) = match (producers, consumers, items) { + (Some(p), Some(c), Some(n)) => (p, c, n), + _ => return Err(None), + }; + if producers < 1 || consumers < 1 { + return Err(Some("--producers and --consumers must be >= 1".to_string())); + } + if items < 0 { + return Err(Some("--items must be >= 0".to_string())); + } + if cap < 1 { + return Err(Some("--cap must be >= 1".to_string())); + } + Ok(Config { + producers, + consumers, + items, + cap: cap as usize, + seed, + buggy, + }) +} + +struct Inner { + q: VecDeque, + closed: bool, +} + +/// A bounded blocking MPMC queue: one Mutex, two Condvars. Producers block on +/// `not_full`, consumers block on `not_empty`; `close` wakes drained consumers. +struct BoundedQueue { + inner: Mutex, + not_full: Condvar, + not_empty: Condvar, + cap: usize, +} + +impl BoundedQueue { + fn new(cap: usize) -> Self { + BoundedQueue { + inner: Mutex::new(Inner { + q: VecDeque::new(), + closed: false, + }), + not_full: Condvar::new(), + not_empty: Condvar::new(), + cap, + } + } + + fn push(&self, v: u64) { + let mut g = self.inner.lock().unwrap(); + while g.q.len() >= self.cap { + g = self.not_full.wait(g).unwrap(); + } + g.q.push_back(v); + drop(g); + self.not_empty.notify_one(); + } + + /// Pop one item; returns None once the queue is closed and drained. + fn pop(&self) -> Option { + let mut g = self.inner.lock().unwrap(); + loop { + if let Some(v) = g.q.pop_front() { + drop(g); + self.not_full.notify_one(); + return Some(v); + } + if g.closed { + return None; + } + g = self.not_empty.wait(g).unwrap(); + } + } + + fn close(&self) { + let mut g = self.inner.lock().unwrap(); + g.closed = true; + drop(g); + self.not_empty.notify_all(); + } +} + +/// A raw pointer the buggy path deliberately shares across threads. Safe Rust +/// forbids this; the `unsafe impl`s are what it takes to opt out of the very +/// protection this chapter demonstrates. +#[derive(Clone, Copy)] +struct Racy(*mut T); +unsafe impl Send for Racy {} +unsafe impl Sync for Racy {} + +fn produce(queue: &BoundedQueue, produced: &AtomicI64, cfg: &Config, p: i64) { + let mut i = p; + while i < cfg.items { + queue.push(payload(cfg.seed, i as u64)); + produced.fetch_add(1, Ordering::Relaxed); + i += cfg.producers; + } +} + +fn run(cfg: &Config) { + let queue = BoundedQueue::new(cfg.cap); + let produced = AtomicI64::new(0); + let mut shared_consumed: i64 = 0; + let mut shared_checksum: u64 = 0; + let start = Instant::now(); + + let queue = &queue; + let produced = &produced; + + let (consumed, checksum) = if cfg.buggy { + let cptr = Racy(&mut shared_consumed as *mut i64); + let kptr = Racy(&mut shared_checksum as *mut u64); + thread::scope(|s| { + let mut prod = Vec::new(); + for p in 0..cfg.producers { + prod.push(s.spawn(move || produce(queue, produced, cfg, p))); + } + let mut cons = Vec::new(); + for _ in 0..cfg.consumers { + cons.push(s.spawn(move || { + // Bind the whole wrappers so the closure captures the + // `Send` `Racy`, not the bare `*mut` field (2024 captures + // disjoint fields otherwise). + let (cptr, kptr) = (cptr, kptr); + while let Some(v) = queue.pop() { + // DATA RACE: two consumers write these with no lock. + unsafe { + *cptr.0 += 1; + *kptr.0 ^= v; + } + } + })); + } + for h in prod { + h.join().unwrap(); + } + queue.close(); + for h in cons { + h.join().unwrap(); + } + }); + (shared_consumed, shared_checksum) + } else { + thread::scope(|s| { + let mut prod = Vec::new(); + for p in 0..cfg.producers { + prod.push(s.spawn(move || produce(queue, produced, cfg, p))); + } + let mut cons = Vec::new(); + for _ in 0..cfg.consumers { + cons.push(s.spawn(move || { + let mut count: i64 = 0; + let mut sum: u64 = 0; + while let Some(v) = queue.pop() { + count += 1; + sum ^= v; + } + (count, sum) + })); + } + for h in prod { + h.join().unwrap(); + } + queue.close(); + let mut consumed = 0i64; + let mut checksum = 0u64; + for h in cons { + let (c, k) = h.join().unwrap(); + consumed += c; + checksum ^= k; + } + (consumed, checksum) + }) + }; + + let ms = start.elapsed().as_millis(); + println!( + "workq: produced={} consumed={} checksum={:016x} ms={}", + produced.load(Ordering::Relaxed), + consumed, + checksum, + ms + ); +} + +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + match parse(&args) { + Ok(cfg) => run(&cfg), + Err(msg) => { + if let Some(m) = msg { + eprintln!("workq: {m}"); + } + eprintln!("{USAGE}"); + std::process::exit(2); + } + } +} diff --git a/examples/25-shared-state-and-the-futex/verify.lua b/examples/25-shared-state-and-the-futex/verify.lua new file mode 100644 index 0000000..06d30df --- /dev/null +++ b/examples/25-shared-state-and-the-futex/verify.lua @@ -0,0 +1,118 @@ +-- Verify workq (25-shared-state-and-the-futex) for LSP_LANG. +-- +-- Asserts the CORRECT path's observable behavior, identical across cpp/go/rust +-- (the --buggy path is a sanitizer/race-detector chapter demo, deliberately +-- NOT exercised here): +-- 1. error contract: no args / unknown flag / missing required / bad integer +-- / missing value all print the usage line and exit 2 +-- 2. a fixed run (P=4 C=4 N=100000, default seed) prints +-- workq: produced=100000 consumed=100000 checksum=42b3746c6cee7465 ms= +-- the checksum is the SAME 64-bit constant for all three languages +-- 3. determinism: the checksum does not depend on P, C, or the queue cap — +-- four different shapes over the same (N, seed) all yield that constant, +-- and every run reports produced==consumed==N +-- 4. a different seed yields a different-but-shared constant, and N=0 yields +-- produced==consumed==0 with an all-zero checksum +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set; the interpreter may be lua or luajit. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- Precomputed cross-language constants (see README "Wire/value contract"). +local CK_DEFAULT = "42b3746c6cee7465" -- N=100000, default seed 0x0123456789ABCDEF +local CK_SEED42 = "fceb5608a4b73dc0" -- N=50000, seed 42 +local N = 100000 + +-- expect_true: route a computed boolean through expect_exit so it lands in the +-- same pass/fail accounting as everything else. +local function expect_true(cond, label) + checks.expect_exit({ exit = cond and 0 or 1 }, 0, label) +end + +local USAGE = "usage: workq %-%-producers P %-%-consumers C %-%-items N" + +-- --------------------------------------------------------------------------- +-- 1. error contract +-- --------------------------------------------------------------------------- + +local noargs = checks.run(demo) +checks.expect_exit(noargs, 2, lang .. ": no arguments exits 2") +checks.expect_match(noargs.out, USAGE, lang .. ": no-args prints the usage line") + +local unknown = checks.run(demo .. " --producers 1 --consumers 1 --items 4 --frob") +checks.expect_exit(unknown, 2, lang .. ": unknown flag exits 2") +checks.expect_match(unknown.out, "workq: unknown flag: %-%-frob", lang .. ": names the unknown flag") + +local missing = checks.run(demo .. " --producers 2 --items 4") +checks.expect_exit(missing, 2, lang .. ": missing --consumers exits 2") +checks.expect_match(missing.out, USAGE, lang .. ": missing-required prints usage") + +local badint = checks.run(demo .. " --producers x --consumers 1 --items 4") +checks.expect_exit(badint, 2, lang .. ": non-integer value exits 2") +checks.expect_match(badint.out, "workq: not an integer: x", lang .. ": names the bad integer") + +local noval = checks.run(demo .. " --producers") +checks.expect_exit(noval, 2, lang .. ": flag without a value exits 2") +checks.expect_match(noval.out, "workq: %-%-producers needs a value", lang .. ": names the value-less flag") + +-- --------------------------------------------------------------------------- +-- 2. fixed run: shared checksum constant, produced==consumed==N +-- --------------------------------------------------------------------------- + +local base = checks.run(demo .. " --producers 4 --consumers 4 --items " .. N) +checks.expect_exit(base, 0, lang .. ": correct run exits 0") +checks.expect_match(base.out, + "workq: produced=" .. N .. " consumed=" .. N .. " checksum=" .. CK_DEFAULT .. " ms=%d+", + lang .. ": produced==consumed==N and checksum matches the cross-language constant") + +-- --------------------------------------------------------------------------- +-- 3. determinism: checksum independent of P, C, cap +-- --------------------------------------------------------------------------- + +local shapes = { + "--producers 1 --consumers 1 --items " .. N .. " --cap 8", + "--producers 8 --consumers 3 --items " .. N .. " --cap 1024", + "--producers 2 --consumers 16 --items " .. N .. " --cap 4", +} +for _, shape in ipairs(shapes) do + local r = checks.run(demo .. " " .. shape) + checks.expect_exit(r, 0, lang .. ": shape [" .. shape .. "] exits 0") + local produced = r.out:match("produced=(%d+)") + local consumed = r.out:match("consumed=(%d+)") + local checksum = r.out:match("checksum=(%x+)") + expect_true(produced == tostring(N) and consumed == tostring(N), + lang .. ": [" .. shape .. "] produced==consumed==N (got p=" .. + tostring(produced) .. " c=" .. tostring(consumed) .. ")") + expect_true(checksum == CK_DEFAULT, + lang .. ": [" .. shape .. "] checksum is order-independent (" .. + tostring(checksum) .. ")") +end + +-- --------------------------------------------------------------------------- +-- 4. a different seed => different shared constant; N=0 => zero checksum +-- --------------------------------------------------------------------------- + +local seeded = checks.run(demo .. " --producers 3 --consumers 3 --items 50000 --seed 42") +checks.expect_exit(seeded, 0, lang .. ": seeded run exits 0") +checks.expect_match(seeded.out, + "workq: produced=50000 consumed=50000 checksum=" .. CK_SEED42 .. " ms=%d+", + lang .. ": seed 42 yields its own cross-language constant") + +local empty = checks.run(demo .. " --producers 3 --consumers 2 --items 0") +checks.expect_exit(empty, 0, lang .. ": N=0 run exits 0") +checks.expect_match(empty.out, + "workq: produced=0 consumed=0 checksum=0000000000000000 ms=%d+", + lang .. ": empty queue drains to an all-zero checksum") + +checks.finish() diff --git a/examples/26-atomics-and-lock-free/README.md b/examples/26-atomics-and-lock-free/README.md new file mode 100644 index 0000000..3519132 --- /dev/null +++ b/examples/26-atomics-and-lock-free/README.md @@ -0,0 +1,138 @@ +# 26-atomics-and-lock-free + +Chapter 26's example: `spscring`, a single-producer/single-consumer lock-free +ring buffer implemented three times — C++23, Go, and Rust — each with that +language's atomics, but one shared observable contract so a single `verify.lua` +covers all three. + +## The program + +``` +spscring --capacity K --items N [--pad on|off] +``` + +A **producer** thread pushes `N` `u64` values (`0, 1, ..., N-1`) through a +bounded ring of `K` slots; the **consumer** (the main thread) pops each value +and sums it. There is no mutex and no lock: the ring is coordinated by two +`u64` atomic counters, `head` (advanced by the consumer) and `tail` (advanced +by the producer), using acquire/release ordering — Lamport's classic SPSC +queue. When the producer has finished all `N` pushes and the consumer all `N` +pops, the program prints exactly one line: + +``` +spscring: items=1000000 sum=499999500000 throughput_mops=37.29 pad=off +``` + +- `items=` — echoes the requested count. +- `sum=` — the consumer's running total. Because the values are `0..N-1`, + a correct run **always** yields `s = N·(N-1)/2`. Any dropped, duplicated, or + torn hand-off changes this number, so the sum is a concurrency oracle. +- `throughput_mops=` — millions of pushes per second, `N / wall_seconds / 1e6`, + printed with two decimals. Its magnitude is machine- and run-dependent; the + book uses it only to *narrate* the false-sharing effect, and `verify.lua` + only asserts that it parses as a positive number. +- `pad=` — echoes the `--pad` mode. + +The three binaries produce the same line shape and the same `items`/`sum`/`pad` +fields for the same arguments. + +**`--pad on|off`** toggles cache-line padding between `head` and `tail` +(default `off`): + +- `off` — the two counters sit adjacent in one 64-byte cache line. The + producer's `store(tail)` and the consumer's `store(head)` then invalidate + each other's copy of the shared line on every hand-off — **false sharing** — + even though neither thread touches the other's counter. +- `on` — each counter is `alignas(64)` / `#[repr(align(64))]` / padded onto its + own cache line, so the two cores stop fighting over one line. + +Both modes compute the same, correct sum; the throughput difference between +them is the chapter's whole point, not a hard assertion (on a lightly loaded +box a single 1M-item run is noisy — the book measures the gap with repeats). + +## The ring protocol (identical across languages) + +`head` and `tail` are **monotonic** `u64` counters that only ever increase; +the physical slot for logical position `p` is `buf[p % K]`. + +``` +push(v): pop() -> v: + t = tail (private to producer) h = head (private to consumer) + while t - head.load(Acquire) == K while h == tail.load(Acquire) + spin # ring full spin # ring empty + buf[t % K] = v v = buf[h % K] + tail.store(t+1, Release) head.store(h+1, Release) +``` + +- The producer owns `tail` and only *reads* `head`; the consumer owns `head` + and only *reads* `tail`. That single-writer-per-counter rule is what makes + the queue safe without a lock. +- `tail.store(Release)` **publishes** the slot write that precedes it; + `tail.load(Acquire)` in the consumer **observes** it, establishing a + happens-before edge so the consumer's read of `buf[h % K]` is not a data + race. `head` carries the symmetric edge that lets the producer safely reuse + a slot only after the consumer has drained it. +- Ring occupancy is `tail - head`; **full** is `tail - head == K`, **empty** + is `head == tail`. Using monotonic counters (rather than wrapped indices) + makes those two states unambiguous without a spare slot. + +## Three atomics toolkits, one contract + +| | index atomics | memory ordering | padding knob | +|---|---|---|---| +| **C++23** | `std::atomic` head/tail in a `Control` struct; producer is a `std::jthread`, consumer is `main` | explicit `memory_order_acquire` on the loads, `memory_order_release` on the stores; the plain `std::vector` slots ride the release/acquire edge | `Control` gives head and tail each `alignas(64)`; `Control` packs them adjacent | +| **Go** | `atomic.Uint64` head/tail; producer goroutine, consumer on `main`, joined over a `chan struct{}` | **no per-op order argument** — Go's `sync/atomic` is sequentially consistent, and the memory model's "an atomic write is synchronized-before the read that observes it" *is* the acquire/release edge, just not relaxable | `ctrlPadded` inserts `[56]byte` after each counter; `ctrlPacked` leaves them adjacent | +| **Rust** | `AtomicU64` head/tail shared across a `std::thread::scope`; slots are `Vec` too | explicit `Ordering::{Acquire, Release}` on head/tail, `Ordering::Relaxed` on the slots — so the data race is ruled out by the type system, no `unsafe` | `CtrlPadded` wraps each counter in `#[repr(align(64))]`; `CtrlPacked` leaves them adjacent | + +**What Go gives you (and doesn't):** C++ and Rust let you name the exact +ordering per operation (`acquire`, `release`, `relaxed`) and could, in +principle, relax the slot accesses further. Go's `sync/atomic` exposes only +sequentially-consistent `Load`/`Store`/`Add` — you cannot ask for a weaker +order, but you also cannot get it wrong: every atomic access already carries +the happens-before edge this queue needs. The Go build runs under `-race` +(see `go/demo.sh`); the detector understands the atomic edge and stays silent +on the plain `buf[]` slot accesses, proving the lock-free hand-off is +race-free rather than merely lucky. + +## Layout + +``` +26-atomics-and-lock-free/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build (GCC or clang), demo.sh +├── go/ # go build -race (toolchain go1.26.5), demo.sh +└── rust/ # cargo build (rustup pin 1.97.1; std only), demo.sh +``` + +## The demo contract + +Every language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary + (e.g. `./demo.sh run --capacity 1024 --items 1000000 --pad on`) +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh` instead of running locally. + +This is a host-only example — pure threads and atomics, no sockets, no VM. + +## Verification + +`verify.lua` runs from this directory with `LSP_LANG` set to one language at a +time and asserts, per language: + +1. `--capacity 1024 --items 1000000 --pad off` exits 0 and prints + `items=1000000 sum=499999500000 throughput_mops= pad=off` — the sum + is the exact triangular number `N·(N-1)/2`, so every one of the million + hand-offs delivered its value once; +2. the same with `--pad on` yields the identical sum with `pad=on`; +3. `--capacity 1 --items 1000` (a one-slot ring — maximum head/tail + contention) still sums to exactly `499500`; +4. the `throughput_mops` field parses as a positive number; +5. no arguments prints `usage: spscring --capacity K --items N [--pad on|off]` + on stderr and exits 2. + +Verified on the Fedora 44 reference host (kernel 7.1.3-200.fc44): all three +languages PASS all 11 checks, Go under the race detector. diff --git a/examples/26-atomics-and-lock-free/cpp/.clang-format b/examples/26-atomics-and-lock-free/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/26-atomics-and-lock-free/cpp/.clang-tidy b/examples/26-atomics-and-lock-free/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/26-atomics-and-lock-free/cpp/CMakeLists.txt b/examples/26-atomics-and-lock-free/cpp/CMakeLists.txt new file mode 100644 index 0000000..5bc14cc --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/CMakeLists.txt @@ -0,0 +1,11 @@ +cmake_minimum_required(VERSION 3.25) +project(spscring CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app src/main.cpp) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/26-atomics-and-lock-free/cpp/CMakePresets.json b/examples/26-atomics-and-lock-free/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/26-atomics-and-lock-free/cpp/demo.sh b/examples/26-atomics-and-lock-free/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/26-atomics-and-lock-free/cpp/src/main.cpp b/examples/26-atomics-and-lock-free/cpp/src/main.cpp new file mode 100644 index 0000000..009ccc8 --- /dev/null +++ b/examples/26-atomics-and-lock-free/cpp/src/main.cpp @@ -0,0 +1,180 @@ +// spscring: a single-producer/single-consumer lock-free ring buffer. +// +// spscring --capacity K --items N [--pad on|off] +// +// A producer thread pushes N u64 values (0, 1, ..., N-1) through a bounded +// ring of K slots; the consumer (this thread) pops and sums them. The head +// and tail indices are std::atomic monotonic counters synchronised +// with explicit acquire/release ordering (Lamport's SPSC queue) — no mutex, +// no lock. --pad places head and tail on separate cache lines to remove the +// false sharing that otherwise ping-pongs the line between the two cores. +// +// Prints exactly one line: +// spscring: items= sum= throughput_mops= pad= + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr std::string_view kUsage = + "usage: spscring --capacity K --items N [--pad on|off]"; + +struct Args { + std::uint64_t capacity = 0; + std::uint64_t items = 0; + bool pad = false; +}; + +[[nodiscard]] std::optional parse_u64(std::string_view s) { + std::uint64_t v = 0; + const char* first = s.data(); + const char* last = s.data() + s.size(); + auto [ptr, ec] = std::from_chars(first, last, v); + if (ec != std::errc{} || ptr != last) { + return std::nullopt; + } + return v; +} + +// Parse argv into Args. Returns nullopt on any malformed input (caller then +// prints the usage line and exits 2). +[[nodiscard]] std::optional parse_args(int argc, char** argv) { + Args a{}; + bool have_cap = false; + bool have_items = false; + for (int i = 1; i < argc; ++i) { + const std::string_view flag = argv[i]; + auto next_value = [&]() -> std::optional { + if (i + 1 >= argc) { + return std::nullopt; + } + return std::string_view{argv[++i]}; + }; + if (flag == "--capacity") { + const auto v = next_value(); + if (!v) return std::nullopt; + const auto n = parse_u64(*v); + if (!n || *n == 0) return std::nullopt; + a.capacity = *n; + have_cap = true; + } else if (flag == "--items") { + const auto v = next_value(); + if (!v) return std::nullopt; + const auto n = parse_u64(*v); + if (!n) return std::nullopt; + a.items = *n; + have_items = true; + } else if (flag == "--pad") { + const auto v = next_value(); + if (!v) return std::nullopt; + if (*v == "on") { + a.pad = true; + } else if (*v == "off") { + a.pad = false; + } else { + return std::nullopt; + } + } else { + return std::nullopt; + } + } + if (!have_cap || !have_items) { + return std::nullopt; + } + return a; +} + +// Control block: the two index atomics. The padded specialisation forces head +// and tail onto separate 64-byte cache lines; the packed one leaves them +// adjacent so the two cores fight over one line (false sharing). +template +struct Control; + +template <> +struct Control { + alignas(64) std::atomic head{0}; + alignas(64) std::atomic tail{0}; +}; + +template <> +struct Control { + std::atomic head{0}; + std::atomic tail{0}; +}; + +struct Result { + std::uint64_t sum; + double mops; +}; + +template +[[nodiscard]] Result run_bench(std::uint64_t cap, std::uint64_t items) { + std::vector buf(cap); + auto ctrl = std::make_unique>(); + auto& head = ctrl->head; + auto& tail = ctrl->tail; + + const auto start = std::chrono::steady_clock::now(); + + // Producer: push 0..items-1. jthread joins on scope exit. + std::jthread producer([&] { + std::uint64_t t = 0; + for (std::uint64_t i = 0; i < items; ++i) { + // Wait until the ring has a free slot (consumer advanced head). + while (t - head.load(std::memory_order_acquire) == cap) { + // spin + } + buf[t % cap] = i; + ++t; + tail.store(t, std::memory_order_release); + } + }); + + // Consumer runs on this thread. + std::uint64_t sum = 0; + std::uint64_t h = 0; + for (std::uint64_t i = 0; i < items; ++i) { + while (h == tail.load(std::memory_order_acquire)) { + // spin: ring empty + } + sum += buf[h % cap]; + ++h; + head.store(h, std::memory_order_release); + } + + producer.join(); + const auto end = std::chrono::steady_clock::now(); + double secs = std::chrono::duration(end - start).count(); + if (secs < 1e-9) { + secs = 1e-9; + } + const double mops = static_cast(items) / secs / 1e6; + return {sum, mops}; +} + +} // namespace + +int main(int argc, char** argv) { + const auto args = parse_args(argc, argv); + if (!args) { + std::println(stderr, "{}", kUsage); + return 2; + } + + const Result r = args->pad ? run_bench(args->capacity, args->items) + : run_bench(args->capacity, args->items); + + std::println("spscring: items={} sum={} throughput_mops={:.2f} pad={}", + args->items, r.sum, r.mops, args->pad ? "on" : "off"); + return 0; +} diff --git a/examples/26-atomics-and-lock-free/demo.sh b/examples/26-atomics-and-lock-free/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/26-atomics-and-lock-free/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/26-atomics-and-lock-free/go/demo.sh b/examples/26-atomics-and-lock-free/go/demo.sh new file mode 100755 index 0000000..0931d6f --- /dev/null +++ b/examples/26-atomics-and-lock-free/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -race -o "$BIN" . # race detector on: this is the atomics/lock-free chapter +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/26-atomics-and-lock-free/go/go.mod b/examples/26-atomics-and-lock-free/go/go.mod new file mode 100644 index 0000000..23ac3bc --- /dev/null +++ b/examples/26-atomics-and-lock-free/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/26-atomics-and-lock-free/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/26-atomics-and-lock-free/go/main.go b/examples/26-atomics-and-lock-free/go/main.go new file mode 100644 index 0000000..c487bf9 --- /dev/null +++ b/examples/26-atomics-and-lock-free/go/main.go @@ -0,0 +1,174 @@ +// spscring: a single-producer/single-consumer lock-free ring buffer. +// +// spscring --capacity K --items N [--pad on|off] +// +// A producer goroutine pushes N u64 values (0..N-1) through a bounded ring of +// K slots; the consumer (main goroutine) pops and sums them. head and tail are +// atomic.Uint64 monotonic counters (Lamport's SPSC queue) — no mutex. +// +// What Go's sync/atomic gives you: unlike C++ or Rust there is NO per-operation +// memory-order argument. Every Load/Store/Add is sequentially consistent, and +// the Go memory model promises that an atomic write is "synchronized before" +// the atomic read that observes it — that happens-before edge is exactly the +// acquire/release pairing we need, just without the ability to relax it. So the +// plain buf[] writes before tail.Store are visible to the consumer that reads +// tail, and the -race detector (see demo.sh) understands that edge and stays +// quiet on this deliberately lock-free code. +// +// --pad puts head and tail on separate cache lines to remove the false sharing +// that otherwise ping-pongs one line between the two cores. +// +// Prints exactly one line: +// +// spscring: items= sum= throughput_mops= pad= +package main + +import ( + "fmt" + "os" + "strconv" + "sync/atomic" + "time" +) + +const usage = "usage: spscring --capacity K --items N [--pad on|off]" + +type args struct { + capacity uint64 + items uint64 + pad bool +} + +func parseArgs(argv []string) (args, bool) { + var a args + haveCap, haveItems := false, false + for i := 0; i < len(argv); i++ { + next := func() (string, bool) { + if i+1 >= len(argv) { + return "", false + } + i++ + return argv[i], true + } + switch argv[i] { + case "--capacity": + v, ok := next() + if !ok { + return a, false + } + n, err := strconv.ParseUint(v, 10, 64) + if err != nil || n == 0 { + return a, false + } + a.capacity, haveCap = n, true + case "--items": + v, ok := next() + if !ok { + return a, false + } + n, err := strconv.ParseUint(v, 10, 64) + if err != nil { + return a, false + } + a.items, haveItems = n, true + case "--pad": + v, ok := next() + if !ok { + return a, false + } + switch v { + case "on": + a.pad = true + case "off": + a.pad = false + default: + return a, false + } + default: + return a, false + } + } + if !haveCap || !haveItems { + return a, false + } + return a, true +} + +// Two control-block layouts. ctrlPadded forces head and tail onto separate +// 64-byte cache lines; ctrlPacked leaves them adjacent (false sharing). +type ctrlPadded struct { + head atomic.Uint64 + _ [56]byte // pad head out to a full cache line + tail atomic.Uint64 + _ [56]byte +} + +type ctrlPacked struct { + head atomic.Uint64 + tail atomic.Uint64 +} + +func bench(capacity, items uint64, pad bool) (sum uint64, mops float64) { + buf := make([]uint64, capacity) + + // Grab *atomic.Uint64 pointers into whichever layout we chose; the struct + // stays alive because these pointers reference it. + var head, tail *atomic.Uint64 + if pad { + c := &ctrlPadded{} + head, tail = &c.head, &c.tail + } else { + c := &ctrlPacked{} + head, tail = &c.head, &c.tail + } + + start := time.Now() + + done := make(chan struct{}) + go func() { // producer + var t uint64 + for i := uint64(0); i < items; i++ { + for t-head.Load() == capacity { // ring full: spin + } + buf[t%capacity] = i + t++ + tail.Store(t) + } + close(done) + }() + + // consumer (main goroutine) + var h uint64 + for i := uint64(0); i < items; i++ { + for h == tail.Load() { // ring empty: spin + } + sum += buf[h%capacity] + h++ + head.Store(h) + } + + <-done + secs := time.Since(start).Seconds() + if secs < 1e-9 { + secs = 1e-9 + } + mops = float64(items) / secs / 1e6 + return sum, mops +} + +func main() { + a, ok := parseArgs(os.Args[1:]) + if !ok { + fmt.Fprintln(os.Stderr, usage) + os.Exit(2) + } + + sum, mops := bench(a.capacity, a.items, a.pad) + + padStr := "off" + if a.pad { + padStr = "on" + } + fmt.Printf("spscring: items=%d sum=%d throughput_mops=%.2f pad=%s\n", + a.items, sum, mops, padStr) +} diff --git a/examples/26-atomics-and-lock-free/rust/Cargo.lock b/examples/26-atomics-and-lock-free/rust/Cargo.lock new file mode 100644 index 0000000..1a50fb6 --- /dev/null +++ b/examples/26-atomics-and-lock-free/rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "spscring" +version = "0.1.0" diff --git a/examples/26-atomics-and-lock-free/rust/Cargo.toml b/examples/26-atomics-and-lock-free/rust/Cargo.toml new file mode 100644 index 0000000..0e52ce7 --- /dev/null +++ b/examples/26-atomics-and-lock-free/rust/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "spscring" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] diff --git a/examples/26-atomics-and-lock-free/rust/demo.sh b/examples/26-atomics-and-lock-free/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/26-atomics-and-lock-free/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/26-atomics-and-lock-free/rust/rust-toolchain.toml b/examples/26-atomics-and-lock-free/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/26-atomics-and-lock-free/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/26-atomics-and-lock-free/rust/src/main.rs b/examples/26-atomics-and-lock-free/rust/src/main.rs new file mode 100644 index 0000000..d345277 --- /dev/null +++ b/examples/26-atomics-and-lock-free/rust/src/main.rs @@ -0,0 +1,169 @@ +//! spscring: a single-producer/single-consumer lock-free ring buffer. +//! +//! ```text +//! spscring --capacity K --items N [--pad on|off] +//! ``` +//! +//! A producer thread pushes N u64 values (0..N-1) through a bounded ring of K +//! slots; the consumer (main thread) pops and sums them. head and tail are +//! `AtomicU64` monotonic counters synchronised with explicit +//! `Ordering::{Acquire, Release}` (Lamport's SPSC queue) — no mutex, no lock. +//! The ring slots are themselves `AtomicU64` accessed `Relaxed`; the head/tail +//! acquire/release pair is what publishes each slot to the other thread, so the +//! data race is ruled out at the type level rather than by `unsafe`. +//! +//! `--pad` puts head and tail on separate cache lines to remove the false +//! sharing that otherwise ping-pongs one line between the two cores. +//! +//! Prints exactly one line: +//! `spscring: items= sum= throughput_mops= pad=` + +use std::process::ExitCode; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::thread; +use std::time::Instant; + +const USAGE: &str = "usage: spscring --capacity K --items N [--pad on|off]"; + +struct Args { + capacity: u64, + items: u64, + pad: bool, +} + +fn parse_args(argv: &[String]) -> Result { + let mut capacity: Option = None; + let mut items: Option = None; + let mut pad = false; + + let mut i = 0; + while i < argv.len() { + let flag = argv[i].as_str(); + let mut next = || -> Result<&String, ()> { + i += 1; + argv.get(i).ok_or(()) + }; + match flag { + "--capacity" => { + let n: u64 = next()?.parse().map_err(|_| ())?; + if n == 0 { + return Err(()); + } + capacity = Some(n); + } + "--items" => { + let n: u64 = next()?.parse().map_err(|_| ())?; + items = Some(n); + } + "--pad" => match next()?.as_str() { + "on" => pad = true, + "off" => pad = false, + _ => return Err(()), + }, + _ => return Err(()), + } + i += 1; + } + + Ok(Args { + capacity: capacity.ok_or(())?, + items: items.ok_or(())?, + pad, + }) +} + +/// A cache-line-aligned atomic: when used for head and tail separately this +/// forces each onto its own 64-byte line, removing false sharing. +#[repr(align(64))] +struct CacheAligned(AtomicU64); + +/// Padded layout: head and tail on separate cache lines. +struct CtrlPadded { + head: CacheAligned, + tail: CacheAligned, +} + +/// Packed layout: head and tail adjacent (false sharing). +struct CtrlPacked { + head: AtomicU64, + tail: AtomicU64, +} + +/// The SPSC benchmark, generic over the two index atomics. `head` is written by +/// the consumer and read by the producer; `tail` the reverse. +fn run(cap: u64, items: u64, head: &AtomicU64, tail: &AtomicU64) -> (u64, f64) { + let buf: Vec = (0..cap).map(|_| AtomicU64::new(0)).collect(); + let buf = buf.as_slice(); + + let start = Instant::now(); + + let sum = thread::scope(|s| { + // Producer: push 0..items-1. + s.spawn(move || { + let mut t: u64 = 0; + for i in 0..items { + while t - head.load(Ordering::Acquire) == cap { + std::hint::spin_loop(); + } + buf[(t % cap) as usize].store(i, Ordering::Relaxed); + t += 1; + tail.store(t, Ordering::Release); + } + }); + + // Consumer on this thread. + let mut h: u64 = 0; + let mut sum: u64 = 0; + for _ in 0..items { + while h == tail.load(Ordering::Acquire) { + std::hint::spin_loop(); + } + sum = sum.wrapping_add(buf[(h % cap) as usize].load(Ordering::Relaxed)); + h += 1; + head.store(h, Ordering::Release); + } + sum + }); + + let mut secs = start.elapsed().as_secs_f64(); + if secs < 1e-9 { + secs = 1e-9; + } + let mops = items as f64 / secs / 1e6; + (sum, mops) +} + +fn bench(cap: u64, items: u64, pad: bool) -> (u64, f64) { + if pad { + let ctrl = CtrlPadded { + head: CacheAligned(AtomicU64::new(0)), + tail: CacheAligned(AtomicU64::new(0)), + }; + run(cap, items, &ctrl.head.0, &ctrl.tail.0) + } else { + let ctrl = CtrlPacked { + head: AtomicU64::new(0), + tail: AtomicU64::new(0), + }; + run(cap, items, &ctrl.head, &ctrl.tail) + } +} + +fn main() -> ExitCode { + let argv: Vec = std::env::args().skip(1).collect(); + let args = match parse_args(&argv) { + Ok(a) => a, + Err(()) => { + eprintln!("{USAGE}"); + return ExitCode::from(2); + } + }; + + let (sum, mops) = bench(args.capacity, args.items, args.pad); + let pad = if args.pad { "on" } else { "off" }; + println!( + "spscring: items={} sum={} throughput_mops={:.2} pad={}", + args.items, sum, mops, pad + ); + ExitCode::SUCCESS +} diff --git a/examples/26-atomics-and-lock-free/verify.lua b/examples/26-atomics-and-lock-free/verify.lua new file mode 100644 index 0000000..cc13535 --- /dev/null +++ b/examples/26-atomics-and-lock-free/verify.lua @@ -0,0 +1,72 @@ +-- Verify spscring for LSP_LANG: the SPSC lock-free ring delivers every item +-- exactly once (sum == N*(N-1)/2) under real producer/consumer concurrency, +-- with both cache-line-padding modes, at a healthy ring size and at capacity 1 +-- (maximum head/tail contention); the throughput line is well formed; and a +-- missing argument prints the usage line and exits 2. +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- N*(N-1)/2 for the values 0..N-1 the producer pushes. +local function triangular(n) + -- n <= 10^6 here, so this stays well inside Lua 5.4 integer range. + return (n * (n - 1)) // 2 +end + +-- Run spscring and assert the output line is byte-shaped as documented and +-- carries the expected items / sum / pad token, with a parseable throughput. +local function check_run(capacity, items, pad_flag, pad_token, label) + local cmd = demo .. " --capacity " .. capacity .. " --items " .. items + if pad_flag then + cmd = cmd .. " --pad " .. pad_flag + end + local r = checks.run(cmd) + checks.expect_exit(r, 0, label .. ": exits 0") + + local want_sum = triangular(items) + -- Full line shape: everything is pinned except the throughput number. + local pat = "^spscring: items=" .. items .. + " sum=" .. want_sum .. + " throughput_mops=%d+%.%d+ pad=" .. pad_token .. "%s*$" + checks.expect_match(r.out, pat, + label .. ": line is 'items=" .. items .. " sum=" .. want_sum .. + " throughput_mops= pad=" .. pad_token .. "'") + + -- Independently confirm the throughput field parses as a positive number. + local mops = r.out:match("throughput_mops=([%d%.]+)") + if mops and tonumber(mops) and tonumber(mops) > 0 then + checks.expect_match("ok", "ok", label .. ": throughput_mops parses as a positive number") + else + checks.expect_match("no", "yes", + label .. ": throughput_mops parses as a positive number (got " .. tostring(mops) .. ")") + end +end + +-- 1. Healthy ring, no padding: correctness of the acquire/release protocol. +check_run(1024, 1000000, "off", "off", lang .. " cap=1024 pad=off") + +-- 2. Healthy ring, cache-line padding on: same delivered sum. +check_run(1024, 1000000, "on", "on", lang .. " cap=1024 pad=on") + +-- 3. Capacity 1: producer and consumer hand off one slot at a time, the +-- worst case for the head/tail handshake; sum must still be exact. +check_run(1, 1000, nil, "off", lang .. " cap=1 default-pad") + +-- 4. CLI contract: no arguments -> usage on stderr, exit 2. +local u = checks.run(demo) +checks.expect_exit(u, 2, lang .. ": missing args exits 2") +checks.expect_match(u.out, "usage: spscring %-%-capacity K %-%-items N", + lang .. ": missing args prints usage") + +checks.finish() diff --git a/examples/27-async-runtimes-and-coroutines/README.md b/examples/27-async-runtimes-and-coroutines/README.md new file mode 100644 index 0000000..fa53e44 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/README.md @@ -0,0 +1,155 @@ +# 27 — async runtimes and coroutines (chatterd v4) + +`chatterd` is the book's recurring peer-to-peer chat daemon. This chapter is the +**async rewrite**: the *same* server, with its connection handling rebuilt on +each language's async model. The observable behaviour is unchanged — a message +from one client is broadcast to every other connected client — so a single +`verify.lua` covers all three languages. + +| language | `--engine async` is… | +|----------|------------------------------------------------------------------| +| **C++** | hand-rolled **C++20 coroutines** over an epoll reactor: each connection is a coroutine that `co_await`s socket readability. The `task`/awaitable machinery lives in `cpp/src/main.cpp`; no external async library is linked. | +| **Rust** | **tokio**: an `async fn` per connection on a `tokio::net::TcpListener`, spawned as tasks, fanning out over per-client `mpsc` channels. | +| **Go** | **unchanged from v1** — goroutines already *are* the async model. `--engine async` selects the same goroutine-per-connection hub. That is the lesson: Go has no separate coroutine layer to add. | + +Every version also keeps the prior **`--engine thread`** engine working +(one OS thread per client), so you can compare the two side by side and confirm +they are byte-for-byte identical on the wire. + +## The wire format (canonical chatterd chat frame) + +This is the **same frame every chatterd chapter uses** — introduced in ch21 +and extended only additively (by adding new `type` values) ever since; the +header never changes shape. Every message on a connection, in both +directions, is a single frame: + +``` + byte: 0 1 2 3 4 5 6 6+L-1 + +----+----+----+----+----+----+=========================+ + |magic(CH)|ver |type|len (u16)| payload (L bytes) | + |0x43 0x48|0x01| |big-end. | UTF-8, type-specific | + +----+----+----+----+----+----+=========================+ +``` + +* `magic` — 2 bytes, always `0x43 0x48` (ASCII `"CH"`). +* `version` — 1 byte, `0x01`. +* `type` — 1 byte: `JOIN=1`, `MSG=2`, `DELIVER=3`, `WELCOME=4`, `PING=5`. +* `length` (`L`) — a 16-bit **big-endian** unsigned integer: the number of + bytes in `payload` (0‥65535). +* `payload` — `L` bytes, UTF-8, type-specific (see below). + +This chapter's engines (thread and async, in all three languages) use only +three of the five types — the same three ch21 introduced: + +* **JOIN** — the **first** frame a client sends after connecting; payload is + the client's nick (e.g. `alice`). +* **MSG** — every **later** frame the client sends; payload is message text. +* **DELIVER** — for a MSG with payload `T` from the client whose nick is `N`, + the server sends every **other** connected client a DELIVER frame whose + payload is exactly `N + 0x00 (NUL) + T`. The sender never receives its own + message. + +`WELCOME` and `PING` are other chapters' additions to this same frame (ch22's +welcome/roster message, ch24's heartbeat) — this chapter never sends them, and +because the header is unchanged, a ch21-era client that doesn't know those +types still exchanges JOIN/MSG/DELIVER correctly with this chapter's server. + +### Byte-exact examples + +``` +JOIN "alice" 43 48 01 01 00 05 61 6c 69 63 65 +MSG "hi" 43 48 01 02 00 02 68 69 +DELIVER "alice\0hi" 43 48 01 03 00 08 61 6c 69 63 65 00 68 69 +``` + +That last line is a real capture: with `alice` and `bob` both connected, +`alice` sending `hi` puts these fourteen bytes on `bob`'s socket — the 6-byte +header (`43 48` magic, `01` version, `03` type DELIVER, `00 08` length) then +the 8-byte payload `alice\0hi` (nick, NUL, text). The framing and the +`N NUL T` payload are identical across C++, Go, and Rust, and across the +`thread` and `async` engines. + +## CLI + +``` +app serve [--engine thread|async] [--host H] [--port P] # run the daemon +app loadtest [--host H] [--port P] [--clients N] # drive a broadcast +``` + +Defaults: `--host 127.0.0.1`, `--port 47100`, `--engine async`, `--clients 20`. + +* `serve` prints `chatterd: listening on H:P engine=E` on stderr once bound, + then serves until it receives `SIGINT` or `SIGTERM`, at which point it logs + `chatterd: shutdown` and exits `0`. +* An unknown engine prints `chatterd: unknown engine '…' (want thread|async)` + and exits `2`; a missing/unknown subcommand prints usage and exits `2`. +* `loadtest` opens `N` receiver connections plus one sender, has the sender + broadcast `hello world`, and checks every receiver got `sender: hello world`. + It prints `loadtest: delivered K/N` and exits `0` iff `K == N`. + +## Demo contract + +Standard across the book: + +``` +./demo.sh # build + run all three +./demo.sh cpp run serve --engine async --port 47100 +./demo.sh go run loadtest --port 47100 --clients 20 +./demo.sh build # build only +``` + +With `TARGET` set, `run` deploys the binary to that lab VM. This example runs +entirely on loopback; it never touches a VM. + +Try it by hand: + +``` +./demo.sh cpp run serve --engine async --port 47100 & # start the daemon +./demo.sh cpp run loadtest --port 47100 --clients 20 # -> loadtest: delivered 20/20 +kill -TERM %1 # -> chatterd: shutdown +``` + +## Verification + +`verify.lua` (driven by `scripts/lib/checks.lua`) starts the daemon in the +background, waits for the port, runs the 20-client `loadtest`, and asserts the +observable behaviour for **both** engines: + +* `chatterd: listening on 127.0.0.1:PORT engine=` announced, +* `loadtest: delivered 20/20` — the broadcast reached all 20 clients, +* the loadtest exits `0`, +* `SIGTERM` produces `chatterd: shutdown` and a `0` exit — a clean shutdown, + +plus the CLI contract (missing subcommand → exit 2, unknown engine → exit 2). +The same assertions pass identically for C++, Go, and Rust. + +## Layout + +``` +27-async-runtimes-and-coroutines/ +├── demo.sh # dispatcher +├── verify.lua # background-daemon broadcast + shutdown checks +├── cpp/src/main.cpp # thread engine + hand-rolled coroutine (async) engine +├── go/main.go # goroutine hub (async == thread here) +└── rust/src/main.rs # std thread engine + tokio (async) engine +``` + +## Notes on the C++ coroutine engine + +`cpp/src/main.cpp` implements the minimum coroutine runtime the async engine +needs, and nothing more: + +* **`Detached`** — a fire-and-forget coroutine (eager start, self-destroying at + `final_suspend`). The acceptor and each per-connection handler are `Detached`. +* **`Task`** — a lazy, awaitable coroutine that returns a value and uses + symmetric transfer to resume its awaiter on completion. `fill()` (one + non-blocking `recv`, suspending on `EAGAIN`) is a `Task`. +* **`Reactor`** — an epoll loop mapping a waited-on fd to the + `std::coroutine_handle` parked on its readability (`EPOLLONESHOT`, re-armed on + each `co_await`). A `co_await ReadAwaitable{reactor, fd}` suspends the current + coroutine until `fd` is readable. + +Every socket is owned by an RAII `Fd`; a `SIGTERM`/`SIGINT` handler flips an +`std::atomic` and kicks an `eventfd` so the reactor (or the `thread` +engine's `poll`) wakes and unwinds cleanly. The `thread` engine uses +`std::jthread` workers joined on shutdown. diff --git a/examples/27-async-runtimes-and-coroutines/cpp/.clang-format b/examples/27-async-runtimes-and-coroutines/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/27-async-runtimes-and-coroutines/cpp/.clang-tidy b/examples/27-async-runtimes-and-coroutines/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/27-async-runtimes-and-coroutines/cpp/CMakeLists.txt b/examples/27-async-runtimes-and-coroutines/cpp/CMakeLists.txt new file mode 100644 index 0000000..2d686cc --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/CMakeLists.txt @@ -0,0 +1,10 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) +find_package(Threads REQUIRED) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/27-async-runtimes-and-coroutines/cpp/CMakePresets.json b/examples/27-async-runtimes-and-coroutines/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/27-async-runtimes-and-coroutines/cpp/demo.sh b/examples/27-async-runtimes-and-coroutines/cpp/demo.sh new file mode 100755 index 0000000..6d53e45 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" # exec so signals sent to this PID reach the app + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp b/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp new file mode 100644 index 0000000..35ee1d2 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/cpp/src/main.cpp @@ -0,0 +1,747 @@ +// chatterd v4 (C++23) — a peer-to-peer chat daemon. +// +// The connection-handling engine is selectable: +// * thread : one std::jthread per client (the ch21 baseline). +// * async : a single-threaded C++20 coroutine reactor over epoll — each +// connection is a coroutine that `co_await`s socket readability. +// The task/awaitable machinery is hand-rolled here; no external +// async library is used. +// +// Both engines speak the SAME length-prefixed frame protocol and are +// observably identical: a broadcast reaches every other connected client. +// +// Usage: +// app serve [--engine thread|async] [--host H] [--port P] +// app loadtest [--host H] [--port P] [--clients N] +// +// Wire format (canonical chatterd chat frame, v1..v4): +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// type: JOIN=1, MSG=2, DELIVER=3, WELCOME=4, PING=5. +// The client's first frame is JOIN (payload = its nick). Every later client +// frame is MSG (payload = message text). For a MSG with text T from the +// client whose nick is N, the server sends every OTHER client a DELIVER +// frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use +// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// the same frame and are simply never sent here. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +// -------------------------------------------------------------------------- +// Shutdown plumbing: a signal sets the flag and kicks the wake eventfd so a +// blocked epoll_wait/poll returns promptly. +// -------------------------------------------------------------------------- +std::atomic g_stop{false}; +volatile std::sig_atomic_t g_wake_fd = -1; + +extern "C" void on_signal(int) { + g_stop.store(true, std::memory_order_relaxed); + if (g_wake_fd >= 0) { + std::uint64_t one = 1; + ssize_t r = ::write(g_wake_fd, &one, sizeof one); + (void)r; // best effort; async-signal-safe + } +} + +void install_signals() { + struct sigaction sa{}; + sa.sa_handler = on_signal; + sigemptyset(&sa.sa_mask); + sa.sa_flags = 0; // no SA_RESTART: interrupt blocking syscalls + sigaction(SIGINT, &sa, nullptr); + sigaction(SIGTERM, &sa, nullptr); + struct sigaction ign{}; + ign.sa_handler = SIG_IGN; + sigemptyset(&ign.sa_mask); + sigaction(SIGPIPE, &ign, nullptr); // never die on a write to a gone peer +} + +// -------------------------------------------------------------------------- +// RAII owner for a file descriptor — closes on scope exit, move-only. +// -------------------------------------------------------------------------- +class Fd { +public: + Fd() noexcept = default; + explicit Fd(int fd) noexcept : fd_(fd) {} + Fd(Fd&& o) noexcept : fd_(std::exchange(o.fd_, -1)) {} + Fd& operator=(Fd&& o) noexcept { + if (this != &o) { + reset(); + fd_ = std::exchange(o.fd_, -1); + } + return *this; + } + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + ~Fd() { reset(); } + + [[nodiscard]] int get() const noexcept { return fd_; } + explicit operator bool() const noexcept { return fd_ >= 0; } + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + fd_ = -1; + } + } + +private: + int fd_ = -1; +}; + +// -------------------------------------------------------------------------- +// Frame helpers — canonical chatterd chat frame (see file header). +// -------------------------------------------------------------------------- +constexpr int kMaxBody = 65535; +constexpr std::size_t kHeaderLen = 6; +constexpr std::uint8_t kMagic0 = 0x43; +constexpr std::uint8_t kMagic1 = 0x48; +constexpr std::uint8_t kVersion = 0x01; +constexpr std::uint8_t kTypeJoin = 1; +constexpr std::uint8_t kTypeMsg = 2; +constexpr std::uint8_t kTypeDeliver = 3; + +struct Frame { + std::uint8_t type; + std::string body; +}; + +std::string encode_frame(std::uint8_t type, std::string_view body) { + std::string out; + out.reserve(kHeaderLen + body.size()); + out.push_back(static_cast(kMagic0)); + out.push_back(static_cast(kMagic1)); + out.push_back(static_cast(kVersion)); + out.push_back(static_cast(type)); + std::uint16_t len = static_cast(body.size()); + out.push_back(static_cast((len >> 8) & 0xff)); + out.push_back(static_cast(len & 0xff)); + out.append(body); + return out; +} + +// Write every byte of `data` to fd, tolerating short writes / EAGAIN / EINTR. +void send_all(int fd, std::string_view data) { + std::size_t off = 0; + while (off < data.size()) { + ssize_t n = ::send(fd, data.data() + off, data.size() - off, MSG_NOSIGNAL); + if (n > 0) { + off += static_cast(n); + continue; + } + if (n < 0 && errno == EINTR) { + continue; + } + if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) { + struct pollfd p{fd, POLLOUT, 0}; + ::poll(&p, 1, 1000); + continue; + } + return; // peer gone or hard error: drop this recipient + } +} + +// Blocking frame reader: pull bytes into `buf` until one whole frame is +// available, hand its type+body back, and consume it. false => EOF / error. +bool read_frame_blocking(int fd, std::string& buf, Frame& out) { + for (;;) { + if (buf.size() >= kHeaderLen) { + std::uint8_t type = static_cast(buf[3]); + std::uint16_t len = (static_cast(buf[4]) << 8) | + static_cast(buf[5]); + if (buf.size() >= kHeaderLen + len) { + out.type = type; + out.body.assign(buf, kHeaderLen, len); + buf.erase(0, kHeaderLen + len); + return true; + } + } + char tmp[4096]; + ssize_t n = ::recv(fd, tmp, sizeof tmp, 0); + if (n > 0) { + buf.append(tmp, static_cast(n)); + continue; + } + if (n < 0 && errno == EINTR) { + continue; + } + return false; + } +} + +// -------------------------------------------------------------------------- +// Shared client registry used by every engine. +// -------------------------------------------------------------------------- +class Server { +public: + void add(int fd, std::string nick) { + std::scoped_lock lk(mu_); + clients_.push_back({fd, std::move(nick)}); + } + void remove(int fd) { + std::scoped_lock lk(mu_); + std::erase_if(clients_, [&](const Client& c) { return c.fd == fd; }); + } + // Deliver a DELIVER frame (payload = nick + NUL + text) to every client + // except the sender. + void broadcast(int from_fd, std::string_view nick, std::string_view text) { + std::string payload; + payload.reserve(nick.size() + 1 + text.size()); + payload.append(nick); + payload.push_back('\0'); + payload.append(text); + std::string frame = encode_frame(kTypeDeliver, payload); + std::scoped_lock lk(mu_); + for (const Client& c : clients_) { + if (c.fd != from_fd) { + send_all(c.fd, frame); + } + } + } + // Wake every blocked client read so thread workers can retire. + void shutdown_all() { + std::scoped_lock lk(mu_); + for (const Client& c : clients_) { + ::shutdown(c.fd, SHUT_RDWR); + } + } + +private: + struct Client { + int fd; + std::string nick; + }; + std::mutex mu_; + std::vector clients_; +}; + +// -------------------------------------------------------------------------- +// Networking setup. +// -------------------------------------------------------------------------- +std::expected make_listener(const std::string& host, std::uint16_t port, + bool nonblock) { + int type = SOCK_STREAM | SOCK_CLOEXEC | (nonblock ? SOCK_NONBLOCK : 0); + Fd s{::socket(AF_INET, type, 0)}; + if (!s) { + return std::unexpected(std::string("socket: ") + std::strerror(errno)); + } + int one = 1; + ::setsockopt(s.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof one); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected("inet_pton: bad host " + host); + } + if (::bind(s.get(), reinterpret_cast(&addr), sizeof addr) < 0) { + return std::unexpected(std::string("bind: ") + std::strerror(errno)); + } + if (::listen(s.get(), 128) < 0) { + return std::unexpected(std::string("listen: ") + std::strerror(errno)); + } + return s; +} + +std::expected connect_to(const std::string& host, std::uint16_t port) { + Fd s{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!s) { + return std::unexpected(std::string("socket: ") + std::strerror(errno)); + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected("inet_pton: bad host " + host); + } + if (::connect(s.get(), reinterpret_cast(&addr), sizeof addr) < 0) { + return std::unexpected(std::string("connect: ") + std::strerror(errno)); + } + return s; +} + +// ========================================================================== +// thread engine — one jthread per client. +// ========================================================================== +void serve_client_blocking(Server& server, Fd conn) { + int fd = conn.get(); + std::string buf, nick; + Frame f; + bool have_nick = false; + while (read_frame_blocking(fd, buf, f)) { + if (!have_nick) { + // First frame is JOIN: its payload is the nick. + nick = f.body; + have_nick = true; + server.add(fd, nick); + } else { + // Every later frame is MSG: its payload is the message text. + server.broadcast(fd, nick, f.body); + } + } + if (have_nick) { + server.remove(fd); + } +} // Fd destructor closes the socket + +std::expected run_thread(const std::string& host, std::uint16_t port) { + auto lfd = make_listener(host, port, /*nonblock=*/true); + if (!lfd) { + return std::unexpected(lfd.error()); + } + Fd listen = std::move(*lfd); + Fd wake{::eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC)}; + if (!wake) { + return std::unexpected(std::string("eventfd: ") + std::strerror(errno)); + } + g_wake_fd = wake.get(); + + std::println(stderr, "chatterd: listening on {}:{} engine=thread", host, port); + + Server server; + std::vector workers; + while (!g_stop.load(std::memory_order_relaxed)) { + struct pollfd pfds[2] = {{listen.get(), POLLIN, 0}, {wake.get(), POLLIN, 0}}; + int pr = ::poll(pfds, 2, -1); + if (pr < 0) { + if (errno == EINTR) { + continue; + } + break; + } + if (pfds[1].revents & POLLIN) { + break; // signalled + } + if (pfds[0].revents & POLLIN) { + for (;;) { + int cfd = ::accept4(listen.get(), nullptr, nullptr, SOCK_CLOEXEC); + if (cfd < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + break; + } + if (errno == EINTR) { + continue; + } + break; + } + workers.emplace_back( + [&server, f = Fd{cfd}]() mutable { serve_client_blocking(server, std::move(f)); }); + } + } + } + server.shutdown_all(); // unblock client reads + return {}; // workers joined by ~jthread +} + +// ========================================================================== +// async engine — hand-rolled coroutine reactor over epoll. +// ========================================================================== + +// A fire-and-forget coroutine: starts eagerly, destroys its own frame when it +// finishes. Used for the top-level acceptor and per-connection coroutines. +struct Detached { + struct promise_type { + Detached get_return_object() noexcept { return {}; } + std::suspend_never initial_suspend() noexcept { return {}; } + std::suspend_never final_suspend() noexcept { return {}; } + void return_void() noexcept {} + void unhandled_exception() { std::terminate(); } + }; +}; + +// A value-returning coroutine that can be co_awaited. Lazy start + symmetric +// transfer back to its awaiter on completion. +template +class Task { +public: + struct promise_type { + std::coroutine_handle<> cont_{}; + T value_{}; + Task get_return_object() noexcept { + return Task{std::coroutine_handle::from_promise(*this)}; + } + std::suspend_always initial_suspend() noexcept { return {}; } + struct Final { + bool await_ready() noexcept { return false; } + std::coroutine_handle<> await_suspend( + std::coroutine_handle h) noexcept { + auto c = h.promise().cont_; + return c ? c : std::noop_coroutine(); + } + void await_resume() noexcept {} + }; + Final final_suspend() noexcept { return {}; } + void return_value(T v) noexcept { value_ = std::move(v); } + void unhandled_exception() { std::terminate(); } + }; + using handle_t = std::coroutine_handle; + + explicit Task(handle_t h) noexcept : h_(h) {} + Task(Task&& o) noexcept : h_(std::exchange(o.h_, {})) {} + Task& operator=(Task&&) = delete; + Task(const Task&) = delete; + ~Task() { + if (h_) { + h_.destroy(); + } + } + + bool await_ready() noexcept { return false; } + handle_t await_suspend(std::coroutine_handle<> caller) noexcept { + h_.promise().cont_ = caller; + return h_; // symmetric transfer: begin the task + } + T await_resume() noexcept { return std::move(h_.promise().value_); } + +private: + handle_t h_; +}; + +// The reactor: maps a waited-on fd to the coroutine parked on its readability. +class Reactor { +public: + Reactor(Fd ep, int wake_fd) : ep_(std::move(ep)), wake_fd_(wake_fd) { + epoll_event ev{}; + ev.events = EPOLLIN; // persistent: the wake fd is never rearmed + ev.data.fd = wake_fd_; + ::epoll_ctl(ep_.get(), EPOLL_CTL_ADD, wake_fd_, &ev); + } + + void wait_readable(int fd, std::coroutine_handle<> h) { + waiters_[fd] = h; + epoll_event ev{}; + ev.events = EPOLLIN | EPOLLONESHOT; + ev.data.fd = fd; + if (armed_.contains(fd)) { + ::epoll_ctl(ep_.get(), EPOLL_CTL_MOD, fd, &ev); + } else { + ::epoll_ctl(ep_.get(), EPOLL_CTL_ADD, fd, &ev); + armed_.insert(fd); + } + } + + void forget(int fd) { + if (armed_.contains(fd)) { + ::epoll_ctl(ep_.get(), EPOLL_CTL_DEL, fd, nullptr); + armed_.erase(fd); + } + waiters_.erase(fd); + } + + void run() { + std::array evs{}; + while (!g_stop.load(std::memory_order_relaxed)) { + int n = ::epoll_wait(ep_.get(), evs.data(), static_cast(evs.size()), -1); + if (n < 0) { + if (errno == EINTR) { + continue; + } + break; + } + for (int i = 0; i < n; ++i) { + int fd = evs[i].data.fd; + if (fd == wake_fd_) { + continue; // loop guard picks up g_stop + } + auto it = waiters_.find(fd); + if (it == waiters_.end()) { + continue; + } + auto h = it->second; + waiters_.erase(it); + h.resume(); + } + } + } + +private: + Fd ep_; + int wake_fd_; + std::unordered_map> waiters_; + std::unordered_set armed_; +}; + +// Suspend the current coroutine until `fd` is readable. +struct ReadAwaitable { + Reactor& reactor; + int fd; + bool await_ready() const noexcept { return false; } + void await_suspend(std::coroutine_handle<> h) const { reactor.wait_readable(fd, h); } + void await_resume() const noexcept {} +}; + +// Pull at least one more byte into `buf`. false => EOF / error. +Task fill(Reactor& reactor, int fd, std::string& buf) { + for (;;) { + char tmp[4096]; + ssize_t n = ::recv(fd, tmp, sizeof tmp, 0); + if (n > 0) { + buf.append(tmp, static_cast(n)); + co_return true; + } + if (n == 0) { + co_return false; // peer closed + } + if (errno == EAGAIN || errno == EWOULDBLOCK) { + co_await ReadAwaitable{reactor, fd}; + continue; + } + if (errno == EINTR) { + continue; + } + co_return false; + } +} + +Detached connection(Reactor& reactor, Server& server, Fd conn) { + int fd = conn.get(); + std::string buf, nick; + bool have_nick = false; + bool alive = true; + while (alive) { + while (alive && buf.size() < kHeaderLen) { + if (!co_await fill(reactor, fd, buf)) { + alive = false; + } + } + if (!alive) { + break; + } + std::uint16_t len = (static_cast(buf[4]) << 8) | + static_cast(buf[5]); + while (alive && buf.size() < kHeaderLen + len) { + if (!co_await fill(reactor, fd, buf)) { + alive = false; + } + } + if (!alive) { + break; + } + std::string body = buf.substr(kHeaderLen, len); + buf.erase(0, kHeaderLen + len); + if (!have_nick) { + // First frame is JOIN: its payload is the nick. + nick = body; + have_nick = true; + server.add(fd, nick); + } else { + // Every later frame is MSG: its payload is the message text. + server.broadcast(fd, nick, body); + } + } + reactor.forget(fd); + if (have_nick) { + server.remove(fd); + } + co_return; // Fd destructor closes the socket +} + +Detached acceptor(Reactor& reactor, Server& server, int listen_fd) { + for (;;) { + co_await ReadAwaitable{reactor, listen_fd}; + for (;;) { + int cfd = ::accept4(listen_fd, nullptr, nullptr, SOCK_NONBLOCK | SOCK_CLOEXEC); + if (cfd < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + break; + } + if (errno == EINTR) { + continue; + } + break; + } + connection(reactor, server, Fd{cfd}); // detached; self-owning + } + } +} + +std::expected run_async(const std::string& host, std::uint16_t port) { + auto lfd = make_listener(host, port, /*nonblock=*/true); + if (!lfd) { + return std::unexpected(lfd.error()); + } + Fd listen = std::move(*lfd); + Fd wake{::eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC)}; + if (!wake) { + return std::unexpected(std::string("eventfd: ") + std::strerror(errno)); + } + g_wake_fd = wake.get(); + Fd ep{::epoll_create1(EPOLL_CLOEXEC)}; + if (!ep) { + return std::unexpected(std::string("epoll_create1: ") + std::strerror(errno)); + } + + std::println(stderr, "chatterd: listening on {}:{} engine=async", host, port); + + Reactor reactor{std::move(ep), wake.get()}; + Server server; + acceptor(reactor, server, listen.get()); // detached + reactor.run(); + return {}; +} + +// ========================================================================== +// loadtest client — drives the broadcast assertion. +// ========================================================================== +int run_loadtest(const std::string& host, std::uint16_t port, int nclients) { + std::vector receivers; + std::vector bufs(static_cast(nclients)); + receivers.reserve(static_cast(nclients)); + + for (int i = 0; i < nclients; ++i) { + auto c = connect_to(host, port); + if (!c) { + std::println(stderr, "loadtest: connect failed: {}", c.error()); + return 1; + } + struct timeval tv{5, 0}; + ::setsockopt(c->get(), SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + std::string hello = encode_frame(kTypeJoin, "r" + std::to_string(i)); + send_all(c->get(), hello); + receivers.push_back(std::move(*c)); + } + + auto sender = connect_to(host, port); + if (!sender) { + std::println(stderr, "loadtest: sender connect failed: {}", sender.error()); + return 1; + } + send_all(sender->get(), encode_frame(kTypeJoin, "sender")); // JOIN + + // Let the server register every client before the broadcast goes out. + std::this_thread::sleep_for(std::chrono::milliseconds(250)); + send_all(sender->get(), encode_frame(kTypeMsg, "hello world")); + + const std::string want_nick = "sender"; + const std::string want_text = "hello world"; + int delivered = 0; + for (int i = 0; i < nclients; ++i) { + Frame f; + bool ok = read_frame_blocking(receivers[i].get(), bufs[i], f) && f.type == kTypeDeliver; + if (ok) { + auto nul = f.body.find('\0'); + ok = nul != std::string::npos && f.body.substr(0, nul) == want_nick && + f.body.substr(nul + 1) == want_text; + } + if (ok) { + ++delivered; + } else { + std::println(stderr, "loadtest: client r{} missed the broadcast", i); + } + } + std::println("loadtest: delivered {}/{}", delivered, nclients); + return delivered == nclients ? 0 : 1; +} + +// -------------------------------------------------------------------------- +// CLI. +// -------------------------------------------------------------------------- +void usage() { + std::println(stderr, + "usage:\n" + " app serve [--engine thread|async] [--host H] [--port P]\n" + " app loadtest [--host H] [--port P] [--clients N]"); +} + +struct Opts { + std::string engine = "async"; + std::string host = "127.0.0.1"; + std::uint16_t port = 47100; + int clients = 20; +}; + +// Returns false on a malformed option. +bool parse_opts(int argc, char** argv, int start, Opts& o) { + for (int i = start; i < argc; ++i) { + std::string_view a = argv[i]; + const char* v = nullptr; + auto next = [&]() -> bool { + if (i + 1 >= argc) { + return false; + } + v = argv[++i]; + return true; + }; + if (a == "--engine" && next()) { + o.engine = v; + } else if (a == "--host" && next()) { + o.host = v; + } else if (a == "--port" && next()) { + o.port = static_cast(std::atoi(v)); + } else if (a == "--clients" && next()) { + o.clients = std::atoi(v); + } else { + return false; + } + } + return true; +} + +} // namespace + +int main(int argc, char** argv) { + install_signals(); + if (argc < 2) { + usage(); + return 2; + } + std::string cmd = argv[1]; + Opts o; + if (!parse_opts(argc, argv, 2, o)) { + usage(); + return 2; + } + + if (cmd == "serve") { + std::expected r; + if (o.engine == "thread") { + r = run_thread(o.host, o.port); + } else if (o.engine == "async") { + r = run_async(o.host, o.port); + } else { + std::println(stderr, "chatterd: unknown engine '{}' (want thread|async)", o.engine); + return 2; + } + if (!r) { + std::println(stderr, "chatterd: {}", r.error()); + return 1; + } + std::println(stderr, "chatterd: shutdown"); + return 0; + } + if (cmd == "loadtest") { + if (o.clients < 1 || o.clients > kMaxBody) { + std::println(stderr, "loadtest: --clients out of range"); + return 2; + } + return run_loadtest(o.host, o.port, o.clients); + } + usage(); + return 2; +} diff --git a/examples/27-async-runtimes-and-coroutines/demo.sh b/examples/27-async-runtimes-and-coroutines/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/27-async-runtimes-and-coroutines/go/demo.sh b/examples/27-async-runtimes-and-coroutines/go/demo.sh new file mode 100755 index 0000000..fabd403 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -race -o "$BIN" . # race detector on: goroutines are this chapter's async engine +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/27-async-runtimes-and-coroutines/go/go.mod b/examples/27-async-runtimes-and-coroutines/go/go.mod new file mode 100644 index 0000000..e9e463c --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/go/go.mod @@ -0,0 +1,7 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/27-async-runtimes-and-coroutines/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 diff --git a/examples/27-async-runtimes-and-coroutines/go/go.sum b/examples/27-async-runtimes-and-coroutines/go/go.sum new file mode 100644 index 0000000..37ee2d4 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/go/go.sum @@ -0,0 +1,2 @@ +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= diff --git a/examples/27-async-runtimes-and-coroutines/go/main.go b/examples/27-async-runtimes-and-coroutines/go/main.go new file mode 100644 index 0000000..101c7fb --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/go/main.go @@ -0,0 +1,389 @@ +// chatterd v4 (Go) — a peer-to-peer chat daemon. +// +// This is the async chapter, and Go's answer is unchanged from v1: goroutines +// already ARE the async model. `serve --engine async` and `--engine thread` +// select the very same goroutine-per-connection server (a hub goroutine fans +// out broadcasts over channels via select + context for shutdown). The engine +// flag exists only for parity with the C++/Rust versions, which rewrite their +// connection handling for this chapter; here there is nothing to rewrite. +// +// Wire format (canonical chatterd chat frame, v1..v4): +// +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// type: JOIN=1, MSG=2, DELIVER=3, WELCOME=4, PING=5. +// The client's first frame is JOIN (payload = its nick). Every later client +// frame is MSG (payload = message text). For a MSG with text T from the +// client whose nick is N, the server sends every OTHER client a DELIVER +// frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use +// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// the same frame and are simply never sent here. +// +// Usage: +// +// app serve [--engine thread|async] [--host H] [--port P] +// app loadtest [--host H] [--port P] [--clients N] +package main + +import ( + "bufio" + "context" + "encoding/binary" + "fmt" + "io" + "net" + "os" + "os/signal" + "strconv" + "strings" + "syscall" + "time" + + "golang.org/x/sys/unix" +) + +// --------------------------------------------------------------------------- +// Frame helpers — canonical chatterd chat frame (see file header). +// --------------------------------------------------------------------------- + +const ( + magic0 = 0x43 + magic1 = 0x48 + version = 0x01 + + typeJoin = 1 + typeMsg = 2 + typeDeliver = 3 +) + +func encodeFrame(typ byte, body string) []byte { + b := make([]byte, 6+len(body)) + b[0] = magic0 + b[1] = magic1 + b[2] = version + b[3] = typ + binary.BigEndian.PutUint16(b[4:6], uint16(len(body))) + copy(b[6:], body) + return b +} + +// readFrame returns the frame's type and payload. +func readFrame(r *bufio.Reader) (byte, string, error) { + var hdr [6]byte + if _, err := io.ReadFull(r, hdr[:]); err != nil { + return 0, "", err + } + typ := hdr[3] + n := binary.BigEndian.Uint16(hdr[4:6]) + buf := make([]byte, n) + if _, err := io.ReadFull(r, buf); err != nil { + return 0, "", fmt.Errorf("read body: %w", err) + } + return typ, string(buf), nil +} + +// --------------------------------------------------------------------------- +// Hub: owns the client set, fans out broadcasts over channels. +// --------------------------------------------------------------------------- + +type client struct { + conn net.Conn + out chan []byte +} + +type broadcastReq struct { + from *client + frame []byte +} + +type hub struct { + register chan *client + unregister chan *client + broadcast chan broadcastReq +} + +func newHub() *hub { + return &hub{ + register: make(chan *client), + unregister: make(chan *client), + broadcast: make(chan broadcastReq), + } +} + +func (h *hub) run(ctx context.Context) { + clients := make(map[*client]struct{}) + for { + select { + case c := <-h.register: + clients[c] = struct{}{} + case c := <-h.unregister: + if _, ok := clients[c]; ok { + delete(clients, c) + close(c.out) + } + case b := <-h.broadcast: + for c := range clients { + if c != b.from { + c.out <- b.frame // buffered; delivery is guaranteed + } + } + case <-ctx.Done(): + for c := range clients { + c.conn.Close() // unblock the reader goroutine + close(c.out) + } + return + } + } +} + +// handleConn reads frames from one client: the first (JOIN) is its nick, each +// later one (MSG) is a message to broadcast as a DELIVER frame. +func handleConn(ctx context.Context, h *hub, conn net.Conn) { + defer conn.Close() + c := &client{conn: conn, out: make(chan []byte, 64)} + r := bufio.NewReader(conn) + + // First frame is JOIN: its payload is the nick. + _, nick, err := readFrame(r) + if err != nil { + return + } + select { + case h.register <- c: + case <-ctx.Done(): + return + } + // Writer goroutine: drains c.out until the hub closes it. + go func() { + for frame := range c.out { + if _, err := conn.Write(frame); err != nil { + return + } + } + }() + + for { + // Every later frame is MSG: its payload is the message text. + _, body, err := readFrame(r) + if err != nil { + break + } + frame := encodeFrame(typeDeliver, nick+"\x00"+body) + select { + case h.broadcast <- broadcastReq{from: c, frame: frame}: + case <-ctx.Done(): + return + } + } + select { + case h.unregister <- c: + case <-ctx.Done(): + } +} + +// listen builds a TCP listener with SO_REUSEADDR set via x/sys/unix, so the +// daemon can rebind its port immediately after a restart. +func listen(ctx context.Context, host string, port int) (net.Listener, error) { + lc := net.ListenConfig{ + Control: func(network, address string, c syscall.RawConn) error { + var serr error + if err := c.Control(func(fd uintptr) { + serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1) + }); err != nil { + return err + } + return serr + }, + } + return lc.Listen(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port))) +} + +func runServe(engine, host string, port int) int { + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + ln, err := listen(ctx, host, port) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: listen: %v\n", err) + return 1 + } + + fmt.Fprintf(os.Stderr, "chatterd: listening on %s:%d engine=%s\n", host, port, engine) + + h := newHub() + go h.run(ctx) + + // Unblock Accept on shutdown. + go func() { + <-ctx.Done() + ln.Close() + }() + + for { + conn, err := ln.Accept() + if err != nil { + break // listener closed on shutdown + } + go handleConn(ctx, h, conn) + } + + fmt.Fprintln(os.Stderr, "chatterd: shutdown") + return 0 +} + +// --------------------------------------------------------------------------- +// loadtest client — drives the broadcast assertion. +// --------------------------------------------------------------------------- + +func runLoadtest(host string, port, nclients int) int { + addr := net.JoinHostPort(host, strconv.Itoa(port)) + receivers := make([]net.Conn, nclients) + readers := make([]*bufio.Reader, nclients) + + for i := 0; i < nclients; i++ { + c, err := net.Dial("tcp", addr) + if err != nil { + fmt.Fprintf(os.Stderr, "loadtest: connect failed: %v\n", err) + return 1 + } + receivers[i] = c + readers[i] = bufio.NewReader(c) + if _, err := c.Write(encodeFrame(typeJoin, fmt.Sprintf("r%d", i))); err != nil { + fmt.Fprintf(os.Stderr, "loadtest: hello failed: %v\n", err) + return 1 + } + } + + sender, err := net.Dial("tcp", addr) + if err != nil { + fmt.Fprintf(os.Stderr, "loadtest: sender connect failed: %v\n", err) + return 1 + } + sender.Write(encodeFrame(typeJoin, "sender")) + + // Let the server register every client before the broadcast goes out. + time.Sleep(250 * time.Millisecond) + sender.Write(encodeFrame(typeMsg, "hello world")) + + const wantNick = "sender" + const wantText = "hello world" + delivered := 0 + for i := 0; i < nclients; i++ { + receivers[i].SetReadDeadline(time.Now().Add(5 * time.Second)) + typ, body, err := readFrame(readers[i]) + ok := err == nil && typ == typeDeliver + if ok { + if idx := strings.IndexByte(body, 0); idx >= 0 { + ok = body[:idx] == wantNick && body[idx+1:] == wantText + } else { + ok = false + } + } + if ok { + delivered++ + } else { + fmt.Fprintf(os.Stderr, "loadtest: client r%d missed the broadcast\n", i) + } + } + fmt.Printf("loadtest: delivered %d/%d\n", delivered, nclients) + if delivered == nclients { + return 0 + } + return 1 +} + +// --------------------------------------------------------------------------- +// CLI. +// --------------------------------------------------------------------------- + +func usage() { + fmt.Fprintln(os.Stderr, "usage:") + fmt.Fprintln(os.Stderr, " app serve [--engine thread|async] [--host H] [--port P]") + fmt.Fprintln(os.Stderr, " app loadtest [--host H] [--port P] [--clients N]") +} + +type opts struct { + engine string + host string + port int + clients int +} + +// parseOpts consumes --key value pairs. Returns false on malformed input. +func parseOpts(args []string, o *opts) bool { + for i := 0; i < len(args); i++ { + next := func() (string, bool) { + if i+1 >= len(args) { + return "", false + } + i++ + return args[i], true + } + var v string + var ok bool + switch args[i] { + case "--engine": + if v, ok = next(); !ok { + return false + } + o.engine = v + case "--host": + if v, ok = next(); !ok { + return false + } + o.host = v + case "--port": + if v, ok = next(); !ok { + return false + } + n, err := strconv.Atoi(v) + if err != nil { + return false + } + o.port = n + case "--clients": + if v, ok = next(); !ok { + return false + } + n, err := strconv.Atoi(v) + if err != nil { + return false + } + o.clients = n + default: + return false + } + } + return true +} + +func main() { + if len(os.Args) < 2 { + usage() + os.Exit(2) + } + cmd := os.Args[1] + o := opts{engine: "async", host: "127.0.0.1", port: 47100, clients: 20} + if !parseOpts(os.Args[2:], &o) { + usage() + os.Exit(2) + } + + switch cmd { + case "serve": + if o.engine != "thread" && o.engine != "async" { + fmt.Fprintf(os.Stderr, "chatterd: unknown engine '%s' (want thread|async)\n", o.engine) + os.Exit(2) + } + os.Exit(runServe(o.engine, o.host, o.port)) + case "loadtest": + if o.clients < 1 || o.clients > 65535 { + fmt.Fprintln(os.Stderr, "loadtest: --clients out of range") + os.Exit(2) + } + os.Exit(runLoadtest(o.host, o.port, o.clients)) + default: + usage() + os.Exit(2) + } +} diff --git a/examples/27-async-runtimes-and-coroutines/rust/Cargo.lock b/examples/27-async-runtimes-and-coroutines/rust/Cargo.lock new file mode 100644 index 0000000..2b24371 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/rust/Cargo.lock @@ -0,0 +1,163 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "chatterd" +version = "0.1.0" +dependencies = [ + "signal-hook", + "tokio", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "mio" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "wasi", + "windows-sys", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "signal-hook" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2a0c28ca5908dbdbcd52e6fdaa00358ab88637f8ab33e1f188dd510eb44b53d" +dependencies = [ + "libc", + "signal-hook-registry", +] + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tokio" +version = "1.53.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d988bcd52dbe076d3d46903332f58c912b87a2c49b1428419a5845154762ffee" +dependencies = [ + "bytes", + "libc", + "mio", + "pin-project-lite", + "signal-hook-registry", + "socket2", + "tokio-macros", + "windows-sys", +] + +[[package]] +name = "tokio-macros" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6328af13490e73a9b4694030fafd93f8c8c6a9dede33e821c3fc63eddf8042ba" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] diff --git a/examples/27-async-runtimes-and-coroutines/rust/Cargo.toml b/examples/27-async-runtimes-and-coroutines/rust/Cargo.toml new file mode 100644 index 0000000..25f21c0 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/rust/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "chatterd" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +# This is the async chapter, so tokio is warranted for the async engine. +tokio = { version = "1.53", features = [ + "rt-multi-thread", + "net", + "io-util", + "sync", + "signal", + "macros", +] } +# std-only signal flag for the thread engine (no tokio runtime involved there). +signal-hook = "0.4.4" diff --git a/examples/27-async-runtimes-and-coroutines/rust/demo.sh b/examples/27-async-runtimes-and-coroutines/rust/demo.sh new file mode 100755 index 0000000..f799724 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" # exec so signals sent to this PID reach the app + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/27-async-runtimes-and-coroutines/rust/rust-toolchain.toml b/examples/27-async-runtimes-and-coroutines/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/27-async-runtimes-and-coroutines/rust/src/main.rs b/examples/27-async-runtimes-and-coroutines/rust/src/main.rs new file mode 100644 index 0000000..132b8b8 --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/rust/src/main.rs @@ -0,0 +1,442 @@ +// chatterd v4 (Rust) — a peer-to-peer chat daemon. +// +// The connection-handling engine is selectable: +// * thread : one std thread per client (the ch21 baseline; std::net + a +// signal-hook stop flag). +// * async : tokio — an async fn per connection on a tokio::net::TcpListener, +// spawned as tasks, broadcasting over per-client mpsc channels. +// +// Both engines speak the SAME length-prefixed frame protocol and are +// observably identical: a broadcast reaches every other connected client. +// std's TcpStream / TcpListener are the OwnedFd-backed RAII socket handles; a +// dropped stream closes its descriptor. +// +// Wire format (canonical chatterd chat frame, v1..v4): +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// type: JOIN=1, MSG=2, DELIVER=3, WELCOME=4, PING=5. +// The client's first frame is JOIN (payload = its nick). Every later client +// frame is MSG (payload = message text). For a MSG with text T from the +// client whose nick is N, the server sends every OTHER client a DELIVER +// frame whose payload is N + 0x00 (NUL) + T. This chapter's engines use +// only JOIN/MSG/DELIVER — WELCOME and PING are other versions' additions to +// the same frame and are simply never sent here. +// +// Usage: +// app serve [--engine thread|async] [--host H] [--port P] +// app loadtest [--host H] [--port P] [--clients N] + +use std::collections::HashMap; +use std::io::{self, Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::process::ExitCode; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::Duration; + +// --------------------------------------------------------------------------- +// Frame helpers — canonical chatterd chat frame (see file header). +// --------------------------------------------------------------------------- + +const HEADER_LEN: usize = 6; +const MAGIC: [u8; 2] = [0x43, 0x48]; +const VERSION: u8 = 0x01; +const TYPE_JOIN: u8 = 1; +const TYPE_MSG: u8 = 2; +const TYPE_DELIVER: u8 = 3; + +fn encode_frame(typ: u8, body: &str) -> Vec { + let len = body.len() as u16; + let mut out = Vec::with_capacity(HEADER_LEN + body.len()); + out.extend_from_slice(&MAGIC); + out.push(VERSION); + out.push(typ); + out.extend_from_slice(&len.to_be_bytes()); + out.extend_from_slice(body.as_bytes()); + out +} + +// Blocking frame read on a stream that has a read timeout set. Returns +// Ok((type, body)), or Err on EOF/timeout/error. +fn read_frame_conn(s: &mut TcpStream) -> io::Result<(u8, String)> { + let mut hdr = [0u8; HEADER_LEN]; + s.read_exact(&mut hdr)?; + let typ = hdr[3]; + let n = u16::from_be_bytes([hdr[4], hdr[5]]) as usize; + let mut buf = vec![0u8; n]; + s.read_exact(&mut buf)?; + Ok((typ, String::from_utf8_lossy(&buf).into_owned())) +} + +// ========================================================================== +// thread engine — one std thread per client. +// ========================================================================== + +type ThreadClients = Arc>>>>; + +enum Frame { + Msg(u8, String), + Closed, +} + +// Read one whole frame, honouring the stop flag between short read timeouts. +fn read_frame_std(reader: &mut TcpStream, buf: &mut Vec, stop: &AtomicBool) -> Frame { + loop { + if buf.len() >= HEADER_LEN { + let typ = buf[3]; + let n = u16::from_be_bytes([buf[4], buf[5]]) as usize; + if buf.len() >= HEADER_LEN + n { + let body = String::from_utf8_lossy(&buf[HEADER_LEN..HEADER_LEN + n]).into_owned(); + buf.drain(0..HEADER_LEN + n); + return Frame::Msg(typ, body); + } + } + if stop.load(Ordering::Relaxed) { + return Frame::Closed; + } + let mut tmp = [0u8; 4096]; + match reader.read(&mut tmp) { + Ok(0) => return Frame::Closed, + Ok(k) => buf.extend_from_slice(&tmp[..k]), + Err(e) + if e.kind() == io::ErrorKind::WouldBlock + || e.kind() == io::ErrorKind::TimedOut + || e.kind() == io::ErrorKind::Interrupted => + { + continue // re-check the stop flag + } + Err(_) => return Frame::Closed, + } + } +} + +fn serve_client( + id: u64, + mut reader: TcpStream, + write_handle: Arc>, + clients: ThreadClients, + stop: Arc, +) { + reader + .set_read_timeout(Some(Duration::from_millis(200))) + .ok(); + let mut buf = Vec::new(); + + // First frame is JOIN: its payload is the nick. + let nick = match read_frame_std(&mut reader, &mut buf, &stop) { + Frame::Msg(_typ, n) => n, + Frame::Closed => return, + }; + clients.lock().unwrap().insert(id, write_handle); + + loop { + match read_frame_std(&mut reader, &mut buf, &stop) { + // Every later frame is MSG: its payload is the message text. + Frame::Msg(_typ, body) => { + let frame = encode_frame(TYPE_DELIVER, &format!("{nick}\0{body}")); + let map = clients.lock().unwrap(); + for (cid, w) in map.iter() { + if *cid != id { + if let Ok(mut s) = w.lock() { + let _ = s.write_all(&frame); + } + } + } + } + Frame::Closed => break, + } + } + clients.lock().unwrap().remove(&id); +} + +fn run_thread(host: &str, port: u16) -> io::Result<()> { + let stop = Arc::new(AtomicBool::new(false)); + signal_hook::flag::register(signal_hook::consts::SIGINT, Arc::clone(&stop))?; + signal_hook::flag::register(signal_hook::consts::SIGTERM, Arc::clone(&stop))?; + + let listener = TcpListener::bind((host, port))?; + listener.set_nonblocking(true)?; + eprintln!("chatterd: listening on {host}:{port} engine=thread"); + + let clients: ThreadClients = Arc::new(Mutex::new(HashMap::new())); + let mut workers = Vec::new(); + let mut next_id = 0u64; + + while !stop.load(Ordering::Relaxed) { + match listener.accept() { + Ok((stream, _)) => { + let reader = stream.try_clone()?; + let write_handle = Arc::new(Mutex::new(stream)); + let id = next_id; + next_id += 1; + let clients = Arc::clone(&clients); + let stop = Arc::clone(&stop); + workers.push(thread::spawn(move || { + serve_client(id, reader, write_handle, clients, stop); + })); + } + Err(e) if e.kind() == io::ErrorKind::WouldBlock => { + thread::sleep(Duration::from_millis(50)); + } + Err(e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(_) => break, + } + } + + for w in workers { + let _ = w.join(); + } + eprintln!("chatterd: shutdown"); + Ok(()) +} + +// ========================================================================== +// async engine — tokio. +// ========================================================================== + +mod async_engine { + use super::{encode_frame, HEADER_LEN, TYPE_DELIVER}; + use std::collections::HashMap; + use std::io; + use std::sync::{Arc, Mutex}; + use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt, BufReader}; + use tokio::net::TcpListener; + use tokio::net::TcpStream; + use tokio::signal::unix::{signal, SignalKind}; + use tokio::sync::{broadcast, mpsc}; + + type Clients = Arc>>>>; + + // Returns (type, payload). + async fn read_frame(r: &mut R) -> io::Result<(u8, String)> { + let mut hdr = [0u8; HEADER_LEN]; + r.read_exact(&mut hdr).await?; + let typ = hdr[3]; + let n = u16::from_be_bytes([hdr[4], hdr[5]]) as usize; + let mut buf = vec![0u8; n]; + r.read_exact(&mut buf).await?; + Ok((typ, String::from_utf8_lossy(&buf).into_owned())) + } + + async fn handle_conn( + id: u64, + sock: TcpStream, + clients: Clients, + mut shutdown: broadcast::Receiver<()>, + ) { + let (rd, mut wr) = sock.into_split(); + let mut rd = BufReader::new(rd); + + // First frame is JOIN: its payload is the nick. + let nick = match read_frame(&mut rd).await { + Ok((_typ, n)) => n, + Err(_) => return, + }; + + let (tx, mut rx) = mpsc::unbounded_channel::>(); + clients.lock().unwrap().insert(id, tx); + + // Writer task: drain this client's queue onto the socket. + let writer = tokio::spawn(async move { + while let Some(frame) = rx.recv().await { + if wr.write_all(&frame).await.is_err() { + break; + } + } + }); + + loop { + tokio::select! { + // Every later frame is MSG: its payload is the message text. + r = read_frame(&mut rd) => match r { + Ok((_typ, body)) => { + let frame = encode_frame(TYPE_DELIVER, &format!("{nick}\0{body}")); + let map = clients.lock().unwrap(); + for (cid, tx) in map.iter() { + if *cid != id { + let _ = tx.send(frame.clone()); + } + } + } + Err(_) => break, + }, + _ = shutdown.recv() => break, + } + } + + clients.lock().unwrap().remove(&id); + writer.abort(); + } + + async fn serve(host: String, port: u16) -> io::Result<()> { + let listener = TcpListener::bind((host.as_str(), port)).await?; + eprintln!("chatterd: listening on {host}:{port} engine=async"); + + let clients: Clients = Arc::new(Mutex::new(HashMap::new())); + let (shutdown_tx, _) = broadcast::channel::<()>(1); + let mut sigint = signal(SignalKind::interrupt())?; + let mut sigterm = signal(SignalKind::terminate())?; + let mut next_id = 0u64; + + loop { + tokio::select! { + accepted = listener.accept() => { + let (sock, _) = accepted?; + let id = next_id; + next_id += 1; + tokio::spawn(handle_conn( + id, sock, Arc::clone(&clients), shutdown_tx.subscribe(), + )); + } + _ = sigint.recv() => break, + _ = sigterm.recv() => break, + } + } + + let _ = shutdown_tx.send(()); + eprintln!("chatterd: shutdown"); + Ok(()) + } + + pub fn run(host: &str, port: u16) -> io::Result<()> { + let rt = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build()?; + rt.block_on(serve(host.to_string(), port)) + } +} + +// ========================================================================== +// loadtest client — drives the broadcast assertion. +// ========================================================================== + +fn run_loadtest(host: &str, port: u16, nclients: usize) -> io::Result { + let addr = format!("{host}:{port}"); + let mut receivers: Vec = Vec::with_capacity(nclients); + + for i in 0..nclients { + let mut s = TcpStream::connect(&addr)?; + s.write_all(&encode_frame(TYPE_JOIN, &format!("r{i}")))?; + s.set_read_timeout(Some(Duration::from_secs(5)))?; + receivers.push(s); + } + + let mut sender = TcpStream::connect(&addr)?; + sender.write_all(&encode_frame(TYPE_JOIN, "sender"))?; // JOIN + + // Let the server register every client before the broadcast goes out. + thread::sleep(Duration::from_millis(250)); + sender.write_all(&encode_frame(TYPE_MSG, "hello world"))?; + + let want_nick = "sender"; + let want_text = "hello world"; + let mut delivered = 0; + for (i, s) in receivers.iter_mut().enumerate() { + let ok = match read_frame_conn(s) { + Ok((typ, body)) if typ == TYPE_DELIVER => match body.split_once('\0') { + Some((n, t)) => n == want_nick && t == want_text, + None => false, + }, + _ => false, + }; + if ok { + delivered += 1; + } else { + eprintln!("loadtest: client r{i} missed the broadcast"); + } + } + println!("loadtest: delivered {delivered}/{nclients}"); + Ok(if delivered == nclients { 0 } else { 1 }) +} + +// --------------------------------------------------------------------------- +// CLI. +// --------------------------------------------------------------------------- + +fn usage() { + eprintln!("usage:"); + eprintln!(" app serve [--engine thread|async] [--host H] [--port P]"); + eprintln!(" app loadtest [--host H] [--port P] [--clients N]"); +} + +struct Opts { + engine: String, + host: String, + port: u16, + clients: usize, +} + +// Returns None on malformed input. +fn parse_opts(args: &[String]) -> Option { + let mut o = Opts { + engine: "async".to_string(), + host: "127.0.0.1".to_string(), + port: 47100, + clients: 20, + }; + let mut i = 0; + while i < args.len() { + let key = &args[i]; + let val = args.get(i + 1)?; + match key.as_str() { + "--engine" => o.engine = val.clone(), + "--host" => o.host = val.clone(), + "--port" => o.port = val.parse().ok()?, + "--clients" => o.clients = val.parse().ok()?, + _ => return None, + } + i += 2; + } + Some(o) +} + +fn main() -> ExitCode { + let args: Vec = std::env::args().collect(); + if args.len() < 2 { + usage(); + return ExitCode::from(2); + } + let cmd = args[1].as_str(); + let opts = match parse_opts(&args[2..]) { + Some(o) => o, + None => { + usage(); + return ExitCode::from(2); + } + }; + + match cmd { + "serve" => { + let r = match opts.engine.as_str() { + "thread" => run_thread(&opts.host, opts.port), + "async" => async_engine::run(&opts.host, opts.port), + other => { + eprintln!("chatterd: unknown engine '{other}' (want thread|async)"); + return ExitCode::from(2); + } + }; + if let Err(e) = r { + eprintln!("chatterd: {e}"); + return ExitCode::from(1); + } + ExitCode::SUCCESS + } + "loadtest" => { + if opts.clients < 1 || opts.clients > 65535 { + eprintln!("loadtest: --clients out of range"); + return ExitCode::from(2); + } + match run_loadtest(&opts.host, opts.port, opts.clients) { + Ok(code) => ExitCode::from(code as u8), + Err(e) => { + eprintln!("loadtest: {e}"); + ExitCode::from(1) + } + } + } + _ => { + usage(); + ExitCode::from(2) + } + } +} diff --git a/examples/27-async-runtimes-and-coroutines/verify.lua b/examples/27-async-runtimes-and-coroutines/verify.lua new file mode 100644 index 0000000..3e162df --- /dev/null +++ b/examples/27-async-runtimes-and-coroutines/verify.lua @@ -0,0 +1,99 @@ +-- Verify chatterd v4 for LSP_LANG: the async engine (C++ coroutines / tokio / +-- Go goroutines) delivers a broadcast to 20 clients, the thread engine still +-- delivers it too (prior engine kept working), the daemon shuts down cleanly on +-- SIGTERM, and the CLI contract holds. Invoked from the example directory with +-- LSP_LANG (cpp|go|rust) and REPO_ROOT set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- Scratch area for the background-daemon driver script. +local tmp = checks.run("mktemp -d").out:gsub("%s+$", "") +if tmp == "" or not tmp:match("^/") then + checks.skip("mktemp -d failed") +end +local function cleanup() + os.execute("rm -rf '" .. tmp .. "'") +end + +-- --------------------------------------------------------------------------- +-- Driver: start chatterd in the background on a chosen engine/port, wait for +-- the port, run the 20-client loadtest, then SIGTERM the daemon and report the +-- daemon's exit code and stderr. All output is echoed on stdout so a single +-- checks.run() captures the whole transcript. Runs under bash for /dev/tcp. +-- --------------------------------------------------------------------------- +local driver = tmp .. "/drive.sh" +do + local f = assert(io.open(driver, "w")) + f:write([[ +#!/usr/bin/env bash +set -u +LANG_SEL="$1"; ENGINE="$2"; PORT="$3" +SRVLOG="$(mktemp)" +./demo.sh "$LANG_SEL" run serve --engine "$ENGINE" --port "$PORT" >/dev/null 2>"$SRVLOG" & +sv=$! +up=0 +for _ in $(seq 1 200); do + if (exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null; then + exec 3>&- 3<&-; up=1; break + fi + sleep 0.05 +done +[ "$up" = 1 ] || echo "SERVER_NEVER_UP" +./demo.sh "$LANG_SEL" run loadtest --port "$PORT" --clients 20 +echo "LOADTEST_EXIT=$?" +kill -TERM "$sv" 2>/dev/null +wait "$sv"; echo "SERVER_EXIT=$?" +echo "----SERVERLOG----" +cat "$SRVLOG" +rm -f "$SRVLOG" +exit 0 +]]) + f:close() +end + +-- Run one engine's broadcast test and assert the observable outcome. +local function broadcast_test(engine, port) + local r = checks.run("bash " .. driver .. " " .. lang .. " " .. engine .. " " .. port) + local tag = lang .. "/" .. engine .. ": " + checks.expect_match(r.out, "chatterd: listening on 127%.0%.0%.1:" .. port .. + " engine=" .. engine, tag .. "daemon announces the listening port and engine") + checks.expect_match(r.out, "loadtest: delivered 20/20", + tag .. "broadcast reaches all 20 clients") + checks.expect_match(r.out, "LOADTEST_EXIT=0", tag .. "loadtest exits 0") + checks.expect_match(r.out, "SERVER_EXIT=0", tag .. "daemon exits 0 on SIGTERM") + checks.expect_match(r.out, "chatterd: shutdown", tag .. "daemon logs a clean shutdown") + if not r.out:find("delivered 20/20") then + print(r.out) -- surface the transcript on failure + end +end + +-- 1. The async engine (this chapter's rewrite) delivers to all 20 clients. +broadcast_test("async", 47180) + +-- 2. The prior thread engine still delivers identically. +broadcast_test("thread", 47181) + +-- --------------------------------------------------------------------------- +-- 3. CLI contract: no subcommand -> usage on stderr, exit 2. +-- --------------------------------------------------------------------------- +local u = checks.run(demo) +checks.expect_exit(u, 2, lang .. ": missing subcommand exits 2") +checks.expect_match(u.out, "usage:", lang .. ": missing subcommand prints usage") + +-- 4. Unknown engine -> exit 2 with a diagnostic (no port bound). +local e = checks.run(demo .. " serve --engine bogus") +checks.expect_exit(e, 2, lang .. ": unknown engine exits 2") +checks.expect_match(e.out, "unknown engine 'bogus'", lang .. ": unknown engine is diagnosed") + +cleanup() +checks.finish() diff --git a/examples/manifest.yaml b/examples/manifest.yaml index 52d0b3e..94366b5 100644 --- a/examples/manifest.yaml +++ b/examples/manifest.yaml @@ -85,3 +85,31 @@ examples: langs: [cpp, go, rust] mode: local timeout: 300 + - id: 21-tcp-sockets + langs: [cpp, go, rust] + mode: local + timeout: 240 + - id: 22-scaling-the-server + langs: [cpp, go, rust] + mode: local + timeout: 300 + - id: 23-udp-and-peer-discovery + langs: [cpp, go, rust] + mode: vm-peer + timeout: 360 + - id: 24-time-timers-deadlines + langs: [cpp, go, rust] + mode: local + timeout: 240 + - id: 25-shared-state-and-the-futex + langs: [cpp, go, rust] + mode: local + timeout: 300 + - id: 26-atomics-and-lock-free + langs: [cpp, go, rust] + mode: local + timeout: 300 + - id: 27-async-runtimes-and-coroutines + langs: [cpp, go, rust] + mode: local + timeout: 360