Willow Client (Experimental Work in Progress)

Warning

Willow Client is highly experimental, incomplete, and under active development. It is not ready for production use; expect breaking changes and unfinished behavior.

Willow Client is a Linux desktop push-to-talk client for the Willow ecosystem. The current vertical slice:

• presents a draggable, always-on-top GPUI overlay on Wayland;
• captures the default PipeWire microphone as 16 kHz mono signed 16-bit little-endian PCM;
• discovers Willow Application Server (WAS) from the system DNS search list;
• obtains the Willow Inference Server (WIS) URL from WAS over a WebSocket;
• posts the captured PCM directly to WIS over HTTP or HTTPS;
• forwards the complete typed WIS result to WAS as a Willow endpoint command;
• displays the WIS transcript and WAS endpoint response; and
• applies WAS's configured None, Chimes, or TTS response-audio policy through PipeWire;
• presents WAS notifications, plays their optional remote WAV audio, and reports notify_done; and
• handles WAS identify/locate requests with a visible indication and repeated local chime.

The client also exposes a StatusNotifierItem tray icon with Show Willow and Quit actions.

Workspace architecture

The repository is a Cargo workspace whose boundaries follow the code's actual portability:

• willow-client-core is Rust 1.88 no_std + alloc. It owns Willow wire construction, validated transport-endpoint interpretation, response selection, streaming HTTP framing, RFC 6455 upgrade and frames, fragmented-message limits, and runtime-independent WAS session state with fixed-size JSON scratch. Callers inject streams, limits, cancellation checkpoints, upgrade nonce entropy, frame-mask entropy, and optionally reusable serialization storage.
• willow-client-audio is a separate host-support crate for whole-file PCM16 WAV decoding, playback buffers, and generated response chimes. Native and the compatibility testkit share it; firmware transport does not depend on it.
• willow-client-native owns Linux adapters: PipeWire capture/playback, edge-net desktop sockets and Rustls, DNS, timers, randomness, WAS discovery, XDG state, and the StatusNotifierItem tray service.
• willow-client-testkit defines deterministic PCM fixtures, scenarios, client-visible observations, and ordered assertions shared by Python WAS and WAS-RS compatibility runs.
• willow-desktop is the GPUI composition root and owns its PTT interaction state. The installed binary remains named willow-client.
• xtask compat runs PTT and server-initiated scenarios through the production native network worker. Its common WAS control contract uses the managed test WAS and test Home Assistant today; WAS-RS must expose the same behavior to run the identical PTT, server-event, and firmware-protocol scenarios.

The firmware remains in its own repository. It consumes a pinned core revision; there is no submodule and no third umbrella Rust repository.

Runtime flow

On startup, the client validates its PipeWire format and connects to WAS. PTT remains disabled until PipeWire is usable and WAS has supplied a valid WIS URL.

Hold HOLD TO TALK to activate microphone capture. Releasing the button stops capture, sends the resulting PCM to WIS, and forwards WIS's response to WAS. Capture is bounded to 30 seconds. Moving the pointer off the PTT button also releases it.

Starting another capture cancels an in-flight TTS download and active response playback. TTS work runs independently of the WAS WebSocket receive loop, so a slow inference server cannot prevent WebSocket ping handling.

WAS notifications replace active response playback and appear in the overlay until dismissed, cancelled by WAS, or their audio finishes. Notification audio honors the configured repeat count and volume, with one second between repeats. Completion sends the typed notify_done event; deployed Python WAS then sends the matching cancellation. Identify/locate uses the same visible overlay but a local five-repeat chime and does not impersonate a server notification ID.

Drag the Willow header to reposition the overlay. The layer-shell surface is initialized with the selected output's reported size, while its Wayland input region is restricted to the visible 440×280 panel; the transparent remainder is click-through. During a drag the input region temporarily expands to the whole surface so pointer tracking can reach every edge. The message area wraps multi-line WIS transcripts.

The overlay position persists as XDG application state in $XDG_STATE_HOME/willow-client/overlay-position.json, falling back to $HOME/.local/state/willow-client/overlay-position.json. The file records logical coordinates and the logical output size, is replaced atomically after a completed drag and during orderly shutdown, and is ignored if malformed. When the output size changes, restoration preserves the distance from the nearest horizontal and vertical edges and clamps the panel to the new output. This keeps the state independent of KDE, GNOME, or another compositor's own configuration format.

GPUI's primary-display query is unavailable on Wayland, so the client waits for output-registry events and then uses the first enumerated Wayland output. The pinned GPUI revision calculates those output bounds with wl_output's integer scale even when wp_fractional_scale_v1 later supplies a fractional surface scale. The client corrects the layer size after that event so, for example, a KWin output at 150% uses its full logical dimensions. It compares the desired size with the live viewport on every resize event so a compositor reset after display power saving is corrected again. Panel layout and drag constraints always use the corrected logical geometry, preventing transient integer-scaled layer resets—such as those caused by opening KRunner—from moving the panel. Starting with a real nonzero size also keeps GPUI's Viewporter requests valid. This preserves layer-shell always-on-top behavior without a GPUI fork.

Runtime diagnostics use structured tracing events. The process-wide formatter emits UTC wall-clock timestamps with microsecond precision together with severity, target, thread name, message, and event fields; ANSI escapes are disabled so redirected logs remain machine-readable.

WAS discovery and transport

By default, the client reads search domains from /etc/resolv.conf and tries was.<search-domain> in order, followed by the resolver's short was name. For example:

search home.example internal.example

produces was.home.example, was.internal.example, then was. The secure endpoint is /ws on port 443.

Two environment variables support development and unusual resolver setups:

• WILLOW_SEARCH_DOMAIN overrides the resolver search list. It accepts whitespace- or comma-separated domains.
• WILLOW_WAS_URL bypasses discovery with an explicit wss://host[:port]/ws or ws://host[:port]/ws URL.

Discovery uses WSS by default. An explicit WAS URL may use WSS or plaintext WS; WS is intended for trusted local networks and development because it provides neither transport encryption nor server authentication. The client sends Willow's typed get_config command. The WIS URL returned by WAS may use HTTP or HTTPS and must use the /api/willow path.

Portable WAS, WIS, and TTS protocol handling lives in willow-client-core and uses edge-http, edge-ws, edge-nal, and embedded-io-async with default features disabled. Core consumes the device-compatible parser and validated WAS/WIS URL types from willow-definitions, producing a common endpoint view with scheme, host, port, Host authority, and origin-form request target for native and firmware. It accepts connected async streams and owns HTTP framing, streamed bodies, RFC 6455 upgrade validation, masked frames, fragmentation, control messages, bounded lifecycle serialization, and the WAS lifecycle. On desktop, edge-nal-std provides DNS and plain TCP while a small adapter supplies Rustls TLS. The native network service adds async-executor/futures-lite scheduling, timers, randomness, and channels; core selects none of them. See docs/TRANSPORT-ADAPTERS.md for the adapter API, ESP-IDF sketch, and remaining firmware blockers.

edge-http currently implements HTTP/1.1. The TLS connector therefore advertises only http/1.1; advertising HTTP/2 without an HTTP/2 implementation would let the server select a protocol the client cannot speak. WAS uses the RFC 6455 HTTP/1.1 Upgrade handshake. HTTP/2 or HTTP/3 WebSockets would require an extended-CONNECT-capable transport and are deliberately outside this prototype.

All TLS connections use Rustls and the operating system's certificate trust policy through rustls-platform-verifier. The dependency graph contains no OpenSSL TLS implementation; the transitive openssl-probe crate only locates Linux CA files.

WIS audio contract

The WIS request is an HTTP(S) POST to /api/willow with the raw PCM bytes as its body and these deployed Willow headers:

x-audio-sample-rate: 16000
x-audio-bits: 16
x-audio-channel: 1
x-audio-codec: pcm

Credentials embedded in the WIS URL are removed from the request URL and sent as HTTP Basic authorization. The response must be HTTP 200 JSON and is bounded to 64 KiB. Willow protocol types come from willow-protocol at the pinned willow-definitions revision. Properties absent from that pinned protocol are rejected instead of being accepted as untyped JSON.

Response audio

The client retains audio_response_type, speaker_volume, wis_tts_url, and wis_tts_url_v2 from the typed WAS configuration. Its selection behavior matches the Rust firmware:

• absent or explicit None response mode remains silent;
• Chimes plays a short locally generated success or error indication; and
• TTS speaks WAS's nonempty speech, falling back to Success or Error.

The v2 TTS URL template takes precedence. Legacy URLs receive the deployed format=WAV, speaker=CLB, and text query arguments. Response text is percent-encoded as a query value. URLs are intentionally omitted from debug output because configuration may include user information or access tokens.

TTS may use HTTP or HTTPS through the same core streaming transport. The desktop adapter currently bounds downloads to 16 MiB and then hands the whole file to willow-client-audio for mono or stereo uncompressed PCM16 WAV decoding. The decoder supports both finalized WAV assets and Python WIS's streamed form, whose initial header declares a zero-length data chunk and is followed by PCM. PipeWire receives the source sample rate and performs any device resampling. Configured speaker volume is clamped to 0–100 and applied digitally without floating-point work in the playback callback. Firmware can instead consume response chunks directly without depending on the host audio crate.

Platform scope

The client is Linux-only, PipeWire-only, and currently targets Wayland. It is initially tested against KDE Plasma. The overlay uses zwlr_layer_shell_v1; ordinary Wayland windows have no portable always-on-top hint.

The tray uses the D-Bus StatusNotifierItem protocol and the Willow SVG mark in apps/willow-desktop/assets/willow-client.svg, copied from willow-application-server-ui/public/static/favicon.svg. Plasma supports StatusNotifierItem natively, while GNOME generally needs an AppIndicator/StatusNotifier extension. There is no XDG Desktop Portal tray API.

Build and run

The repository pins Rust and the reviewed GPUI revision. Native development packages are required for PipeWire, Wayland, Vulkan, font rendering, and GPUI's bindgen use. On Gentoo, install the corresponding development-enabled packages; on binary distributions, package names commonly include libpipewire-0.3-dev, Wayland/XKB development headers, Vulkan loader/driver development files, and Clang.

cargo build --workspace --locked
cargo run --package willow-desktop --locked

Run from an active Wayland desktop session with XDG_RUNTIME_DIR, WAYLAND_DISPLAY, and DBUS_SESSION_BUS_ADDRESS available.

Verification

The normal gate is:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked
cargo test --workspace --all-targets --locked
cargo doc --workspace --no-deps --locked --document-private-items

The workspace manifest denies Rust warnings and enables Clippy's all and pedantic groups at deny, so no command-line lint flags are needed. The managed Python WAS acceptance command is documented in docs/COMPATIBILITY-TESTING.md. It requires a reviewed spoken PCM fixture and expected WIS and Home Assistant results; the managed deployment itself is not created or destroyed by the test runner.

The portable crate can additionally be checked without std, using its MSRV:

cargo +1.88.0 check --package willow-client-core \
  --target thumbv7em-none-eabi --locked

The tests include a local Rustls WSS server and plain HTTP WIS server that use the shared testkit contract to verify the complete WAS → WIS → WAS exchange, including exact PCM bytes. A second typed WSS scenario verifies identify plus the notification delivery → notify_done → cancellation lifecycle. A third scenario verifies config/NVS typed persistence round-trips, restart intent, exact OTA URL dispatch, and the complete wake arbitration exchange without performing any host restart or firmware download.

One ignored test requires a live desktop PipeWire graph. It negotiates the production format and performs an actual short capture:

cargo test --locked \
  audio::tests::live_pipewire_negotiates_and_captures_wis_pcm \
  -- --ignored --exact --nocapture

Deliberate boundaries

• No desktop wake-word engine yet. Espressif ESP-SR wake-word binaries cannot be reused on desktop.
• No ALSA or PulseAudio compatibility backend.
• No acoustic echo cancellation or simultaneous capture/playback yet.
• No packaging, autostart, global shortcut, or X11 fallback yet.
• No WAS or WIS implementation is embedded in this repository.

See docs/DEFERRED-SHARING.md for the current shared-code boundary.