Embedded client hardening: define and close the ESP32-S3 production gaps

[jason-shen]

Problem

The ESP32-S3 firmware at streamcoreai/esp32 connects to this server over WHIP and works on the bench. It is listed as not production-ready, and the roadmap says so without saying what that means, which makes it impossible for anyone else to help close the gap.

The firmware is a Rust SDK over Espressif's native esp_peer C library, running on ESP-IDF/FreeRTOS with no tokio and no full OS networking. It already has a fair surface: Agent handle with connect/disconnect, Capturer/Renderer traits with I2S default implementations, callbacks for state, transcripts, responses and errors, RPC registration so the agent can call device tools, plus camera, display and AFE modules.

What "not production-ready" needs to become is a list of specific, checkable gaps. From the code as it stands, the candidates are:

• Recovery. The server ships an ICE-restart-then-resume ladder and every full SDK implements it. The ESP32 client is the one most likely to lose its network — battery, wifi, a device carried around a building — and the least able to recover by having the user click reconnect. If src/whip.rs does not implement PATCH ICE restart and resume redial, it is the highest-value gap on this list.
• Memory. No documented heap or stack headroom figure. On an S3 with PSRAM the WebRTC stack plus Opus plus an audio pipeline is tight, and "works until it doesn't" is the failure mode. A soak test with a high-water mark is what turns this from a worry into a number.
• Audio path. AEC/AGC/NS behaviour through the AFE pipeline, and how it interacts with server-side barge-in. Confirmed working with a specific mic and speaker pair, or documented as the caller's problem.
• Long-run stability. Nothing establishes that a device stays connected for 24 hours, or what it does after the wifi AP reboots.
• TLS and credentials. Certificate handling for https:// WHIP, and where a token lives on a device that a user can physically hold.
• Provisioning and OTA. Wifi credentials are env!() macros in the example, meaning they are baked into the binary at build time. Fine for a demo, not shippable.
• Board coverage. Which boards are actually verified, with pin maps, rather than one board and a hope.

Proposed change

Turn the list above into a checklist in the esp32 repo README, verify each item, and fix or document what fails. Concretely, for this repo's part:

• A hardening tracking issue in streamcoreai/esp32 with one checkbox per gap, so contributors can take one.
• The recovery ladder implemented and tested against a real drop (pull the AP, wait, restore), matching the behaviour documented in docs/protocol.md.
• A published soak result: N hours connected, heap high-water mark, reconnects survived.
• A verified-boards table with pin maps.
• Wifi provisioning that is not compile-time, and a documented OTA path.
• docs/ in this repo linking to that status honestly, so the capability table stays accurate either way.

This is a large, multi-PR effort and splitting it is expected. The immediate deliverable is the checklist and the verified answers — a firmware that says "24h soak, 41 KB free heap at the low-water mark, survives AP reboot" is production-ready in the only sense that matters, and one that cannot answer those questions is not, regardless of how it behaves on a desk.

Acceptance criteria

• Tracking issue in streamcoreai/esp32 enumerating the gaps, each independently claimable.
• ICE restart and resume implemented in the firmware and verified against a real network drop.
• Published soak result with heap high-water mark.
• Verified-boards table with pin maps.
• Runtime wifi provisioning, and a documented OTA story.
• docs/capabilities.md and the roadmap reflect the verified state, with zh-CN mirrors updated.

Pointers

• In streamcoreai/esp32: src/whip.rs and src/webrtc.rs — signalling and the peer, where recovery lands
• src/afe_pipeline.rs, src/audio_processing.rs — AEC/AGC/NS
• src/wifi.rs — connect_sta, currently fed from env!() at build time
• docs/protocol.md — the recovery ladder to match
• Related: VAD needs an echo reference on the SIP/PCMU path (agent barges on its own voice) #48 (barge-in without an echo reference — same class of problem on a different transport)

[jason-shen]

Audited the firmware and broke this down into concrete issues. Tracking checklist with the ordering and dependencies: streamcoreai/esp32#12

The gaps this issue guessed at are mostly real, and the audit turned up two that were not on the list:

• No HTTPS at all (No HTTPS: no CA bundle attached, so the API key and JWT go over the wire in cleartext esp32#1). src/whip.rs never attaches a CA bundle, so any https:// endpoint fails the handshake and .env.example uses http:// for both WHIP and /token. The API key and the JWT it returns travel in cleartext on every connect.
• No OTA slots (No OTA: the partition table has a single factory slot, so fielded devices can never be updated esp32#6). partitions.csv has a single factory app partition. Every other fix on the list reaches a fielded device only over USB, and the partition table itself cannot be changed without a serial flash — so this one cannot be retrofitted to anything already handed out.

On the items this issue did list:

• Recovery (No connection recovery: no ICE restart, no resume, no retry esp32#2) — confirmed absent. src/whip.rs implements POST and DELETE only; no PATCH, no resume token. On failure the worker reports Failed and blocks waiting for the app to call connect() again, which opens a fresh session and replays the greeting.
• Audio (Barge-in is impossible: the mic is gated during playback and AEC is disabled esp32#5) — worse than "AEC behaviour unconfirmed". AEC is compiled in but switched off: src/capture.rs:138 passes has_reference: false, and voiceagent_afe.c:123 gates aec_init on it. The workaround is a half-duplex mic gate at src/agent.rs:625 that drops mic frames for the whole TTS utterance, so barge-in cannot work on device at all. Same root cause as VAD needs an echo reference on the SIP/PCMU path (agent barges on its own voice) #48, different transport.
• Memory (No memory headroom figure and no soak result esp32#9) — no instrumentation anywhere; grep free_heap over src/ is empty. Thread stacks (32K/16K/32K) are unmeasured and the task watchdog is at 30 s.
• Provisioning (WiFi credentials and API key are compiled into the binary — no runtime provisioning esp32#7), boards (No verified-board table, pin maps, or flashing guide esp32#10) — as described. The README quick start has three commented-out pin placeholders and there is no board table.

Also filed: STUN-only ICE with no TURN (streamcoreai/esp32#4), no CI or tests at all (streamcoreai/esp32#8), and crate metadata pointing at a 404 (streamcoreai/esp32#11).