Skip to content

fix(esp): raise loopTask stack so the OTA manifest TLS handshake fits - #277

Merged
cofade merged 4 commits into
mainfrom
fix/276-loop-task-stack-tls-panic
Sep 1, 2026
Merged

fix(esp): raise loopTask stack so the OTA manifest TLS handshake fits#277
cofade merged 4 commits into
mainfrom
fix/276-loop-task-stack-tls-panic

Conversation

@cofade

@cofade cofade commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Addresses #276 — the one-line fix is verified on hardware, but the issue's follow-up questions (a measured stack bound, and whether setup()'s TLS call sites should share a task with an explicit stack) are deliberately not answered here, so this should not auto-close the issue.

What was broken

Current main firmware could not complete setup() on real hardware. A field module joined Wi-Fi, printed an IP, then panicked and rebooted — a ~5.7 s crash loop — so it never reached registration, heartbeat, geolocation or capture, and never appeared on the dashboard.

[WIFI] (re)connected, IP: 192.168.178.120
[STAGE] arduino_ota_begin took=15ms
Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception).
Debug exception reason: Stack canary watchpoint triggered (loopTask)

Next boot: [BOOT] last_stage_before_reboot=ota:manifest_fetch.

setup() runs on the Arduino core's loopTask, whose stack is 8192 bytes by default (ARDUINO_LOOP_STACK_SIZE, in the core's main.cpp — a default that no repo file configures or references). That is not enough for an mbedTLS handshake that parses and verifies a certificate chain against the pinned ISRG Root X1, which is what ota.cpp's httpOtaCheckAndApply does. It dies inside the handshake, before the manifest's status line is read — which reads as a network failure and is not one.

Neither build path raised the default, so the merged firmware.bin the setup wizard flashes had the same defect as the PlatformIO build.

The change

One line at file scope in ESP32-CAM.ino, plus a comment explaining the budget:

SET_LOOP_TASK_STACK_SIZE(16384);

Verification (hardware, not just CI)

ESP32-CAM on COM5, module hive-01 / b0696ef23a08, built without DEV_SERVER_HOST so it targets production.

Before: stack canary panic on every boot, no exceptions, across many reboots.

After — same board, same config, whole path green against highfive.schutera.com:

[OTA] manifest HTTP 404                 <- #275, now survived instead of fatal
[STAGE] http_ota_check took=4279ms
[getGeolocation] success on attempt 1 (lat=48.200001 lng=11.770000 acc=19.1)
Response: {"id":"b0696ef23a08","message":"Module added successfully","name":"hive-01"}
[heartbeat] HTTP/1.1 200 OK
---- warm-up frame 1 OK (21244 bytes)
---- https://highfive.schutera.com/upload responded with status: 200
------ Success

Registration, heartbeat and first image upload all 200. The module is live on production now.

Also run: pio test -e native291/291 pass; pio run -e esp32cam links clean; scripts/check-doc-citations.sh → 7 OK, 0 problems.

The uncomfortable part

Every automated gate this repo runs on firmware was green while the firmware could not boot. pio test -e native has no Arduino runtime and no TLS; pio run -e esp32cam only proves it links. A green CI board on ESP32-CAM/ means "compiles and its pure helpers pass", never "boots". The only thing that closes that gap is a smoke-flash on real hardware before a SEQUENCE bump, and it is manual — now decided: it is a named step in the firmware release checklist (step 2, loopTask stack re-verify).

Second-order finding, in the docs and worth a reviewer's attention: during the crash loop the OTA boot gate counted pv=1/3 → 2/3 → 3/3 and tried to save the board — [OTA] rollback ... — reverting slot — and could not: esp_ota_ops: Rollback is not possible, do not have any suitable apps in slots. The ADR-008 / #148 rollback design assumes a good slot to revert to, which a fleet OTA has and a freshly wizard-flashed factory slot does not. The rollback net protects the fleet from a bad release; it does not protect the onboarding path, which is exactly where this defect lived.

Docs

Per CLAUDE.md's documentation gate:

  • docs/11-risks-and-technical-debt/README.md — lessons learned: the stack was the one resource with no guard, no log line and no comment, while heap headroom and handshake timeouts (each of which had already caused an incident) were treated as first-class.
  • docs/06-runtime-view/esp-reliability.md — new section 9, loopTask stack budget, framed as a budget the TLS call sites share rather than a constant.
  • docs/troubleshooting.md — three entries: the crash-loop signature; the production variant of the wizard's "/firmware.bin not found" (setup wizard cannot flash on production: /firmware.bin, /firmware.app.bin and /firmware.json are all missing from the live host #275, where Firmware: Local is the tell and there is no operator fix from a laptop); and an obsolete PlatformIO core shadowing a good one (Unknown development platform 'espressif32'), which cost time in this session.

Not in scope

🤖 Generated with Claude Code

cofade and others added 2 commits September 1, 2026 01:48
setup() and loop() run on the Arduino core's loopTask, whose stack is
8192 bytes by default (ARDUINO_LOOP_STACK_SIZE, in the core's main.cpp).
That is not enough for an mbedTLS handshake that parses and verifies a
certificate chain against the pinned ISRG Root X1, so ota.cpp's
httpOtaCheckAndApply tripped the stack canary on every boot and panicked
at breadcrumb `ota:manifest_fetch`, before the manifest's status line was
ever read. A real field module joined WiFi, printed an IP, and then
crash-looped every ~5.7 s without ever reaching registration.

Neither build path raised the default -- there is no SET_LOOP_TASK_STACK_SIZE,
ARDUINO_LOOP_STACK_SIZE or CONFIG_ARDUINO_LOOP_STACK_SIZE in platformio.ini,
build.sh or the sources -- so the merged firmware.bin the setup wizard flashes
carried the same defect as the PlatformIO build.

Verified on hardware (ESP32-CAM on COM5, module hive-01 / b0696ef23a08).
Before: stack canary panic on every boot. After: the board walks the whole
path against production -- OTA manifest check (a survived HTTP 404, which is
the separate artefact-publishing gap), geolocation, new_module registration,
boot heartbeat and first image upload, all 200.

16384 clears the observed overflow but is not a measured bound; instrumenting
the real high-water mark with uxTaskGetStackHighWaterMark is tracked as a
follow-up on the issue, along with the question of whether the four TLS call
sites in setup() should share a task with an explicit stack instead.

Worth stating because it shaped the docs added here: no automated gate covers
this. `pio test -e native` has no Arduino runtime and no TLS, and
`pio run -e esp32cam` only proves the firmware links -- both were green while
the firmware could not finish setup() on hardware.

Docs: chapter 11 lessons learned (the uncosted stack budget, and why the OTA
rollback net does not cover a freshly flashed factory slot), a new
esp-reliability section 9 for the budget itself, and three troubleshooting
entries -- the crash-loop signature, the production variant of the wizard's
"/firmware.bin not found", and an obsolete PlatformIO core shadowing a good
one, which cost time in this session.

Addresses #276. Ships nothing to the field on its own: reaching the fleet
needs a SEQUENCE-bumped release per firmware-release.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- correct the TLS call-site count: geolocation handshake runs only on NVS
  geo cache miss (~1 in 14 boots, kGeoCacheMaxBoots), so setup() does TLS
  up to four times, not 'at least four'
- drop the false 'appears nowhere in this repo' claim (the sentence itself
  and an unrelated 8192 chunk size exist); say 'no repo file configures
  or references it'
- merge the obsolete-PIO-core entry into the existing PlatformIO section
  and reattach the orphaned explicit-interpreter tail
- the Makefile/CI do not 'always write python -m platformio' - reword to
  the true claim (a bare pio resolves to whatever pio.exe is on PATH)
- chapter 11: link the fix to the SEQUENCE-bumped release requirement
  (#150/#132 silent no-ops) and spell out that free heap != free stack
- troubleshooting: note the addr2line addresses are dump-specific
@cofade
cofade force-pushed the fix/276-loop-task-stack-tls-panic branch from 27ba86b to bf57bd0 Compare September 1, 2026 00:33
review added 2 commits September 1, 2026 02:34
Formerly 'worth deciding' in the #276 lesson; decided here: the bench
boot in release step 2 doubles as the tripwire for the 16 KB budget,
and the smoke-flash is the only gate that notices a silently-broken
SET_LOOP_TASK_STACK_SIZE override after a core upgrade.
…hecklist

Every build since the #276 fix carries SET_LOOP_TASK_STACK_SIZE(16384);
the conditional read as though some releases might not.
@cofade
cofade marked this pull request as ready for review September 1, 2026 08:36
@cofade
cofade merged commit 58d6ab8 into main Sep 1, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant