fix(esp): raise loopTask stack so the OTA manifest TLS handshake fits - #277
Merged
Conversation
setup() and loop() run on the Arduino core's loopTask, whose stack is 8192 bytes by default (ARDUINO_LOOP_STACK_SIZE, in the core's main.cpp). That is not enough for an mbedTLS handshake that parses and verifies a certificate chain against the pinned ISRG Root X1, so ota.cpp's httpOtaCheckAndApply tripped the stack canary on every boot and panicked at breadcrumb `ota:manifest_fetch`, before the manifest's status line was ever read. A real field module joined WiFi, printed an IP, and then crash-looped every ~5.7 s without ever reaching registration. Neither build path raised the default -- there is no SET_LOOP_TASK_STACK_SIZE, ARDUINO_LOOP_STACK_SIZE or CONFIG_ARDUINO_LOOP_STACK_SIZE in platformio.ini, build.sh or the sources -- so the merged firmware.bin the setup wizard flashes carried the same defect as the PlatformIO build. Verified on hardware (ESP32-CAM on COM5, module hive-01 / b0696ef23a08). Before: stack canary panic on every boot. After: the board walks the whole path against production -- OTA manifest check (a survived HTTP 404, which is the separate artefact-publishing gap), geolocation, new_module registration, boot heartbeat and first image upload, all 200. 16384 clears the observed overflow but is not a measured bound; instrumenting the real high-water mark with uxTaskGetStackHighWaterMark is tracked as a follow-up on the issue, along with the question of whether the four TLS call sites in setup() should share a task with an explicit stack instead. Worth stating because it shaped the docs added here: no automated gate covers this. `pio test -e native` has no Arduino runtime and no TLS, and `pio run -e esp32cam` only proves the firmware links -- both were green while the firmware could not finish setup() on hardware. Docs: chapter 11 lessons learned (the uncosted stack budget, and why the OTA rollback net does not cover a freshly flashed factory slot), a new esp-reliability section 9 for the budget itself, and three troubleshooting entries -- the crash-loop signature, the production variant of the wizard's "/firmware.bin not found", and an obsolete PlatformIO core shadowing a good one, which cost time in this session. Addresses #276. Ships nothing to the field on its own: reaching the fleet needs a SEQUENCE-bumped release per firmware-release.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- correct the TLS call-site count: geolocation handshake runs only on NVS geo cache miss (~1 in 14 boots, kGeoCacheMaxBoots), so setup() does TLS up to four times, not 'at least four' - drop the false 'appears nowhere in this repo' claim (the sentence itself and an unrelated 8192 chunk size exist); say 'no repo file configures or references it' - merge the obsolete-PIO-core entry into the existing PlatformIO section and reattach the orphaned explicit-interpreter tail - the Makefile/CI do not 'always write python -m platformio' - reword to the true claim (a bare pio resolves to whatever pio.exe is on PATH) - chapter 11: link the fix to the SEQUENCE-bumped release requirement (#150/#132 silent no-ops) and spell out that free heap != free stack - troubleshooting: note the addr2line addresses are dump-specific
cofade
force-pushed
the
fix/276-loop-task-stack-tls-panic
branch
from
September 1, 2026 00:33
27ba86b to
bf57bd0
Compare
added 2 commits
September 1, 2026 02:34
Formerly 'worth deciding' in the #276 lesson; decided here: the bench boot in release step 2 doubles as the tripwire for the 16 KB budget, and the smoke-flash is the only gate that notices a silently-broken SET_LOOP_TASK_STACK_SIZE override after a core upgrade.
…hecklist Every build since the #276 fix carries SET_LOOP_TASK_STACK_SIZE(16384); the conditional read as though some releases might not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #276 — the one-line fix is verified on hardware, but the issue's follow-up questions (a measured stack bound, and whether
setup()'s TLS call sites should share a task with an explicit stack) are deliberately not answered here, so this should not auto-close the issue.What was broken
Current
mainfirmware could not completesetup()on real hardware. A field module joined Wi-Fi, printed an IP, then panicked and rebooted — a ~5.7 s crash loop — so it never reached registration, heartbeat, geolocation or capture, and never appeared on the dashboard.Next boot:
[BOOT] last_stage_before_reboot=ota:manifest_fetch.setup()runs on the Arduino core'sloopTask, whose stack is 8192 bytes by default (ARDUINO_LOOP_STACK_SIZE, in the core'smain.cpp— a default that no repo file configures or references). That is not enough for an mbedTLS handshake that parses and verifies a certificate chain against the pinned ISRG Root X1, which is whatota.cpp'shttpOtaCheckAndApplydoes. It dies inside the handshake, before the manifest's status line is read — which reads as a network failure and is not one.Neither build path raised the default, so the merged
firmware.binthe setup wizard flashes had the same defect as the PlatformIO build.The change
One line at file scope in
ESP32-CAM.ino, plus a comment explaining the budget:Verification (hardware, not just CI)
ESP32-CAM on COM5, module
hive-01/b0696ef23a08, built withoutDEV_SERVER_HOSTso it targets production.Before: stack canary panic on every boot, no exceptions, across many reboots.
After — same board, same config, whole path green against
highfive.schutera.com:Registration, heartbeat and first image upload all 200. The module is live on production now.
Also run:
pio test -e native→ 291/291 pass;pio run -e esp32camlinks clean;scripts/check-doc-citations.sh→ 7 OK, 0 problems.The uncomfortable part
Every automated gate this repo runs on firmware was green while the firmware could not boot.
pio test -e nativehas no Arduino runtime and no TLS;pio run -e esp32camonly proves it links. A green CI board onESP32-CAM/means "compiles and its pure helpers pass", never "boots". The only thing that closes that gap is a smoke-flash on real hardware before aSEQUENCEbump, and it is manual — now decided: it is a named step in the firmware release checklist (step 2,loopTaskstack re-verify).Second-order finding, in the docs and worth a reviewer's attention: during the crash loop the OTA boot gate counted
pv=1/3 → 2/3 → 3/3and tried to save the board —[OTA] rollback ... — reverting slot— and could not:esp_ota_ops: Rollback is not possible, do not have any suitable apps in slots. The ADR-008 / #148 rollback design assumes a good slot to revert to, which a fleet OTA has and a freshly wizard-flashed factory slot does not. The rollback net protects the fleet from a bad release; it does not protect the onboarding path, which is exactly where this defect lived.Docs
Per CLAUDE.md's documentation gate:
docs/11-risks-and-technical-debt/README.md— lessons learned: the stack was the one resource with no guard, no log line and no comment, while heap headroom and handshake timeouts (each of which had already caused an incident) were treated as first-class.docs/06-runtime-view/esp-reliability.md— new section 9,loopTaskstack budget, framed as a budget the TLS call sites share rather than a constant.docs/troubleshooting.md— three entries: the crash-loop signature; the production variant of the wizard's "/firmware.bin not found" (setup wizard cannot flash on production: /firmware.bin, /firmware.app.bin and /firmware.json are all missing from the live host #275, whereFirmware: Localis the tell and there is no operator fix from a laptop); and an obsolete PlatformIO core shadowing a good one (Unknown development platform 'espressif32'), which cost time in this session.Not in scope
SEQUENCE-bumped release per firmware-release.md. Given firmware: setTimeout() unit bug (seconds on core 2.0.17), captive-portal loops without deadline/WDT feed, and a fixed OTA download deadline — one SEQUENCE-bumped release #231 is also waiting on a release, these are probably worth landing together as one OTA rather than two.senior-reviewergate has been run (4 rounds, 2026-09-01): it surfaced a factual doc error (theloopTaskTLS call-site count — geolocation is NVS-cache-gated, sosetup()does TLS up to four times, not at least four), a PIO-troubleshooting section that had split its own tail, a false "the repo always writespython -m platformio" claim, and wording traps (fake-quoted log text, a dangling clause, dump-specific addr2line addresses). All findings are fixed on the rebased branch; doc-citation and firmware-build gates pass, and CI is green.🤖 Generated with Claude Code