firmware: module code in PSRAM, a 41% faster blit, a crash record, and a console that outlives a hung pattern

.github/workflows/firmware-checks.yml

@@ -21,6 +21,23 @@ name: Core knows no features
 #   what every pattern built afterwards renders — and reaches only newly built
 #   modules, since an installed .pfm carries its own compiled copy.
 #
+#   check_parser.py — the vendored HTTP request parser, the real Parsing.cpp
+#   against a scripted socket and a fake clock. It runs on pf-net, where a loop
+#   that waits for the peer without sleeping starves IDLE0 and the Core-0
+#   watchdog reboots the board 5 s later. That has happened three times
+#   (src/webserver/VENDORED.md) and nothing on a PC stood in the way of any of
+#   them. Truncated, stalled and trickled requests are replayed here, every
+#   cut point of each, and a wait that neither reads a byte nor sleeps fails -
+#   as does a request still being parsed after its timeouts have run out.
+#
+#   check_crash.py — what a boot makes of the reset before it (src/core_crash.h):
+#   a breadcrumb in RAM that only some resets keep, a core dump in flash that
+#   only some resets write. A mistake there crashes nothing - it reports last
+#   week's backtrace as today's - and a panel cannot be made to take every
+#   reset on demand, so the header is booted here through reset reason x
+#   breadcrumb x dump, with a module's code at an internal-RAM address and at
+#   the 0x43xxxxxx a PC carries for code in PSRAM.
+#
 # The blit, thumbnail mailbox, network recovery and ELF bounds also run on a
 # host C++ compiler with ASan/UBSan. The four-composition Xtensa compile stays
 # local (`firmware/bundles/build.sh all`) and remains required for core edits.
@@ -82,8 +99,12 @@ jobs:
         run: python firmware/toolchain/check_network.py --sanitize
       - name: Runtime admission and frame-boundary lifecycle
         run: python firmware/toolchain/check_runtime.py --sanitize
+      - name: Crash record tells this reset's dump from an older one
+        run: python firmware/toolchain/check_crash.py --sanitize
       - name: HTTP backpressure and bounded send attempts
         run: python firmware/toolchain/check_send.py --sanitize
+      - name: HTTP request parser sleeps in every wait and gives up on a request that stopped
+        run: python firmware/toolchain/check_parser.py --sanitize
       - name: MIDI event ownership and remote-input echo suppression
         run: python firmware/toolchain/check_midi.py --sanitize
       - name: No mangled literals or raw control bytes

CHANGELOG.md

@@ -6,6 +6,15 @@ All notable changes to Patternflow will be documented in this file, newest first
 
 ### Firmware
 
+- **A panel that crashed says where.** `resetReason` in `/api/status` said `"panic"` or `"task_wdt"` and nothing else - not where, and not which of forty installed patterns was on. The SDK had been writing a core dump to its own flash partition on every panic all along, and nothing read it. At boot the firmware now reads that dump's summary - the task, the cause, the faulting address, the PC and up to sixteen return addresses, and the hash of the image that crashed - and it keeps a breadcrumb through the reset: the pattern's slug, which of its calls was running (loading, constructors, `setup`, `update`, `draw`, or none of them) and where the module's code was executing from. Status gains a `crash` object with both. With the two together an address inside a module comes out as an offset (`"+0x4e"`) that the `.pfm`'s own symbol table resolves, whether the code ran from PSRAM or from internal RAM. `DELETE /api/crash` clears it, and the same record is printed on serial at boot (`[CRASH] ...`). It reports and does nothing else: no pattern is skipped, no reboot is forced, and what the panel does after a crash is what it did before. The breadcrumb covers the boot that follows a panic or a watchdog and is gone with the power; the dump stays in flash, so a board that panicked on an older firmware shows that dump, marked `fromThisReset: false`, until it is cleared. Checked on a board: a module stored through a null pointer in `draw()` while running from PSRAM, and after the reboot status named the pattern, the phase, `pc` `+0x4e` and the caller at `+0xb6` - the offsets the module's own symbols give for those two functions - with the dump read in 62 ms; a dump left by an earlier firmware appeared as `fromThisReset: false`. It costs 296 bytes of internal RAM. `docs/rest-api.md` ("The crash record") says how to read one and decode it, and `firmware/toolchain/check_crash.py`, a new host check, boots the logic through every reset reason against every state of the breadcrumb and the dump; CI runs it.
+- **Every pattern is 2.45 ms a frame faster.** Copying a finished frame to the panel's bit-planes was the largest fixed cost in the loop: 5.96 ms of every frame, nearly half of Origin's. The kernel was one long loop with more live values than the CPU has registers and a plane count it only learned at run time; it is now three short passes with the 8-bit case written in, 390 instructions a column pair where there were 635, emitting exactly the same words (`check_blit.py` compares 70 million of them). Measured on two boards, alternating images: `presentUs` 5,960 -> 3,496 µs on the default and Audio editions alike, so Origin goes from 12.1 to 9.6 ms a frame and a 30 ms pattern to 27.6. With white balance or gamma tuned away from the defaults it is 3.8 ms. It also takes 1 KB less internal RAM than the loop it replaces. `/api/status` gains `loopStackMin`. Not yet checked by eye on a panel.
+- **A pattern's code no longer competes with the network for memory.** A loaded module's code used to need one contiguous block of internal executable RAM - the same RAM the console, lwIP and every feature run on - because "the S3 cannot execute from PSRAM". That is true only of the address the heap hands back: on the S3 the instruction and data buses share one MMU table, so the same PSRAM page can be fetched at that address + `0x06000000`. The loader now places code in PSRAM, writes and relocates it through the heap's pointer, and runs it through the alias (`src/core_module_memory.h`, `execAddress()` in `src/core_module_loader.h`). On the Audio edition a module with 23 KB or 32 KB of code was refused and now loads, and internal heap with a module resident is about 27.8 KB whatever the size of its code, where a 10 KB one used to leave 21-23 KB. On a second panel a community pattern with 8 KB of code was being refused one load in four for want of 528 bytes; it now loads every time and leaves 29 KB free instead of 22. Measured on two boards, same boot, alternating: catalogue and community modules within -0.1..+1.2% of their old frame time; modules built to be the worst case (10-32 KB of code, all of it executed every frame) 2.6-5.3% slower; 400 switches between modules of different sizes with no reset and no refusal. The code block is given whole cache lines to itself, the load reads it back through the instruction bus before calling it, and a unit on which that ever fails goes back to internal RAM until reboot (`moduleMemory.codePolicy`). `load.code` in `/api/status` says where the code is. Module files and the ABI are unchanged.
+- **A request that stops half-way can no longer reboot the panel.** A multipart `POST` that ended after its first boundary - to any address, not only an upload - left the web server's form parser reading empty lines for ever with nothing to wait for, and the Core-0 watchdog reset the board five seconds later. The parser now knows a line that never ended from a blank one and gives the form up, tells an upload handler whose file had already arrived that it was aborted, and leaves none of the form's fields behind to answer for the next request (`src/webserver/VENDORED.md`, Fix 4). Reproduced on a panel before, gone after.
+- **Three more requests that could reboot it, found on a PC first.** `curl -X POST` to `/api/patterns` or `/update` with no form in it called the upload handler with no upload to read and panicked the board; a multipart request naming a 9,000-character boundary overflowed the network task's stack; and a raw `PUT /update?size=N` never saw its own `size`. All three are fixed in the vendored server (`VENDORED.md`, Fix 5), and all three came out of `firmware/toolchain/check_parser.py`, a new host check that replays cut-off, stalled and malformed requests - 13,953 of them - through the real parser and fails if it ever spins or holds; CI runs it.
+- **A quote in a pattern's name no longer breaks the pattern list.** `/api/patterns` wrote names into its reply unescaped, so one `"` or `\` in a title made the whole reply unparseable and the `/patterns` page could not list, select or delete anything - including the pattern responsible. Names are escaped now, a sidecar name is read as the JSON string it is - an escaped quote no longer ends it, and `\uXXXX` becomes the character, so "Dynamic Moiré" is that before the pattern has ever been loaded and not only after - and a name too long for its slot is cut between characters rather than through one. The hotspot password got the same escaping in `/api/hotspot`.
+- **A module's global constructors can use the host.** They ran before the module had been handed its host API, so a pattern with a namespace-scope initialiser that allocated, logged or asked for a random number (`float* trail = PFMem::allocFloats(n);`) crashed the panel every time it was picked, having built and uploaded cleanly. Constructors now run after the entry point and before `setup()`.
+- **Lanes keep working past day 25.** The five seconds a knob keeps to itself after a hand touches it was timed with a signed compare meant to survive the 49-day millisecond wrap; what it did was move the wrap to 24.9 days, after which a lane on any knob not touched since was held off for the following 24.9. A panel left alone would have stopped answering its microphone (Audio) or its weather (Performance) on day 25. Found by reading, not by waiting: the hold is a flag that ends once now.
+- **A pattern that never returns no longer takes the console with it.** A module's `draw()` runs on the render loop, and the render loop is on no watchdog: one `while (a > PI) a -= 2 * PI` on a value that has reached infinity - which float32 does where the browser's doubles did not - and the panel holds its last frame for good. The first console request that needed the loop then waited for it without limit, and the `/patterns` page opens with one; the server answers one connection at a time, so `/api/status`, `/update` and Reboot, which need nothing from the loop, went silent behind it, and a frozen panel could be neither diagnosed nor restarted without pulling the plug. A request now gives the loop up once it has not reached its frame boundary for 20 s (`PF_LOOP_STALL_MS` - measured from the loop's own stamp, not from how long the request has waited, so an upload waiting out a slow `setup()` is not mistaken for it) and answers `503 render loop is not answering`; the request is taken back with one atomic exchange, so the loop can never run it on a stack frame that has already returned. `/api/status` carries `loopAgeMs`, `loopStalled` and `loopSyncGaveUp`, read without the loop's help, and every handler in the core and in the features answers the refusal instead of using what its body never set. The Status page says "not answering for 47s" in red, where it would have gone on showing the frame rate from before the stop under "awake". Nothing restarts the panel by itself: this reports and survives, and the Reboot button does the rest. Twenty seconds is clear of one feature lookup with the internet down and not of two in a row, so on Performance with a dead uplink a healthy panel can read "not answering" for some seconds and then carry on - which is why it does not say "hung". The console has to have come up once for any of this: its routes are registered by the loop on the first network link. Found by reading - no stock pattern hangs - then reproduced on a panel with a module that hangs on purpose: before, the pattern list got no reply and after it neither did status, until the board was reset over USB; after, the list answered 503 in 18 s, sixty more requests were refused at once with the heap unchanged, the Status, Update and home pages still loaded, and Reboot brought the panel back. Held by a host test that races the two sides against each other; `firmware/toolchain/tests/modules/_hang_probe` is a module that hangs on purpose, for the bench (`src/core_loop_sync.h`).
 - **No more stray letters on the panel's own screens.** A portrait line holds ten characters, and Adafruit GFX wraps an eleventh onto the next line by itself: the KNOB MAP's "TURN = SHOW" put a lone "W" in front of "K3 = EXIT", the install screen's "web console" did the same, and the default host name "patternflow" lost its "w" on the hotspot and UPDATE screens. The labels are shorter now ("TURN=SHOW", "console"), and names that can be longer than a line are broken into lines of ten by `drawCenteredFit` — "pattern" / "flow-a1b2", "pattern" / "flow.local" — keeping every line where it was.
 - **A deck sent from the community's dock keeps its order.** The console's one-click install from a link (`/patterns?src=`) took only `.pfm` and `.json` from a build's file list, so the `catalog.txt` every deck build carries was left behind and the deck landed in alphabetical order. It now takes exactly what a dropped zip does (`catalog.txt` and `.pfs` too) - one rule, `installable()`, for every way in.
 - **A firmware upload survives a Wi-Fi hitch.** The `/update` connection now waits up to two minutes for a stalled upload instead of five seconds, and the page gives up after five minutes with a message instead of waiting forever. Checked on a panel with a 12 s stall halfway through a flash. `PUT /update` takes a raw image, for `curl -T`. From Simone Majocchi ([#450](https://github.com/engmung/Patternflow/pull/450)).

FEATURE_GUIDE.md

@@ -116,7 +116,12 @@ section fully before writing code.
   using right now — starts or stops something your `loop` hook is ticking,
   reconnects a client it polls, frees a buffer it reads — wraps that part in
   `PFLoopSync::run([&] { ... })` (`src/core_loop_sync.h`): the body runs on
-  the loop task at the frame boundary and the handler waits for it.
+  the loop task at the frame boundary and the handler waits for it. `run()`
+  returns `false` when the body did not run and never will - the render loop
+  has stopped coming round, usually a pattern that never returns from
+  `draw()` - and the handler then answers
+  `PatternflowHttp::sendLoopStalled()` (a 503) and changes nothing. A handler
+  that ignores the result sends no reply at all in that case.
   [`show/`](firmware/patternflow/features/show/), `mqtt/` and `weather/`
   show the shape.