diff --git a/.github/workflows/firmware-checks.yml b/.github/workflows/firmware-checks.yml index 1449a40b..6f1a8be8 100644 --- a/.github/workflows/firmware-checks.yml +++ b/.github/workflows/firmware-checks.yml @@ -21,6 +21,23 @@ name: Core knows no features # what every pattern built afterwards renders — and reaches only newly built # modules, since an installed .pfm carries its own compiled copy. # +# check_parser.py — the vendored HTTP request parser, the real Parsing.cpp +# against a scripted socket and a fake clock. It runs on pf-net, where a loop +# that waits for the peer without sleeping starves IDLE0 and the Core-0 +# watchdog reboots the board 5 s later. That has happened three times +# (src/webserver/VENDORED.md) and nothing on a PC stood in the way of any of +# them. Truncated, stalled and trickled requests are replayed here, every +# cut point of each, and a wait that neither reads a byte nor sleeps fails - +# as does a request still being parsed after its timeouts have run out. +# +# check_crash.py — what a boot makes of the reset before it (src/core_crash.h): +# a breadcrumb in RAM that only some resets keep, a core dump in flash that +# only some resets write. A mistake there crashes nothing - it reports last +# week's backtrace as today's - and a panel cannot be made to take every +# reset on demand, so the header is booted here through reset reason x +# breadcrumb x dump, with a module's code at an internal-RAM address and at +# the 0x43xxxxxx a PC carries for code in PSRAM. +# # The blit, thumbnail mailbox, network recovery and ELF bounds also run on a # host C++ compiler with ASan/UBSan. The four-composition Xtensa compile stays # local (`firmware/bundles/build.sh all`) and remains required for core edits. @@ -82,8 +99,12 @@ jobs: run: python firmware/toolchain/check_network.py --sanitize - name: Runtime admission and frame-boundary lifecycle run: python firmware/toolchain/check_runtime.py --sanitize + - name: Crash record tells this reset's dump from an older one + run: python firmware/toolchain/check_crash.py --sanitize - name: HTTP backpressure and bounded send attempts run: python firmware/toolchain/check_send.py --sanitize + - name: HTTP request parser sleeps in every wait and gives up on a request that stopped + run: python firmware/toolchain/check_parser.py --sanitize - name: MIDI event ownership and remote-input echo suppression run: python firmware/toolchain/check_midi.py --sanitize - name: No mangled literals or raw control bytes diff --git a/CHANGELOG.md b/CHANGELOG.md index a5e2471a..d0736245 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ All notable changes to Patternflow will be documented in this file, newest first ### Firmware +- **A panel that crashed says where.** `resetReason` in `/api/status` said `"panic"` or `"task_wdt"` and nothing else - not where, and not which of forty installed patterns was on. The SDK had been writing a core dump to its own flash partition on every panic all along, and nothing read it. At boot the firmware now reads that dump's summary - the task, the cause, the faulting address, the PC and up to sixteen return addresses, and the hash of the image that crashed - and it keeps a breadcrumb through the reset: the pattern's slug, which of its calls was running (loading, constructors, `setup`, `update`, `draw`, or none of them) and where the module's code was executing from. Status gains a `crash` object with both. With the two together an address inside a module comes out as an offset (`"+0x4e"`) that the `.pfm`'s own symbol table resolves, whether the code ran from PSRAM or from internal RAM. `DELETE /api/crash` clears it, and the same record is printed on serial at boot (`[CRASH] ...`). It reports and does nothing else: no pattern is skipped, no reboot is forced, and what the panel does after a crash is what it did before. The breadcrumb covers the boot that follows a panic or a watchdog and is gone with the power; the dump stays in flash, so a board that panicked on an older firmware shows that dump, marked `fromThisReset: false`, until it is cleared. Checked on a board: a module stored through a null pointer in `draw()` while running from PSRAM, and after the reboot status named the pattern, the phase, `pc` `+0x4e` and the caller at `+0xb6` - the offsets the module's own symbols give for those two functions - with the dump read in 62 ms; a dump left by an earlier firmware appeared as `fromThisReset: false`. It costs 296 bytes of internal RAM. `docs/rest-api.md` ("The crash record") says how to read one and decode it, and `firmware/toolchain/check_crash.py`, a new host check, boots the logic through every reset reason against every state of the breadcrumb and the dump; CI runs it. +- **Every pattern is 2.45 ms a frame faster.** Copying a finished frame to the panel's bit-planes was the largest fixed cost in the loop: 5.96 ms of every frame, nearly half of Origin's. The kernel was one long loop with more live values than the CPU has registers and a plane count it only learned at run time; it is now three short passes with the 8-bit case written in, 390 instructions a column pair where there were 635, emitting exactly the same words (`check_blit.py` compares 70 million of them). Measured on two boards, alternating images: `presentUs` 5,960 -> 3,496 µs on the default and Audio editions alike, so Origin goes from 12.1 to 9.6 ms a frame and a 30 ms pattern to 27.6. With white balance or gamma tuned away from the defaults it is 3.8 ms. It also takes 1 KB less internal RAM than the loop it replaces. `/api/status` gains `loopStackMin`. Not yet checked by eye on a panel. +- **A pattern's code no longer competes with the network for memory.** A loaded module's code used to need one contiguous block of internal executable RAM - the same RAM the console, lwIP and every feature run on - because "the S3 cannot execute from PSRAM". That is true only of the address the heap hands back: on the S3 the instruction and data buses share one MMU table, so the same PSRAM page can be fetched at that address + `0x06000000`. The loader now places code in PSRAM, writes and relocates it through the heap's pointer, and runs it through the alias (`src/core_module_memory.h`, `execAddress()` in `src/core_module_loader.h`). On the Audio edition a module with 23 KB or 32 KB of code was refused and now loads, and internal heap with a module resident is about 27.8 KB whatever the size of its code, where a 10 KB one used to leave 21-23 KB. On a second panel a community pattern with 8 KB of code was being refused one load in four for want of 528 bytes; it now loads every time and leaves 29 KB free instead of 22. Measured on two boards, same boot, alternating: catalogue and community modules within -0.1..+1.2% of their old frame time; modules built to be the worst case (10-32 KB of code, all of it executed every frame) 2.6-5.3% slower; 400 switches between modules of different sizes with no reset and no refusal. The code block is given whole cache lines to itself, the load reads it back through the instruction bus before calling it, and a unit on which that ever fails goes back to internal RAM until reboot (`moduleMemory.codePolicy`). `load.code` in `/api/status` says where the code is. Module files and the ABI are unchanged. +- **A request that stops half-way can no longer reboot the panel.** A multipart `POST` that ended after its first boundary - to any address, not only an upload - left the web server's form parser reading empty lines for ever with nothing to wait for, and the Core-0 watchdog reset the board five seconds later. The parser now knows a line that never ended from a blank one and gives the form up, tells an upload handler whose file had already arrived that it was aborted, and leaves none of the form's fields behind to answer for the next request (`src/webserver/VENDORED.md`, Fix 4). Reproduced on a panel before, gone after. +- **Three more requests that could reboot it, found on a PC first.** `curl -X POST` to `/api/patterns` or `/update` with no form in it called the upload handler with no upload to read and panicked the board; a multipart request naming a 9,000-character boundary overflowed the network task's stack; and a raw `PUT /update?size=N` never saw its own `size`. All three are fixed in the vendored server (`VENDORED.md`, Fix 5), and all three came out of `firmware/toolchain/check_parser.py`, a new host check that replays cut-off, stalled and malformed requests - 13,953 of them - through the real parser and fails if it ever spins or holds; CI runs it. +- **A quote in a pattern's name no longer breaks the pattern list.** `/api/patterns` wrote names into its reply unescaped, so one `"` or `\` in a title made the whole reply unparseable and the `/patterns` page could not list, select or delete anything - including the pattern responsible. Names are escaped now, a sidecar name is read as the JSON string it is - an escaped quote no longer ends it, and `\uXXXX` becomes the character, so "Dynamic Moiré" is that before the pattern has ever been loaded and not only after - and a name too long for its slot is cut between characters rather than through one. The hotspot password got the same escaping in `/api/hotspot`. +- **A module's global constructors can use the host.** They ran before the module had been handed its host API, so a pattern with a namespace-scope initialiser that allocated, logged or asked for a random number (`float* trail = PFMem::allocFloats(n);`) crashed the panel every time it was picked, having built and uploaded cleanly. Constructors now run after the entry point and before `setup()`. +- **Lanes keep working past day 25.** The five seconds a knob keeps to itself after a hand touches it was timed with a signed compare meant to survive the 49-day millisecond wrap; what it did was move the wrap to 24.9 days, after which a lane on any knob not touched since was held off for the following 24.9. A panel left alone would have stopped answering its microphone (Audio) or its weather (Performance) on day 25. Found by reading, not by waiting: the hold is a flag that ends once now. +- **A pattern that never returns no longer takes the console with it.** A module's `draw()` runs on the render loop, and the render loop is on no watchdog: one `while (a > PI) a -= 2 * PI` on a value that has reached infinity - which float32 does where the browser's doubles did not - and the panel holds its last frame for good. The first console request that needed the loop then waited for it without limit, and the `/patterns` page opens with one; the server answers one connection at a time, so `/api/status`, `/update` and Reboot, which need nothing from the loop, went silent behind it, and a frozen panel could be neither diagnosed nor restarted without pulling the plug. A request now gives the loop up once it has not reached its frame boundary for 20 s (`PF_LOOP_STALL_MS` - measured from the loop's own stamp, not from how long the request has waited, so an upload waiting out a slow `setup()` is not mistaken for it) and answers `503 render loop is not answering`; the request is taken back with one atomic exchange, so the loop can never run it on a stack frame that has already returned. `/api/status` carries `loopAgeMs`, `loopStalled` and `loopSyncGaveUp`, read without the loop's help, and every handler in the core and in the features answers the refusal instead of using what its body never set. The Status page says "not answering for 47s" in red, where it would have gone on showing the frame rate from before the stop under "awake". Nothing restarts the panel by itself: this reports and survives, and the Reboot button does the rest. Twenty seconds is clear of one feature lookup with the internet down and not of two in a row, so on Performance with a dead uplink a healthy panel can read "not answering" for some seconds and then carry on - which is why it does not say "hung". The console has to have come up once for any of this: its routes are registered by the loop on the first network link. Found by reading - no stock pattern hangs - then reproduced on a panel with a module that hangs on purpose: before, the pattern list got no reply and after it neither did status, until the board was reset over USB; after, the list answered 503 in 18 s, sixty more requests were refused at once with the heap unchanged, the Status, Update and home pages still loaded, and Reboot brought the panel back. Held by a host test that races the two sides against each other; `firmware/toolchain/tests/modules/_hang_probe` is a module that hangs on purpose, for the bench (`src/core_loop_sync.h`). - **No more stray letters on the panel's own screens.** A portrait line holds ten characters, and Adafruit GFX wraps an eleventh onto the next line by itself: the KNOB MAP's "TURN = SHOW" put a lone "W" in front of "K3 = EXIT", the install screen's "web console" did the same, and the default host name "patternflow" lost its "w" on the hotspot and UPDATE screens. The labels are shorter now ("TURN=SHOW", "console"), and names that can be longer than a line are broken into lines of ten by `drawCenteredFit` — "pattern" / "flow-a1b2", "pattern" / "flow.local" — keeping every line where it was. - **A deck sent from the community's dock keeps its order.** The console's one-click install from a link (`/patterns?src=`) took only `.pfm` and `.json` from a build's file list, so the `catalog.txt` every deck build carries was left behind and the deck landed in alphabetical order. It now takes exactly what a dropped zip does (`catalog.txt` and `.pfs` too) - one rule, `installable()`, for every way in. - **A firmware upload survives a Wi-Fi hitch.** The `/update` connection now waits up to two minutes for a stalled upload instead of five seconds, and the page gives up after five minutes with a message instead of waiting forever. Checked on a panel with a 12 s stall halfway through a flash. `PUT /update` takes a raw image, for `curl -T`. From Simone Majocchi ([#450](https://github.com/engmung/Patternflow/pull/450)). diff --git a/FEATURE_GUIDE.md b/FEATURE_GUIDE.md index 039ff177..96db5e76 100644 --- a/FEATURE_GUIDE.md +++ b/FEATURE_GUIDE.md @@ -116,7 +116,12 @@ section fully before writing code. using right now — starts or stops something your `loop` hook is ticking, reconnects a client it polls, frees a buffer it reads — wraps that part in `PFLoopSync::run([&] { ... })` (`src/core_loop_sync.h`): the body runs on - the loop task at the frame boundary and the handler waits for it. + the loop task at the frame boundary and the handler waits for it. `run()` + returns `false` when the body did not run and never will - the render loop + has stopped coming round, usually a pattern that never returns from + `draw()` - and the handler then answers + `PatternflowHttp::sendLoopStalled()` (a 503) and changes nothing. A handler + that ignores the result sends no reply at all in that case. [`show/`](firmware/patternflow/features/show/), `mqtt/` and `weather/` show the shape. diff --git a/docs/rest-api.md b/docs/rest-api.md index a07557e1..965176fb 100644 --- a/docs/rest-api.md +++ b/docs/rest-api.md @@ -60,6 +60,7 @@ The numbers that explain a device when something is off. Requires `PF_STATUS_HTT "consolePaused": false, "busy": "", "frameUs": 16400, "presentUs": 3100, "loopCore": 1, "httpCore": 0, "netStackMin": 6200, "loopSyncServed": 3, "loopSyncMaxUs": 9800, + "loopAgeMs": 9, "loopStalled": false, "loopSyncGaveUp": 0, "colorBits": 6, "refreshHz": 121, "loadError": "", "load": { "total": 0, "read": 0, "relocate": 0, "setup": 0, "internal": 0, "psram": 0 }, "mqttRole": "off", "mqttState": "idle", "mqttConnected": false @@ -72,6 +73,7 @@ The numbers that explain a device when something is off. Requires `PF_STATUS_HTT | `build` | Which image exactly, as eight lowercase hex digits: the start of the firmware ELF's SHA-256 (a hash of the compile time and version on an image built without it). Two images with the same `version` still differ here. Console page URLs carry it as `?v=` (see [Page caching](#transport)); a page open under another build is stale. Since 1.5. | | `uptime` | Seconds since boot. | | `resetReason` | Why the board is running, `esp_reset_reason()` by name: `"poweron"` (the plug), `"sw"` (the reboot button, a finished update), `"panic"`, `"task_wdt"`, `"int_wdt"`, `"wdt"`, `"brownout"`, `"deepsleep"`, `"ext"`, `"sdio"` or `"unknown"`. Read it together with `uptime`: a small uptime and anything but `"poweron"` or `"sw"` means the board restarted on its own — which from the network otherwise looks exactly like a power cut. The same word is printed once on serial at boot (`[BOOT] reset reason: …`). Since 1.4. | +| `crash` | Since 1.5. Where the last death happened; **absent** when there is nothing to report, which is the normal case. See [The crash record](#the-crash-record). | | `panel` | Physical matrix, `"x"`. The closest thing to a model number. | | `host` | mDNS hostname, i.e. `PF_OTA_HOSTNAME`. **Not unique** — every device ships as `"patternflow"`. See [Identifying a device](#identifying-a-device). | | `network` | Since 1.5, when Wi-Fi is compiled in: `disconnects` counts sampled connected→disconnected edges; `retries` counts explicit retry attempts (including initial join failures); `reconnectMs` is the last sampled downtime, zero before a reconnect. `namesReady` means the core's local mDNS/service/alias and NetBIOS registrations succeeded; probing may still be in progress and it does not prove client reachability. `announcements` counts successful registration passes. Counters reset on boot. | @@ -95,18 +97,66 @@ The numbers that explain a device when something is off. Requires `PF_STATUS_HTT | `moduleMemory` | Since 1.5. `reserve` is the module allocator's internal byte-addressable RAM floor, not a guarantee about allocations by other subsystems. `serviceFree` is current internal byte-addressable free RAM; `execLargest` is the largest contiguous executable block. `runtimeBytes` and lifetime `runtimePeakBytes` track retained module dynamic allocations; `runtimeLimit` defaults to 4 MiB. Sections and temporary ELF storage are admitted separately. `refusals` counts allocation refusals, including recovered attempts; `loaderRetries` counts the asynchronous loader's bounded retries after such failures. `loaderStackBytes` reports the reusable worker's reserved stack, and `loaderStackMin` is its minimum free space after jobs (0 until measured). | | `netMaintenance` | Since 1.5. Lifetime `calls` and `maxGapMs` for the Wi-Fi/name maintenance hook on the network worker. It runs approximately every 25 ms when serviced, including page-body sends and parser/loop-request waits. Other blocking work, including filesystem calls, can still produce longer gaps. | | `loopCore` / `httpCore` | Which core renders (1) and which answered this request — `0` on 3.9.1 and later, `1` when the network task could not be created and `loop()` is serving. | +| `loopStackMin` | Since 1.5: the same for the loop task, which runs every pattern's `draw()` and the panel blit inside it. | | `netStackMin` | The network task's stack high-water mark in bytes: the least it has ever had free. Falling toward zero is the thing to watch after a firmware update. | | `loopSyncServed` / `loopSyncMaxUs` | How many handlers had to run on the loop task (a module eviction, a show start, an MQTT reconnect) and the longest one waited for a frame boundary. | +| `loopAgeMs` / `loopStalled` / `loopSyncGaveUp` | Since 1.5. Milliseconds since `loop()` was last at its frame boundary, read without the loop's help — a frame or two on a running panel. `loopStalled` turns `true` at 20 s (`PF_LOOP_STALL_MS`): the render loop has not come round, which in practice is a pattern that never returns from `draw()` — `active` names it. It can also be a healthy loop held up: with the internet down one feature lookup holds the loop 14 s, two features' lookups in the same iteration (Performance: weather and an MQTT broker named by host) 28-30 s, and a TLS handshake longer still, so `true` there is not yet a hang. What tells a hang from a loop that is only held up is the next poll: a hung loop's age has grown by exactly the time between the two, a busy one's is back to a few milliseconds with `loopStalled` `false` (`runtime.maxUs` is the longest it has been held since boot). The panel holds its last frame; this endpoint and `POST /api/wifi/reboot` keep answering, as does `/update` while it is armed without the UPDATE screen (the default), and every route that waits for the loop answers `503` (see *Conventions*). Two limits: the console's routes are registered by the loop on the first network link, so a loop that hangs before the panel has ever joined a network or raised its hotspot has no console to outlive it; and routes that only queue work for the loop - `POST /api/params`, `/api/sleep`, a brightness change - still answer `ok` on such a panel and do nothing until it returns. `loopSyncGaveUp` counts the requests refused that way. Nothing reboots the panel on its own, and a reboot goes back to the remembered pattern unless that pattern hung within 15 s of the boot that restored it — pick another one before it hangs again. | | `colorBits` / `refreshHz` | What the HUB75 driver actually settled on — it trades colour depth against the requested refresh rate, so these are read back rather than configured. | | `loadError` | Why the last module load failed, empty when it did not. Without it a refusal is invisible from the network. | | `loading` | `true` while a module is being loaded on the network core (since 1.4 a switch no longer holds the frame; a heavy pattern's `setup()` takes seconds). `active` reads `"-"` meanwhile. | | `thumbs` | Since 1.5: boot counters `captures` (cache refreshed), `reads` (disk image adopted), `writes` (save completed). `captureMaxUs` is the maximum loop-side capture duration; `ioMaxUs` is the maximum background file-operation wall time, including scheduling/filesystem contention. Both are µs. The cache and its temporary I/O snapshot use PSRAM only. | -| `load` | The last module load, in µs: `total`, `read` (flash), `relocate`, `setup` (the pattern's own). Since 1.4 also where its sections landed, in bytes: `internal` and `psram`. A big `internal` is the number to look at when the console dies the moment a pattern is chosen — the loader now sends data sections over 16 KB, or any that would leave under 24 KB of internal heap, to PSRAM (`PF_MODULE_DATA_INTERNAL_MAX`, `PF_MODULE_INTERNAL_RESERVE`). | +| `load` | The last module load, in µs: `total`, `read` (flash), `relocate`, `setup` (the pattern's own). Since 1.4 also where its sections landed, in bytes: `internal` and `psram`. A big `internal` is the number to look at when the console dies the moment a pattern is chosen — the loader now sends data sections over 16 KB, or any that would leave under 24 KB of internal heap, to PSRAM (`PF_MODULE_DATA_INTERNAL_MAX`, `PF_MODULE_INTERNAL_RESERVE`). Since 1.5 `code` says where the module's code runs from, `"psram"` or `"internal"`: in PSRAM it is counted in `psram` and takes nothing from the internal heap, and `"internal"` on a module means PSRAM could not hold it or `moduleMemory.codePolicy` is `0` - which it becomes, until reboot, if code placed in PSRAM ever fails to read back on this unit. | | `variant` | Which firmware this is: `"core"`, or a variant's own name. What the site's variant list matches, and what stops the update banner offering a core build on top of someone's chosen firmware. | | `caps` | What this build can do. **Probe this rather than assuming a feature exists.** The default build reports `["patterns","params","sleep"]` and nothing else; each [edition](EDITIONS.md) adds its own — Audio adds `osc` and `audio`, Performance adds `weather`, `mqtt` and `shows`. `patterns` and `params` are on every build. | | `mqttRole` | `"off"`, `"publisher"` or `"subscriber"`. Decides whether the device obeys knob and pattern topics — see [Knobs](#knobs-and-parameters). | | `featureNav` | `[path, label, one-line description]` per console page the loaded features serve, e.g. `[["/audio-in","Audio","The panel hears the room…"]]`. What the console header and home screen build their feature links from; empty on the default build. | +### The crash record + +`resetReason` says that the board restarted on its own. `crash`, since 1.5, says where. It reports and nothing else: no pattern is forgotten and no reboot is forced because of it. + +```json +"crash": { + "pattern": "cell_ripple", "phase": "draw", + "code": { "base": "0x43c81000", "size": 4660 }, + "dump": { + "fromThisReset": true, "task": "loopTask", "cause": 28, "vaddr": "0x00000000", + "pc": "+0x1a1", + "backtrace": ["+0x1a1", "+0x9c", "0x4200f1a3", "0x4200e907", "0x42010b52"], + "corrupted": false, "build": "3f9a0c1e5d7b2a40", "bytes": 23108 + } +} +``` + +It has two halves, kept apart because they do not live equally long. + +**`pattern`, `phase`, `code`** are a breadcrumb the firmware keeps in a corner of RAM that no boot initialises, which a panic or watchdog reset does not clear and a power cut does. They are present only on the boot that follows a `panic`, `task_wdt`, `int_wdt` or `wdt` reset, and describe the reset `resetReason` names. + +| Field | Meaning | +|---|---| +| `pattern` | Slug of the pattern that was resident (a module's file name without `.pfm`; a preset's name, slugified). `""` when none was. | +| `phase` | What the device was in the middle of: `"loading"` (reading and relocating a module, through its entry point), `"constructors"`, `"setup"`, `"update"`, `"draw"`, or `"idle"` — anywhere outside pattern code: the blit, a feature, the network. `"dump-read"` means the previous boot died while reading the core dump, and this boot left it unread. A pattern chosen while the panel is running loads on a task of its own, `pf-load`, while `loopTask` keeps drawing, so read `phase` beside `dump.task`: `"loading"`, `"constructors"` or `"setup"` with a crash in `pf-load` is a crash in the load, and with a crash in another task it is one that happened while a module was loading, not in it. The exception is the pattern restored at boot, which is loaded on `loopTask` before the first frame — there, and for a preset's `setup()`, which also runs at boot, `"setup"` with `loopTask` is a crash in `setup()`. | +| `code` | Modules only: the address the CPU fetched the module's code from, and its size in bytes. `0x43…` is PSRAM as the instruction bus sees it (`load.code` `"psram"`); `0x40…` is internal RAM. | + +**`dump`** is the summary of the core dump the SDK writes to the `coredump` partition on every panic. It survives power cycles and reflashing, and stays until the next panic overwrites it or [`DELETE /api/crash`](#delete-apicrash) erases it. The SDK was writing these dumps before any firmware read them, so a board that panicked on an earlier firmware still holds that dump: after the update it is reported with `fromThisReset` `false`, on every boot, until it is cleared. It is not a new crash. + +| Field | Meaning | +|---|---| +| `fromThisReset` | Whether the dump was written by the reset this boot followed. Only then do the breadcrumb and the dump describe one event. `false` after a power cycle, and after a death that wrote no dump of its own — an RTC watchdog reset, or a dump too large for the 64 KB partition — which leaves an earlier one in place. | +| `task` | Name of the task that took the exception. | +| `cause` | Xtensa `EXCCAUSE` as recorded: `0` illegal instruction, `6` integer divide by zero, `20` instruction fetch prohibited (a jump to nowhere), `28` load prohibited, `29` store prohibited (a bad pointer read or written). `29` with `vaddr` `0x00000000` and `panic_abort` at the top of the backtrace is not a bad pointer but the SDK stopping on purpose, which it does by storing to address zero: `abort()`, a failed `assert`, the stack-smashing check and the task watchdog all look like this. A value of 64 or above is not an Xtensa cause but one of the SDK's own, offset by 64: `69` and `70` are the interrupt watchdog on core 0 and 1, `71` a cache access error. | +| `vaddr` | The address that faulted, for `28` and `29`. | +| `pc`, `backtrace` | Where it happened and up to 16 return addresses, innermost first, as the SDK's own summary gives them — the `Backtrace:` line of a serial panic, including its convention that every address is three bytes short of the real one, so that a return address resolves to the call that made it. `backtrace[0]` is `pc`. An address inside the module `code` describes is written as an offset into it, `"+0x1a1"`, when `fromThisReset` is true; every other address is `"0x…"`. After a `task_wdt` or `int_wdt` reset the backtrace is the interrupt's stack — the watchdog's handler, not the code that was stuck — and the breadcrumb is the useful half. | +| `corrupted` | The SDK could not walk the stack to its end; the addresses up to that point still hold. | +| `build` | Sixteen hex digits of the ELF hash of the image that crashed. Its first eight are that image's `build`; when they are not this reply's `build`, the dump predates an update. | +| `bytes` | Size of the dump image in flash. | + +**Decoding.** A `"0x…"` address is firmware code: `xtensa-esp32s3-elf-addr2line -pfiaC -e firmware.elf 0x4200f1a3 …` against the ELF of the image named by `build` (its SHA-256 starts with those digits). A `"+0x…"` address is an offset into the module's `.text`, and a `.pfm` carries its own symbol table: `xtensa-esp32s3-elf-nm -nC cell_ripple.pfm` lists its functions by offset, and the one at or below the address is where it was. Without `fromThisReset` a module address stays raw: nothing recorded where that module had been loaded. The same record is printed on serial at boot as `[CRASH] …` lines. The raw dump is not served: it is every task's stack, and whatever was on one. + +### `DELETE /api/crash` + +Erases the core dump and drops the record; `crash` is absent from the next status reply. `{"ok":true}`, or `404` with `{"ok":false,"error":"no crash recorded"}` when there was nothing to clear. The erase is 64 KB of flash, and the panel holds its last frame while it runs. + ### `POST /api/params` Write the absolute parameter bus — the four channels a pattern reads as @@ -478,13 +528,14 @@ In short: HTTP is the management and state transport, OSC and MIDI are the low-l - JSON lives under `/api/`; anything else on this server is a page for a person. - Parameters are query-string or form-encoded. There is no JSON request body anywhere, and adding one would need a body parser the firmware does not have. - Errors are `{"ok":false,"error":""}` with a `4xx`/`5xx` status. Success bodies vary; the ones that report a mutation start with `"ok":true`. +- A route whose handler has to run on the loop task answers `503 {"ok":false,"error":"render loop is not answering"}` when the loop has stopped coming round (`loopStalled` in status) — the pattern list, select, install, delete, format, the knob settings, and a feature's routes that reconfigure what its loop hook is ticking. Nothing was changed. Retrying helps only once `loopStalled` reads `false` again, which a pattern stuck in `draw()` never lets it: that panel has to be restarted (`POST /api/wifi/reboot` still answers). A request made in the first twenty seconds after the loop stops waits out the rest of them for that reply; later ones get it at once. A new handler that uses `PFLoopSync::run()` answers a `false` the same way, with `PatternflowHttp::sendLoopStalled()`. - A handler must never touch FATFS, the ELF loader, the DMA engine or the CPU clock. Queue the work and let `loop()` do it; the reply then reports the pre-transition state, which callers already expect. - A feature's endpoints exist only when that feature is composed in — the route itself is absent otherwise, so clients probe `caps` rather than expecting a soft 404. Core endpoints are gated by a core `PF_*_ENABLED` flag and must leave the sketch compiling at `0`. - Nothing here may stream per-frame pixel data. That has been tried. ## Version history -- **1.5** (unreleased) — the hotspot (`hotspot` in status, `GET`/`POST /api/hotspot`); status gains `network` and `thumbs` diagnostics, `fsError` (why storage is not mounted) and `flashId`; a failed `POST /api/patterns/format` returns the reason as its `error`; `POST /api/wifi/reconnect` reconnects without a reboot; core name registration retries partial failures and preserves feature-owned services; the MIDI edition adds a `midiUsb` block to status ([`midi-spec.md`](midi-spec.md)); `GET`/`POST /api/knobs` and the `/knobs` page (encoder direction and edges per click, per knob, persisted); `PUT /update` takes a raw image, and an upload survives a stall of up to two minutes; `GET /api/wifi` gains `join` (what became of the last network asked for) and `GET /api/hotspot` gains `seen` (the names in range); a network added on the hotspot with no station link is tried at once; console pages are cached by build (`?v=` immutable, bare URLs `no-cache` with an `ETag` and `304`, another build's `v` redirected with `302`), the chrome by its CRC (`?h=`), only for a `Host` that can only be the panel; `GET /favicon.ico` answers `204`; status gains `build`, `viaHotspot` and `busy`. +- **1.5** (unreleased) — the hotspot (`hotspot` in status, `GET`/`POST /api/hotspot`); status gains `network` and `thumbs` diagnostics, `fsError` (why storage is not mounted) and `flashId`; a failed `POST /api/patterns/format` returns the reason as its `error`; `POST /api/wifi/reconnect` reconnects without a reboot; core name registration retries partial failures and preserves feature-owned services; the MIDI edition adds a `midiUsb` block to status ([`midi-spec.md`](midi-spec.md)); `GET`/`POST /api/knobs` and the `/knobs` page (encoder direction and edges per click, per knob, persisted); `PUT /update` takes a raw image, and an upload survives a stall of up to two minutes; `GET /api/wifi` gains `join` (what became of the last network asked for) and `GET /api/hotspot` gains `seen` (the names in range); a network added on the hotspot with no station link is tried at once; console pages are cached by build (`?v=` immutable, bare URLs `no-cache` with an `ETag` and `304`, another build's `v` redirected with `302`), the chrome by its CRC (`?h=`), only for a `Host` that can only be the panel; `GET /favicon.ico` answers `204`; status gains `build`, `viaHotspot` and `busy`; `load` gains `code` (where the module's code runs from); pattern names in `GET /api/patterns` and `/api/patterns/select` are JSON-escaped, so a quote or backslash in a title no longer makes the reply unparseable; a multipart request that stops mid-form is dropped - the connection is closed with no reply - instead of spinning the server into its watchdog, and leaves none of its fields behind for later requests; `moduleMemory` gains `codePolicy`; status gains `loopStackMin`; the hotspot password is JSON-escaped in `/api/hotspot`; a `POST` that is not multipart to `/api/patterns` or `/update` is answered instead of crashing the panel, a multipart boundary longer than 70 characters is refused, and a raw `PUT` sees its own query string; status gains `crash` (the core dump's summary, and the pattern and phase the reset interrupted) and `DELETE /api/crash` clears it; status gains `loopAgeMs`, `loopStalled` and `loopSyncGaveUp`, and a route that needs the render loop answers `503` (`render loop is not answering`) when a pattern never returns from `draw()`, where it used to take the whole server down with it. - **1.4** (2026-09-06) — `GET /api/patterns/file` gains `ext=thumb`; `GET /api/display` takes `brightness` and status reports it; status gains `resetReason` and `load.internal`/`load.psram`; console pages are served gzip-compressed (`Content-Encoding: gzip`); the page sender no longer truncates on a slow link; the server no longer trips the Core-0 watchdog on a request that stalls mid-header. - **1.3** (2026-09-04) — `GET`/`POST /api/clock` (Utility edition) and the `clock` block in status; `caps` gains `"clock"`. - **1.2** (2026-09-03) — the server is serviced on Core 0 (the one-connection rule stands; the render-pays rule is history); status gains `httpCore`, `netStackMin`, `loopSyncServed`/`loopSyncMaxUs`; `POST /api/params` documents `d1`..`d4` and how a held value reaches a legacy pattern; `GET /api/patterns/select` gains `step`; `GET`/`POST /api/audio` (Audio-React) are documented; `featureNav`'s microphone label is *Audio*. diff --git a/firmware/README.md b/firmware/README.md index 638ce617..a2bdfa81 100644 --- a/firmware/README.md +++ b/firmware/README.md @@ -52,10 +52,13 @@ build keeps so switching stays safe. Read it before writing a feature. > | core 3.3.8 (IDF 5.5) | 15,320 B | 7,668 B | > | core 2.0.17 (IDF 4.4) | **98,708 B** | **90,100 B** | > -> That largest-block number is the hard ceiling on a loadable module: a `.pfm`'s -> `.text` has to land in one contiguous internal executable block, because the -> S3 cannot execute loaded code from PSRAM. On core 3.x a 12 KB pattern is -> simply refused; on core 2.x it loads and runs. See +> When this was measured that largest-block number was the hard ceiling on a +> loadable module: a `.pfm`'s `.text` had to land in one contiguous internal +> executable block, so on core 3.x a 12 KB pattern was simply refused and on +> core 2.x it loaded. That ceiling is gone - a module's code now runs from +> PSRAM (`src/core_module_memory.h` says how) - but the headroom is still the +> reason for this choice: it is what the console, lwIP and every feature run +> on. See > [Internal RAM is the budget](#internal-ram-is-the-budget-everything-else-is-roomy). ### Build with PlatformIO (recommended) diff --git a/firmware/patternflow/console/status.html b/firmware/patternflow/console/status.html index f6a37768..0d9e7325 100644 --- a/firmware/patternflow/console/status.html +++ b/firmware/patternflow/console/status.html @@ -144,11 +144,22 @@ // states are worth naming, because the numbers below them look identical // either way — and asleep or paused, the frame rate is from before the // panel went dark. - var dark=d.sleep||d.consolePaused; - put('pwr',d.sleep?'asleep':d.consolePaused?'paused for storage':'awake',dark?'warn':''); + // + // And a third state that is not dark at all: the render loop has stopped + // coming round (loopStalled), so the panel shows its last frame and every + // number below is from before it stopped. Left alone this page said + // "awake, 60 fps" about a frozen panel, on the screenshot people send when + // they ask for help. It goes first: a frozen panel is not asleep, whatever + // the flag it can no longer clear says. + var frozen=!!d.loopStalled; + var dark=d.sleep||d.consolePaused||frozen; + put('pwr',frozen?'not answering for '+dur(Math.round(d.loopAgeMs/1000)): + d.sleep?'asleep':d.consolePaused?'paused for storage':'awake',frozen?'bad':dark?'warn':''); // The console no longer pauses the pattern for being open: the pause is // an upload batch or a format holding the pattern's memory while it writes. - $('pwrnote').textContent=d.sleep + $('pwrnote').textContent=frozen + ?'The render loop has not come round - most likely the active pattern is not returning from a frame; with the internet down it can also be a feature waiting on a lookup, which clears by itself within a minute or two. If the time above keeps growing, knobs and pattern switching are dead until the panel restarts; Reboot on the Wi-Fi page still works. It comes back on the same pattern, so pick another before it stops again.' + :d.sleep ?'Panel off, still on the network. Any knob or button wakes it, as does the switch on the console home page.' :d.consolePaused?'Patterns are being written to storage; the pattern resumes when that is done.':''; diff --git a/firmware/patternflow/features/clock/core_clock_http.h b/firmware/patternflow/features/clock/core_clock_http.h index 1d53b2c0..3ddb9ce1 100644 --- a/firmware/patternflow/features/clock/core_clock_http.h +++ b/firmware/patternflow/features/clock/core_clock_http.h @@ -212,7 +212,9 @@ inline void configOnLoop() { sendState(200); } -inline void handleConfig() { PFLoopSync::run([] { configOnLoop(); }); } +inline void handleConfig() { + if (!PFLoopSync::run([] { configOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} inline void begin() { if (initialized) return; diff --git a/firmware/patternflow/features/mqtt/core_mqtt_http.h b/firmware/patternflow/features/mqtt/core_mqtt_http.h index 1e0601d2..9a59a282 100644 --- a/firmware/patternflow/features/mqtt/core_mqtt_http.h +++ b/firmware/patternflow/features/mqtt/core_mqtt_http.h @@ -309,12 +309,24 @@ inline void forgetOnLoop() { // PubSubClient is not thread-safe, and the feature loop on the render core // calls client.loop() every frame. Anything here that reconnects, changes // mode or clears the config runs on the loop task (core_loop_sync.h). -inline void handlePost() { PFLoopSync::run([] { postOnLoop(); }); } -inline void handleConfig() { PFLoopSync::run([] { configOnLoop(); }); } -inline void handleDirector() { PFLoopSync::run([] { directorOnLoop(); }); } -inline void handleFlowLocal() { PFLoopSync::run([] { flowLocalOnLoop(); }); } -inline void handleMode() { PFLoopSync::run([] { modeOnLoop(); }); } -inline void handleForget() { PFLoopSync::run([] { forgetOnLoop(); }); } +inline void handlePost() { + if (!PFLoopSync::run([] { postOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleConfig() { + if (!PFLoopSync::run([] { configOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleDirector() { + if (!PFLoopSync::run([] { directorOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleFlowLocal() { + if (!PFLoopSync::run([] { flowLocalOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleMode() { + if (!PFLoopSync::run([] { modeOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleForget() { + if (!PFLoopSync::run([] { forgetOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} inline void begin() { if (initialized) return; diff --git a/firmware/patternflow/features/show/core_show_http.h b/firmware/patternflow/features/show/core_show_http.h index 7cb12c07..db459951 100644 --- a/firmware/patternflow/features/show/core_show_http.h +++ b/firmware/patternflow/features/show/core_show_http.h @@ -589,9 +589,15 @@ inline void handlePutDone() { // these start, stop, reconfigure or delete what it is ticking. The HTTP // server runs on the network core, so the bodies above are handed to the // loop task and run at the frame boundary (core_loop_sync.h). -inline void handleControl() { PFLoopSync::run([] { controlOnLoop(); }); } -inline void handleSchedule() { PFLoopSync::run([] { scheduleOnLoop(); }); } -inline void handleDelete() { PFLoopSync::run([] { deleteOnLoop(); }); } +inline void handleControl() { + if (!PFLoopSync::run([] { controlOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleSchedule() { + if (!PFLoopSync::run([] { scheduleOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleDelete() { + if (!PFLoopSync::run([] { deleteOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} inline void begin() { if (initialized) return; diff --git a/firmware/patternflow/features/weather/core_weather_http.h b/firmware/patternflow/features/weather/core_weather_http.h index 74870af3..b36ccca8 100644 --- a/firmware/patternflow/features/weather/core_weather_http.h +++ b/firmware/patternflow/features/weather/core_weather_http.h @@ -232,9 +232,15 @@ inline void activateOnLoop() { // manual fetch is left on the network core on purpose — a blocking HTTPS // round trip is exactly what should not hold a frame — guarded by the // same in-flight flag the scheduled fetch honours. -inline void handleConfig() { PFLoopSync::run([] { configOnLoop(); }); } -inline void handleForget() { PFLoopSync::run([] { forgetOnLoop(); }); } -inline void handleActivate() { PFLoopSync::run([] { activateOnLoop(); }); } +inline void handleConfig() { + if (!PFLoopSync::run([] { configOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleForget() { + if (!PFLoopSync::run([] { forgetOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} +inline void handleActivate() { + if (!PFLoopSync::run([] { activateOnLoop(); })) PatternflowHttp::sendLoopStalled(); +} inline void begin() { if (initialized) return; diff --git a/firmware/patternflow/pattern_registry.h b/firmware/patternflow/pattern_registry.h index 8d20c2bc..958ee528 100644 --- a/firmware/patternflow/pattern_registry.h +++ b/firmware/patternflow/pattern_registry.h @@ -296,6 +296,101 @@ inline void displayNameFromSlug(const char* slug, char* out, size_t outSize) { out[n] = '\0'; } +// A JSON string's value as UTF-8, read from just past its opening quote. +// False when the string never closes or decodes to nothing. +// +// This was `indexOf('"')`: stop at the next quote, copy what lies between. A +// name written as Say \"hi\" therefore ended at the first escaped quote, as +// a name ending in a backslash - which then ate the closing quote of every +// JSON reply it was written into, and /api/patterns stopped parsing. And the +// sidecars the build writes spell every non-ASCII character as \uXXXX, so a +// name like "Dynamic Moiré" was kept as backslash-u text: harmless while it +// was passed through raw and the browser decoded it, wrong the moment the +// reply escapes backslashes as it has to. +// +// So this decodes: \" \\ \/ lose their backslash, \uXXXX (and a surrogate +// pair) become UTF-8, the control escapes are dropped. The stored name is +// then the same text the module's own NAME carries once it has loaded. A +// character that will not fit is left out whole - the old copy could cut a +// multi-byte sequence in half and hand a strict client invalid UTF-8. +inline bool jsonStringValue(const char* text, char* out, size_t outSize) { + if (!text || !out || outSize == 0) return false; + auto hex4 = [](const char* h, uint32_t& value) { + value = 0; + for (int i = 0; i < 4; ++i) { + const char c = h[i]; + uint32_t digit; + if (c >= '0' && c <= '9') digit = (uint32_t)(c - '0'); + else if (c >= 'a' && c <= 'f') digit = (uint32_t)(c - 'a' + 10); + else if (c >= 'A' && c <= 'F') digit = (uint32_t)(c - 'A' + 10); + else return false; + value = (value << 4) | digit; + } + return true; + }; + size_t n = 0; + bool full = false; // once a character has not fitted, nothing later is added + auto put = [&](const char* bytes, size_t count) { + if (full || n + count + 1 > outSize) { full = true; return; } + memcpy(out + n, bytes, count); + n += count; + }; + bool closed = false; + while (*text) { + const uint8_t c = (uint8_t)*text; + if (c == '"') { closed = true; break; } + if (c == '\\') { + const char e = text[1]; + if (e == '\0') break; + if (e == 'u') { + uint32_t code; + if (!hex4(text + 2, code)) break; + text += 6; + if (code >= 0xD800 && code <= 0xDBFF && text[0] == '\\' && text[1] == 'u') { + uint32_t low; + if (hex4(text + 2, low) && low >= 0xDC00 && low <= 0xDFFF) { + code = 0x10000 + ((code - 0xD800) << 10) + (low - 0xDC00); + text += 6; + } + } + if (code >= 0xD800 && code <= 0xDFFF) code = 0xFFFD; // a lone surrogate + char utf8[4]; + size_t count; + if (code < 0x20) continue; // control: dropped + if (code < 0x80) { utf8[0] = (char)code; count = 1; } + else if (code < 0x800) { + utf8[0] = (char)(0xC0 | (code >> 6)); + utf8[1] = (char)(0x80 | (code & 0x3F)); count = 2; + } else if (code < 0x10000) { + utf8[0] = (char)(0xE0 | (code >> 12)); + utf8[1] = (char)(0x80 | ((code >> 6) & 0x3F)); + utf8[2] = (char)(0x80 | (code & 0x3F)); count = 3; + } else { + utf8[0] = (char)(0xF0 | (code >> 18)); + utf8[1] = (char)(0x80 | ((code >> 12) & 0x3F)); + utf8[2] = (char)(0x80 | ((code >> 6) & 0x3F)); + utf8[3] = (char)(0x80 | (code & 0x3F)); count = 4; + } + put(utf8, count); + continue; + } + text += 2; + if (e == '"' || e == '\\' || e == '/') put(&e, 1); + continue; // \n \t \b \f \r: dropped + } + // Raw UTF-8 in the file: take the whole sequence or none of it. + size_t count = c < 0x80 ? 1 : (c & 0xE0) == 0xC0 ? 2 : (c & 0xF0) == 0xE0 ? 3 + : (c & 0xF8) == 0xF0 ? 4 : 1; + for (size_t i = 1; i < count; ++i) { + if (((uint8_t)text[i] & 0xC0) != 0x80) { count = 1; break; } + } + if (c >= 0x20 && !(count == 1 && c >= 0x80)) put(text, count); + text += count; + } + out[n] = '\0'; + return closed && n > 0; +} + // Deliberately a substring scan rather than a JSON parser: the sidecar is our // own generated file. One open for both facts it holds: the display name // (left as it was when the sidecar has none) and whether the module was @@ -320,9 +415,12 @@ inline void readSidecar(const char* modulePath, char* nameOut, size_t nameSize, if (key >= 0) { int colon = json.indexOf(':', key + 6); int open = colon < 0 ? -1 : json.indexOf('"', colon + 1); - int close = open < 0 ? -1 : json.indexOf('"', open + 1); - if (open >= 0 && close > open + 1) { - snprintf(nameOut, nameSize, "%s", json.substring(open + 1, close).c_str()); + if (open >= 0) { + char value[MODULE_NAME_BYTES]; + const size_t room = nameSize < sizeof(value) ? nameSize : sizeof(value); + if (jsonStringValue(json.c_str() + open + 1, value, room)) { + snprintf(nameOut, nameSize, "%s", value); + } } } @@ -884,18 +982,43 @@ inline int findPatternByName(const char* name) { return -1; } +// A module is named on the crash breadcrumb (src/core_crash.h) by the load +// that brings it in. A preset has no load, so the first call into one names +// it; after that this is one pointer compare a frame. Asked of the breadcrumb +// rather than remembered here, because a module load in between renames it. +// By index, because the sketch calls every preset's setup() at boot, before +// any of them is the active one. +inline void namePresetForCrash(int index) { + const PatternEntry& entry = patterns[index]; + if (PFCrash::isRunning(entry.name)) return; + char slug[MODULE_NAME_BYTES]; + patternSlugAt(index, slug, sizeof(slug)); + PFCrash::running(entry.name, slug); +} + inline void updateActivePattern(float dt, const InputFrame& input) { if (activePatternIdx < 0) return; const PatternEntry& entry = patterns[activePatternIdx]; if (entry.modulePath) PFModuleLoader::update(dt, input); - else if (entry.update) entry.update(dt, input); + else if (entry.update) { + // The same pair of stores the loader puts around a module's update(). + namePresetForCrash(activePatternIdx); + PFCrash::enter(PFCrash::UPDATE); + entry.update(dt, input); + PFCrash::enter(PFCrash::IDLE); + } } inline void drawActivePattern() { if (activePatternIdx < 0) return; const PatternEntry& entry = patterns[activePatternIdx]; if (entry.modulePath) PFModuleLoader::draw(); - else if (entry.draw) entry.draw(); + else if (entry.draw) { + namePresetForCrash(activePatternIdx); + PFCrash::enter(PFCrash::DRAW); + entry.draw(); + PFCrash::enter(PFCrash::IDLE); + } } #undef PATTERN_ENTRY diff --git a/firmware/patternflow/patternflow.ino b/firmware/patternflow/patternflow.ino index 8e2ea235..5ab561c8 100644 --- a/firmware/patternflow/patternflow.ino +++ b/firmware/patternflow/patternflow.ino @@ -467,6 +467,11 @@ void setup() { // /api/status carries the same word as resetReason. Serial.printf("[BOOT] reset reason: %s\n", PatternflowStatusHttp::resetReasonName()); + // And, when that word is a bug, where: the core dump the SDK wrote on the + // way down and the breadcrumb of which pattern was in which call. Read here, + // before anything below can load a pattern - the first one overwrites the + // breadcrumb. Reports only; nothing later in this file acts on it. + PFCrash::begin(); reportHeap("boot"); initEncoders(); @@ -533,9 +538,22 @@ void setup() { // version has stopDMAoutput() but no way back, so that trick is unavailable — // the loader's yield() every 64 relocations is the mitigation. Revisit if a // watchdog reset actually shows up on hardware. + // + // Each one between the two stores to the crash breadcrumb (src/core_crash.h) + // that the loader puts around a module's setup(). Without them a board that + // died here came back saying "idle" with no pattern named, which reads as a + // crash in the device itself - and a preset that dies in setup() dies at + // every boot, the one case where the name is all there is to go on. Nothing + // is on the panel yet when the loop ends, so the breadcrumb stops naming the + // last of them. for (int i = 0; i < NUM_PATTERNS; i++) { - if (!patterns[i].modulePath) patterns[i].setup(); + if (patterns[i].modulePath) continue; + namePresetForCrash(i); + PFCrash::enter(PFCrash::SETUP); + patterns[i].setup(); + PFCrash::enter(PFCrash::IDLE); } + PFCrash::forget(); // The calibration test card lives outside the pattern list (it is an overlay // summoned by /api/display, not art) but still bakes its tables once here. CalibPattern::setup(); @@ -1295,12 +1313,26 @@ void applyLaneMotion(InputFrame& input, bool enabled) { static bool wasActive[4] = {false, false, false, false}; static float prevValue[4] = {0.0f, 0.0f, 0.0f, 0.0f}; static float residual[4] = {0.0f, 0.0f, 0.0f, 0.0f}; - static uint32_t handsOffUntil[4] = {0, 0, 0, 0}; + // A flag and the moment of the touch, not a deadline. This used to keep + // `handsOffUntil` and test (int32_t)(until - now) > 0 "so the wrap at 49 + // days is a non-event" - but a signed compare does not remove the wrap, it + // moves it to half the range: 24.9 days after a knob was last touched (or + // after boot, for one never touched) the difference turned positive and + // stayed positive for the next 24.9, and the lane on that knob was held + // off the whole time. An unattended panel stopped answering sound on day + // 25. The hold ends once, here, and only another delta on that knob + // starts it again. + static bool handsOn[4] = {false, false, false, false}; + static uint32_t touchedAtMs[4] = {0, 0, 0, 0}; for (int i = 0; i < 4; i++) { - if (input.knobDeltas[i] != 0) handsOffUntil[i] = input.now + LANE_HANDS_OFF_MS; - // Signed compare so the wrap at 49 days is a non-event. - const bool heldByHand = (int32_t)(handsOffUntil[i] - input.now) > 0; + if (input.knobDeltas[i] != 0) { + handsOn[i] = true; + touchedAtMs[i] = input.now; + } else if (handsOn[i] && (uint32_t)(input.now - touchedAtMs[i]) >= LANE_HANDS_OFF_MS) { + handsOn[i] = false; + } + const bool heldByHand = handsOn[i]; if (!enabled || !input.knobAudioActive[i] || heldByHand) { wasActive[i] = false; diff --git a/firmware/patternflow/platformio.ini b/firmware/patternflow/platformio.ini index 01fa6cec..926cd781 100644 --- a/firmware/patternflow/platformio.ini +++ b/firmware/patternflow/platformio.ini @@ -8,10 +8,13 @@ ; more code in IRAM, which shares physical SRAM with DRAM on the S3, plus ~3.5 KB ; of .data/.bss); the rest is heap taken during IDF startup. ; -; The headroom is what lets big .pfm modules load - a module's .text needs one -; contiguous internal executable block - and keeps the console alive with a heavy -; module resident. Measured 2026-08-18, same board: core3 15,320/7,668 vs core2 -; 98,708/90,100 (free/largest after services), muybridge_35mm refused vs 48.5 fps. +; The headroom is what keeps the console alive with a heavy module resident, and +; when this was measured it was also what let big .pfm modules load at all - a +; module's .text needed one contiguous internal executable block. Measured +; 2026-08-18, same board: core3 15,320/7,668 vs core2 98,708/90,100 (free/largest +; after services), muybridge_35mm refused vs 48.5 fps. Since 2026-10 module code +; runs from PSRAM (src/core_module_memory.h), so the second half is history; the +; first half is not, and it is the same RAM every feature and lwIP live in. ; ; The build tool is incidental: PlatformIO's classic espressif32 platform pins ; core 2.0.17; the Arduino IDE here has 3.3.8. pioarduino would give core 3 under diff --git a/firmware/patternflow/src/core_crash.h b/firmware/patternflow/src/core_crash.h new file mode 100644 index 00000000..25fca799 --- /dev/null +++ b/firmware/patternflow/src/core_crash.h @@ -0,0 +1,406 @@ +// ═══════════════════════════════════════════════════════════ +// PatternFlow - what a board that died leaves behind +// +// A crash used to leave one word. resetReason said "panic" or "task_wdt", and +// nothing said where, or in which of forty installed patterns. Two records +// answer that, and they are kept apart here because they do not live equally +// long and do not always describe the same death. +// +// THE CORE DUMP. The SDK is built with CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH and +// the partition table has always carried a 64 KB `coredump` partition - the +// table was copied to stay identical to the Arduino package and the line came +// along unexamined - so every panic (an exception, an abort(), a task or +// interrupt watchdog) was already set up to end with the SDK writing an ELF +// image of every task's stack there. It survives a power cycle and a reflash +// (the flasher writes nothing at 0xFF0000), and nothing ever read it. begin() +// reads its summary once per boot: the task, the exception, the PC, up to +// sixteen return addresses, and the hash of the image that crashed - the same +// hash /api/status publishes as `build`, so a report names the ELF that +// decodes it. +// +// THE BREADCRUMB. A backtrace says where the CPU was. It does not say which +// pattern was on the panel, and for a module it cannot be decoded at all: a +// .pfm's code is placed when it is loaded - in PSRAM, where the CPU fetches it +// through the instruction-bus alias at 0x43xxxxxx, or in internal RAM at +// 0x40xxxxxx when PSRAM cannot take it - so it sits at an address no +// firmware.elf has heard of. So the pattern's slug, which of its entry points +// was executing and where its code is executed from are kept in a corner of +// RAM that no boot initialises (.noinit, at the variable below), which a +// panic or a watchdog reset does not clear. With those an address inside the +// module becomes an offset into the .pfm, and the .pfm's own symbol table +// resolves that. +// +// The dump outlives the breadcrumb: flash keeps it until the next panic +// overwrites it or somebody clears it, while RAM is gone with the power. And +// a death does not always write a dump - a reset by the RTC watchdog never +// reaches the panic handler, and a dump too large for the partition is +// refused before the old one is erased, so the old one stays. A breadcrumb +// from today next to a backtrace from last week would be a confident lie, so +// the record says whether the two belong together (dumpFromThisReset), and +// only then are module addresses turned into offsets. +// +// What this file does not do is act on any of it. No pattern is forgotten, no +// reboot is forced, the boot latch in the sketch is untouched: this is the +// half that reports. It costs two word stores per call into a pattern and 296 +// bytes of internal RAM (heap start moved from 0x3fcaa7d8 to 0x3fcaa900 on +// the default build). 64 of those are the breadcrumb and eight are this +// file's two pointers. The other 224 are four error strings that come with +// esp_core_dump_get_summary(): the SDK's core dump code keeps its log text in +// DRAM so it can print with the flash cache off, and that holds for the one +// function in it that only ever runs at boot. Reading the dump without the +// SDK's parser would get them back, at the price of a parser. The record +// itself is allocated in PSRAM, and only on a board that has something to +// report. +// +// License: MIT +// ═══════════════════════════════════════════════════════════ +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "core_mem.h" +#include "core_module_elf.h" // ELF_MAGIC + +namespace PFCrash { + +// What the device was doing. IDLE is everything that is not pattern code - +// the blit, the features, the network - so "idle" beside a slug reads "that +// pattern was resident, and this was not inside it". +enum Phase : uint32_t { + IDLE = 0, + LOADING, // reading and relocating a .pfm, up to and including its entry point + CONSTRUCTORS, // the module's .init_array + SETUP, + UPDATE, + DRAW, + // Not a pattern's phase: begin() itself, inside the SDK's dump parser. See + // begin() for why reading a record of a crash is a thing to leave a trail + // around. + READING_DUMP, + PHASE_COUNT +}; + +inline const char* phaseName(uint32_t phase) { + switch (phase) { + case LOADING: return "loading"; + case CONSTRUCTORS: return "constructors"; + case SETUP: return "setup"; + case UPDATE: return "update"; + case DRAW: return "draw"; + case READING_DUMP: return "dump-read"; + default: return "idle"; + } +} + +constexpr size_t SLUG_BYTES = 40; // MODULE_NAME_BYTES; a longer slug is cut +constexpr uint32_t TRAIL_MAGIC = 0x50464331; // "PFC1": bump when the layout changes + +// The breadcrumb. Plain data with no initialisers on purpose: a constructor +// would run at every boot and wipe the one thing this exists to carry across. +struct Trail { + uint32_t magic; + // Over everything below `phase`. This memory is noise after a power-on, + // another image's variables after an update, and intact after a panic, and + // the only way to tell is to check. + uint32_t check; + // Outside the checksum because it is written twice per call into a pattern, + // and the budget for that is one store each way. It needs no seal of its + // own: a block whose magic and checksum hold was written by this firmware, + // and then so was this word. + uint32_t phase; + // Where the module's code is EXECUTED, and how much of it there is: the + // range a crash PC inside the module falls in. Zero for a preset, whose + // code is in firmware.elf like everything else. + uint32_t codeBase; + uint32_t codeSize; + // The checksum word of the core dump that was in flash when this boot + // looked; 0 for none. The next boot compares, and a dump that is not this + // one was written by the reset in between. + uint32_t dumpSeen; + char slug[SLUG_BYTES]; +}; + +// .noinit is a NOLOAD region of internal DRAM that the startup code neither +// copies nor zeroes, which is the whole point. It keeps its contents through +// every reset this file acts on - a panic, the watchdogs, a software restart +// - because none of them takes the power away, and loses them with the plug. +// +// Not RTC memory, which is where this first went (RTC_NOINIT_ATTR) and where +// a thing meant to outlive a reset looks like it belongs. .rtc_noinit is not +// empty: the SDK keeps the wall clock's bookkeeping there (s_rtc_last_ticks, +// s_esp_rtc_time_us, 16 bytes) and initialises it only on a power-on. The +// sketch's object links ahead of the SDK's, so a struct placed there sits +// first and pushes those two words 64 bytes along - and an update over the +// air is a warm reset into an image that then reads them from memory nothing +// wrote. That holds in both directions: upgrading into the image with the +// struct, and going back from it to any older one. The libc clock is garbage +// on that boot until SNTP sets it, and src/core_clock.h hands that clock to +// whatever draws the time or schedules by it. (From the SDK's code - +// esp_rtc_get_time_us zeroes the two words only while the slow-clock +// calibration register reads 0 - and never let onto a panel.) DRAM .noinit +// is empty in every image this project has shipped, so nothing is displaced +// by being here. +// +// What the move costs is 64 bytes of internal RAM, and what it leaves is that +// another image's .noinit need not start at the same address. Nothing has to +// be done about that: after an update this struct is read from wherever the +// old image kept something else, and the magic and the FNV seal reject a +// stale or shifted block exactly as they reject power-on noise. +inline __NOINIT_ATTR Trail trail; + +// Who the breadcrumb currently names, as a pointer nobody dereferences: a +// preset's name literal, a module's path. Ordinary RAM, because it only has +// to answer "is this still the pattern I named" within one boot. +inline const void* owner = nullptr; + +inline uint32_t seal() { + // FNV-1a over the fields that change when the pattern does. + const uint8_t* p = reinterpret_cast(&trail.codeBase); + const uint8_t* end = reinterpret_cast(&trail) + sizeof(trail); + uint32_t h = 2166136261u; + for (; p < end; ++p) { + h ^= *p; + h *= 16777619u; + } + return h; +} + +// The store around a call into pattern code. Nothing else belongs in here. +inline void enter(Phase phase) { trail.phase = phase; } + +inline bool isRunning(const void* who) { return owner == who; } + +// Name the pattern. Takes a module's path or a preset's slug and keeps the +// leaf without its extension, which for "/patterns/cell_ripple.pfm" is the +// slug and for a slug is itself. +inline void running(const void* who, const char* pathOrSlug) { + const char* leaf = pathOrSlug ? strrchr(pathOrSlug, '/') : nullptr; + leaf = leaf ? leaf + 1 : (pathOrSlug ? pathOrSlug : ""); + memset(trail.slug, 0, sizeof(trail.slug)); + snprintf(trail.slug, sizeof(trail.slug), "%s", leaf); + char* extension = strrchr(trail.slug, '.'); + if (extension) memset(extension, 0, sizeof(trail.slug) - (size_t)(extension - trail.slug)); + trail.codeBase = 0; + trail.codeSize = 0; + trail.check = seal(); + owner = who; +} + +// Where the named module's code runs from. Must be the address the CPU +// fetches it at - the one a crash PC will carry - not merely the address it +// was written through. +inline void code(uintptr_t base, uint32_t bytes) { + trail.codeBase = (uint32_t)base; + trail.codeSize = bytes; + trail.check = seal(); +} + +// Nothing is resident: a module left, or a load failed. Without this the +// breadcrumb keeps naming the last pattern through a PATTERN FAILED screen +// and a crash there is pinned on something that was not running. +inline void forget() { + memset(trail.slug, 0, sizeof(trail.slug)); + trail.codeBase = 0; + trail.codeSize = 0; + trail.phase = IDLE; + trail.check = seal(); + owner = nullptr; +} + +// What begin() found, for /api/status. Two halves, each with its own flag. +struct Record { + // The breadcrumb, when this boot followed a panic or a watchdog and the + // memory still held it. The reset it belongs to is this boot's resetReason. + bool trailed; + uint8_t phase; + char slug[SLUG_BYTES]; + uint32_t codeBase; + uint32_t codeSize; + + // The core dump, when the partition holds a valid one. + bool dumped; + // The dump was written by the reset this boot followed, so the breadcrumb + // above describes the same death. False for a dump left by an earlier one. + bool dumpFromThisReset; + bool corrupted; // the SDK's verdict on the backtrace it walked + uint8_t depth; + char task[16]; + uint32_t cause; // Xtensa EXCCAUSE as the SDK recorded it + uint32_t vaddr; + uint32_t pc; + uint32_t frames[16]; + char build[APP_ELF_SHA256_SZ]; + uint32_t bytes; +}; + +inline Record* record = nullptr; + +// An address from the dump that lies inside the code the breadcrumb named, +// as an offset into it. Only meaningful when both halves are one death. +inline bool inModule(const Record& r, uint32_t address, uint32_t& offset) { + if (!r.trailed || !r.dumpFromThisReset || r.codeSize == 0) return false; + offset = address - r.codeBase; + return offset < r.codeSize; +} + +inline const esp_partition_t* dumpPartition() { + return esp_partition_find_first(ESP_PARTITION_TYPE_DATA, + ESP_PARTITION_SUBTYPE_DATA_COREDUMP, nullptr); +} + +// Whether the partition holds something the SDK's summary parser can safely +// be handed, and that image's size and checksum word. +// +// esp_core_dump_get_summary() maps as many bytes as the first word says and +// reads an ELF header twenty bytes in, checking neither (IDF 4.4.7, +// disassembled: no bounds test before esp_partition_mmap, `addi a5, a6, 20`). +// The partition is not this firmware's alone - a board that ran a build on +// another SDK generation keeps that build's dump, and a later generation's +// header is a word longer - so the size is bounded and the ELF magic is +// looked for where THIS parser will look, before anything is parsed. +inline bool dumpPresent(const esp_partition_t* part, uint32_t& bytes, uint32_t& checkWord) { + uint32_t head[6]; + if (!part || esp_partition_read(part, 0, head, sizeof(head)) != ESP_OK) return false; + bytes = head[0]; + if (bytes == 0xFFFFFFFFu) return false; // erased: never panicked, or cleared + if (bytes < sizeof(head) + sizeof(uint32_t) || bytes > part->size) return false; + if (head[5] != PFModuleLoader::ELF_MAGIC) return false; + // CONFIG_ESP_COREDUMP_CHECKSUM_CRC32: the image ends in its own CRC. + return esp_partition_read(part, bytes - sizeof(uint32_t), &checkWord, + sizeof(checkWord)) == ESP_OK; +} + +// Once, first thing in setup(): before any pattern is loaded, because the +// first one overwrites the breadcrumb this reads. +inline void begin() { + const esp_reset_reason_t why = esp_reset_reason(); + const bool died = why == ESP_RST_PANIC || why == ESP_RST_INT_WDT || + why == ESP_RST_TASK_WDT || why == ESP_RST_WDT; + const bool kept = trail.magic == TRAIL_MAGIC && trail.check == seal() && + trail.phase < PHASE_COUNT; + const Trail last = trail; + + // This boot's own trail starts here, valid from the first line that could + // crash - which is the dump parser, a few lines down. + if (!kept) memset(&trail, 0, sizeof(trail)); + trail.magic = TRAIL_MAGIC; + forget(); + + const bool trailed = kept && died; + // The last boot died in the parser below. Reading a crash record at boot is + // the one place this file could turn a board that crashed once into a board + // that cannot start, and setup() runs on the task no watchdog covers. So + // the parse leaves a trail like a pattern does, and a boot that finds it + // does not try again. (The panic it took will normally have replaced the + // dump that caused it, so the boot after this one reads a good one.) + const bool choked = trailed && last.phase == READING_DUMP; + + const uint32_t startedMs = millis(); + const esp_partition_t* part = dumpPartition(); + uint32_t bytes = 0, checkWord = 0; + const bool present = dumpPresent(part, bytes, checkWord); + bool dumped = false; + // Zeroed, because the parser returns ESP_OK without having filled the task + // name or the backtrace when it does not find the crashed task's stack. + esp_core_dump_summary_t summary = {}; + if (present && !choked) { + enter(READING_DUMP); + // The whole image against its CRC first: a dump cut short by a power loss + // has a believable first word and anything at all after it. + dumped = esp_core_dump_image_check() == ESP_OK && + esp_core_dump_get_summary(&summary) == ESP_OK; + enter(IDLE); + } + const uint32_t readMs = millis() - startedMs; + + // A dump this boot has not seen before, after a reset that writes one. Only + // a kept trail can say what "before" was; after a power-on the dump is + // simply an old one. + const bool fresh = dumped && trailed && checkWord != last.dumpSeen; + trail.dumpSeen = present ? checkWord : 0; + trail.check = seal(); + + if (choked) { + Serial.println("[CRASH] the last boot died reading the core dump - not reading it again this boot"); + } + if (!trailed && !dumped) return; + + record = static_cast(PFMem::alloc(sizeof(Record))); + if (!record) return; + if (trailed) { + record->trailed = true; + record->phase = (uint8_t)last.phase; + memcpy(record->slug, last.slug, sizeof(record->slug)); + record->slug[sizeof(record->slug) - 1] = '\0'; + record->codeBase = last.codeBase; + record->codeSize = last.codeSize; + Serial.printf("[CRASH] the reset came during %s", phaseName(last.phase)); + if (record->slug[0]) Serial.printf(" of \"%s\"", record->slug); + else Serial.print(", no pattern resident"); + if (last.codeSize) { + Serial.printf(", module code at 0x%08x (%u B)", (unsigned)last.codeBase, + (unsigned)last.codeSize); + } + Serial.println(); + } + if (dumped) { + record->dumped = true; + record->dumpFromThisReset = fresh; + record->corrupted = summary.exc_bt_info.corrupted; + record->cause = summary.ex_info.exc_cause; + record->vaddr = summary.ex_info.exc_vaddr; + record->pc = summary.exc_pc; + record->bytes = bytes; + snprintf(record->task, sizeof(record->task), "%.15s", summary.exc_task); + snprintf(record->build, sizeof(record->build), "%.*s", (int)sizeof(record->build) - 1, + reinterpret_cast(summary.app_elf_sha256)); + const uint32_t depth = summary.exc_bt_info.depth; + record->depth = (uint8_t)(depth < 16 ? depth : 16); + for (uint8_t i = 0; i < record->depth; ++i) record->frames[i] = summary.exc_bt_info.bt[i]; + + Serial.printf("[CRASH] core dump%s: task %s, cause %u at 0x%08x, build %s, %u B, read in %u ms\n", + fresh ? "" : " (from an earlier reset)", record->task, + (unsigned)record->cause, (unsigned)record->vaddr, record->build, + (unsigned)bytes, (unsigned)readMs); + // The same shape as the SDK's own "Backtrace:" line, so the tools people + // already paste that into take this one too. A frame inside the module is + // written as an offset, which no firmware.elf could have decoded anyway. + Serial.print("[CRASH] backtrace:"); + for (uint8_t i = 0; i < record->depth; ++i) { + uint32_t offset; + if (inModule(*record, record->frames[i], offset)) { + Serial.printf(" %s+0x%x", record->slug, (unsigned)offset); + } else { + Serial.printf(" 0x%08x", (unsigned)record->frames[i]); + } + } + Serial.println(record->corrupted ? " |<-CORRUPTED" : ""); + } +} + +// Erase the dump and drop the record. True when there was anything to clear. +// +// The SDK erases the whole 64 KB rather than the first sector, which is the +// right amount - a dump is every task's stack, and "cleared" should mean gone +// - and costs a flash erase with both cores' caches off: the panel holds its +// last frame for a moment. Called from an HTTP handler, on the task that +// also reads `record`. +inline bool clear() { + uint32_t first = 0xFFFFFFFFu; + const esp_partition_t* part = dumpPartition(); + const bool stored = part && + esp_partition_read(part, 0, &first, sizeof(first)) == ESP_OK && + first != 0xFFFFFFFFu; + if (!stored && !record) return false; + if (stored) esp_core_dump_image_erase(); + free(record); + record = nullptr; + return true; +} + +} // namespace PFCrash diff --git a/firmware/patternflow/src/core_display.h b/firmware/patternflow/src/core_display.h index c9e1fc1f..d471cb7d 100644 --- a/firmware/patternflow/src/core_display.h +++ b/firmware/patternflow/src/core_display.h @@ -58,7 +58,11 @@ inline void initDisplay() { // docs/investigations/2026-08-the-panel-clock-and-the-wifi-radio.md. // Short version: 8 MHz is a real improvement and still is not shipped, // because every min_refresh_rate that keeps the panel bright bands on - // video. If you lower i2sspeed, lower min_refresh_rate with it. + // video. If you lower i2sspeed, lower min_refresh_rate with it - and read + // "The blit must stay slower than the scan" in src/hub75/VENDORED.md first: + // at a slower pixel clock the scan falls behind the blit, which then writes + // rows of the buffer that is still on the panel, and needs a wait it does + // not have. mxconfig.i2sspeed = HUB75_I2S_CFG::HZ_15M; mxconfig.min_refresh_rate = 240; mxconfig.latch_blanking = 2; diff --git a/firmware/patternflow/src/core_hotspot.h b/firmware/patternflow/src/core_hotspot.h index f0a61d7a..fdbd0c26 100644 --- a/firmware/patternflow/src/core_hotspot.h +++ b/firmware/patternflow/src/core_hotspot.h @@ -474,8 +474,10 @@ inline void registerRoutes() { json += "{\"ok\":true,"; appendStatus(json); appendSeen(json); + // Escaped: a password is checked for length only, and one quote in it + // made this reply - and so the page's hotspot card - unreadable. json += "\"pass\":\""; - json += pass; + PatternflowHttp::appendJsonText(json, pass); json += "\"}"; PatternflowHttp::server().sendHeader("Cache-Control", "no-store"); PatternflowHttp::server().send(200, "application/json", json); @@ -511,7 +513,7 @@ inline void registerRoutes() { appendStatus(json); appendSeen(json); json += "\"pass\":\""; - json += pass; + PatternflowHttp::appendJsonText(json, pass); json += "\"}"; srv.sendHeader("Cache-Control", "no-store"); srv.send(200, "application/json", json); diff --git a/firmware/patternflow/src/core_http.h b/firmware/patternflow/src/core_http.h index 48783a82..6bba6e8c 100644 --- a/firmware/patternflow/src/core_http.h +++ b/firmware/patternflow/src/core_http.h @@ -38,6 +38,18 @@ inline WebServer& server() { return httpServer; } inline bool started = false; +// Text nobody here chose - a pattern's name, a network's, an error message - +// goes into a hand-assembled JSON reply through this. One quote or backslash +// in a community pattern's title is otherwise one reply that does not parse, +// and for /api/patterns that is a page that can no longer list, select or +// delete anything - including the pattern that broke it. +inline void appendJsonText(String& json, const char* s) { + for (; s && *s; s++) { + if (*s == '"' || *s == '\\') { json += '\\'; json += *s; } + else if ((uint8_t)*s >= 0x20) json += *s; + } +} + // Registered once, by the first begin() that runs — the console chrome is // not any one page's property. inline bool chromeRegistered = false; @@ -86,4 +98,23 @@ inline void handle() { if (started) httpServer.handleClient(); } +// What a handler answers when PFLoopSync::run() came back false: the request +// needed the render loop, and the render loop has stopped coming round - a +// pattern that never returns from draw(), most likely (core_loop_sync.h). +// 503 because nothing was wrong with the request and the console itself is +// alive: /api/status says how long the loop has been gone (loopAgeMs), and +// Reboot needs nothing from it - nor does /update, as long as it is armed +// without the UPDATE screen (PF_WEBUPDATE_ALWAYS_ARMED 1, the default; the +// screen is drawn and its knob read by the loop). One wording for every route, so a +// page or a script can recognise it without knowing which handler refused. +constexpr char LOOP_STALLED_ERROR[] = "render loop is not answering"; + +inline void sendLoopStalled() { + String body = "{\"ok\":false,\"error\":\""; + body += LOOP_STALLED_ERROR; + body += "\"}"; + httpServer.sendHeader("Cache-Control", PFSend::CC_NO_STORE); + httpServer.send(503, "application/json", body); +} + } // namespace PatternflowHttp diff --git a/firmware/patternflow/src/core_knobs_http.h b/firmware/patternflow/src/core_knobs_http.h index c36b59e2..98ad439e 100644 --- a/firmware/patternflow/src/core_knobs_http.h +++ b/firmware/patternflow/src/core_knobs_http.h @@ -86,11 +86,17 @@ inline void handlePost() { return; } // On the frame task: the rebase reads the counters the frame is reading. - PFLoopSync::run([&] { + const bool applied = PFLoopSync::run([&] { for (int i = 0; i < 4; i++) { if (changed[i]) setKnobSettings(i, inv[i], sub[i]); } }); + // No frame, no change - and nothing saved either, or NVS would hold a + // setting the reply said was refused. + if (!applied) { + PatternflowHttp::sendLoopStalled(); + return; + } if (!saveKnobSettings()) { sendJson(500, "{\"ok\":false,\"error\":\"applied, but could not save to NVS\"}"); return; diff --git a/firmware/patternflow/src/core_loop_sync.h b/firmware/patternflow/src/core_loop_sync.h index 526d8e1d..1c2331f1 100644 --- a/firmware/patternflow/src/core_loop_sync.h +++ b/firmware/patternflow/src/core_loop_sync.h @@ -23,6 +23,22 @@ // holding it during a page send would stall the render exactly as it did // before the task existed. A request executed at a known point is smaller. // +// And not a wait without end. The loop task is on no watchdog, and a +// module's draw() runs on it: `while (a > PI) a -= 2 * PI` never returns +// once a is inf, and float32 gets there where the browser's doubles did +// not. The wait in runRaw() used to have no limit, so the first console +// request that needed the loop parked the network task behind the hung +// frame - and the server is one connection, so /api/status, /update and +// Reboot, which need nothing from the loop, went silent with it. A frozen +// panel took its own diagnosis and its own remote restart down too, and +// only the plug was left. Found by reading, in the 2026-10 core review, and +// not on a panel: the stock patterns all return, so the bench had never +// shown it (toolchain/tests/modules/_hang_probe is the module that does). +// Now a caller gives up once the loop has stopped ARRIVING +// (PF_LOOP_STALL_MS below), run() returns false, and the handler answers +// 503. That is everything this file does about a hung loop: it reports and +// it survives. Nothing here restarts anything. +// // License: MIT // ═══════════════════════════════════════════════════════════ #pragma once @@ -32,9 +48,61 @@ #include "core_runtime.h" #include "core_net_maintenance.h" +// How long the loop may go without reaching service() before a waiting +// caller stops believing in it. Measured from the loop's own stamp, not +// from how long the caller has waited: a storage transaction waits whole +// seconds for a module's setup() through runWhen() while the loop tests it +// every frame, and a loop doing that is healthy. +// +// What a healthy loop has been seen to do: 0.27 s frames during an install +// (patternflow.ino, at DT_MAX_MS) and one 2.02 s iteration when an upload +// met a load (docs/investigations/2026-09-firmware-runtime.md; that wait has +// since left the loop). What it can do on a bad day is longer and nobody +// has measured it: a feature's loop hook that fetches over HTTPS blocks in +// the name lookup until lwIP gives up, 14 s, when the router is up and the +// internet behind it is not (WiFiGenericClass::hostByName; the connect and +// the reply have 5 s each after that). The panel stands still for those +// seconds and then carries on, and that is not a hang. Twenty is clear of +// ONE such lookup and of nothing more. On an edition with two features that +// resolve names, both hooks run in the same iteration with no service() +// between them and each can sit out its own lookup - 28 to 30 s with a +// resolver that does not answer - and a TLS handshake has a limit of its own +// (120 s in WiFiClientSecure) that an HTTP timeout does not lower. So with +// the internet down a healthy panel can be called stalled until its loop +// comes back: a request gets a 503 it can retry, the Status page says "not +// answering", and nothing is lost. That is why the word everywhere is +// "not answering" and not "hung". runtime.maxUs in /api/status is the longest +// iteration since boot - the number to read off a panel on every edition +// before this is lowered, or before anything is ever allowed to ACT on it. +// +// Three long things are not in that reckoning because none of them is a +// gap between two visits to service(). The boot restore runs a module's +// setup() on this task, seconds of it for a heavy pattern, but from +// setup(), and no handler can be waiting: the server is not serviced until +// loop() itself has registered the routes (servicesReady, core_net_task.h). +// A pattern picked later is set up on the loader task, not here. And a +// format, an upload or a firmware image is written from the network task: +// the flash driver stops this core for one erase at a time and lets it +// through between them (the SDK is built with +// CONFIG_SPI_FLASH_YIELD_DURING_ERASE; read there, not timed on a panel). +// +// Too low costs a request a false 503 while the loop is merely busy; it is +// retried and nothing is lost. Too high costs only the FIRST request after +// a real hang - once the stamp is this old, every later request is turned +// away within one 25 ms slice. +#ifndef PF_LOOP_STALL_MS +#define PF_LOOP_STALL_MS 20000 +#endif + namespace PFLoopSync { inline TaskHandle_t loopTask = nullptr; // captured by attach() +// The request, and whose it is. Non-null: posted, and still its caller's to +// take back. Null: nothing is posted, or the loop has taken it. Both sides +// move it with one atomic exchange, so a posted request goes to exactly one +// of them - to service() to run, or back to its caller - and the other reads +// null. pendingArg points into the caller's stack frame, and that exchange +// is the whole of what decides whether the frame may still be touched. inline void (*volatile pendingFn)(void*) = nullptr; inline bool (*pendingAttempt)(void*) = nullptr; inline void* volatile pendingArg = nullptr; @@ -45,12 +113,21 @@ inline SemaphoreHandle_t callerLock = nullptr; // one request at a time // waited. A frame is ~16 ms; a module load or a long page can hold one. inline volatile uint32_t served = 0; inline volatile uint32_t maxWaitUs = 0; +// ...and how many were taken back because the loop had stopped arriving. +inline volatile uint32_t gaveUp = 0; +// millis() when the loop was last at its service point, coming or going. +inline volatile uint32_t beatMs = 0; + +inline void beat() { + __atomic_store_n(&beatMs, (uint32_t)millis(), __ATOMIC_RELEASE); +} // From the loop task, once, before any other task can call run(). inline void attach() { loopTask = xTaskGetCurrentTaskHandle(); if (!doneSignal) doneSignal = xSemaphoreCreateBinary(); if (!callerLock) callerLock = xSemaphoreCreateMutex(); + beat(); } // True before attach() as well: with no loop task known there is nobody to @@ -59,25 +136,54 @@ inline bool onLoopTask() { return loopTask == nullptr || xTaskGetCurrentTaskHandle() == loopTask; } +// Milliseconds since the loop was last at its service point. Any task may +// ask and none needs the loop to answer - /api/status publishes it, and it +// is the one number that tells a frozen panel from a slow one. +inline uint32_t loopAgeMs() { + if (!loopTask) return 0; + // The stamp first, the clock second. The other way round, a beat that + // lands between the two reads is in this reading's future, and the + // unsigned difference comes out at 49 days: a stalled loop, on the word + // of a loop that has just proved it is running. + const uint32_t beatAt = __atomic_load_n(&beatMs, __ATOMIC_ACQUIRE); + return (uint32_t)millis() - beatAt; +} + +inline bool stalled() { return loopAgeMs() >= PF_LOOP_STALL_MS; } + // From loop(), at the frame boundary: run whatever is waiting. inline void service() { - void (*fn)(void*) = __atomic_load_n(&pendingFn, __ATOMIC_ACQUIRE); + beat(); + // Take it. From this exchange until the request is put back or signalled + // done, its caller cannot leave: the caller's own exchange reads null. + void (*fn)(void*) = + __atomic_exchange_n(&pendingFn, (void (*)(void*))nullptr, __ATOMIC_ACQ_REL); if (!fn) return; // A conditional request owns its caller's storage until it completes. // Not ready means another frame, never a wait inside this task. const uint32_t startedUs = micros(); if (pendingAttempt && !pendingAttempt(pendingArg)) { PFRuntime::noteSync(micros() - startedUs); + // Back on the table, where its caller may take it back. Nothing after + // this store may read pendingArg or pendingAttempt: by the next line + // they can belong to a frame that has returned. + __atomic_store_n(&pendingFn, fn, __ATOMIC_RELEASE); return; } fn(pendingArg); PFRuntime::noteSync(micros() - startedUs); pendingAttempt = nullptr; - __atomic_store_n(&pendingFn, (void (*)(void*))nullptr, __ATOMIC_RELEASE); + // Stamp on the way out as well. A body can be long - a page sent from + // here, a client reconnecting - and the stamp from the way in is that + // old by now; the next caller's first look would find a stalled loop one + // frame before it answered. + beat(); xSemaphoreGive(doneSignal); } // Run fn(arg) on the loop task and wait for it. Inline when already there. +// False: it did not run. From another task that means the loop has stopped +// arriving and the request was taken back - it will not run later either. inline bool runRaw(void (*fn)(void*), void* arg, bool (*attempt)(void*) = nullptr) { if (onLoopTask()) { if (attempt && !attempt(arg)) return false; @@ -91,35 +197,63 @@ inline bool runRaw(void (*fn)(void*), void* arg, bool (*attempt)(void*) = nullpt __atomic_store_n(&pendingFn, fn, __ATOMIC_RELEASE); // Wait in slices so a loop that has stopped servicing is visible on // Serial rather than a silent hang of the console. + bool ran = true; uint32_t lastLogUs = t0; while (xSemaphoreTake(doneSignal, pdMS_TO_TICKS(25)) != pdTRUE) { PFNetMaintenance::poll(); + // Giving up is taking the request back, and nothing short of that. If + // the exchange reads null the loop has it - is inside fn on this stack + // frame right now, or has finished and is about to signal - and there + // is no leaving before doneSignal, however old the stamp. A loop that + // is running our call is not a dead loop in any case. + if (stalled() && + __atomic_exchange_n(&pendingFn, (void (*)(void*))nullptr, __ATOMIC_ACQ_REL)) { + ran = false; + break; + } if (micros() - lastLogUs >= 2000000) { Serial.println("[LOOP-SYNC] still waiting for the loop task"); lastLogUs = micros(); } } - const uint32_t waited = micros() - t0; - if (waited > maxWaitUs) maxWaitUs = waited; // Not served++: C++20 deprecates increment on a volatile-qualified type, // and CI builds the host tests as C++20 with -Werror. A separate read and - // write is the same single-writer store this always was - only this task - // writes it, and the reader is a status page. - served = served + 1; + // write is the same single-writer store this always was - only the task + // holding callerLock writes it, and the reader is a status page. + if (ran) { + const uint32_t waited = micros() - t0; + if (waited > maxWaitUs) maxWaitUs = waited; + served = served + 1; + } else { + gaveUp = gaveUp + 1; + Serial.printf("[LOOP-SYNC] the loop has not come round in %u ms - request withdrawn\n", + (unsigned)loopAgeMs()); + } xSemaphoreGive(callerLock); - return true; + return ran; } // Any callable, captures included: PFLoopSync::run([&] { ... }); +// False means the body did not run and never will (see runRaw). A handler +// answers that with PatternflowHttp::sendLoopStalled() and leaves whatever +// the body was going to change as it found it. +// +// [[nodiscard]] because this returned nothing while the wait had no end. A +// handler written then still compiles, and on a stalled loop it goes on to +// use what its body never set - a list with no rows in it, an index still +// at -1 - or sends no reply at all. The warning is the compiler finding +// those, in this tree and in a bundle kept outside it. template -inline void run(F&& f) { +[[nodiscard]] inline bool run(F&& f) { using Fn = typename std::remove_reference::type; - runRaw([](void* p) { (*static_cast(p))(); }, (void*)&f); + return runRaw([](void* p) { (*static_cast(p))(); }, (void*)&f); } // Retry a short, transactional attempt at each frame boundary. Returning // false must leave the operation uncommitted. A caller already on the loop // cannot wait for itself: it receives false and must retry later or reply busy. +// A caller on another task receives false only when the loop has stopped +// arriving; stalled() is how a handler tells the two apart. template inline bool runWhen(F&& attempt) { using Fn = typename std::remove_reference::type; diff --git a/firmware/patternflow/src/core_module_loader.h b/firmware/patternflow/src/core_module_loader.h index e708b182..81e0fe94 100644 --- a/firmware/patternflow/src/core_module_loader.h +++ b/firmware/patternflow/src/core_module_loader.h @@ -20,6 +20,7 @@ #include "config.h" #include "abi/pf_abi.h" #include "core_canvas.h" +#include "core_crash.h" #include "core_module_memory.h" #include "core_encoders.h" #include "core_mem.h" @@ -81,10 +82,38 @@ struct LoadedSection { uint32_t elfAddress = 0; uint32_t size = 0; uint8_t* memory = nullptr; + // Where the CPU fetches this section from. Equal to `memory` unless the + // code sits in PSRAM, where the heap's pointer is the data-bus view and + // this is the instruction-bus view of the same bytes (see execAddress). + uintptr_t exec = 0; + // What to free. Code in PSRAM starts at a cache-line boundary inside its + // allocation (PFModuleMemory::code), so `memory` is not the heap's pointer + // there; everywhere else this is null and `memory` is. + void* block = nullptr; bool executable = false; bool initArray = false; }; +// The address a block of loaded code is CALLED at. +// +// On the S3 the instruction bus and the data bus index one MMU table, so the +// PSRAM page the heap hands out at 0x3Dxxxxxx is the same page at +// 0x43xxxxxx on the instruction bus - the app's own .flash.text and +// .flash.rodata share that table, which is why the linker script carries a +// dummy section to keep them apart. Everything that WRITES the code (the +// copy, the relocations) keeps using the heap's pointer; everything that +// names the code for the CPU - a function pointer in the descriptor, a +// literal a callx8 loads, an .init_array entry, the entry point - gets this. +// Internal executable RAM is one address for both, so it maps to itself. +inline uintptr_t execAddress(const uint8_t* memory) { +#if defined(CONFIG_IDF_TARGET_ESP32S3) + if (esp_ptr_external_ram(memory)) { + return (uintptr_t)memory + (SOC_IROM_LOW - SOC_DROM_LOW); + } +#endif + return (uintptr_t)memory; +} + inline LoadedSection sections[MAX_SECTIONS]; inline int sectionCount = 0; @@ -100,6 +129,9 @@ inline uint32_t lastPsramBytes = 0; // it went on to load. Published beside the budget it was weighed against, so a // refusal is arithmetic anyone can redo from the console. inline uint32_t lastCodeBytes = 0; +// Whether the resident module's code runs from PSRAM (through the +// instruction-bus alias) rather than from internal executable RAM. +inline bool lastCodeExternal = false; inline void* moduleAllocs[MAX_MODULE_ALLOCS] = {}; inline int moduleAllocCount = 0; inline uint32_t runtimeBytes = 0; @@ -155,24 +187,35 @@ inline bool isInitArraySection(const Elf32Shdr& section, const char* names, // arbitrary patterns people upload from the community site. // // Runs after relocation and after the I-cache sync, because each entry is a -// pointer into the module's freshly patched .text. +// pointer into the module's freshly patched .text - and after the module's +// entry point, because that is what gives a constructor a host to call. inline void runInitArray() { for (int i = 0; i < sectionCount; ++i) { if (!sections[i].initArray) continue; size_t count = sections[i].size / sizeof(void (*)()); auto** constructors = reinterpret_cast(sections[i].memory); + // Their own phase on the crash breadcrumb (core_crash.h), and afterwards + // back to whichever one the caller was in, so the mark does not depend on + // where in load() this is called from. + const PFCrash::Phase caller = (PFCrash::Phase)PFCrash::trail.phase; + PFCrash::enter(PFCrash::CONSTRUCTORS); for (size_t c = 0; c < count; ++c) { uintptr_t function = reinterpret_cast(constructors[c]); if (function == 0 || function == (uintptr_t)-1) continue; // ld padding constructors[c](); } + PFCrash::enter(caller); } } inline uintptr_t mapDefinedSymbol(const Elf32Sym& symbol) { LoadedSection* section = sectionByIndex(symbol.shndx); if (!section || symbol.value > section->size) return 0; - return (uintptr_t)section->memory + symbol.value; + // A symbol in code resolves to where the code runs, not to where it was + // written: a pointer to the data-bus view is a pointer that cannot be called. + const uintptr_t base = + section->executable ? section->exec : (uintptr_t)section->memory; + return base + symbol.value; } // A module reaching for raw malloc would take memory the loader never gets back @@ -445,12 +488,15 @@ inline PFHostAPI hostAPI = { inline void unload() { active = nullptr; + // Every way a module leaves comes through here, so this is where the crash + // breadcrumb stops naming it. + PFCrash::forget(); for (int i = 0; i < moduleAllocCount; ++i) free(moduleAllocs[i]); memset(moduleAllocs, 0, sizeof(moduleAllocs)); moduleAllocCount = 0; runtimeBytes = 0; for (int i = 0; i < sectionCount; ++i) { - free(sections[i].memory); + free(sections[i].block ? sections[i].block : sections[i].memory); sections[i] = {}; } sectionCount = 0; @@ -458,6 +504,7 @@ inline void unload() { // module that left, or the partial footprint of one that never arrived. lastInternalBytes = 0; lastPsramBytes = 0; + lastCodeExternal = false; PFModuleMemory::endLoad(); } @@ -479,9 +526,55 @@ inline bool copyExecutable(uint8_t* destination, const uint8_t* source, size_t b // code into it, the instruction fetch path can still see stale lines — on // ESP32-S3 that shows up as a silent TG0WDT reboot the moment we call into // the module. Write-back + invalidate before the first call. -inline void syncExecutable(uint8_t* memory, size_t bytes) { - if (!memory || bytes == 0) return; +// +// For code in PSRAM this is not a precaution, it is the mechanism. The bytes +// were written through the data cache and will be fetched through the +// instruction cache, two caches over one memory: until the data cache writes +// its lines back the PSRAM still holds whatever was there, and until the +// instruction cache drops its lines for the alias it still holds the PREVIOUS +// module's code - a new module regularly lands on the address the last one +// freed. Write back the data-bus range, then invalidate the instruction-bus +// range, then read the alias back against the copy: a load that cannot be +// trusted is refused here rather than discovered as an IllegalInstruction. +// +// In slices under a critical section, because a flash operation started on +// the other core suspends both caches through IPC and must not land in the +// middle of a cache operation; a slice keeps interrupts off this core (the +// network core, when the loader worker runs it) for well under a millisecond. +inline bool syncExecutable(const LoadedSection& section) { + uint8_t* memory = section.memory; + size_t bytes = section.size; + if (!memory || bytes == 0) return true; size_t aligned = (bytes + 3) & ~size_t(3); +#if defined(CONFIG_IDF_TARGET_ESP32S3) + if (section.exec != (uintptr_t)memory) { + // Whole cache lines. The block owns every line it touches - that is what + // PFModuleMemory::code() allocates, and core_module_memory.h says why a + // line shared with a neighbour must not be written back from here - so + // the rounding below lands exactly on the block's own start and end. + constexpr uint32_t LINE = (uint32_t)PFModuleMemory::CODE_LINE; + constexpr uint32_t SLICE = 2048; + const uint32_t alias = (uint32_t)(section.exec - (uintptr_t)memory); + const uint32_t low = (uint32_t)(uintptr_t)memory & ~(LINE - 1); + const uint32_t high = + ((uint32_t)(uintptr_t)memory + (uint32_t)aligned + LINE - 1) & ~(LINE - 1); + static portMUX_TYPE cacheMux = portMUX_INITIALIZER_UNLOCKED; + for (uint32_t at = low; at < high; at += SLICE) { + const uint32_t span = high - at < SLICE ? high - at : SLICE; + portENTER_CRITICAL(&cacheMux); + Cache_WriteBack_Addr(at, span); + Cache_Invalidate_Addr(at + alias, span); + portEXIT_CRITICAL(&cacheMux); + } + const volatile uint32_t* written = reinterpret_cast(memory); + const volatile uint32_t* fetched = + reinterpret_cast(section.exec); + for (size_t i = 0; i < aligned / 4; ++i) { + if (written[i] != fetched[i]) return false; + } + return true; + } +#endif #if defined(CONFIG_IDF_TARGET_ESP32S3) || defined(CONFIG_IDF_TARGET_ESP32S2) Cache_WriteBack_Addr((uint32_t)memory, aligned); Cache_Invalidate_Addr((uint32_t)memory, aligned); @@ -491,6 +584,7 @@ inline void syncExecutable(uint8_t* memory, size_t bytes) { #else __asm__ __volatile__("memw" ::: "memory"); #endif + return true; } // Phase timings from the last successful load(). Switching to a module is the @@ -536,10 +630,27 @@ inline bool looksLikeModule(fs::FS& filesystem, const char* path, char* why, siz return true; } +// The module on the crash breadcrumb (core_crash.h) for as long as load() +// runs. An object rather than calls, because load() has well over a dozen ways +// out and the breadcrumb has to be right after every one of them: a load that +// failed leaves nothing resident, and a board that dies later on the PATTERN +// FAILED screen must not be reported as having died loading this. +struct LoadMark { + explicit LoadMark(const char* path) { + PFCrash::running(path, path); + PFCrash::enter(PFCrash::LOADING); + } + ~LoadMark() { + if (active) PFCrash::enter(PFCrash::IDLE); + else PFCrash::forget(); + } +}; + inline bool load(fs::FS& filesystem, const char* path) { unload(); lastError[0] = '\0'; Serial.printf("[MODULE] loading %s\n", path); + const LoadMark mark(path); const uint32_t startedUs = micros(); File file = filesystem.open(path, FILE_READ); @@ -650,8 +761,9 @@ inline bool load(fs::FS& filesystem, const char* path) { const bool executable = (section.flags & SHF_EXECINSTR) != 0; if (executable != (pass == 0)) continue; const size_t allocationSize = plannedSize[q]; + void* block = nullptr; uint8_t* memory = executable - ? static_cast(PFModuleMemory::code(allocationSize)) + ? static_cast(PFModuleMemory::code(allocationSize, &block)) : static_cast(PFModuleMemory::data( allocationSize, true, allocationSize > PF_MODULE_DATA_INTERNAL_MAX)); if (!memory) { @@ -659,7 +771,9 @@ inline bool load(fs::FS& filesystem, const char* path) { // after the cleanup and contradicts the failure it is explaining. const unsigned freeNow = (unsigned)PFModuleMemory::serviceFree(); const unsigned largest = (unsigned)heap_caps_get_largest_free_block( - executable ? PFModuleMemory::internalCode : PFModuleMemory::internalData); + !executable ? PFModuleMemory::internalData + : PFModuleMemory::codeExternal ? PFModuleMemory::externalData + : PFModuleMemory::internalCode); free(image); unload(); snprintf(lastError, sizeof(lastError), "no %s RAM: %u B, free %u, blk %u", @@ -675,7 +789,20 @@ inline bool load(fs::FS& filesystem, const char* path) { loaded.elfAddress = section.addr; loaded.size = section.size; loaded.memory = memory; + loaded.block = block; + loaded.exec = executable ? execAddress(memory) : 0; loaded.executable = executable; + if (executable) lastCodeExternal = loaded.exec != (uintptr_t)memory; + if (executable && PFModuleMemory::codeExternal && !lastCodeExternal) { + // Admission chose PSRAM and this block has no instruction-bus view: + // a target without the alias, or a heap that reaches outside the + // range execAddress() knows. Calling it would be a fetch fault. + free(image); + unload(); + PFModuleMemory::codeDemoted = true; + ++PFModuleMemory::refusals; + return fail("code in PSRAM has no instruction-bus address here"); + } loaded.initArray = isInitArraySection(section, sectionNames, sectionNamesSize); if (section.type != SHT_NOBITS) { if (executable) copyExecutable(memory, image + section.offset, section.size); @@ -686,6 +813,22 @@ inline bool load(fs::FS& filesystem, const char* path) { // Placement is over; setup()'s api->alloc() must not spend the load budget. PFModuleMemory::endLoad(); + // From here a crash PC can be inside this module, at an address that means + // nothing without the range it was loaded into - so the breadcrumb carries + // the range. module.ld collapses a module's code into one .text, so the + // first executable section is all of it. + // + // `exec`, not `memory`. A PC is an instruction-bus address, and for code in + // PSRAM that is 0x43xxxxxx while `memory` is the 0x3Dxxxxxx the loader + // wrote it through: recorded as `memory`, no frame of any crash would ever + // have fallen inside the range, and the offset this range exists to produce + // would never have been printed. For code in internal RAM the two are equal. + for (int i = 0; i < sectionCount; ++i) { + if (!sections[i].executable) continue; + PFCrash::code(sections[i].exec, sections[i].size); + break; + } + const Elf32Sym* symbols = nullptr; size_t symbolCount = 0; const char* strings = nullptr; @@ -799,19 +942,42 @@ inline bool load(fs::FS& filesystem, const char* path) { // Relocations may have patched literals inside .text — publish those // writes to the instruction side before the first call into the module. for (int i = 0; i < sectionCount; ++i) { - if (sections[i].executable) { - syncExecutable(sections[i].memory, sections[i].size); + if (sections[i].executable && !syncExecutable(sections[i])) { + // Counted as a refusal so the worker retries - and demoted first, so + // the retry is a different experiment. The allocator would hand the + // same block straight back, and a unit on which that block does not + // verify would otherwise fail this pattern six times and then every + // other one, where internal RAM ran all of them. + PFModuleMemory::codeDemoted = true; + ++PFModuleMemory::refusals; + unload(); + return fail("code in PSRAM did not read back through the instruction bus"); } } - runInitArray(); - - lastRelocateUs = micros() - startedUs - lastReadUs; - Serial.printf("[MODULE] entering %s...\n", path); using Entry = const PFPatternModule* (*)(const PFHostAPI*); Entry entry = reinterpret_cast(entryAddress); + // Entry, constructors, entry again. The entry point is what hands the + // module its host API, so it has to come before any constructor can run: + // constructors used to run first, and a namespace-scope initialiser that + // allocates, asks for a random number or logs + // (`float* trail = PFMem::allocFloats(n);`, a global object whose + // constructor uses `new`) dereferenced a null PFHost::api on the loader + // task and took the board down every time that pattern was picked, having + // built and uploaded cleanly. But the entry also copies NAME and the knob + // labels into the descriptor by value, and those may themselves be + // dynamically initialised - so it is asked again once they exist. A module + // that rejects the host on the first call has no constructors run at all. + if (!entry(&hostAPI)) { + unload(); + return fail("module rejected host ABI or panel size"); + } + runInitArray(); active = entry(&hostAPI); + + lastRelocateUs = micros() - startedUs - lastReadUs; + // Descriptor version 1 (pre-absolute) and 2 (reads the appended // absolute-param InputFrame fields) both run here — the host always fills // the extended frame. Anything else is a layout we do not provide. @@ -847,6 +1013,7 @@ inline bool load(fs::FS& filesystem, const char* path) { return fail("module name is empty or unterminated (reloc bug)"); } } + PFCrash::enter(PFCrash::SETUP); Serial.printf("[MODULE] setup %s...\n", active->name); const uint32_t setupStartedUs = micros(); active->setup(); @@ -859,14 +1026,24 @@ inline bool load(fs::FS& filesystem, const char* path) { return true; } +// The two calls a frame makes into the module, each between two stores to +// the crash breadcrumb (core_crash.h): after a reset, which of them never +// came back is the difference between "a pattern crashed" and "this one, in +// draw()". One word each way and nothing else - this runs every frame. inline void update(float dt, const InputFrame& input) { if (active) { + PFCrash::enter(PFCrash::UPDATE); active->update(dt, reinterpret_cast(&input)); + PFCrash::enter(PFCrash::IDLE); } } inline void draw() { - if (active) active->draw(); + if (active) { + PFCrash::enter(PFCrash::DRAW); + active->draw(); + PFCrash::enter(PFCrash::IDLE); + } } inline const char* error() { diff --git a/firmware/patternflow/src/core_module_memory.h b/firmware/patternflow/src/core_module_memory.h index fc8fcb68..a17ca429 100644 --- a/firmware/patternflow/src/core_module_memory.h +++ b/firmware/patternflow/src/core_module_memory.h @@ -10,9 +10,11 @@ // fail a load. // // Code is PRICED, once, for the whole module, from its section headers, -// before anything is allocated. .text is the only part with no second home, -// so it is the only thing admission can honestly be about - and it must be -// charged its own size. +// before anything is allocated. While internal RAM was the only place code +// could run, .text was the one part with no second home, so it was the only +// thing admission could honestly be about - and it had to be charged its own +// size. It has a second home now (PF_MODULE_CODE_POLICY, below); the pricing +// is unchanged for whatever still lands internally. // // Both rules were broken. The previous policy refused an executable allocation // whenever total free was under the reserve WITHOUT looking at the requested @@ -39,6 +41,45 @@ #define PF_MODULE_RUNTIME_MAX_BYTES (4u * 1024u * 1024u) #endif +// Where a module's code may live. +// +// "The S3 cannot execute from PSRAM" was the axiom everything above was built +// around, and it is true only of the pointer malloc() hands back. On the S3 +// the instruction bus (0x42000000) and the data bus (0x3C000000) index ONE MMU +// table, so a PSRAM page the heap reaches at 0x3Dxxxxxx is also fetchable at +// that address + 0x06000000. The heap does not know: it tags the region as +// data and returns the data-bus address, which faults when jumped to. The +// loader writes and relocates through that pointer and calls through the +// alias (core_module_loader.h, execAddress). +// +// 0 internal executable RAM only - the rule until 2026-10 +// 1 internal while the budget allows, PSRAM when it does not +// 2 PSRAM whenever it can take the code, internal only when it cannot +// +// 2 is what ships, and the reason is the services rather than the pattern. +// Measured 2026-10-01 on one board, A-B-B-A on the same boot: nine catalogue +// modules ran within -0.1..+0.8% of their internal frame time; the worst case +// that could be built - 10, 23 and 32 KB of code with every byte executed +// every frame, twice what the 16 KB instruction cache holds - cost 2.6..5.3% +// (0.3..0.6 ms). Against that, on the Audio edition a module with 10 KB of +// code left 21..23 KB of internal heap under rule 0 and rule 1, which is where +// lwIP was once seen to stop sending for good, and 27.5..27.9 KB under rule +// 2, whether the module carried 5 KB of code or 32. Rule 1 changes nothing for +// what already loads, which also means it leaves that where it was, and it +// makes the PSRAM path the one that only ever runs for unusual modules. +// Rule 2 runs it on every load. +#define PF_MODULE_CODE_INTERNAL 0 +#define PF_MODULE_CODE_PSRAM_FALLBACK 1 +#define PF_MODULE_CODE_PSRAM_FIRST 2 +#ifndef PF_MODULE_CODE_POLICY +#if defined(CONFIG_IDF_TARGET_ESP32S3) +#define PF_MODULE_CODE_POLICY PF_MODULE_CODE_PSRAM_FIRST +#else +// The alias is a property of the S3's MMU. Nothing else is assumed to have it. +#define PF_MODULE_CODE_POLICY PF_MODULE_CODE_INTERNAL +#endif +#endif + namespace PFModuleMemory { constexpr uint32_t internalData = MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT; constexpr uint32_t internalCode = @@ -47,6 +88,35 @@ constexpr uint32_t externalData = MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT; inline uint32_t refusals = 0; +// A variable, not the macro, so the host test can walk every rule and a bench +// build can compare placements on one boot. +inline uint8_t codePolicy = PF_MODULE_CODE_POLICY; +// The verdict admitCode() reached for the module being loaded: all of its +// executable sections go to the same home. +inline bool codeExternal = false; +// Set when code placed in PSRAM once failed to read back through the +// instruction bus. Retrying would only repeat it - the allocator hands back +// the block it was just given - so from then until reboot this unit keeps +// code where it always worked. Published as moduleMemory.codePolicy 0. +inline bool codeDemoted = false; +inline uint8_t codeRule() { + return codeDemoted ? (uint8_t)PF_MODULE_CODE_INTERNAL : codePolicy; +} + +// Code in PSRAM is given whole cache lines, start and length. The bytes are +// written through the data cache and have to be written back to the chip +// before they can be fetched, and the S3's write-back has an erratum: a line +// that someone else touches while it is being written back can come back +// wrong (ESP_ROM_HAS_CACHE_WRITEBACK_BUG; the SDK's patched routine protects +// only the unaligned edges of a range). An ordinary heap block shares its +// first and last line with the allocator's own headers and with whatever +// block sits next to it, which the other core may be splitting or merging at +// that moment. A block that owns its lines has no such neighbour. +constexpr size_t CODE_LINE = 64; // covers every cache line size the S3 offers +inline size_t codeBlockBytes(size_t bytes) { + return (bytes + CODE_LINE - 1) & ~(CODE_LINE - 1); +} + // Internal bytes this load may still place in data sections: what the reserve // leaves after the module's executable image is priced. Zero outside a load, // so setup()'s api->alloc() and the temporary ELF image stay PSRAM-first. @@ -85,10 +155,35 @@ inline void spend(size_t bytes) { // placed. codeBytes is the SUM of the module's executable sections, taken from // the section headers, so the verdict is a property of the module and not of // the order its sections happen to be walked in. +// +// With a second home for code the verdict is where, and only fails when there +// is nowhere: code in PSRAM is charged nothing against the internal budget, +// so all of it is left for the module's small data. +inline bool codeFitsExternal(size_t codeBytes) { + // Each executable section is rounded up to whole lines and aligned to one; + // eight lines of slack covers that for any module the loader will take. + return codeBytes && codeBlockBytes(codeBytes) + 8 * CODE_LINE <= + heap_caps_get_largest_free_block(externalData); +} inline bool admitCode(size_t codeBytes) { const size_t room = budget(); - dataBudget = codeBytes > room ? 0 : room - codeBytes; - return codeBytes <= room; + const bool fitsInternal = codeBytes <= room; + const uint8_t rule = codeRule(); + // Under the fallback rule "does not fit" has to mean what code() will find: + // room in total is not a block of that size, and a module refused for + // fragmentation is exactly the one the second home is for. + const bool placeable = + fitsInternal && codeBytes <= heap_caps_get_largest_free_block(internalCode); + codeExternal = + (rule == PF_MODULE_CODE_PSRAM_FIRST && codeFitsExternal(codeBytes)) || + (rule == PF_MODULE_CODE_PSRAM_FALLBACK && !placeable && + codeFitsExternal(codeBytes)); + if (codeExternal) { + dataBudget = room; + return true; + } + dataBudget = fitsInternal ? room - codeBytes : 0; + return fitsInternal; } inline void endLoad() { dataBudget = 0; } @@ -114,8 +209,29 @@ inline void* data(size_t bytes, bool zero, bool preferExternal) { return p; } -inline void* code(size_t bytes) { - void* p = internal(bytes, internalCode); +// The home admitCode() chose. A PSRAM block is returned at its data-bus +// address; the caller owns the translation to the address it is fetched from. +// +// Returns where the code goes; `*block` is what to free. They differ in PSRAM: +// the allocation carries one line of slack and the code starts at the first +// line boundary inside it, so every line it occupies lies wholly within the +// allocation. Aligned by hand rather than with heap_caps_aligned_alloc() +// because that call was the allocator's aligned path's only user in the +// image, and the path is IRAM: 1.5 KB of the internal RAM this placement +// exists to give back (measured from the linked sections, 2026-10-02). +inline void* code(size_t bytes, void** block) { + void* p = nullptr; + if (codeExternal) { + void* raw = heap_caps_malloc(codeBlockBytes(bytes) + CODE_LINE, externalData); + *block = raw; + if (raw) { + p = reinterpret_cast(((uintptr_t)raw + CODE_LINE - 1) & + ~(uintptr_t)(CODE_LINE - 1)); + } + } else { + p = internal(bytes, internalCode); + *block = p; + } if (!p) ++refusals; return p; } diff --git a/firmware/patternflow/src/core_patterns_http.h b/firmware/patternflow/src/core_patterns_http.h index c7b09090..748e9509 100644 --- a/firmware/patternflow/src/core_patterns_http.h +++ b/firmware/patternflow/src/core_patterns_http.h @@ -61,6 +61,10 @@ inline char uploadSlug[MODULE_NAME_BYTES] = {}; inline char uploadPath[MODULE_PATH_BYTES] = {}; inline char uploadError[96] = {}; inline bool uploadFailed = false; +// The upload was refused because the render loop has stopped coming round, +// not because anything was wrong with the file: handleUploadDone answers 503 +// for that, where every other failure here is the sender's and a 400. +inline bool uploadLoopStalled = false; inline bool uploadCreated = false; inline volatile bool storageOperationActive = false; struct StorageOperation { @@ -311,11 +315,27 @@ inline void sendJsonAndClose(int code, const String& body) { sendJson(code, body); } +// captureSelectionOnce() said no. There are two reasons, and they ask +// different things of whoever is at the page: a pattern still in setup() +// clears by itself, so "retry" is true; a render loop that has stopped +// coming round does not, and telling a person to retry that is telling them +// to keep pressing a dead button. +inline const char* captureRefusal(bool loopStalled) { + return loopStalled ? PatternflowHttp::LOOP_STALLED_ERROR : "pattern still loading; retry"; +} + +inline void sendCaptureRefused() { + String body = "{\"ok\":false,\"error\":\""; + body += captureRefusal(PFLoopSync::stalled()); + body += "\"}"; + sendJson(503, body); +} + // Explicit, destructive, button-initiated. See formatModuleStorage. inline void handleFormat() { StorageOperation operation; if (!captureSelectionOnce()) { - sendJson(503, "{\"ok\":false,\"error\":\"pattern still loading; retry\"}"); + sendCaptureRefused(); return; } bool ok = formatModuleStorage(); @@ -361,13 +381,13 @@ inline void handleList() { json += ",\"patterns\":["; // The list is rebuilt by tick() on the loop task after an upload; walk it // there, not underneath that. - PFLoopSync::run([&] { + const bool walked = PFLoopSync::run([&] { for (int i = 0; i < NUM_PATTERNS; i++) { if (i) json += ','; json += "{\"index\":"; json += i; json += ",\"name\":\""; - json += patterns[i].name; + PatternflowHttp::appendJsonText(json, patterns[i].name); json += "\",\"module\":"; if (patterns[i].modulePath) { char slug[MODULE_NAME_BYTES]; @@ -384,6 +404,14 @@ inline void handleList() { json += '}'; } }); + // This is the request the /patterns page opens with, so it is the one that + // used to park the network task behind a pattern stuck in draw() - and + // with it the status and Reboot routes that would have explained and + // ended the hang. Half a list is worse than an honest refusal. + if (!walked) { + PatternflowHttp::sendLoopStalled(); + return; + } json += "],\"pendingRev\":"; json += PatternflowPackSelect::rev; json += ",\"pending\":["; @@ -503,7 +531,7 @@ inline void handleDeleteMany() { } if (!captureSelectionOnce()) { - sendJson(503, "{\"ok\":false,\"error\":\"pattern still loading; retry\"}"); + sendCaptureRefused(); return; } @@ -592,7 +620,7 @@ inline void handleDelete() { // Drop it out of executable RAM before the file goes, in case it is running. if (!captureSelectionOnce()) { - sendJson(503, "{\"ok\":false,\"error\":\"pattern still loading; retry\"}"); + sendCaptureRefused(); return; } if (!removeModuleFiles(slug)) { @@ -624,6 +652,7 @@ inline void handleUpload() { __atomic_store_n(&storageOperationActive, true, __ATOMIC_RELEASE); uploadCreated = false; uploadFailed = false; + uploadLoopStalled = false; uploadError[0] = '\0'; uploadBytes = 0; uploadPath[0] = '\0'; @@ -669,7 +698,8 @@ inline void handleUpload() { if (!captureSelectionOnce()) { uploadFailed = true; uploadPath[0] = '\0'; // no file opened: do not delete an original on failure - snprintf(uploadError, sizeof(uploadError), "pattern still loading; retry"); + uploadLoopStalled = PFLoopSync::stalled(); + snprintf(uploadError, sizeof(uploadError), "%s", captureRefusal(uploadLoopStalled)); return; } @@ -732,6 +762,7 @@ inline void handlePutBody() { __atomic_store_n(&storageOperationActive, true, __ATOMIC_RELEASE); uploadCreated = false; uploadFailed = false; + uploadLoopStalled = false; uploadError[0] = '\0'; uploadBytes = 0; uploadPath[0] = '\0'; @@ -775,7 +806,8 @@ inline void handlePutBody() { if (!captureSelectionOnce()) { uploadFailed = true; uploadPath[0] = '\0'; // no file opened: do not delete an original on failure - snprintf(uploadError, sizeof(uploadError), "pattern still loading; retry"); + uploadLoopStalled = PFLoopSync::stalled(); + snprintf(uploadError, sizeof(uploadError), "%s", captureRefusal(uploadLoopStalled)); return; } @@ -836,7 +868,7 @@ inline void handleUploadDone() { String body = "{\"ok\":false,\"error\":\""; body += uploadError[0] ? uploadError : "upload failed"; body += "\"}"; - sendJsonAndClose(400, body); + sendJsonAndClose(uploadLoopStalled ? 503 : 400, body); return; } if (!uploadPath[0] || uploadBytes == 0) { @@ -913,7 +945,7 @@ inline void handleSelect() { int index = -1; String name; - PFLoopSync::run([&] { + const bool resolved = PFLoopSync::run([&] { if (byStep) { // ?step=+1|-1: the next (or previous) pattern that is not hidden, // wrapping, from wherever the panel is now. @@ -951,6 +983,16 @@ inline void handleSelect() { restorePending = false; } }); + // Not "no such pattern": nothing was looked up, and nothing was queued. A + // switch is exactly what a person tries when the panel has frozen, so the + // reply has to say that switching is not what will fix it. The header goes + // out here too - this route is called from other origins, which cannot + // read the reason without it. + if (!resolved) { + server().sendHeader("Access-Control-Allow-Origin", "*"); + PatternflowHttp::sendLoopStalled(); + return; + } server().sendHeader("Cache-Control", "no-store"); server().sendHeader("Access-Control-Allow-Origin", "*"); @@ -961,7 +1003,7 @@ inline void handleSelect() { String body = "{\"ok\":true,\"index\":"; body += index; body += ",\"name\":\""; - body += name; + PatternflowHttp::appendJsonText(body, name.c_str()); body += "\"}"; server().send(200, "application/json", body); } diff --git a/firmware/patternflow/src/core_status_http.h b/firmware/patternflow/src/core_status_http.h index b2bd5807..8fd9352f 100644 --- a/firmware/patternflow/src/core_status_http.h +++ b/firmware/patternflow/src/core_status_http.h @@ -39,6 +39,7 @@ #include "core_http.h" #include "core_build.h" // PFBuild::id() +#include "core_crash.h" // the `crash` object and DELETE /api/crash #include "core_bus.h" #include "core_canvas.h" // presentUs #include "core_send.h" @@ -58,6 +59,8 @@ extern bool nvsUsable; extern uint32_t nvsFailures; // Panel brightness as set; the sketch owns it (K1 and /api/display move it). extern uint8_t currentBrightness; +// The Arduino core's handle for the task loop() runs on (cores/esp32/main.cpp). +extern TaskHandle_t loopTaskHandle; namespace PatternflowStatusHttp { @@ -128,6 +131,86 @@ inline void appendText(String& json, const char* s) { } } +// One address out of a core dump. An address inside the module the breadcrumb +// named is written "+0x": the raw value is wherever +// that module's code was placed on that boot - internal RAM at 0x40xxxxxx, or +// PSRAM seen through the instruction bus at 0x43xxxxxx - which no ELF can +// decode, while the offset is what the .pfm's own symbol table resolves. +// Everything else is firmware code and goes out as the address addr2line +// wants. +inline void appendCrashAddress(String& json, const PFCrash::Record& r, uint32_t address) { + char text[12]; + uint32_t offset; + if (PFCrash::inModule(r, address, offset)) { + snprintf(text, sizeof(text), "+0x%x", (unsigned)offset); + } else { + snprintf(text, sizeof(text), "0x%08x", (unsigned)address); + } + json += '"'; + json += text; + json += '"'; +} + +// `crash`: what the last death left (core_crash.h), absent on a board with +// nothing to report - which is nearly every board, so nearly every reply is +// unchanged. Two halves that come and go separately. pattern/phase/code are +// the breadcrumb and are only there on the boot that follows a panic or a +// watchdog, describing the reset `resetReason` names. `dump` is the core dump +// and stays through power cycles until the next panic replaces it or DELETE +// /api/crash erases it; fromThisReset says whether the two are one event. +// +// The task name, the slug and the build hash are escaped like any other text +// nobody here chose: they come out of flash and uninitialised RAM, and a dump +// written by some other firmware is only probably well-formed. +inline void appendCrash(String& json) { + const PFCrash::Record* r = PFCrash::record; + if (!r) return; + char hex[12]; + json += "\"crash\":{"; + if (r->trailed) { + json += "\"pattern\":\""; + PatternflowHttp::appendJsonText(json, r->slug); + json += "\",\"phase\":\""; + json += PFCrash::phaseName(r->phase); + json += '"'; + if (r->codeSize) { + snprintf(hex, sizeof(hex), "0x%08x", (unsigned)r->codeBase); + json += ",\"code\":{\"base\":\""; + json += hex; + json += "\",\"size\":"; + json += r->codeSize; + json += '}'; + } + } + if (r->dumped) { + if (r->trailed) json += ','; + json += "\"dump\":{\"fromThisReset\":"; + json += r->dumpFromThisReset ? "true" : "false"; + json += ",\"task\":\""; + PatternflowHttp::appendJsonText(json, r->task); + json += "\",\"cause\":"; + json += r->cause; + snprintf(hex, sizeof(hex), "0x%08x", (unsigned)r->vaddr); + json += ",\"vaddr\":\""; + json += hex; + json += "\",\"pc\":"; + appendCrashAddress(json, *r, r->pc); + json += ",\"backtrace\":["; + for (uint8_t i = 0; i < r->depth; i++) { + if (i) json += ','; + appendCrashAddress(json, *r, r->frames[i]); + } + json += "],\"corrupted\":"; + json += r->corrupted ? "true" : "false"; + json += ",\"build\":\""; + PatternflowHttp::appendJsonText(json, r->build); + json += "\",\"bytes\":"; + json += r->bytes; + json += '}'; + } + json += "},"; +} + // What is keeping the device from answering promptly, as a word an open // console slows its polling for: "update" while a firmware image is arriving // or the reboot after it is due, "storage" while the pattern volume is being @@ -193,6 +276,8 @@ inline void handleStatus() { json += "\"resetReason\":\""; json += resetReasonName(); json += "\","; + // And when that word is a bug, where it happened. + appendCrash(json); json += "\"panel\":\""; json += PANEL_RES_W; json += 'x'; @@ -362,6 +447,12 @@ inline void handleStatus() { json += ",\"netMaintenance\":{\"calls\":"; json += PFNetMaintenance::calls; json += ",\"maxGapMs\":"; json += PFNetMaintenance::maxGapMs; json += "}"; + // The loop task's stack, the least it has ever had free. The blit runs + // inside whatever frame a pattern's draw() is holding and takes about 640 B + // of it; a module with a large local in draw() is what would bring this + // down, and until this field nothing reported it. + json += ",\"loopStackMin\":"; + json += loopTaskHandle ? (uint32_t)uxTaskGetStackHighWaterMark(loopTaskHandle) : 0; json += ",\"netStackMin\":"; json += PatternflowNetTask::stackMinFree; // Handlers that had to run on the loop task, and the longest one waited. @@ -377,6 +468,21 @@ inline void handleStatus() { json += PFLoopSync::served; json += ",\"loopSyncMaxUs\":"; json += PFLoopSync::maxWaitUs; + // How long since loop() was last at its frame boundary, read straight off + // the stamp and never through the loop: this is the number that says the + // loop is the thing that is stuck, so it cannot wait on the loop to be + // produced. A frame or two is a running panel. Seconds, and larger on the + // next poll by the time between the polls, is a pattern that is not coming + // back from draw() - `active` above names it, and Reboot still answers. + // One reading for both fields, so a reply never says an age under the + // limit and stalled in the same breath. + const uint32_t loopAge = PFLoopSync::loopAgeMs(); + json += ",\"loopAgeMs\":"; + json += loopAge; + json += ",\"loopStalled\":"; + json += loopAge >= PF_LOOP_STALL_MS ? "true" : "false"; + json += ",\"loopSyncGaveUp\":"; + json += PFLoopSync::gaveUp; json += ",\"colorBits\":"; json += dma_display->getCfg().getPixelColorDepthBits(); json += ",\"refreshHz\":"; @@ -402,7 +508,11 @@ inline void handleStatus() { json += PFModuleLoader::lastInternalBytes; json += ",\"psram\":"; json += PFModuleLoader::lastPsramBytes; - json += "}"; + // Where the resident module's code runs from. In PSRAM it is counted in + // `psram` above and costs the services nothing. + json += ",\"code\":\""; + json += PFModuleLoader::lastCodeExternal ? "psram" : "internal"; + json += "\"}"; json += ",\"moduleMemory\":{\"reserve\":"; json += PF_MODULE_INTERNAL_RESERVE; json += ",\"runtimeBytes\":"; json += PFModuleLoader::runtimeBytes; json += ",\"runtimePeakBytes\":"; json += PFModuleLoader::runtimePeakBytes; @@ -416,6 +526,9 @@ inline void handleStatus() { // codeBytes <= budget, and both numbers are readable while it is running. json += ",\"budget\":"; json += (uint32_t)PFModuleMemory::budget(); json += ",\"codeBytes\":"; json += PFModuleLoader::lastCodeBytes; + // The rule in force for where code goes (core_module_memory.h): 2 as built, + // 0 once a PSRAM placement has failed to verify on this unit since boot. + json += ",\"codePolicy\":"; json += PFModuleMemory::codeRule(); json += ",\"execLargest\":"; json += heap_caps_get_largest_free_block(PFModuleMemory::internalCode); json += ",\"refusals\":"; json += PFModuleMemory::refusals; @@ -445,6 +558,29 @@ inline void handleStatus() { server().send(200, "application/json", json); } +// DELETE /api/crash +// +// Erases the core dump and drops the record, so `crash` leaves the status +// reply. Nothing else clears it and that is deliberate: the dump has to +// outlive the power cycle a person does before they think to look, so it +// stays until a newer panic replaces it or somebody says they have read it. +// 404 when there was nothing, as DELETE /api/wifi does for an unknown name. +// +// The erase is 64 KB of flash and holds both cores for as long as that takes; +// the panel keeps its last frame meanwhile. It runs here and not through the +// loop task because it touches nothing a frame is using. +inline void handleCrashClear() { + PatternflowPatternsHttp::noteConsoleApiCall(); + server().sendHeader("Cache-Control", "no-store"); + if (!PFCrash::clear()) { + server().send(404, "application/json", + "{\"ok\":false,\"error\":\"no crash recorded\"}"); + return; + } + Serial.println("[CRASH] core dump erased, record cleared"); + server().send(200, "application/json", "{\"ok\":true}"); +} + inline void handleIndex() { if (PatternflowPatternsHttp::noteConsolePageOpened()) { PatternflowPatternsHttp::sendConsoleWakePage(); @@ -588,6 +724,7 @@ inline void begin() { server().on("/status", HTTP_GET, handleIndex); server().on("/api/status", HTTP_GET, handleStatus); + server().on("/api/crash", HTTP_DELETE, handleCrashClear); server().on("/api/sleep", HTTP_POST, handleSleep); server().on("/api/params", HTTP_POST, handleParams); diff --git a/firmware/patternflow/src/core_thumbs.h b/firmware/patternflow/src/core_thumbs.h index 08e45f32..431756a7 100644 --- a/firmware/patternflow/src/core_thumbs.h +++ b/firmware/patternflow/src/core_thumbs.h @@ -76,6 +76,17 @@ inline uint32_t ioMaxUs = 0; inline uint32_t captureMaxUs = 0; inline uint32_t nextGeneration = 0; +// Set by a forget() or forgetAll() that could not reach the loop because the +// loop has stopped coming round (core_loop_sync.h). The file is removed all +// the same; what cannot be done from the network task is the PSRAM half, +// because the slots are the loop's and a loop that looks dead can wake up. +// So the loop drops them itself, in service(), whenever it next runs - every +// picture rather than the one slug, since one flag cannot name a slug and a +// cache may always be emptied. Until then the disk worker writes nothing: a +// save queued before the stall would put a deleted pattern's picture straight +// back on the volume, and a stale .thumb is the bug that survives a reboot. +inline bool dropAllDeferred = false; + // One immutable job, at most one extra frame of PSRAM. The loop owns the // cache and submits snapshots; the existing network task owns disk I/O. // No extra task/stack and no lock held across a file read or write. @@ -163,7 +174,8 @@ inline void serviceDisk() { __ATOMIC_ACQ_REL, __ATOMIC_ACQUIRE)) return; const uint32_t started = micros(); io.ok = false; - if (__atomic_load_n(&io.slot->generation, __ATOMIC_ACQUIRE) == io.generation) + if (__atomic_load_n(&io.slot->generation, __ATOMIC_ACQUIRE) == io.generation && + !__atomic_load_n(&dropAllDeferred, __ATOMIC_ACQUIRE)) io.ok = io.write ? writeToDisk(io) : readFromDisk(io); const uint32_t elapsed = micros() - started; if (elapsed > ioMaxUs) ioMaxUs = elapsed; @@ -212,8 +224,24 @@ inline bool queueIO(Slot& s, bool write) { return true; } +// Loop task only: every copy in PSRAM is stale. +inline void dropAll() { + for (int i = 0; i < slotCount; i++) { + __atomic_store_n(&slots[i].generation, ++nextGeneration, __ATOMIC_RELEASE); + if (slots[i].px) free(slots[i].px); + slots[i].px = nullptr; + } + slotCount = 0; +} + inline void service() { collectIO(); + // Drop, then lower the flag: the generations are bumped before the disk + // worker is let back in, so a job from before the stall is stale by then. + if (__atomic_load_n(&dropAllDeferred, __ATOMIC_ACQUIRE)) { + dropAll(); + __atomic_store_n(&dropAllDeferred, false, __ATOMIC_RELEASE); + } for (int i = 0; i < slotCount; ++i) { Slot& s = slots[i]; if (s.savePending && !s.savedThisBoot && s.px && queueIO(s, true)) { @@ -272,7 +300,7 @@ inline void paint(const uint16_t* px) { // The slot stays (slugs are few) and will look at the volume again if the // same slug is ever installed back. inline void forget(const char* slug) { - PFLoopSync::run([&] { + const bool dropped = PFLoopSync::run([&] { if (Slot* s = find(slug)) { __atomic_store_n(&s->generation, ++nextGeneration, __ATOMIC_RELEASE); if (s->px) free(s->px); @@ -282,6 +310,7 @@ inline void forget(const char* slug) { s->savePending = false; } }); + if (!dropped) __atomic_store_n(&dropAllDeferred, true, __ATOMIC_RELEASE); char path[64]; pathFor(slug, path, sizeof(path)); if (FFat.exists(path)) FFat.remove(path); @@ -289,14 +318,9 @@ inline void forget(const char* slug) { // The volume was formatted: every file is gone, so every copy is stale. inline void forgetAll() { - PFLoopSync::run([] { - for (int i = 0; i < slotCount; i++) { - __atomic_store_n(&slots[i].generation, ++nextGeneration, __ATOMIC_RELEASE); - if (slots[i].px) free(slots[i].px); - slots[i].px = nullptr; + if (!PFLoopSync::run([] { dropAll(); })) { + __atomic_store_n(&dropAllDeferred, true, __ATOMIC_RELEASE); } - slotCount = 0; - }); } } // namespace PFThumbs diff --git a/firmware/patternflow/src/core_web_update.h b/firmware/patternflow/src/core_web_update.h index 733768d0..810a79e3 100644 --- a/firmware/patternflow/src/core_web_update.h +++ b/firmware/patternflow/src/core_web_update.h @@ -71,6 +71,12 @@ inline bool armed = false; // set only from the on-device UPDATE screen inline bool uploading = false; inline bool rejected = false; // this POST arrived while not armed inline bool completedOk = false; +// Whether THIS request carried an image. The three flags above are written +// by the body callback, and a request with no image in it - a POST that is +// not a form, a form with no file part - never reaches that callback: its +// completion handler would otherwise answer with whatever the last upload +// left behind ("locked", or an old error, or nothing at all). +inline bool bodySeen = false; inline bool bootMarkedValid = false; inline unsigned progressPct = 0; inline size_t expectedBytes = 0; @@ -126,6 +132,7 @@ inline void handleUpload() { switch (up.status) { case UPLOAD_FILE_START: { uploadAttempts++; + bodySeen = true; rejected = !isArmed() || uploading || rebootAtMs != 0; if (rejected) { Serial.println("[UPDATE] upload refused (not armed)"); @@ -180,6 +187,7 @@ inline void handleUpload() { break; } case UPLOAD_FILE_ABORTED: { + bodySeen = false; if (uploading) { Update.abort(); uploading = false; @@ -198,6 +206,7 @@ inline void handleRawBody() { switch (raw.status) { case RAW_START: { uploadAttempts++; + bodySeen = true; rejected = !isArmed() || uploading || rebootAtMs != 0; if (rejected) { Serial.println("[UPDATE] raw upload refused (not armed)"); @@ -247,6 +256,7 @@ inline void handleRawBody() { break; } case RAW_ABORTED: { + bodySeen = false; if (uploading) { Update.abort(); uploading = false; @@ -260,6 +270,12 @@ inline void handleRawBody() { // Completion handler — runs after the whole POST body is consumed. inline void handleUploadDone() { + if (!bodySeen) { + server().send(400, "application/json", + "{\"error\":\"no firmware image in the request\"}"); + return; + } + bodySeen = false; if (rejected) { server().send(403, "application/json", "{\"error\":\"locked - on the device: hold K2 for NETWORK, then turn K4 for UPDATE\"}"); diff --git a/firmware/patternflow/src/hub75/ESP32-HUB75-MatrixPanel-I2S-DMA.cpp b/firmware/patternflow/src/hub75/ESP32-HUB75-MatrixPanel-I2S-DMA.cpp index 5a32ecbb..ba668277 100644 --- a/firmware/patternflow/src/hub75/ESP32-HUB75-MatrixPanel-I2S-DMA.cpp +++ b/firmware/patternflow/src/hub75/ESP32-HUB75-MatrixPanel-I2S-DMA.cpp @@ -556,7 +556,25 @@ static constexpr uint8_t PF_SPREAD_MAX_DEPTH = 10; // Two adjacent columns share one aligned 32-bit word in every plane. #define PF_CLEAR32 (((uint32_t)BITMASK_RGB12_CLEAR << 16) | (uint32_t)BITMASK_RGB12_CLEAR) -static void pfBuildSpread(uint8_t depth) +// An empty asm statement that claims to rewrite its operand. The value is +// untouched; the optimiser just stops knowing where it came from. Each use +// says what this GCC does with that knowledge. MSVC, which the host test may +// be built with, has no such statement and needs none. +#if defined(__GNUC__) +#define PF_OPAQUE(v) __asm__("" : "+r"(v)) +#define PF_NOINLINE __attribute__((noinline)) +#else +#define PF_OPAQUE(v) ((void)0) +#define PF_NOINLINE +#endif +// PF_NOINLINE keeps a function out of blitRGB888. For the passes further down +// that is the whole idea: folded back in, they would share one set of registers +// again. For the rest it is about where the code lives - whatever is inlined +// into an IRAM function is in IRAM, and that is internal RAM. + +// Not inlined: this runs once per depth change, and inlined into blitRGB888 it +// sat in IRAM for nothing. +static PF_NOINLINE void pfBuildSpread(uint8_t depth) { const uint8_t maskOffset = 16 - depth; for (int v = 0; v < 256; v++) @@ -613,6 +631,316 @@ static void pfBuildSpread(uint8_t depth) // Rec.601 luma pivot for the saturation boost, then the caller's gamma/WB LUT. // Kept as a macro so it stays inside the pixel loop with no call overhead. #define PF_POST_PIXEL(src, dst) { int _r = (src)[0], _g = (src)[1], _b = (src)[2]; int _y = (_r * 77 + _g * 150 + _b * 29) >> 8; _r = _y + (((_r - _y) * satBoostQ8) >> 8); _g = _y + (((_g - _y) * satBoostQ8) >> 8); _b = _y + (((_b - _y) * satBoostQ8) >> 8); if (_r < 0) _r = 0; else if (_r > 255) _r = 255; if (_g < 0) _g = 0; else if (_g > 255) _g = 255; if (_b < 0) _b = 0; else if (_b > 255) _b = 255; (dst)[0] = lutR[_r]; (dst)[1] = lutG[_g]; (dst)[2] = lutB[_b]; } + +// --------------------------------------------------------------------------- +// The kernel as three short passes (2026-10). +// +// Counted in the linked image, the one long loop this replaces - it is still +// here, as pfBlitRowPair - ran 635 instructions per column pair in 691 +// cycles: compute-bound, not waiting on the DMA engine as its own comment +// said. Three things in the disassembly made up the excess, and none of them +// was the arithmetic: +// +// - depth was a run-time value, so every plane word paid for a multiply by +// six, two variable shifts and a plane pointer reloaded from a stack +// array - 25 instructions where 10 do; +// - this compiler (GCC 8.4 at -Os) synthesises x77, x150 and x29 as +// shift/add chains, about 20 instructions a pixel where three multiplies +// do; +// - one 400-instruction body keeps more values alive than the 14 registers +// a windowed call leaves it, so every column pair made 48 reloads from the +// stack and 12 literal loads. +// +// Each pass below is short enough to keep its state in registers, and the +// plane pass has the depth written into it: +// +// 1a pfPostRow saturation + gamma/WB LUT RGB888 -> 3 bytes a pixel +// pfPostRowRaw the same, when the LUTs are the identity +// 1b pfSpreadRow on-time + plane spread 6 bytes -> (lo, hi) a column +// 2 pfStoreRow8 plane words 2 x (lo, hi) -> 8 plane words +// +// 426 instructions per column pair, counted the same way - 390 when the LUTs +// are the identity, which is what config.h ships. Same tables, same DMA +// words, same on-time sum: check_blit.py holds this to its scalar reference +// exactly as it held the loop. What the split costs is a store and a reload +// of every intermediate, through scratch on the caller's stack. +// +// THE NEXT SPEED-UP HAS A FLOOR THAT IS NOT THE CPU. flipDMABuffer() returns +// before the swap happens: the retired buffer stays on the panel until the DMA +// finishes its pass, up to 3.3 ms, and the next blit writes into that very +// buffer. Nothing shows, for one reason - the scan takes 96 us a row pair +// (12 buffers of 128 words at 16 MHz), it is already ahead when the blit +// starts, and a writer slower than the reader never catches it. At 390 +// instructions a column pair this blit cannot go under 104 us a row pair. A +// kernel that does get under 96 - 3.07 ms a frame - or one split across the +// two cores, overtakes the scan and tears; it has to wait for the swap first. + +// GCC's count-register pass rewrites `while (p != end)` as a down-counter, +// meant for Xtensa's zero-overhead loop instruction. In passes 1a and 1b it +// then finds no register left for the counter and keeps it on the stack - +// load, decrement, store, branch, every time round. With the pass off the test +// is a compare against a pointer. +#if defined(__XTENSA__) +#define PF_NO_COUNT_REG __attribute__((optimize("no-branch-count-reg"))) +#else +#define PF_NO_COUNT_REG +#endif + +// Columns per trip through the passes. The scratch between them is 192 + 256 +// bytes at this size, and it lives on the stack of whoever calls blitRGB888 - +// not in static storage, so a blit split by row pair across the two cores +// would have nothing shared to trip over. +#define PF_BLIT_CHUNK 32 + +// What pass 1b hands pass 2 for one column: every plane's R1 G1 B1 R2 G2 B2. +struct PfPlanes { uint32_t lo, hi; }; + +// One pixel of pass 1a as far as the clamp, written once for its two forms. +// +// (c*q + y*(256-q)) >> 8 is PF_POST_PIXEL's y + (((c-y)*q) >> 8) with the y*256 +// carried inside the shift - exact for any q, because the shift is arithmetic. +// It costs one multiply shared by the three channels where the other form +// costs a subtract in each. +#define PF_SATURATE(src) \ + const int r = (src)[0], g = (src)[1], b = (src)[2]; \ + const int k = ((r * wr + g * wg + b * wb) >> 8) * kq; \ + int R = (r * q + k) >> 8, G = (g * q + k) >> 8, B = (b * q + k) >> 8; \ + if (R < 0) R = 0; else if (R > 255) R = 255; \ + if (G < 0) G = 0; else if (G > 255) G = 255; \ + if (B < 0) B = 0; else if (B > 255) B = 255; +// What it reads besides the pixel. The weights are hidden, or the three +// multiplies come back as the shift/add chains. +#define PF_SATURATE_CONSTANTS(q) \ + const int kq = 256 - (q); \ + int wr = 77, wg = 150, wb = 29; \ + PF_OPAQUE(wr); PF_OPAQUE(wg); PF_OPAQUE(wb); + +// Pass 1a. `n` pixels of one canvas row -> the three bytes the tables are +// indexed by, written six apart: the other row of the pair fills the gaps, and +// pass 1b walks one pointer instead of two. +static PF_NOINLINE PF_NO_COUNT_REG void IRAM_ATTR +pfPostRow(const uint8_t *src, unsigned n, uint8_t *out, + const uint8_t *lutR, const uint8_t *lutG, const uint8_t *lutB, int q) +{ + PF_SATURATE_CONSTANTS(q) + const uint8_t *const end = src + (size_t)n * 3; + do + { + PF_SATURATE(src) + out[0] = lutR[R]; out[1] = lutG[G]; out[2] = lutB[B]; + src += 3; out += 6; + } while (src != end); +} + +// Pass 1a when all three LUTs are the identity - gamma 1.0 and white balance +// 1/1/1, which is what config.h ships. The lookups would hand each index back +// unchanged, at two instructions apiece and three table pointers this loop has +// no registers for: 41 instructions a pixel with them, 32 without. +static PF_NOINLINE PF_NO_COUNT_REG void IRAM_ATTR +pfPostRowRaw(const uint8_t *src, unsigned n, uint8_t *out, int q) +{ + PF_SATURATE_CONSTANTS(q) + const uint8_t *const end = src + (size_t)n * 3; + do + { + PF_SATURATE(src) + out[0] = (uint8_t)R; out[1] = (uint8_t)G; out[2] = (uint8_t)B; + src += 3; out += 6; + } while (src != end); +} + +static bool IRAM_ATTR pfLutIsIdentity(const uint8_t *lut) +{ + for (unsigned v = 0; v < 256; v++) + if (lut[v] != v) return false; + return true; +} + +// Pass 1b. Six index bytes per column - the top row's three, then the bottom +// row's - become that column's planes, and their on-time is returned. +static PF_NOINLINE PF_NO_COUNT_REG uint32_t IRAM_ATTR +pfSpreadRow(const uint8_t *idx, unsigned n, PfPlanes *out) +{ + const uint32_t *const spreadLo = pfSpreadLo, *const spreadHi = pfSpreadHi; + const uint16_t *const cie = pfCie; + uint32_t on = 0; + const PfPlanes *const end = out + n; + do + { + const unsigned t0 = idx[0], t1 = idx[1], t2 = idx[2], m0 = idx[3], m1 = idx[4], m2 = idx[5]; + on += (uint32_t)cie[t0] + cie[t1] + cie[t2] + cie[m0] + cie[m1] + cie[m2]; + out->lo = spreadLo[t0] | (spreadLo[t1] << 1) | (spreadLo[t2] << 2) | + (spreadLo[m0] << 3) | (spreadLo[m1] << 4) | (spreadLo[m2] << 5); + out->hi = spreadHi[t0] | (spreadHi[t1] << 1) | (spreadHi[t2] << 2) | + (spreadHi[m0] << 3) | (spreadHi[m1] << 4) | (spreadHi[m2] << 5); + idx += 6; out++; + } while (out != end); + return on; +} + +static inline uint32_t pfSwapHalves(uint32_t v) { return (v << 16) | (v >> 16); } + +#if defined(SPIRAM_DMA_BUFFER) +#define PF_PLANE_WRITEBACK(p) Cache_WriteBack_Addr((uint32_t)(p), sizeof(uint32_t)) +#else +#define PF_PLANE_WRITEBACK(p) ((void)0) +#endif + +// One plane's word for two adjacent columns: the ten control bits of each stay, +// the six colour bits of each are replaced. `fieldA` is already in bits 0-5 and +// `fieldB` in bits 16-21; which column the DMA sends first is the FIFO order's +// business, as in the loop. The word access is spelled the way the loop spells +// it, for the reason given there. +#define PF_STORE_PLANE(fieldA, fieldB) \ + { \ + void *aligned = __builtin_assume_aligned(p, sizeof(uint32_t)); \ + uint32_t word; \ + memcpy(&word, aligned, sizeof(word)); \ + word = (word & PF_CLEAR32) | \ + (aLow ? ((fieldA) | (fieldB)) : pfSwapHalves((fieldA) | (fieldB))); \ + memcpy(aligned, &word, sizeof(word)); \ + PF_PLANE_WRITEBACK(p); \ + p = (ESP32_I2S_DMA_STORAGE_TYPE *)((uint8_t *)p + stride); \ + } + +// Pass 2, for a depth of exactly 8. `pairs` column pairs of (lo, hi) go into +// all eight planes of the row pair; `col` is plane 0 at the first column and +// `stride` the bytes from one plane to the next. With the depth known every +// field's position is a constant - an extract and a shift - where the loop +// computes 6 * d and shifts by a register. +static PF_NOINLINE void IRAM_ATTR +pfStoreRow8(const PfPlanes *in, unsigned pairs, ESP32_I2S_DMA_STORAGE_TYPE *col, size_t stride) +{ + const bool aLow = (ESP32_TX_FIFO_POSITION_ADJUST(0) == 0); + const uint32_t fieldBMask = 0x003F0000u; + const PfPlanes *const end = in + (size_t)pairs * 2; + do + { + const uint32_t loA = in[0].lo, hiA = in[0].hi, loB = in[1].lo, hiB = in[1].hi; + // Hidden: shown that p starts at col, GCC walks a pointer of its own for + // half the planes and keeps stride multiples on the stack for the rest, + // three instructions a column pair more than stepping p by the stride. + ESP32_I2S_DMA_STORAGE_TYPE *p = col; + PF_OPAQUE(p); + PF_STORE_PLANE(loA & 0x3Fu, (loB << 16) & fieldBMask); + PF_STORE_PLANE((loA >> 6) & 0x3Fu, (loB << 10) & fieldBMask); + PF_STORE_PLANE((loA >> 12) & 0x3Fu, (loB << 4) & fieldBMask); + PF_STORE_PLANE((loA >> 18) & 0x3Fu, (loB >> 2) & fieldBMask); + PF_STORE_PLANE((loA >> 24) & 0x3Fu, (loB >> 8) & fieldBMask); + PF_STORE_PLANE(hiA & 0x3Fu, (hiB << 16) & fieldBMask); + PF_STORE_PLANE((hiA >> 6) & 0x3Fu, (hiB << 10) & fieldBMask); + PF_STORE_PLANE((hiA >> 12) & 0x3Fu, (hiB << 4) & fieldBMask); + in += 2; col += 2; + } while (in != end); +} + +// The loop the passes replaced, kept whole for what they do not cover: an odd +// width, where alternate planes are not word-aligned, and any depth but 8. +// One row pair per call; returns its on-time. Nothing this firmware ships +// reaches it, so it is the one part of the blit that is not in IRAM - and not +// inlined, or it would follow blitRGB888 back there. +static PF_NOINLINE uint64_t +pfBlitRowPair(const uint8_t *top, const uint8_t *bot, + ESP32_I2S_DMA_STORAGE_TYPE *plane0, uint16_t w, uint8_t depth, + const uint8_t *lutR, const uint8_t *lutG, const uint8_t *lutB, int satBoostQ8) +{ + uint64_t onTime = 0; + + // Row pointers do not change across the pixels of a row — hoist all of + // them once instead of recomputing per pixel per plane. A row's planes + // are one allocation, `w` words apart (rowBitStruct::getDataPtr). + ESP32_I2S_DMA_STORAGE_TYPE *plane[16]; + for (uint8_t d = 0; d < depth; d++) + plane[d] = plane0 + (size_t)d * w; + + // Two columns per step. x and x + 1 are adjacent uint16_t words in every + // plane, so both halves of the panel for both columns — two pixel pairs — + // are one aligned 32-bit read-modify-write per plane instead of two. + // (This used to add that every access arbitrates with the DMA engine and + // that this is the cost. Counted, it is not: see the passes above.) + // Which column takes the low half is the FIFO order's business, decided + // once by the macro at compile time. + const bool aLow = (ESP32_TX_FIFO_POSITION_ADJUST(0) == 0); + uint16_t x = 0; + // DMA allocation is word-aligned. With an even width, every plane and + // every two-column offset is too. Tell the compiler that fact below: + // memcpy on a uint16_t* otherwise becomes four byte loads, a stack + // spill, and four byte stores on Xtensa, not the intended word access. + // Odd widths use the scalar path since alternate planes are unaligned. + const uint16_t pairedWidth = (w & 1) ? 0 : w; + for (; x + 1 < pairedWidth; x += 2) + { + uint32_t loA, hiA, loB, hiB; + PF_PLANES_FOR(top + (size_t)x * 3, bot + (size_t)x * 3, loA, hiA, onTime); + PF_PLANES_FOR(top + (size_t)(x + 1) * 3, bot + (size_t)(x + 1) * 3, loB, hiB, onTime); + + uint8_t d = depth; + while (d > 5) + { + --d; + const uint32_t bA = (hiA >> (6 * (d - 5))) & 0x3Fu; + const uint32_t bB = (hiB >> (6 * (d - 5))) & 0x3Fu; + const uint32_t both = aLow ? (bA | (bB << 16)) : (bB | (bA << 16)); + ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + x; + void *aligned = __builtin_assume_aligned(p, sizeof(uint32_t)); + uint32_t word; + memcpy(&word, aligned, sizeof(word)); + word = (word & PF_CLEAR32) | both; + memcpy(aligned, &word, sizeof(word)); +#if defined(SPIRAM_DMA_BUFFER) + Cache_WriteBack_Addr((uint32_t)p, sizeof(word)); +#endif + } + while (d) + { + --d; + const uint32_t bA = (loA >> (6 * d)) & 0x3Fu; + const uint32_t bB = (loB >> (6 * d)) & 0x3Fu; + const uint32_t both = aLow ? (bA | (bB << 16)) : (bB | (bA << 16)); + ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + x; + void *aligned = __builtin_assume_aligned(p, sizeof(uint32_t)); + uint32_t word; + memcpy(&word, aligned, sizeof(word)); + word = (word & PF_CLEAR32) | both; + memcpy(aligned, &word, sizeof(word)); +#if defined(SPIRAM_DMA_BUFFER) + Cache_WriteBack_Addr((uint32_t)p, sizeof(word)); +#endif + } + } + + // Scalar fallback for odd widths, one column at a time. + for (; x < w; ++x) + { + uint32_t lo, hi; + PF_PLANES_FOR(top + (size_t)x * 3, bot + (size_t)x * 3, lo, hi, onTime); + const uint16_t idx = ESP32_TX_FIFO_POSITION_ADJUST(x); + uint8_t d = depth; + while (d > 5) + { + --d; + const uint16_t bits = (uint16_t)((hi >> (6 * (d - 5))) & 0x3Fu); + ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + idx; + *p = (*p & BITMASK_RGB12_CLEAR) | bits; +#if defined(SPIRAM_DMA_BUFFER) + Cache_WriteBack_Addr((uint32_t)p, sizeof(ESP32_I2S_DMA_STORAGE_TYPE)); +#endif + } + while (d) + { + --d; + const uint16_t bits = (uint16_t)((lo >> (6 * d)) & 0x3Fu); + ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + idx; + *p = (*p & BITMASK_RGB12_CLEAR) | bits; +#if defined(SPIRAM_DMA_BUFFER) + Cache_WriteBack_Addr((uint32_t)p, sizeof(ESP32_I2S_DMA_STORAGE_TYPE)); +#endif + } + } + + return onTime; +} + void IRAM_ATTR MatrixPanel_I2S_DMA::blitRGB888(const uint8_t *rgb, const uint8_t *lutR, const uint8_t *lutG, const uint8_t *lutB, int satBoostQ8) @@ -646,102 +974,47 @@ void IRAM_ATTR MatrixPanel_I2S_DMA::blitRGB888(const uint8_t *rgb, // the bitplane writes underneath it. See lastFrameOnTime(). uint64_t onTime = 0; + // The passes need every plane word-aligned (an even width) and the depth + // pass 2 is written for. Everything else goes to the loop they replaced. + const bool passes = (depth == 8) && !(w & 1); + // Asked of the tables on every frame, not remembered: they are the caller's, + // and it rebuilds them in place whenever gamma or white balance is tuned. + // 768 compares a frame, against 36 instructions on each of its column pairs. + const bool rawLut = passes && pfLutIsIdentity(lutR) && pfLutIsIdentity(lutG) && pfLutIsIdentity(lutB); + // A row's planes are one allocation, `w` words apart + // (rowBitStruct::getDataPtr), so plane 0 and a stride reach all of them. + const size_t stride = (size_t)w * sizeof(ESP32_I2S_DMA_STORAGE_TYPE); + uint8_t idx[PF_BLIT_CHUNK * 6]; + PfPlanes planes[PF_BLIT_CHUNK]; + // A two-scan panel lights row `r` and row `r + ROWS_PER_FRAME` together, and // both live in the same uint16_t. Walk the pairs, not the rows. for (uint8_t row = 0; row < rows; row++) { const uint8_t *top = rgb + (size_t)row * w * 3; const uint8_t *bot = rgb + ((size_t)row + rows) * w * 3; + ESP32_I2S_DMA_STORAGE_TYPE *plane0 = getRowDataPtr(row, 0); - // Row pointers do not change across the pixels of a row — hoist all of - // them once instead of recomputing per pixel per plane. - ESP32_I2S_DMA_STORAGE_TYPE *plane[16]; - for (uint8_t d = 0; d < depth; d++) - plane[d] = getRowDataPtr(row, d); - - // Two columns per step. x and x + 1 are adjacent uint16_t words in every - // plane, so both halves of the panel for both columns — two pixel pairs — - // are one aligned 32-bit read-modify-write per plane instead of two. The - // DMA engine is reading this memory the whole time, and every access that - // has to arbitrate with it is the cost; halving them is the point. - // Which column takes the low half is the FIFO order's business, decided - // once by the macro at compile time. - const bool aLow = (ESP32_TX_FIFO_POSITION_ADJUST(0) == 0); - uint16_t x = 0; - // DMA allocation is word-aligned. With an even width, every plane and - // every two-column offset is too. Tell the compiler that fact below: - // memcpy on a uint16_t* otherwise becomes four byte loads, a stack - // spill, and four byte stores on Xtensa, not the intended word access. - // Odd widths use the scalar path since alternate planes are unaligned. - const uint16_t pairedWidth = (w & 1) ? 0 : w; - for (; x + 1 < pairedWidth; x += 2) + if (!passes) { - uint32_t loA, hiA, loB, hiB; - PF_PLANES_FOR(top + (size_t)x * 3, bot + (size_t)x * 3, loA, hiA, onTime); - PF_PLANES_FOR(top + (size_t)(x + 1) * 3, bot + (size_t)(x + 1) * 3, loB, hiB, onTime); - - uint8_t d = depth; - while (d > 5) - { - --d; - const uint32_t bA = (hiA >> (6 * (d - 5))) & 0x3Fu; - const uint32_t bB = (hiB >> (6 * (d - 5))) & 0x3Fu; - const uint32_t both = aLow ? (bA | (bB << 16)) : (bB | (bA << 16)); - ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + x; - void *aligned = __builtin_assume_aligned(p, sizeof(uint32_t)); - uint32_t word; - memcpy(&word, aligned, sizeof(word)); - word = (word & PF_CLEAR32) | both; - memcpy(aligned, &word, sizeof(word)); -#if defined(SPIRAM_DMA_BUFFER) - Cache_WriteBack_Addr((uint32_t)p, sizeof(word)); -#endif - } - while (d) - { - --d; - const uint32_t bA = (loA >> (6 * d)) & 0x3Fu; - const uint32_t bB = (loB >> (6 * d)) & 0x3Fu; - const uint32_t both = aLow ? (bA | (bB << 16)) : (bB | (bA << 16)); - ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + x; - void *aligned = __builtin_assume_aligned(p, sizeof(uint32_t)); - uint32_t word; - memcpy(&word, aligned, sizeof(word)); - word = (word & PF_CLEAR32) | both; - memcpy(aligned, &word, sizeof(word)); -#if defined(SPIRAM_DMA_BUFFER) - Cache_WriteBack_Addr((uint32_t)p, sizeof(word)); -#endif - } + onTime += pfBlitRowPair(top, bot, plane0, w, depth, lutR, lutG, lutB, satBoostQ8); + continue; } - - // Scalar fallback for odd widths, one column at a time. - for (; x < w; ++x) + for (unsigned x = 0; x < w; x += PF_BLIT_CHUNK) { - uint32_t lo, hi; - PF_PLANES_FOR(top + (size_t)x * 3, bot + (size_t)x * 3, lo, hi, onTime); - const uint16_t idx = ESP32_TX_FIFO_POSITION_ADJUST(x); - uint8_t d = depth; - while (d > 5) + const unsigned n = (w - x < PF_BLIT_CHUNK) ? (w - x) : PF_BLIT_CHUNK; + if (rawLut) { - --d; - const uint16_t bits = (uint16_t)((hi >> (6 * (d - 5))) & 0x3Fu); - ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + idx; - *p = (*p & BITMASK_RGB12_CLEAR) | bits; -#if defined(SPIRAM_DMA_BUFFER) - Cache_WriteBack_Addr((uint32_t)p, sizeof(ESP32_I2S_DMA_STORAGE_TYPE)); -#endif + pfPostRowRaw(top + (size_t)x * 3, n, idx, satBoostQ8); + pfPostRowRaw(bot + (size_t)x * 3, n, idx + 3, satBoostQ8); } - while (d) + else { - --d; - const uint16_t bits = (uint16_t)((lo >> (6 * d)) & 0x3Fu); - ESP32_I2S_DMA_STORAGE_TYPE *p = plane[d] + idx; - *p = (*p & BITMASK_RGB12_CLEAR) | bits; -#if defined(SPIRAM_DMA_BUFFER) - Cache_WriteBack_Addr((uint32_t)p, sizeof(ESP32_I2S_DMA_STORAGE_TYPE)); -#endif + pfPostRow(top + (size_t)x * 3, n, idx, lutR, lutG, lutB, satBoostQ8); + pfPostRow(bot + (size_t)x * 3, n, idx + 3, lutR, lutG, lutB, satBoostQ8); } + onTime += pfSpreadRow(idx, n, planes); + pfStoreRow8(planes, n / 2, plane0 + x, stride); } } diff --git a/firmware/patternflow/src/hub75/VENDORED.md b/firmware/patternflow/src/hub75/VENDORED.md index 38707d23..d457c7a3 100644 --- a/firmware/patternflow/src/hub75/VENDORED.md +++ b/firmware/patternflow/src/hub75/VENDORED.md @@ -64,6 +64,129 @@ saturation values and both FIFO orders (over 30 million words). See the [bench report](../../../../docs/investigations/2026-09-firmware-runtime.md) for measurements. +**2026-10-01: three short passes.** The instructions the September kernel +executes were counted - its code taken from the linked image and run one +instruction at a time against a synthetic panel - and came to 635 per column +pair, in the 691 cycles a 5.90 ms frame works out to. That is 1.09 cycles an +instruction. The blit was compute-bound, and the DMA contention the paragraph +above still half believed in was never the cost. The excess was in three +places, none of them the arithmetic: + +- depth was a run-time value, so each plane word paid for a multiply by six, + two variable shifts and a plane pointer reloaded from a stack array - 25 + instructions where 10 do; +- GCC 8.4 at `-Os` builds x77, x150 and x29 out of shifts and adds, about 20 + instructions a pixel where three multiplies do; +- one 400-instruction loop body holds more live values than the 14 registers + it has, so every column pair made 48 reloads from the stack and 12 literal + loads. + +The kernel is now three passes over 32 columns at a time, each short enough to +keep its state in registers: `pfPostRow` (saturation and the LUTs, RGB888 to +three index bytes a pixel), `pfSpreadRow` (on-time and the spread tables, to a +`(lo, hi)` per column) and `pfStoreRow8` (the plane words, with the depth of 8 +written in, so every field position is a constant). Saturation is computed as +`(c*q + y*(256-q)) >> 8`, which is `y + (((c-y)*q) >> 8)` exactly: the shift is +arithmetic. Same tables, same DMA words, same on-time sum. + +| | September | now | +|---|---|---| +| instructions per column pair, from the image | 635 | 426, or 390 with identity LUTs | +| IRAM the blit holds | 2,440 B | 1,430 B | +| `blitRGB888` stack frame | 224 B | 592 B, and 48 more inside a pass | + +**When all three LUTs are the identity, pass 1a skips them.** Gamma 1.0 and +white balance 1/1/1 are what `config.h` ships, and then the three lookups hand +each index back unchanged - at two instructions apiece, and with three table +pointers the loop has no registers for. `pfPostRowRaw` is the same pass without +them, 32 instructions a pixel instead of 41. `blitRGB888` asks the tables on +every frame rather than remembering the answer, because they are the caller's +and it rebuilds them in place whenever gamma or white balance is tuned: 768 +compares, against 36 instructions on each of 2,048 column pairs. So tuning +white balance away from 1/1/1 costs about a third of a millisecond a frame; +that is the LUT being used, not a regression. + +On a board (2026-10-02, default composition, A-B-B-A between the two images): +`presentUs` **5,960 -> 3,496 µs**, the same on Origin, Wave Cascade and +Two-stream, and every pattern's `frameUs` fell by the same 2.45 ms (Origin +12.08 -> 9.61 ms). With white balance tuned off the identity (`wb_r=0.95`) +it reads 3,802 µs: the LUT path costs 306 µs. That is a little better than +the instruction count predicted (5.90 ms x 390 / 635 = 3.62), so cycles per +instruction did not rise with the intermediates going through memory. + +The loop the passes replace is kept whole, as `pfBlitRowPair`: an odd width and +every depth other than 8 still go through it, and `blit_test.cpp` still runs it +(depths 2-7, 9, 10, and width 127 at depth 8). It is no longer in IRAM and +neither is `pfBuildSpread` - no shipped configuration executes the first and +the second runs once per depth change - which is where the 1.2 KB of internal +RAM comes back from. + +Three things in the passes are there for this compiler alone, and each says so +where it is used: `PF_OPAQUE`, an empty asm statement that hides where a value +came from (the luma weights, or the multiplies come back as chains; and the +plane pointer); `PF_NOINLINE`; and `optimize("no-branch-count-reg")` on passes +1a and 1b, where GCC otherwise invents a down-counter, has no register for it +and keeps it on the stack. None of them changes a result, and MSVC builds the +host test without any of them. + +The scratch between the passes - 448 bytes - is on the caller's stack and not +in static storage, so that a blit split by row pair across the two cores finds +nothing shared. + +Roads that were compiled and did not pay, written down so nobody walks them +twice: `-O2` on the kernel (about a third more instructions, all spills); +reusing the left neighbour's result where a colour repeats (the miss path grows +by more than the hit path saves on anything but flat fills); per-channel tables +with the LUT and the CIE curve folded in (9 % fewer instructions, 9 KB of +DRAM). One road was not taken for a different reason: this GCC never emits +`addx`, so every table index is a shift and an add, and interleaving +`pfSpreadLo` with `pfSpreadHi` into one table of pairs would save 14 +instructions a column pair. `check_blit.py` finds the kernel by the first +table's declaration, and the two were left as they are. + +#### The blit must stay slower than the scan + +`flipDMABuffer()` returns before the swap takes effect. +`flip_dma_output_buffer()` (`platforms/esp32s3/gdma_lcd_parallel16.cpp`) only +rewrites the `next` pointer of each chain's last descriptor - upstream's wait +is commented out - and `back_buffer_id` moves at once. So the buffer just +retired stays on the panel until the DMA reaches the end of its pass, up to +3.33 ms later, and the next `blitRGB888` writes into exactly that buffer. + +Nothing shows, and what prevents it is a rate, not a wait. The scan and the +blit both walk row pairs 0 to 31 in order. The scan spends twelve buffers of +128 words on a row pair: 1,536 clocks, 96 µs at 16 MHz, 3.07 ms for all 32 (the +261 µs of padding comes after the last row). It is already somewhere down the +frame when the flip happens; the blit starts later, at row 0, and as long as it +needs more than 96 µs a row pair it falls further behind with every row and +never writes a row that is still to be shown. The September kernel took 184 µs. +This one cannot take less than 104 - 390 instructions, 64 column pairs, one +cycle each at 240 MHz - and measures 109 (3,496 µs over 32 row pairs), 119 on +the LUT path. + +About one flip in six hundred lands inside the last padding descriptor, after +the DMA has already fetched its `next`, and the old buffer gets one more whole +pass. The same argument covers it: that pass begins within 6 µs of the flip, +still ahead of the blit. + +**The line is the scan's row time: buffers per row x words per row / pixel +clock.** For the shipped configuration - 12 buffers, 128 words, 16 MHz - that +is 96 µs per row pair, 3.07 ms per frame, and it is a property of the +configuration, not of the kernel: at the 10 MHz the S3 really runs when +`i2sspeed` is lowered (`core_display.h`), the same 12 buffers take 154 µs a row +pair and this kernel is the faster of the two. The September loop was slower +than the scan at every clock the driver can pick; this one needs the pixel +clock to stay above about 14.8 MHz at 12 buffers (11.1 at 9, 9.9 at 8). The +rate argument also assumes the first plane store comes at least one scan row +after the flip, which today the pattern's own draw supplies and nothing in +the driver enforces. A blit faster than the line +catches the scan from behind unless it starts late: at B µs a row pair it must +not begin sooner than 3,072 - 31 B µs after the flip - 1.1 ms at B = 62, the +whole pass as B goes to zero. Crossing it tears: the rows past the point where +the two meet show the next frame one refresh early, and the row where they meet +mixes old and new planes. This kernel split across both cores crosses it. Any +change that does has to wait for the swap before its first write. + ### 2. `resumeDMAoutput()` — the way back from `stopDMAoutput()` Upstream's `stopDMAoutput()` is a one-way trip ("Screen will forever be black diff --git a/firmware/patternflow/src/status_index.h b/firmware/patternflow/src/status_index.h index 02fba6d4..4ac1891c 100644 --- a/firmware/patternflow/src/status_index.h +++ b/firmware/patternflow/src/status_index.h @@ -155,11 +155,22 @@ function paint(d){ // states are worth naming, because the numbers below them look identical // either way — and asleep or paused, the frame rate is from before the // panel went dark. - var dark=d.sleep||d.consolePaused; - put('pwr',d.sleep?'asleep':d.consolePaused?'paused for storage':'awake',dark?'warn':''); + // + // And a third state that is not dark at all: the render loop has stopped + // coming round (loopStalled), so the panel shows its last frame and every + // number below is from before it stopped. Left alone this page said + // "awake, 60 fps" about a frozen panel, on the screenshot people send when + // they ask for help. It goes first: a frozen panel is not asleep, whatever + // the flag it can no longer clear says. + var frozen=!!d.loopStalled; + var dark=d.sleep||d.consolePaused||frozen; + put('pwr',frozen?'not answering for '+dur(Math.round(d.loopAgeMs/1000)): + d.sleep?'asleep':d.consolePaused?'paused for storage':'awake',frozen?'bad':dark?'warn':''); // The console no longer pauses the pattern for being open: the pause is // an upload batch or a format holding the pattern's memory while it writes. - $('pwrnote').textContent=d.sleep + $('pwrnote').textContent=frozen + ?'The render loop has not come round - most likely the active pattern is not returning from a frame; with the internet down it can also be a feature waiting on a lookup, which clears by itself within a minute or two. If the time above keeps growing, knobs and pattern switching are dead until the panel restarts; Reboot on the Wi-Fi page still works. It comes back on the same pattern, so pick another before it stops again.' + :d.sleep ?'Panel off, still on the network. Any knob or button wakes it, as does the switch on the console home page.' :d.consolePaused?'Patterns are being written to storage; the pattern resumes when that is done.':''; @@ -268,366 +279,392 @@ PF.status(paint); )HTML"; // GENERATED by firmware/toolchain/console_pages.py build — do not edit. -// gzip -9 of STATUS_INDEX_HTML (14087 bytes → 5725 bytes), served with Content-Encoding: gzip. +// gzip -9 of STATUS_INDEX_HTML (15075 bytes → 6144 bytes), served with Content-Encoding: gzip. static const uint8_t STATUS_INDEX_HTML_GZ[] PROGMEM = { 0x1f,0x8b,0x08,0x00,0x00,0x00,0x00,0x00,0x02,0xff,0xc5,0x5b,0x6b,0x72,0xdb,0x48, - 0x92,0xfe,0xaf,0x53,0x94,0xe5,0x69,0x01,0x18,0x81,0x10,0x25,0xbf,0x41,0x53,0x5a, - 0xcb,0x8f,0x99,0x9e,0x6e,0xf7,0x78,0x25,0xf7,0x4e,0x6c,0x7b,0x3b,0x36,0x40,0xa0, - 0x20,0xa2,0x05,0xa2,0x10,0x55,0xa0,0x28,0x0e,0xc5,0x88,0x3e,0xc4,0xfc,0xd9,0x0b, - 0xec,0x15,0xf6,0xff,0x1e,0xa5,0x4f,0xb2,0x5f,0x66,0x15,0x48,0x90,0x92,0x6d,0x75, - 0xcc,0x8f,0x9d,0xe8,0xb1,0x50,0x8f,0xcc,0xca,0xca,0x77,0x26,0xc0,0x97,0x0f,0x32, - 0x95,0x36,0xf3,0x5a,0x8a,0x71,0x33,0x29,0x8f,0x77,0x5e,0xd2,0x1f,0x51,0x26,0xd5, - 0xc5,0x70,0x57,0x56,0xbb,0x34,0x21,0x93,0xec,0xf8,0xe5,0x44,0x36,0x89,0x48,0xc7, - 0x89,0x36,0xb2,0x19,0xee,0x4e,0x9b,0xbc,0xf7,0x9c,0x16,0x4d,0xaa,0x8b,0xba,0x11, - 0x46,0xa7,0xc3,0xdd,0x83,0x3a,0xef,0xa5,0xaa,0x32,0xaa,0x94,0xd1,0x2f,0xe6,0x64, - 0x3c,0xcc,0xfa,0x47,0xa3,0x47,0x8f,0x8e,0x8e,0x76,0x8f,0x5f,0x1e,0xd8,0x8d,0xc7, - 0x0e,0xe0,0xb8,0xc8,0xfd,0x07,0xb3,0xa2,0xca,0xd4,0x2c,0xfa,0xf0,0x2e,0xf0,0xf3, - 0x69,0x95,0x36,0x85,0xaa,0xfc,0x60,0x71,0x95,0x68,0xf1,0xb7,0xa1,0x5d,0x0b,0x7f, - 0x1c,0x7a,0x07,0x49,0x5d,0x1c,0x98,0x26,0x69,0xa6,0xc6,0x0b,0xbf,0x1f,0x7e,0xfa, - 0x39,0x7c,0x3f,0x3c,0x92,0x2f,0xc2,0xf3,0xf0,0xa7,0xf0,0xc3,0xf0,0x6f,0x00,0x1f, - 0x2e,0x68,0x59,0xc6,0x5e,0x59,0x5c,0x49,0x2f,0xcc,0x0a,0xdd,0xcc,0xe3,0x07,0x87, - 0xe1,0x55,0x5c,0x4d,0xcb,0x32,0xbc,0x90,0x4d,0xfc,0xa7,0xb0,0xa8,0xf2,0xb2,0xb8, - 0x18,0x37,0xf1,0x0f,0xd3,0xc9,0x48,0xea,0x70,0xac,0xca,0x2c,0xfe,0xeb,0xe8,0x17, - 0x99,0x36,0xa1,0x49,0xe6,0xf1,0x8a,0x82,0x26,0xbc,0x0c,0x65,0xb0,0x90,0x7b,0x7b, - 0xbe,0x8c,0x1a,0x79,0xdd,0xbc,0x56,0x55,0x23,0xab,0x66,0xd8,0x04,0xcb,0x70,0x34, - 0x35,0x9d,0xad,0xa3,0xb0,0x0e,0x16,0xed,0x48,0xe4,0x20,0x7e,0x14,0x65,0x85,0x49, - 0x46,0xa5,0xcc,0x86,0x0f,0x0e,0x97,0xdd,0x51,0x7f,0x50,0x47,0xcd,0x58,0x56,0x7e, - 0x1e,0xe6,0xc1,0x40,0xcb,0x66,0xaa,0x2b,0x51,0x2f,0xc3,0x5a,0x95,0xe5,0x1a,0xa3, - 0x09,0x27,0x61,0x1a,0x2a,0xcb,0x85,0x26,0xcc,0xc3,0x8b,0xe1,0xe1,0x60,0x75,0xc2, - 0x19,0x4e,0xb8,0xd8,0xdb,0x7b,0x90,0x83,0xb6,0x7c,0x78,0x18,0xa6,0xa5,0x4c,0xf4, - 0xc7,0x62,0x22,0xd5,0xb4,0xf1,0x9b,0x20,0xf4,0x4d,0x94,0x26,0x65,0x79,0x62,0xfc, - 0x20,0xfe,0x93,0x6f,0x82,0xc0,0x9d,0xd8,0x62,0xcf,0x82,0x45,0xea,0x67,0x21,0x71, - 0x05,0x77,0x59,0x4d,0x4b,0x9a,0x66,0x56,0xc9,0x60,0xe9,0x60,0xbe,0x0b,0xbf,0x0b, - 0x82,0xe5,0xea,0xe4,0xef,0x70,0x72,0x3e,0xec,0x0f,0xb6,0x4e,0x1c,0x80,0x1a,0xbf, - 0x19,0x42,0x25,0xda,0xb9,0xb3,0x70,0x02,0x38,0xb5,0xb7,0xa7,0xa2,0xbc,0xd0,0xa6, - 0x19,0x0e,0xc1,0x88,0x13,0x80,0xc7,0x20,0xde,0x5d,0x1b,0xe2,0x52,0x75,0xdc,0x11, - 0xf9,0xc5,0x6d,0xcc,0xcb,0x10,0x48,0xd7,0x7b,0xaa,0x60,0x31,0x19,0x56,0x83,0xfc, - 0xe6,0x06,0xa8,0x96,0x61,0xa5,0x66,0xf1,0xd9,0x12,0x7b,0x58,0x2d,0xd6,0xdb,0x52, - 0xdc,0x64,0x6f,0xef,0xfb,0xa8,0x9e,0x9a,0x31,0x06,0x83,0xf3,0x13,0x0c,0x53,0xff, - 0x3c,0x88,0x7f,0x3a,0xf9,0x29,0x02,0x14,0xe8,0xf8,0x69,0xf8,0x21,0x22,0xa6,0xfb, - 0x3f,0x86,0xef,0xc3,0x37,0x40,0x36,0x4b,0x9a,0x74,0x7c,0xbe,0x85,0x69,0x12,0x2c, - 0x3e,0x44,0x16,0x3d,0xe8,0xfe,0x29,0x02,0x35,0xfe,0xfb,0xe1,0xfb,0xa4,0x19,0x47, - 0x93,0x02,0xcb,0xe1,0xfb,0x80,0x41,0x8b,0xe6,0x9d,0xd2,0x6f,0xe4,0x55,0x91,0xca, - 0x35,0x30,0xe4,0xa7,0x86,0xea,0xe6,0x66,0xb1,0x6c,0x05,0x5d,0xc9,0x99,0xf8,0xa0, - 0xd5,0xa4,0x30,0x72,0x2d,0x8e,0x79,0x58,0x59,0x41,0x8f,0xd7,0x24,0x1d,0xc9,0x47, - 0x6b,0xc1,0x98,0x60,0x61,0x20,0x6e,0x5f,0x45,0xa3,0x69,0x51,0x66,0x7b,0x7b,0xc6, - 0x3e,0x0c,0x87,0x6e,0x26,0x00,0xfb,0xc7,0x11,0xb1,0xd3,0x0f,0xc2,0x39,0x49,0x7c, - 0x19,0x0c,0x3a,0xd2,0xe8,0xf0,0xb8,0xdd,0x36,0xa8,0xfc,0xb7,0x5a,0x2b,0xed,0x7b, - 0x8d,0xdd,0xe4,0xd1,0x45,0x54,0xe4,0x46,0x37,0x37,0x2f,0xe4,0x63,0x60,0x59,0x2e, - 0xd7,0x6a,0xf7,0x27,0x7f,0x4a,0x2a,0xe9,0xae,0x92,0x4b,0xb0,0x8b,0x67,0xb6,0xb4, - 0x4b,0x07,0x0b,0xb2,0x6a,0x1d,0xa9,0xcb,0xa0,0x19,0x6b,0x35,0x13,0xf6,0x20,0xed, - 0xf8,0xb8,0xd2,0x7a,0xd6,0x0e,0xb2,0xac,0x13,0xcd,0x7f,0x20,0x14,0x0d,0x87,0x41, - 0x64,0x2e,0x3b,0x3a,0xf7,0x86,0xee,0x0f,0x8c,0xf8,0xf7,0x1c,0x76,0x5e,0xe7,0x56, - 0x46,0x43,0x33,0xf8,0x3e,0xca,0x95,0x7e,0x9b,0x80,0x8c,0xd5,0xe1,0x39,0xf4,0x13, - 0x3b,0x89,0xf0,0x65,0xe0,0x07,0x2b,0x7f,0xb3,0x63,0x1d,0x57,0x95,0x4c,0xe4,0x70, - 0xf7,0xaa,0x90,0xb3,0x5a,0xe9,0x66,0x57,0xa4,0xce,0xa6,0x77,0x67,0x45,0xd6,0xc0, - 0x51,0xb1,0x00,0x7b,0x3c,0x80,0xa7,0x28,0x9a,0x22,0x29,0x7b,0x06,0xb6,0x24,0x87, - 0x87,0xe4,0xe5,0x9a,0xa2,0x29,0xe5,0xf1,0x87,0xa4,0x69,0xa4,0x86,0x1f,0xc1,0xd5, - 0x7a,0xc2,0x52,0xf3,0xf2,0xc0,0xae,0xc1,0x13,0x36,0x73,0xfa,0x1b,0x6b,0xa5,0x9a, - 0x45,0xaf,0x97,0x6a,0x99,0x4c,0xe2,0x87,0xfd,0xd7,0xfd,0xd3,0xfe,0x8b,0x41,0xaf, - 0x57,0x54,0x97,0xf1,0xc3,0xb7,0x6f,0xde,0x3e,0x7b,0x73,0x8a,0xd1,0x64,0xda,0xc8, - 0x2c,0x7e,0xf8,0xfc,0xd5,0xf3,0xa3,0x67,0x47,0x18,0xe7,0x49,0x51,0x35,0xf1,0xc3, - 0x27,0xaf,0x9e,0x3c,0x79,0xfc,0x74,0xb0,0xd3,0xeb,0xe9,0x69,0x29,0xe3,0x87,0x47, - 0x8f,0x8f,0x0e,0x0f,0x9f,0x0f,0xec,0xb0,0x67,0x54,0x8e,0x3d,0x87,0xa7,0x87,0x2f, - 0x0e,0x1f,0x63,0xae,0x24,0x0c,0xef,0xde,0x3d,0x79,0x7d,0xf4,0x16,0x23,0x05,0xf4, - 0x4f,0x9e,0x9d,0x3e,0x7f,0xf6,0x0e,0x83,0x59,0xa2,0xab,0xf8,0xe1,0x9b,0x17,0xaf, - 0xfa,0x8f,0xde,0x11,0xb6,0x3a,0xa9,0x64,0x09,0xd0,0x47,0x87,0x87,0x87,0x7d,0x9a, - 0x30,0x49,0x65,0x62,0xef,0x5b,0x30,0x41,0x7b,0xe1,0xb4,0xe0,0x71,0xcf,0x48,0x5d, - 0xe4,0xa1,0x99,0x9b,0x46,0x4e,0x7a,0xd3,0x22,0x5c,0x4f,0x12,0xc8,0x44,0x55,0x2a, - 0xf6,0xfe,0x22,0x9b,0x53,0x0d,0x62,0x8d,0x78,0x8f,0x31,0xc3,0xd2,0x82,0xa9,0x93, - 0x54,0x86,0xe7,0xef,0x68,0xb2,0x77,0x26,0x2f,0xa6,0x65,0xa2,0xc3,0xf7,0xb2,0x2a, - 0x55,0xb8,0x5a,0x5e,0xee,0xfc,0x71,0x31,0x52,0xd7,0x3d,0x53,0xfc,0xbd,0xa8,0x2e, - 0xe2,0x91,0xd2,0x99,0xd4,0x3d,0xcc,0x2c,0x77,0x46,0x2a,0x9b,0x2f,0x26,0x89,0xbe, - 0x28,0xaa,0xb8,0x3f,0x18,0x25,0xe9,0xe5,0x85,0x56,0xd3,0x2a,0x8b,0x61,0x1d,0xbe, - 0xe3,0x65,0x30,0x48,0x55,0xa9,0xb4,0x9b,0x02,0x3b,0x83,0x41,0x0e,0x29,0x82,0x75, - 0x93,0xa2,0x9c,0xbb,0x69,0x22,0x39,0x18,0xec,0x94,0x45,0x25,0x7b,0x63,0xc9,0xfe, - 0xfe,0x30,0x7a,0x32,0xe8,0xcd,0xe4,0xe8,0xb2,0xc0,0x5e,0x02,0x30,0x13,0x48,0x68, - 0x4c,0x24,0x24,0x15,0x09,0xba,0x48,0x8c,0xcc,0x96,0x3b,0xd1,0x4c,0x27,0x35,0x88, - 0xb8,0xb6,0x6a,0x10,0x3f,0x7d,0xde,0xaf,0xaf,0x07,0x2d,0x51,0x22,0x99,0x36,0x6a, - 0x50,0x27,0x59,0x46,0x80,0x8f,0x8e,0xea,0x6b,0x71,0x84,0x75,0xf1,0xf4,0x71,0x0d, - 0xfa,0x29,0x44,0x4a,0xbd,0x80,0xb7,0xaf,0x4b,0x8a,0x25,0xa5,0xbc,0x1e,0x00,0xf1, - 0x45,0xd5,0x2b,0xc0,0x4b,0x13,0xa7,0x92,0x38,0x3d,0xb8,0x48,0xea,0xf8,0x39,0x90, - 0x3a,0x34,0xb8,0x7b,0xd3,0xa8,0x49,0x7c,0x08,0x6c,0x83,0x9d,0x15,0x3f,0xec,0x1c, - 0x70,0x23,0x90,0x16,0x99,0xb0,0x17,0x23,0x05,0x08,0x40,0x64,0x06,0xe5,0xb2,0xf4, - 0x3d,0x03,0x90,0xbb,0x22,0x3d,0xde,0x62,0x1a,0xd4,0x23,0x18,0x38,0xa4,0x3a,0xc9, - 0x0a,0x78,0xb7,0x43,0xa6,0xf5,0x70,0x61,0xd9,0x50,0xfc,0x5d,0xc6,0x87,0x4f,0x00, - 0xca,0xc3,0x99,0x45,0xf5,0xb4,0xdf,0x5f,0xdd,0x79,0x40,0xf7,0x88,0x0f,0x71,0xaa, - 0x99,0x8e,0x16,0xb7,0x99,0x4d,0x92,0x75,0x42,0xb0,0xc8,0x80,0x7f,0x43,0x48,0xac, - 0xd5,0xa0,0xda,0x48,0x36,0x52,0x27,0xe1,0x1e,0x79,0xff,0xa3,0xa7,0x4c,0xcb,0xd1, - 0x62,0x0b,0x7c,0x9b,0x96,0x52,0x92,0xcd,0xf5,0x48,0x81,0x88,0xf1,0x51,0xff,0x85, - 0x9c,0x0c,0xc8,0x69,0xf4,0x1a,0x0d,0x59,0xc3,0x0f,0x4c,0xe2,0x69,0x5d,0x4b,0x9d, - 0x42,0x8a,0x83,0x9d,0xee,0xe9,0x6c,0x63,0xc1,0x5a,0x82,0x7d,0xf1,0x9c,0xce,0xcc, - 0xca,0x8e,0xa6,0x59,0xf6,0x10,0x41,0x77,0x31,0x9c,0x2d,0x8e,0xb8,0x0e,0x67,0xf6, - 0x79,0xe1,0x8e,0x70,0x32,0x29,0x1c,0x8b,0x97,0x65,0xd9,0xaa,0xc9,0x33,0xd2,0x92, - 0xfb,0xc9,0xb6,0x3d,0x2a,0x6b,0x16,0x96,0xeb,0xc2,0xa9,0x5c,0x87,0x3f,0x8f,0xb6, - 0xd8,0xbb,0x75,0x41,0x00,0x67,0xeb,0xab,0x21,0x4a,0x39,0x45,0xee,0x0f,0xee,0x21, - 0x3a,0x22,0x93,0xd9,0xca,0x37,0x8b,0x35,0x09,0x60,0xa0,0xae,0xa4,0x26,0x6f,0xd7, - 0x23,0xd3,0x80,0xb5,0xcc,0x67,0x63,0xa9,0x25,0x9d,0x13,0x8d,0x8a,0x8b,0xc5,0x16, - 0x69,0x5b,0xa2,0x03,0xdb,0xd4,0xe5,0xa2,0x4b,0x2f,0xe2,0xc2,0x32,0x22,0xd7,0xb4, - 0x31,0x4b,0x13,0x98,0x1f,0x25,0xd9,0xc6,0x34,0xa9,0x2f,0x50,0x8c,0x12,0xbd,0x70, - 0x5a,0xfe,0xe8,0x2e,0x2d,0x5f,0xf3,0x6e,0xd0,0xd1,0x2e,0xd2,0x6a,0x7b,0xf9,0xc3, - 0x7e,0xff,0x1b,0x8b,0x46,0x14,0x2b,0x11,0x8e,0x4a,0x95,0x5e,0xb6,0xc6,0x43,0x3b, - 0xee,0xb6,0x1e,0xc0,0x55,0xaa,0x91,0x8b,0x2d,0x36,0xdd,0xd6,0xf0,0x56,0x04,0xd0, - 0x2f,0x88,0x0d,0x4a,0xbb,0xe1,0x7e,0x1e,0x3f,0x01,0xa2,0x46,0xa9,0xd2,0xdc,0xcb, - 0x41,0xf0,0x19,0x9d,0xbb,0x1c,0xb2,0x87,0x19,0x4d,0xa1,0x3a,0xd5,0xa2,0x56,0xa6, - 0x20,0x53,0x8a,0xb5,0x2c,0x93,0x06,0xe9,0xee,0xe0,0x9e,0x66,0x79,0xcb,0x8e,0x9e, - 0x7f,0xd1,0x8e,0x5a,0x0d,0x86,0x95,0x0a,0x26,0xa8,0xc3,0x20,0x06,0xa8,0x13,0x0d, - 0x92,0xef,0x52,0x47,0xab,0xeb,0x77,0x3b,0xb0,0x2d,0x7f,0xc4,0xdc,0x9c,0x6a,0x03, - 0x14,0xb5,0x2a,0x88,0x03,0xed,0x45,0xe3,0x31,0xe9,0xde,0xe2,0x96,0xc7,0x77,0xf0, - 0xb7,0x15,0xe5,0x5f,0x26,0x32,0x2b,0x12,0xdf,0xa1,0x89,0x53,0x45,0x85,0x0a,0xf2, - 0x71,0x8b,0x2d,0x4e,0xf2,0x86,0xd1,0x71,0xd4,0x8f,0x77,0x77,0x07,0x2b,0x46,0x26, - 0x23,0x50,0x09,0xca,0x07,0xc4,0xeb,0x1e,0x79,0x67,0x67,0xa5,0xfc,0x5c,0x4a,0x84, - 0xdc,0x1e,0x24,0x30,0x60,0x8b,0xe0,0xc7,0x25,0xc4,0x39,0x31,0x17,0xff,0x8c,0x43, - 0x24,0x78,0x24,0x34,0xbd,0xbb,0x0c,0xc4,0x2d,0x49,0xad,0xef,0xb0,0x87,0x87,0x69, - 0x4d,0x42,0xdb,0xd2,0xe4,0xb5,0xaa,0xaf,0x94,0xba,0x13,0xbc,0xac,0x16,0xf5,0x3b, - 0x7e,0x89,0x6e,0xa6,0x25,0x93,0x08,0x2e,0x37,0x05,0x12,0x1d,0xf8,0xa8,0x6d,0x13, - 0xe0,0xa4,0xe1,0x8e,0xa8,0xfb,0x3b,0x05,0x4c,0xec,0x60,0x4e,0x1c,0x90,0x29,0x88, - 0x0e,0xa7,0x96,0x3b,0x9f,0xc6,0x45,0x96,0xc9,0xea,0xe7,0xd5,0x7d,0x2a,0x55,0xc9, - 0x07,0xc5,0x84,0x92,0x34,0x04,0x67,0x30,0x8a,0x98,0x94,0xe7,0x64,0x50,0x02,0xde, - 0x4d,0x91,0xfa,0xa2,0xb8,0x8b,0x60,0x52,0x39,0xa2,0x38,0x64,0xda,0xb9,0xe3,0xa3, - 0x8e,0xef,0xb5,0x97,0x66,0xcd,0xfd,0xb2,0x8f,0x47,0xb6,0xf0,0x4f,0x08,0x32,0x59, - 0xdc,0xb6,0x80,0xe5,0x0e,0xb2,0x4f,0x4e,0x06,0x5f,0x1e,0xd8,0xda,0x99,0xd2,0x9b, - 0xe3,0x97,0x59,0x71,0x25,0xd2,0x32,0x31,0x06,0x59,0x27,0xfc,0x69,0x5b,0x5a,0x4b, - 0x8d,0xa2,0x18,0xbc,0x6e,0xd7,0x10,0xe1,0xb9,0x5e,0xc6,0xd4,0xf1,0xcb,0xf1,0xe1, - 0x71,0x9b,0x67,0xe2,0x71,0x63,0x1f,0x62,0xf2,0xae,0x28,0x32,0xd4,0xe2,0x40,0xd5, - 0x6b,0x01,0x0e,0x1c,0xca,0x9d,0x9d,0x97,0x0f,0x7a,0x3d,0xf1,0x71,0x5c,0x18,0x51, - 0x27,0x17,0x52,0xe0,0xef,0x6c,0x9c,0x34,0x02,0xd5,0xb0,0x11,0x48,0x8e,0xa5,0xac, - 0xcc,0x18,0x9a,0x2e,0xb3,0x50,0x28,0x8d,0x3d,0x86,0x1f,0xe1,0xe1,0x2b,0xf0,0x68, - 0x22,0x89,0x66,0x91,0x98,0x4b,0x23,0xe0,0x1b,0x76,0x04,0xfd,0x6f,0x2c,0xcb,0x3a, - 0x16,0x48,0xfa,0x85,0x35,0x2b,0x64,0xd1,0x75,0x21,0x8d,0x90,0x50,0xa2,0xb9,0xc8, - 0x0b,0x59,0x66,0xa2,0x53,0xad,0x8b,0x71,0x62,0x42,0x01,0x0f,0x2a,0x7e,0x99,0x9a, - 0x86,0xe1,0x10,0x47,0x4d,0x24,0x7a,0x3d,0x5c,0xbd,0xc3,0x0d,0xf6,0x8d,0xb8,0xb3, - 0xc3,0x4a,0xcd,0x88,0xe1,0xae,0x1d,0xd8,0x1b,0xe2,0x9c,0xf9,0xee,0xf1,0x6b,0xfc, - 0x2b,0x60,0xe4,0x17,0xc8,0x27,0xa1,0xb3,0x60,0x89,0xdd,0xb3,0xc9,0x16,0x98,0x8f, - 0x03,0xaa,0xe9,0x71,0xc5,0xc9,0x03,0x1c,0x48,0x99,0x3c,0xac,0x07,0xae,0x2b,0x71, - 0x5b,0x68,0xb8,0x2b,0x30,0xce,0x54,0x55,0xce,0x85,0x55,0x47,0x91,0xe8,0x22,0xe9, - 0x95,0xc9,0x48,0x96,0xc3,0xdd,0x37,0xeb,0x03,0x09,0x59,0x0b,0x4f,0x0c,0x76,0x69, - 0x0d,0xa4,0x74,0x74,0x6c,0x4b,0x40,0xb0,0xff,0x08,0x82,0x2e,0x8f,0xc1,0xb0,0xee, - 0x0d,0x71,0x6f,0x00,0x67,0xcd,0xf1,0xbb,0x42,0x4f,0x10,0xf0,0xb0,0x11,0x83,0x97, - 0x59,0xc6,0x64,0xe4,0x33,0x16,0x60,0x96,0xb5,0x54,0x7e,0x06,0xf8,0x94,0x4a,0xbe, - 0x0d,0xc8,0x51,0x99,0xdd,0x0f,0xf4,0x7b,0x88,0x17,0xd7,0x84,0x60,0x74,0xb3,0x81, - 0x41,0xeb,0xfb,0x9e,0x6d,0xe6,0x9b,0x47,0x63,0xe2,0x7e,0xa0,0x1f,0xc8,0x8f,0x08, - 0xb2,0xa3,0x0d,0x04,0x75,0xb5,0x05,0x8e,0x3f,0x25,0x89,0xcb,0x71,0x75,0x8b,0xc1, - 0x67,0xb2,0x82,0x5e,0x7f,0x95,0xc1,0x7c,0xd8,0xe6,0x39,0xb3,0x7b,0x5e,0xf1,0x9d, - 0x46,0x91,0x28,0x74,0xd2,0xac,0xe9,0x74,0x5b,0x90,0xec,0x58,0x9d,0xca,0x6b,0xf3, - 0x7b,0x70,0x51,0x4d,0xbd,0x29,0xe9,0xe6,0x7e,0xe0,0xaf,0x52,0x8a,0xed,0x30,0x4a, - 0x2e,0x38,0x37,0x50,0x24,0x69,0x73,0x17,0xdf,0x76,0x5e,0xd6,0x2d,0x16,0xca,0x59, - 0x76,0xdb,0xab,0xf3,0x00,0x3b,0xeb,0xcf,0x73,0xd6,0x95,0xb5,0xe6,0xab,0xbc,0xfd, - 0xa8,0x9a,0x64,0x8b,0xb7,0xcd,0xfd,0x35,0xb7,0x11,0x70,0x99,0x0a,0x6e,0xe2,0x0e, - 0x0b,0xa8,0xeb,0x7b,0xaa,0xb1,0x82,0x73,0xcb,0x44,0xae,0xd5,0x44,0x98,0x46,0x69, - 0x38,0xb6,0x4d,0x3c,0x93,0xdf,0x61,0x0e,0x25,0x90,0x09,0x49,0x2d,0x8a,0x0d,0x1c, - 0xa5,0xbc,0x37,0x7f,0x09,0xc3,0x3d,0x18,0xfc,0x5e,0x4e,0x94,0x9e,0x7f,0x95,0xbd, - 0x5c,0x8d,0x57,0x49,0x09,0x3f,0x9b,0xd4,0xb8,0xa3,0xdc,0xbc,0xdb,0xb8,0xb8,0xef, - 0xdd,0xf4,0x05,0x0c,0x9d,0x11,0x08,0xce,0x0f,0x36,0xd1,0x94,0xf7,0xb4,0xda,0xf3, - 0xb3,0x57,0xef,0xef,0xa0,0xa2,0xbe,0x0f,0x77,0xb6,0xee,0x82,0xd8,0x43,0xde,0xdf, - 0xa4,0x89,0x4e,0xa5,0x40,0x88,0x17,0x3d,0x9e,0xf8,0xfe,0xed,0x1b,0xc1,0x59,0x86, - 0x67,0xc4,0x9b,0xf7,0xaf,0x10,0x56,0xf2,0x5c,0x6a,0xb3,0x43,0x7d,0x5c,0xda,0xa0, - 0x65,0x24,0x5e,0xb5,0x56,0x20,0x26,0x2a,0x43,0xbc,0x66,0xb1,0x19,0xc1,0xee,0xba, - 0xc8,0x45,0x01,0x39,0xca,0xe4,0x0a,0x51,0xc8,0x46,0x01,0xf6,0x69,0xe6,0x6a,0xf7, - 0xf8,0xe8,0xb1,0xf8,0xee,0xd4,0x39,0xff,0x1d,0xc5,0x1b,0xe9,0x32,0x03,0xa1,0xa7, - 0x15,0x70,0xcc,0xa4,0x16,0x49,0x95,0x31,0x19,0x95,0x6c,0x66,0x4a,0x5f,0xb6,0x64, - 0x72,0xdf,0x52,0x70,0x87,0x40,0x40,0x61,0x2f,0x14,0x45,0xc5,0xa2,0x64,0x8a,0x76, - 0x98,0x5c,0x71,0x29,0x65,0x6d,0x44,0xa6,0x93,0x19,0x36,0x45,0x5f,0x14,0xfe,0x79, - 0xab,0xa6,0x5f,0x75,0x5c,0xf6,0x96,0x77,0xa9,0x75,0x7e,0x4f,0xaf,0xe3,0xce,0xba, - 0x43,0xa9,0x73,0x73,0xa7,0x56,0x77,0xb1,0xa0,0x44,0x02,0x96,0xc2,0x6d,0xa7,0x91, - 0xe0,0x2c,0xc6,0xf5,0xc8,0xe2,0x3e,0x69,0x79,0xe1,0xc0,0x37,0xae,0xcb,0x89,0xc6, - 0x4c,0x09,0x64,0x69,0x97,0xc4,0x42,0x64,0x19,0x29,0x44,0x31,0x92,0x62,0x5a,0x0b, - 0x0c,0x54,0x95,0xca,0x78,0x83,0xd3,0xf4,0x6c,0x39,0xf9,0x0b,0x12,0x76,0x58,0xb4, - 0x4f,0xbc,0xe5,0x24,0x83,0x72,0x07,0xe0,0x0d,0x56,0xd2,0x69,0x15,0x44,0xcd,0x2a, - 0x81,0x84,0xc5,0xd4,0xaa,0x89,0x04,0x35,0xff,0x6c,0x2a,0x53,0x34,0x76,0xa9,0x93, - 0x5a,0x74,0xd9,0xff,0x83,0x3d,0xf1,0xab,0xec,0x5f,0xed,0xeb,0x30,0xcd,0xdc,0x97, - 0xed,0xa8,0xe3,0xb6,0x1c,0xa3,0xbe,0x27,0xe8,0xab,0x2c,0x43,0x54,0x36,0x1b,0xb0, - 0xc5,0x3d,0x9d,0xe1,0x0f,0x08,0x31,0x9b,0x90,0xd5,0x3d,0xdd,0xdf,0x9f,0x2d,0x1b, - 0x37,0x0d,0xdb,0xfc,0x2e,0x58,0xb2,0xc6,0x2d,0xcf,0x60,0x7e,0xdf,0xe9,0x98,0x2f, - 0x50,0x8c,0x99,0x2d,0x24,0xe9,0xfd,0x90,0x9c,0x49,0x68,0x00,0x14,0xe7,0xaa,0x48, - 0x36,0x10,0x60,0xfc,0xd5,0xb4,0xc2,0x56,0x0e,0xc7,0x2f,0x13,0x31,0xd6,0x32,0x1f, - 0xee,0x1e,0xec,0x82,0x26,0x8a,0xd4,0xc9,0xb1,0xd8,0x9b,0x20,0x05,0x54,0xcd,0x40, - 0xac,0x57,0x9d,0xff,0x01,0x7f,0xd6,0x91,0x32,0x01,0x46,0x87,0x66,0xa7,0x3d,0xa7, - 0xed,0x38,0xaf,0x1a,0xd8,0x7f,0xf0,0x8b,0x55,0xd7,0x3c,0x53,0xe9,0x74,0x82,0xeb, - 0x46,0x50,0xda,0xb7,0xa5,0xa4,0xc7,0xd3,0xf9,0xb7,0x19,0x76,0x2c,0xd7,0x00,0x97, - 0x23,0x7f,0xb4,0x82,0x18,0x1d,0x0f,0x0f,0xfb,0x8f,0x9f,0x3f,0x79,0xf6,0xf4,0xc4, - 0x1f,0x1d,0xb8,0xc7,0x20,0x6a,0xd4,0xbb,0xe2,0x5a,0x66,0xfe,0x51,0xb0,0xef,0x89, - 0xf7,0xa7,0x5e,0xcc,0xef,0x24,0xb8,0x6e,0xe3,0x6d,0x47,0x8f,0x69,0xe1,0xbb,0x53, - 0xaf,0x83,0x37,0x9b,0x6a,0xea,0xa2,0x83,0x97,0xf4,0xb6,0x21,0xb3,0xaf,0x31,0xf2, - 0x52,0x29,0x4c,0x1f,0x3c,0x7f,0xfa,0xb8,0xdf,0x0f,0xc2,0xf1,0xc6,0xec,0x37,0x3c, - 0x7b,0xf0,0xe8,0x29,0x2d,0x59,0xc3,0x14,0x93,0xcd,0x1d,0xb4,0x76,0xf0,0x14,0xcb, - 0xd7,0x43,0xf3,0xcd,0xd3,0xfe,0x00,0xbb,0x8a,0xdc,0xcf,0x82,0xf6,0xc2,0xfb,0x5e, - 0x26,0xbc,0xfd,0xf1,0xbe,0x37,0xf6,0x06,0x58,0x18,0xb7,0x0b,0x34,0x83,0x85,0xc9, - 0xbe,0x37,0xf1,0x1c,0xd0,0xa4,0x5d,0xa3,0x49,0xac,0x5d,0xef,0x7b,0xc6,0x6b,0xdf, - 0x16,0xd8,0xc1,0x4e,0xe7,0x3a,0x69,0x69,0x7c,0x59,0x86,0x69,0xb0,0x90,0x65,0xc4, - 0x8a,0x41,0xa6,0x30,0xf4,0x90,0xa6,0x01,0x38,0xed,0xec,0xac,0xa7,0x8d,0x5f,0x64, - 0x61,0x43,0x7b,0xe9,0xea,0x72,0x08,0x99,0x64,0xc1,0x60,0xeb,0xc5,0x1e,0xc6,0x6b, - 0x34,0xe9,0xcd,0x8d,0x07,0xe6,0x1d,0x1c,0x08,0x69,0xea,0xff,0x84,0x75,0xca,0x06, - 0xff,0x26,0xfc,0x7a,0x02,0x05,0x51,0x1b,0x22,0x38,0x89,0xe1,0xf7,0x0a,0xe4,0x83, - 0x42,0x24,0x37,0x02,0x0e,0x24,0x83,0x0f,0xfa,0x38,0xe6,0x18,0x67,0x44,0x82,0x80, - 0x76,0x41,0x88,0xd2,0x64,0x6a,0x68,0x0c,0x00,0x2d,0x33,0xb8,0x42,0x38,0x4b,0x8d, - 0x0a,0x4b,0x91,0xef,0x82,0x73,0x14,0x33,0x90,0x21,0xa6,0x15,0x62,0x27,0x8a,0x88, - 0x0c,0x9e,0x93,0x21,0x6c,0x9e,0xd3,0xf1,0x9b,0x84,0x2b,0x11,0xa9,0x4e,0xcc,0x18, - 0x21,0x4c,0xc1,0xd9,0xca,0x6b,0xe4,0x83,0x88,0x84,0x65,0x71,0x29,0xb1,0x34,0x26, - 0xa7,0x49,0x25,0x14,0xf9,0xcd,0x72,0x7a,0x11,0xed,0xd0,0xad,0xcf,0xce,0x86,0x8b, - 0x9a,0x22,0x9e,0xaa,0x62,0x8f,0x1f,0xb0,0xc7,0x0b,0xc1,0x81,0xd8,0xe3,0xfb,0x89, - 0xba,0xc0,0xd8,0xcc,0x62,0x8f,0x3a,0x69,0x33,0x4b,0x27,0xd7,0x0a,0x5e,0x08,0xf7, - 0x5b,0xa4,0xb1,0x67,0x0f,0xf5,0x79,0x14,0x78,0xa4,0x12,0xc8,0xe6,0xfe,0x73,0x96, - 0x01,0x05,0xbf,0x3f,0xcb,0xd4,0x85,0xf0,0xb9,0xf7,0xa2,0xa7,0x75,0x83,0x1d,0x0d, - 0x6a,0xc7,0xed,0x75,0x9a,0xc3,0xd2,0xc6,0x2c,0xe3,0xca,0x28,0x9e,0x96,0xf8,0x87, - 0xe6,0x2f,0xdd,0xc5,0x69,0x52,0xf0,0xac,0x17,0x8e,0x60,0xf8,0x95,0x9a,0x02,0xae, - 0x7d,0x02,0x2d,0x7c,0x93,0xac,0xa8,0x03,0x6f,0x39,0xe0,0x8b,0x9e,0xfe,0x78,0xfe, - 0xef,0xf4,0x32,0x98,0xa3,0x20,0x50,0xe9,0xa2,0xa1,0x20,0xde,0x1a,0xb0,0x17,0x4e, - 0xeb,0x8c,0x5f,0x14,0x6b,0x99,0xca,0xe2,0x8a,0xd6,0x5a,0x49,0x12,0x0a,0xc6,0x01, - 0x3d,0x68,0x86,0xf4,0x32,0x74,0xd0,0x51,0x23,0xaa,0xf6,0xe9,0xed,0x29,0x48,0xe5, - 0xf5,0x8c,0x14,0xf7,0x0f,0xbe,0x37,0xad,0xbd,0x60,0x43,0x95,0x30,0x03,0x15,0x24, - 0x7b,0xcb,0x22,0xf0,0x01,0xf9,0x7f,0x40,0x5b,0x21,0xb7,0xfb,0x54,0xe2,0x9f,0xa9, - 0xbe,0xb9,0xf0,0x0e,0x31,0x6f,0x31,0x21,0x93,0x19,0x93,0x1e,0x2a,0x61,0x92,0x39, - 0xa5,0x26,0x08,0x85,0x2b,0x85,0xe4,0xe8,0xbb,0x46,0x4a,0x87,0xa9,0x3c,0x12,0xaf, - 0x15,0xd6,0xb0,0xdd,0x50,0x3d,0x4e,0x99,0x8d,0x45,0x05,0xca,0x75,0x22,0x7e,0xfb, - 0xf5,0x1f,0x84,0xd4,0xa5,0x3f,0x99,0xcc,0x93,0x29,0xb2,0xf5,0x55,0x76,0xa4,0x0a, - 0xc3,0x44,0x53,0x25,0x0f,0x2d,0x6c,0xc6,0xd8,0x1c,0x01,0x9e,0x8c,0xcb,0xcb,0x67, - 0x5e,0x98,0x45,0x57,0xc8,0xda,0xc0,0xa8,0x7d,0x1f,0xd7,0xbe,0xa2,0xea,0xb9,0x6a, - 0xf6,0xf6,0x56,0x8f,0x0f,0x86,0x43,0x2f,0x05,0x01,0x5e,0x70,0xe2,0x89,0xff,0xfd, - 0x1f,0x62,0x50,0xbb,0xb4,0xbf,0x06,0xf8,0x37,0x8b,0x03,0x5b,0x3a,0xeb,0x6e,0x32, - 0xf6,0xbc,0x80,0xfe,0xcf,0xcc,0xe4,0x73,0x51,0xf9,0xd2,0xc1,0xfc,0x16,0x14,0x06, - 0xdb,0xf3,0xd6,0x4b,0x5a,0x7b,0xe1,0xd9,0xd9,0xa7,0x2c,0x62,0xdd,0x3e,0x63,0xd3, - 0xfd,0xf9,0xe6,0x66,0x63,0xcc,0x20,0xe1,0x01,0xab,0xf3,0x0d,0xf4,0xf1,0xa6,0x55, - 0xab,0x03,0x88,0xd3,0x40,0xd6,0xdd,0xcd,0x20,0x7b,0x94,0x64,0x1e,0x11,0xb0,0x3e, - 0x1f,0xe5,0xaf,0x25,0xc0,0xcc,0x4f,0x7c,0x52,0xbc,0x4f,0x76,0xc0,0x27,0xd1,0x03, - 0x08,0xa6,0x17,0x64,0x6e,0x8b,0x47,0x6d,0xee,0x4d,0x14,0x75,0x45,0xab,0x9c,0xde, - 0xb4,0x37,0xb0,0x52,0xc1,0xa9,0x59,0xcf,0xf6,0x23,0x90,0x02,0x0f,0x6c,0xee,0x3c, - 0x2b,0x60,0x2f,0x82,0x32,0x63,0xd3,0x5a,0xb8,0xfb,0xd6,0x02,0x59,0xd1,0x44,0xb2, - 0x6a,0x45,0xe2,0x14,0xd2,0x15,0x59,0x02,0x57,0xc1,0x98,0xf8,0xbb,0x08,0xeb,0x78, - 0xac,0xe4,0xe0,0xaf,0x20,0xfc,0x70,0xe5,0x61,0x58,0xc0,0xfc,0x41,0x84,0xc1,0x1c, - 0xbd,0xd8,0xc4,0xcc,0x84,0x9d,0x0b,0xc2,0x29,0x34,0x9a,0x7a,0x83,0x4e,0x57,0x0a, - 0xca,0xc8,0xc5,0x0c,0x3a,0x47,0x0a,0x43,0xea,0x91,0xb0,0x75,0xda,0xf6,0x11,0xb0, - 0x65,0xa1,0x75,0x8e,0xab,0x3a,0x9a,0xb4,0x86,0x4d,0x79,0x24,0x73,0xa5,0x6d,0x02, - 0xcd,0xb8,0x6c,0xee,0xc7,0x5e,0x8f,0x88,0x8d,0xda,0xc8,0x84,0xe7,0x61,0x16,0x31, - 0x56,0x62,0xa3,0xbb,0xe0,0x07,0x46,0xbe,0xe6,0xdb,0x4c,0x13,0xe3,0x78,0xd7,0x89, - 0x67,0x69,0xf0,0xe2,0xad,0xdd,0x27,0x9e,0x25,0x89,0xed,0xc7,0x79,0x04,0xb0,0x3f, - 0x21,0xf7,0x02,0x68,0x1c,0xb4,0x25,0x12,0x36,0xd1,0x35,0x4f,0x2b,0xa4,0xb3,0xaa, - 0xba,0x90,0xee,0x6a,0xc6,0xa5,0xa2,0x36,0x47,0x27,0x9c,0x23,0x49,0xee,0x43,0xd5, - 0xb2,0x8a,0xdd,0xda,0x94,0xcd,0xc4,0xa2,0x42,0xfa,0x3b,0xad,0xb9,0xc0,0x1c,0x91, - 0x9f,0x23,0x16,0x25,0x04,0x36,0x81,0x75,0xd2,0x97,0x27,0x5c,0x5b,0xac,0x31,0x22, - 0xbd,0x9d,0x70,0x91,0xe8,0x2a,0x0d,0x58,0x23,0x39,0x2f,0x69,0x22,0xeb,0x67,0x5c, - 0x21,0xbf,0xe5,0x6c,0x1c,0x0f,0x38,0x3c,0x9f,0x78,0xb6,0xcd,0xa2,0xf2,0x1c,0x9e, - 0xa2,0x29,0xca,0xb2,0xd5,0x12,0x17,0x3b,0x50,0x49,0x55,0x73,0x71,0x59,0xa9,0x11, - 0x11,0xe3,0x5a,0x6d,0xc4,0x0d,0x1b,0xbe,0xe0,0x50,0x32,0xe5,0xae,0xe9,0x74,0xed, - 0xb3,0x5a,0xe6,0xf1,0x89,0xb7,0x39,0xde,0xa6,0x47,0xac,0x6f,0x96,0x3f,0x74,0x0b, - 0xd0,0xca,0xce,0xca,0x0a,0x61,0xb0,0xc1,0x49,0x58,0xd9,0x94,0x22,0x28,0x3b,0x3e, - 0xf6,0x5d,0x05,0x11,0x52,0xe1,0x10,0x08,0x86,0x0d,0x82,0x14,0x23,0xaf,0x0d,0x6e, - 0xcb,0x8a,0xf5,0xa3,0x39,0x39,0x94,0x4f,0x0f,0x56,0xa3,0xb8,0xef,0x7c,0x31,0xf6, - 0xdc,0xe2,0x4f,0x0b,0xe1,0x5b,0x81,0x43,0x6f,0xbd,0x18,0xfb,0x56,0x49,0xd4,0x21, - 0xe5,0x4a,0x0c,0x18,0xc3,0x00,0x09,0x11,0xe5,0x16,0x2d,0x32,0xa7,0x26,0x0c,0x72, - 0x3c,0x7c,0xfc,0xe4,0xc4,0x53,0x97,0x6e,0x70,0xf4,0x64,0xa5,0x3e,0xe4,0x1a,0x82, - 0x96,0x86,0xe6,0x73,0x24,0xec,0xed,0x3d,0x60,0x6c,0xfe,0x6a,0x06,0xb9,0x1a,0x92, - 0xab,0xcd,0x7c,0x6e,0x62,0xbc,0x96,0x10,0x60,0x43,0x68,0xbf,0x85,0x8e,0x74,0x0a, - 0x8c,0x25,0xef,0x9a,0x70,0x93,0x68,0x08,0xe7,0xda,0xf3,0x4e,0x3c,0xb7,0xf0,0xdb, - 0xaf,0xff,0xed,0xc5,0x2c,0x9f,0x76,0x03,0xb9,0x58,0xfb,0xf4,0xad,0x79,0xcf,0x25, - 0x34,0x5c,0xac,0xf0,0x6d,0x35,0x1d,0xe0,0x3c,0x0c,0x46,0xae,0x3f,0x13,0x38,0x27, - 0x44,0x2a,0x77,0xfb,0xec,0x36,0x8e,0x0e,0x68,0xb9,0xbe,0xbd,0xcc,0x2e,0xd3,0x38, - 0xe2,0xeb,0xc9,0xad,0x0d,0xf6,0x48,0xb3,0x32,0xe4,0x92,0x2c,0x91,0x6f,0xc4,0x1f, - 0x8a,0xc0,0x09,0x52,0xfb,0x7f,0x63,0xae,0xeb,0x79,0x9d,0x2e,0x94,0x99,0x63,0x03, - 0x7f,0x5f,0x63,0x13,0xc8,0x32,0x03,0x1f,0x9b,0xa4,0xe4,0x08,0xcd,0xc7,0xb7,0xad, - 0x99,0xed,0xf8,0xcc,0x9d,0x9f,0x4e,0x27,0x21,0x46,0xb4,0x59,0x81,0x6f,0x89,0x04, - 0xda,0xe1,0x52,0x5f,0x92,0x8c,0x10,0x43,0xc1,0x9d,0x62,0x07,0x41,0x8f,0x5f,0x00, - 0xd8,0xc7,0xde,0x52,0xa5,0xe4,0x05,0xdb,0xfd,0x76,0xf8,0x45,0x18,0x30,0x90,0x33, - 0x08,0x02,0xe0,0xe7,0xdb,0xbb,0x59,0x4b,0xe8,0xde,0x4b,0x21,0x4b,0xb8,0x9c,0xaf, - 0x5e,0xf9,0x07,0xd5,0xbd,0x30,0x5c,0xa2,0x29,0x28,0xe7,0xa4,0xf4,0x33,0xb2,0x88, - 0x76,0xd6,0xfe,0x8f,0xb7,0x68,0x57,0x6e,0x73,0xaa,0xdf,0xaa,0x8a,0xed,0x62,0x71, - 0x5c,0xd4,0x57,0x32,0x68,0xd3,0x05,0x7e,0x35,0xc3,0x96,0x3b,0x61,0xf5,0x30,0xa1, - 0x0b,0x3d,0xca,0x79,0x8f,0x52,0x4d,0x35,0xa5,0x31,0x25,0x45,0x17,0x38,0x36,0x04, - 0x06,0x8a,0x23,0x00,0xa9,0xc8,0x27,0xd2,0xb5,0x5c,0x08,0x8a,0x29,0x3b,0xe6,0x86, - 0x81,0x46,0x32,0x0c,0x32,0xda,0x7c,0x07,0xee,0x89,0xd6,0x05,0xfc,0x12,0x72,0x17, - 0xf1,0x1c,0xe5,0x8d,0xe0,0x96,0x4e,0x1b,0x38,0xb4,0xb9,0x1a,0xfa,0xad,0x82,0x59, - 0x42,0xc9,0x42,0xee,0x24,0xfc,0xe6,0xe6,0xe8,0x31,0x2a,0x29,0xa7,0xa6,0x80,0xdc, - 0xe2,0x17,0x8a,0x30,0x4c,0xae,0xc3,0xf4,0xb8,0xf0,0x42,0xcc,0x65,0x11,0xb5,0xb2, - 0xda,0xbe,0x16,0x7c,0xc3,0xc6,0xf8,0x18,0x20,0xfb,0xcf,0x0f,0x5f,0x1c,0x59,0x1f, - 0x71,0x7b,0xf1,0x4e,0x77,0x31,0x2e,0x6f,0x9f,0x6d,0x41,0x5d,0x13,0x6f,0xb5,0xb1, - 0xfe,0xdc,0xc6,0x0f,0x46,0xd3,0xa7,0x31,0x3b,0xae,0xfa,0x8a,0x72,0xd8,0xf8,0xb4, - 0xa2,0xd7,0x50,0x56,0x2f,0x88,0x3b,0xe4,0x9d,0xc9,0x13,0x99,0x1f,0x29,0x46,0x43, - 0xd3,0x79,0xc0,0xdd,0xdc,0x01,0x6f,0xb2,0x99,0x9c,0xe1,0x7b,0x62,0xb9,0x47,0x00, - 0xec,0x19,0xa9,0x85,0x08,0x46,0x7b,0xfb,0x76,0xc1,0xe6,0x5f,0xd6,0xd3,0x51,0x87, - 0x08,0x44,0x71,0x8b,0x28,0xb2,0x5f,0x51,0xd1,0x96,0x13,0x82,0x3d,0xc0,0xc3,0x1f, - 0xa1,0xb8,0x71,0x1f,0x58,0xbe,0xd9,0x50,0xd6,0xf5,0x59,0x5e,0xc5,0x5d,0x04,0x26, - 0x16,0x23,0xcb,0x97,0xcf,0xa1,0xf6,0xfa,0xde,0x00,0x7a,0xba,0x22,0x95,0xbd,0x44, - 0x6e,0xb6,0xfd,0x86,0x9b,0xd9,0xf2,0x1a,0xd0,0xa1,0x73,0xdb,0x55,0xb2,0xbd,0xbd, - 0x48,0xfc,0xad,0xa0,0x5c,0x48,0xad,0xfa,0x51,0xae,0x0b,0x85,0xa8,0xc8,0x4a,0xeb, - 0xda,0x4c,0xd4,0xbe,0xf2,0x6d,0x7e,0x0d,0x65,0x1f,0xab,0x99,0xc5,0xd5,0x74,0x33, - 0x79,0xed,0xfa,0x10,0xf4,0x29,0x5c,0x10,0xae,0x42,0x58,0xe2,0x2c,0x99,0x13,0x30, - 0xfb,0x12,0x0c,0xba,0x4e,0x79,0x75,0xab,0xb2,0x63,0x8a,0x68,0xee,0xa0,0xd6,0x93, - 0xf1,0xe5,0x8c,0xa1,0x9b,0xcc,0x8a,0xbc,0x38,0x81,0x0f,0x30,0x45,0x06,0x6d,0xba, - 0x2a,0x92,0x3f,0xb7,0x5b,0xc7,0x06,0x95,0xc5,0x89,0xd7,0xd2,0x48,0x79,0x22,0xae, - 0xea,0x5e,0x92,0x7a,0xa1,0xf3,0xfd,0x0c,0xef,0x74,0xf1,0x0e,0xe8,0x3b,0x94,0x51, - 0xdf,0x0e,0x9f,0x8e,0x08,0x0d,0x2a,0xa0,0x0d,0xd9,0xe9,0xa4,0x1b,0x9d,0x8a,0xfa, - 0x73,0xfb,0x8b,0xba,0xdd,0xe6,0x9c,0xca,0x48,0x25,0x3a,0x73,0x3e,0x85,0xbf,0xb7, - 0xb2,0x72,0x88,0x3b,0x55,0x07,0x35,0x85,0x21,0x10,0xc5,0xbe,0xc1,0xa6,0x87,0x49, - 0x65,0x66,0x94,0x97,0x36,0xaa,0xe5,0x59,0x35,0x19,0x7e,0xfa,0xb9,0x6d,0x34,0x80, - 0x79,0xa6,0x79,0x45,0xd8,0x82,0x6a,0x62,0xbf,0xf8,0xec,0xcc,0xed,0x7b,0x11,0xb9, - 0xdb,0xd2,0x5e,0x6f,0xb5,0x9f,0xbc,0x02,0xfd,0x45,0x49,0xf2,0x79,0x04,0x9b,0xb0, - 0xb8,0x6a,0xb5,0x1d,0xcb,0xb0,0x9f,0x54,0xc6,0xb7,0xe5,0x4c,0xc0,0xe9,0xfb,0xda, - 0x61,0x40,0x80,0x8e,0xcd,0xd0,0x38,0xae,0x77,0x30,0x24,0x49,0xee,0xfb,0x78,0x28, - 0xea,0xb6,0x0a,0xe2,0x81,0x2d,0x6d,0x20,0xbe,0x35,0xd0,0xe5,0x2a,0x29,0x25,0xd3, - 0x37,0xdb,0x87,0x63,0x1b,0x35,0xdf,0x56,0x87,0xf2,0xa6,0xf4,0xf6,0x26,0xe0,0xc2, - 0xbf,0xae,0xcb,0xd6,0x11,0x1c,0xd4,0xe1,0x96,0xe4,0xd6,0x2a,0x72,0xe2,0xad,0x9a, - 0xae,0xbf,0xfd,0xfa,0x5f,0xa6,0xb5,0x05,0x90,0xd4,0x49,0x26,0xbd,0xd6,0xae,0x10, - 0x5b,0x72,0x99,0x34,0x53,0x6d,0xdf,0xed,0x0a,0xdb,0xb3,0x5f,0xd7,0xba,0x63,0x84, - 0x4b,0xf8,0xf9,0xf7,0xff,0xfa,0xf1,0x23,0x47,0x8a,0x50,0xa4,0x49,0x9d,0x8c,0x8a, - 0xb2,0x68,0xe6,0xbd,0x8b,0x84,0x6a,0xde,0x9e,0xcb,0x98,0x05,0x95,0x88,0x16,0x4a, - 0x4b,0xfa,0x00,0x80,0xf3,0x6b,0xa8,0x85,0x3b,0x00,0xfa,0xc3,0xd6,0x24,0x46,0x73, - 0x6e,0xbb,0xd0,0xab,0x81,0xad,0x25,0x8b,0xa9,0x53,0x23,0xc1,0x18,0xd7,0x5b,0x48, - 0xfb,0x08,0x7d,0xc8,0x58,0xc1,0x97,0xf4,0x72,0xdd,0x61,0xb1,0xaf,0xcd,0x23,0xe1, - 0xd3,0x5b,0x66,0x8b,0xa7,0xf3,0xaa,0xd9,0x65,0xd4,0x69,0xa2,0x75,0x21,0x6d,0xf9, - 0x4e,0x2f,0xbd,0xed,0x3b,0x6f,0xd7,0x12,0xb2,0xe7,0xc0,0xf4,0x6b,0xd4,0x04,0x59, - 0x14,0xec,0x20,0xac,0x02,0xcb,0x6f,0xff,0xf8,0x15,0xff,0x89,0xed,0x97,0xd7,0x6e, - 0xfe,0xff,0xeb,0x3f,0xa2,0xec,0x6d,0xe7,0xb5,0x3d,0x9c,0x3c,0x07,0x73,0x2a,0x17, - 0x1b,0xe1,0xde,0xdf,0xc3,0x4e,0x77,0x2f,0xe5,0x3c,0x86,0xed,0x95,0x53,0xb9,0xcb, - 0x02,0xa4,0x37,0xfa,0x92,0x3e,0x15,0x10,0x8a,0xbf,0x9d,0x37,0x84,0x29,0x73,0xdf, - 0x11,0x20,0xe6,0x27,0xf6,0x4b,0x02,0xfb,0xae,0x8f,0xde,0xc7,0x18,0x33,0x95,0xb6, - 0xfc,0x49,0xb9,0x35,0x41,0x3d,0x08,0xfe,0x60,0xc0,0xbe,0x60,0x49,0x3a,0xcd,0x0a, - 0xdb,0x2a,0xa3,0x2e,0x58,0xb4,0xee,0xbd,0xe4,0xa0,0xc9,0x57,0x61,0x1d,0xa2,0x38, - 0xe7,0x98,0x86,0x2a,0xca,0x27,0x67,0x70,0x49,0x2d,0x37,0xd5,0x09,0x73,0x57,0x43, - 0xf5,0xe9,0xf2,0xe7,0xb0,0x1a,0xd6,0x27,0x35,0x4c,0x18,0x41,0x2b,0xbe,0xb4,0xf1, - 0x0a,0xe6,0x7f,0xb5,0xb7,0x47,0xdf,0x15,0xe0,0x9e,0x57,0x94,0x31,0x5b,0xe2,0x3d, - 0x64,0xe5,0xaf,0xb4,0x4e,0xe6,0x51,0x61,0xf8,0xaf,0x7f,0x15,0x04,0x7c,0xe0,0x55, - 0x58,0xf1,0x81,0x16,0x9e,0x43,0x17,0x86,0xd6,0x49,0x54,0xfb,0x1e,0x67,0x8d,0xdb, - 0x90,0x27,0x57,0xd1,0x24,0xa9,0xd7,0x9f,0x24,0x5f,0xaf,0x5a,0xb9,0xd7,0xab,0xc3, - 0xaf,0x3b,0x87,0x9f,0xfc,0xe5,0xfc,0xaf,0x3f,0x20,0xca,0x69,0xf0,0xa1,0xc8,0xe7, - 0xd8,0x1f,0x5f,0x2f,0x03,0xe7,0x58,0x42,0xb8,0x95,0xf8,0xca,0x06,0x5c,0x8a,0x7b, - 0xed,0xa7,0xd3,0xd3,0x86,0x5a,0xa1,0x60,0xd4,0xb8,0x69,0xea,0x98,0xb2,0x23,0xe3, - 0xa2,0x8b,0x91,0x29,0x99,0x21,0x7f,0xe3,0x74,0xdd,0xb0,0x30,0xaa,0xe4,0xaa,0x80, - 0x91,0x29,0x4d,0x5e,0xa0,0x66,0x2f,0x4c,0xfb,0xa7,0x66,0x9a,0x94,0xe5,0x5c,0x4c, - 0x20,0x1c,0x6a,0x05,0x01,0x8b,0xaa,0x58,0x72,0xf6,0x4d,0x4b,0x62,0x5f,0x4a,0x0c, - 0x84,0xbc,0x96,0xe9,0x6b,0x35,0x99,0xa0,0xd6,0xf7,0x3d,0xfa,0x08,0xc3,0x0b,0xd6, - 0x16,0x63,0x0d,0xc8,0x1a,0x06,0xf9,0x04,0xd4,0x91,0x15,0x77,0x4d,0x59,0xad,0xa8, - 0x9f,0x62,0x6c,0x9a,0xb7,0xca,0x0c,0x23,0xf1,0x6d,0x8e,0x58,0x80,0xd0,0x9b,0x27, - 0x45,0xc9,0x48,0x88,0x54,0x22,0x09,0xe2,0x9f,0x71,0x95,0x48,0xa7,0x90,0x85,0x53, - 0x27,0xb3,0xa3,0x05,0xa5,0xbc,0x48,0xd2,0x39,0x19,0x91,0xdf,0xac,0x1a,0xd9,0xc9, - 0x70,0xd5,0x55,0xa7,0x2f,0x84,0x1b,0xe9,0x1a,0xeb,0xbe,0xd7,0x7e,0xa5,0x81,0x1a, - 0x4e,0x5d,0x0e,0xf3,0xa4,0xa4,0x6f,0xda,0x84,0x48,0x22,0xd6,0xe4,0x61,0x33,0x48, - 0x28,0xbb,0x7e,0xd5,0x80,0xf7,0x88,0xf9,0x12,0xc1,0xcf,0x7d,0x01,0x82,0x44,0xc4, - 0x7a,0xde,0xc4,0xa5,0x1f,0xa9,0x31,0x1f,0x81,0x6c,0xe8,0xad,0x3e,0x18,0xe3,0x8c, - 0x95,0xbf,0x16,0xeb,0xdb,0xcf,0xc3,0xfa,0x83,0xf6,0xb3,0xa4,0x3e,0xbb,0xd8,0x15, - 0x51,0xd4,0xc4,0x8b,0xac,0x3b,0x78,0x8d,0xb2,0x3e,0xf3,0x93,0x16,0xb5,0x2c,0xa1, - 0x00,0x7e,0x60,0xc9,0x38,0xe7,0x11,0x50,0x9f,0x25,0xd5,0x85,0xf4,0xfb,0x61,0x13, - 0x95,0xb2,0xba,0x68,0xc6,0xbc,0xbb,0xd1,0xf3,0x05,0xee,0xb0,0x42,0x7a,0x87,0x4c, - 0x96,0x29,0xf5,0x18,0xe8,0x67,0x1c,0xcb,0x5b,0xc7,0x6b,0x24,0xbf,0x57,0xb2,0x7b, - 0x3c,0x7d,0xa3,0xbf,0xf5,0x85,0x3e,0xe3,0x81,0x8a,0xe5,0x94,0xbc,0xd1,0xfd,0xbb, - 0xcd,0x76,0x2c,0x11,0x07,0x1c,0xdf,0x01,0xed,0x7e,0xb6,0x53,0x98,0x73,0xd6,0xb8, - 0xd7,0x56,0xe1,0xf6,0xf6,0xee,0xd0,0xb6,0x3b,0x27,0x23,0x6e,0x6d,0x10,0xce,0x80, - 0xad,0xab,0xfd,0x89,0xc4,0x97,0x76,0xe2,0x74,0xfb,0x6b,0x03,0xfe,0x99,0x4a,0xe7, - 0x67,0x0d,0x1b,0x8a,0xb1,0xe4,0x0b,0x7e,0xf1,0x27,0x17,0x5b,0x10,0x83,0xb9,0xcf, - 0x50,0xcb,0x9d,0x3f,0xb4,0xdc,0x8c,0xa0,0xfe,0x70,0x8d,0x55,0xf3,0x7d,0x01,0x47, - 0x56,0x49,0x62,0x0f,0xa9,0xba,0xd7,0x3d,0xd6,0x69,0xe0,0x64,0x48,0x60,0xf4,0xa1, - 0x11,0x34,0x2d,0xb1,0x03,0x62,0xc6,0x2a,0xeb,0x78,0x40,0xed,0xe2,0x60,0xf1,0xe1, - 0x5d,0x04,0xc7,0x87,0x44,0x42,0xb5,0xbe,0x75,0x2e,0x1b,0xe8,0x9a,0xa4,0x5e,0xe5, - 0xa4,0xfd,0x61,0xc4,0xd2,0x61,0x85,0xb6,0x75,0x7f,0x6a,0xe0,0x20,0x38,0x65,0xa0, - 0x4b,0xbd,0x81,0xa6,0xfb,0x54,0x12,0x7e,0x7b,0xfe,0xd7,0x73,0xf6,0x1e,0x3e,0x25, - 0xeb,0xbc,0xce,0x25,0x26,0x48,0x74,0x79,0xcc,0x7f,0x54,0xde,0x3e,0xfb,0x33,0x22, - 0x03,0xaa,0x1d,0x7e,0xfa,0xb9,0x75,0x30,0x58,0x62,0x22,0x3b,0xf2,0xdd,0xfa,0x3d, - 0x47,0xb0,0x48,0x22,0xfb,0x41,0xd4,0xb0,0xd1,0x53,0x39,0x68,0x2f,0xc1,0x5f,0x7d, - 0x51,0xca,0x8e,0x44,0x05,0xc4,0x2f,0xb7,0xf8,0xb2,0x69,0x66,0x0e,0x81,0x35,0xc1, - 0x24,0xca,0xa1,0x9c,0xc6,0x29,0xbe,0xb3,0x01,0x06,0x69,0x71,0xdb,0xd9,0xb5,0x6b, - 0xb0,0x3d,0x49,0xea,0x35,0xb2,0x8a,0x16,0xc4,0xb4,0xa2,0xca,0x15,0x73,0x8d,0x5c, - 0x23,0xc9,0x8e,0xb2,0x7c,0x20,0xe8,0xfc,0x8e,0xc7,0x3f,0xa2,0xaa,0x79,0x40,0xb3, - 0xee,0x37,0x3c,0xb5,0xfd,0x50,0x77,0x67,0xfd,0x43,0xb4,0x03,0xfb,0x55,0xde,0x81, - 0xfd,0xd9,0xdb,0xff,0x01,0xaa,0xe5,0x89,0x7b,0x07,0x37,0x00,0x00, + 0x92,0xfe,0xaf,0x53,0x94,0xe5,0x6e,0x01,0x18,0x81,0x10,0x25,0xbf,0x41,0x53,0x1a, + 0xc9,0x8f,0x99,0x9e,0xb6,0x7a,0xbc,0x96,0x7b,0x27,0xb6,0xbd,0x1d,0x13,0x45,0xa0, + 0x40,0xa2,0x05,0x02,0x08,0x14,0x28,0x8a,0x43,0x31,0xa2,0x0f,0x31,0x7f,0xf6,0x02, + 0x7b,0x85,0xfd,0xbf,0x47,0xe9,0x93,0xec,0x97,0x59,0x05,0x10,0xa4,0x64,0x5b,0x1d, + 0xf3,0x63,0x27,0x7a,0x2c,0xd6,0x2b,0x2b,0x2b,0xdf,0x99,0x55,0x78,0xf9,0x20,0x2e, + 0xa2,0x7a,0x51,0x2a,0x31,0xa9,0xa7,0xd9,0xf1,0xce,0x4b,0xfa,0x23,0x32,0x99,0x8f, + 0x87,0xbb,0x2a,0xdf,0xa5,0x0e,0x25,0xe3,0xe3,0x97,0x53,0x55,0x4b,0x11,0x4d,0x64, + 0xa5,0x55,0x3d,0xdc,0x9d,0xd5,0x49,0xef,0x39,0x0d,0xea,0xa8,0x4a,0xcb,0x5a,0xe8, + 0x2a,0x1a,0xee,0x1e,0x94,0x49,0x2f,0x2a,0x72,0x5d,0x64,0x2a,0xf8,0x45,0x9f,0x4c, + 0x86,0x71,0xff,0x68,0xf4,0xe8,0xd1,0xd1,0xd1,0xee,0xf1,0xcb,0x03,0x33,0xf1,0xd8, + 0x2e,0x38,0x4e,0x13,0xf7,0xc1,0x3c,0xcd,0xe3,0x62,0x1e,0xbc,0x7f,0xeb,0xb9,0xc9, + 0x2c,0x8f,0xea,0xb4,0xc8,0x5d,0x6f,0x79,0x25,0x2b,0xf1,0xb7,0xa1,0x19,0xf3,0x7f, + 0x1c,0x3a,0x07,0xb2,0x4c,0x0f,0x74,0x2d,0xeb,0x99,0x76,0xfc,0x77,0xc3,0x4f,0x3f, + 0xfb,0xe7,0xc3,0x23,0xf5,0xc2,0xbf,0xf0,0x7f,0xf2,0xdf,0x0f,0xff,0x86,0xe5,0xc3, + 0x25,0x0d,0xab,0xd0,0xc9,0xd2,0x2b,0xe5,0xf8,0x71,0x5a,0xd5,0x8b,0xf0,0xc1,0xa1, + 0x7f,0x15,0xe6,0xb3,0x2c,0xf3,0xc7,0xaa,0x0e,0xff,0xe4,0xa7,0x79,0x92,0xa5,0xe3, + 0x49,0x1d,0xfe,0x30,0x9b,0x8e,0x54,0xe5,0x4f,0x8a,0x2c,0x0e,0xff,0x3a,0xfa,0x45, + 0x45,0xb5,0xaf,0xe5,0x22,0x6c,0x31,0xa8,0xfd,0x4b,0x5f,0x79,0x4b,0xb5,0xb7,0xe7, + 0xaa,0xa0,0x56,0xd7,0xf5,0xab,0x22,0xaf,0x55,0x5e,0x0f,0x6b,0x6f,0xe5,0x8f,0x66, + 0xba,0x33,0x75,0xe4,0x97,0xde,0xb2,0x69,0x89,0x04,0xc8,0x8f,0x82,0x38,0xd5,0x72, + 0x94,0xa9,0x78,0xf8,0xe0,0x70,0xd5,0x6d,0xf5,0x07,0x65,0x50,0x4f,0x54,0xee,0x26, + 0x7e,0xe2,0x0d,0x2a,0x55,0xcf,0xaa,0x5c,0x94,0x2b,0xbf,0x2c,0xb2,0x6c,0x0d,0x51, + 0xfb,0x53,0x3f,0xf2,0x0b,0x43,0x85,0xda,0x4f,0xfc,0xf1,0xf0,0x70,0xd0,0xee,0xf0, + 0x01,0x3b,0x8c,0xf7,0xf6,0x1e,0x24,0xc0,0x2d,0x19,0x1e,0xfa,0x51,0xa6,0x64,0xf5, + 0x31,0x9d,0xaa,0x62,0x56,0xbb,0xb5,0xe7,0xbb,0x3a,0x88,0x64,0x96,0x9d,0x68,0xd7, + 0x0b,0xff,0xe4,0x6a,0xcf,0xb3,0x3b,0x36,0xd0,0x63,0x6f,0x19,0xb9,0xb1,0x4f,0x54, + 0xc1,0x59,0xda,0x6e,0x45,0xdd,0x4c,0x2a,0xe5,0xad,0xec,0x9a,0xef,0xfd,0xef,0x3d, + 0x6f,0xd5,0xee,0xfc,0x3d,0x76,0x4e,0x86,0xfd,0xc1,0xd6,0x8e,0x03,0x60,0xe3,0xd6, + 0x43,0x88,0x44,0xd3,0xf7,0xc1,0x9f,0x62,0x5d,0xb1,0xb7,0x57,0x04,0x49,0x5a,0xe9, + 0x7a,0x38,0x04,0x21,0x4e,0xb0,0x3c,0x04,0xf2,0xf6,0xd8,0x60,0x57,0x51,0x86,0x1d, + 0x96,0x8f,0x6f,0x43,0x5e,0xf9,0x00,0xba,0x9e,0x93,0x7b,0xcb,0xe9,0x30,0x1f,0x24, + 0x37,0x37,0x00,0xb5,0xf2,0xf3,0x62,0x1e,0x7e,0x58,0x61,0x0e,0x8b,0xc5,0x7a,0x5a, + 0x84,0x93,0xec,0xed,0xbd,0x0b,0xca,0x99,0x9e,0xa0,0x31,0xb8,0x38,0x41,0x33,0x72, + 0x2f,0xbc,0xf0,0xa7,0x93,0x9f,0x02,0xac,0x02,0x1e,0x3f,0x0d,0xdf,0x07,0x44,0x74, + 0xf7,0x47,0xff,0xdc,0x7f,0x0d,0x60,0x73,0x59,0x47,0x93,0x8b,0x2d,0x48,0x53,0x6f, + 0xf9,0x3e,0x30,0xe0,0x81,0xf7,0x4f,0x01,0xb0,0x71,0xcf,0x87,0xe7,0xb2,0x9e,0x04, + 0xd3,0x14,0xc3,0xfe,0xb9,0xc7,0x4b,0xd3,0xfa,0x6d,0x51,0xbd,0x56,0x57,0x69,0xa4, + 0xd6,0x8b,0xc1,0xbf,0x62,0x58,0xdc,0xdc,0x2c,0x57,0x0d,0xa3,0x73,0x35,0x17,0xef, + 0xab,0x62,0x9a,0x6a,0xb5,0x66,0xc7,0xc2,0xcf,0x0d,0xa3,0x27,0x6b,0x94,0x8e,0xd4, + 0xa3,0x35,0x63,0xb4,0xb7,0xd4,0x60,0xb7,0x5b,0x04,0xa3,0x59,0x9a,0xc5,0x7b,0x7b, + 0xda,0xfc,0x18,0x0e,0x6d,0x8f,0x07,0xf2,0x4f,0x02,0x22,0xa7,0xeb,0xf9,0x0b,0xe2, + 0xf8,0xca,0x1b,0x74,0xb8,0xd1,0xa1,0x71,0x33,0x6d,0x90,0xbb,0x6f,0xaa,0xaa,0xa8, + 0x5c,0xa7,0x36,0x93,0x1c,0x3a,0x48,0x11,0xd8,0xd6,0xcd,0xcd,0x0b,0xf5,0x18,0x50, + 0x56,0xab,0xb5,0xd8,0xfd,0xc9,0x9d,0x91,0x48,0xda,0xa3,0x24,0x0a,0xe4,0xe2,0x9e, + 0x2d,0xe9,0xaa,0xbc,0x25,0x69,0x75,0x15,0x14,0x97,0x5e,0x3d,0xa9,0x8a,0xb9,0x30, + 0x1b,0x55,0x96,0x8e,0xad,0xd4,0xb3,0x74,0x90,0x66,0x9d,0x54,0xfc,0x07,0x4c,0xa9, + 0x60,0x30,0x08,0xcd,0x55,0x47,0xe6,0x5e,0xd3,0xf9,0x01,0x11,0xff,0x5e,0x40,0xcf, + 0xcb,0xc4,0xf0,0x68,0xa8,0x07,0xef,0x82,0xa4,0xa8,0xde,0x48,0xa0,0xd1,0x6e,0x9e, + 0x40,0x3e,0x31,0x93,0x10,0x5f,0x79,0xae,0xd7,0xda,0x9b,0x1d,0x63,0xb8,0x72,0x39, + 0x55,0xc3,0xdd,0xab,0x54,0xcd,0xcb,0xa2,0xaa,0x77,0x45,0x64,0x75,0x7a,0x77,0x9e, + 0xc6,0x35,0x0c,0x15,0x33,0xb0,0xc7,0x0d,0x58,0x8a,0xb4,0x4e,0x65,0xd6,0xd3,0xd0, + 0x25,0x35,0x3c,0x24,0x2b,0x57,0xa7,0x75,0xa6,0x8e,0xdf,0xcb,0xba,0x56,0x15,0xec, + 0x08,0x8e,0xd6,0x13,0x06,0x9b,0x97,0x07,0x66,0x0c,0x96,0xb0,0x5e,0xd0,0xdf,0xb0, + 0x2a,0x8a,0x7a,0xd9,0xeb,0x45,0x95,0x92,0xd3,0xf0,0x61,0xff,0x55,0xff,0xac,0xff, + 0x62,0xd0,0xeb,0xa5,0xf9,0x65,0xf8,0xf0,0xcd,0xeb,0x37,0xcf,0x5e,0x9f,0xa1,0x35, + 0x9d,0xd5,0x2a,0x0e,0x1f,0x3e,0x3f,0x7d,0x7e,0xf4,0xec,0x08,0xed,0x44,0xa6,0x79, + 0x1d,0x3e,0x7c,0x72,0xfa,0xe4,0xc9,0xe3,0xa7,0x83,0x9d,0x5e,0xaf,0x9a,0x65,0x2a, + 0x7c,0x78,0xf4,0xf8,0xe8,0xf0,0xf0,0xf9,0xc0,0x34,0x7b,0xba,0x48,0x30,0xe7,0xf0, + 0xec,0xf0,0xc5,0xe1,0x63,0xf4,0x65,0x04,0xe1,0xed,0xdb,0x27,0xaf,0x8e,0xde,0xa0, + 0x55,0x00,0xfc,0x93,0x67,0x67,0xcf,0x9f,0xbd,0x45,0x63,0x2e,0xab,0x3c,0x7c,0xf8, + 0xfa,0xc5,0x69,0xff,0xd1,0x5b,0x82,0x56,0xca,0x5c,0x65,0x58,0xfa,0xe8,0xf0,0xf0, + 0xb0,0x4f,0x1d,0x5a,0xe6,0x3a,0x74,0xbe,0x03,0x11,0x2a,0xc7,0x9f,0xa5,0xdc,0xee, + 0x69,0x55,0xa5,0x89,0xaf,0x17,0xba,0x56,0xd3,0xde,0x2c,0xf5,0xd7,0x9d,0xb4,0x64, + 0x5a,0xe4,0x45,0xe8,0xfc,0x45,0xd5,0x67,0x15,0x90,0xd5,0xe2,0x1c,0x6d,0x5e,0x4b, + 0x03,0xba,0x94,0x91,0xf2,0x2f,0xde,0x52,0x67,0xef,0x83,0x1a,0xcf,0x32,0x59,0xf9, + 0xe7,0x2a,0xcf,0x0a,0xbf,0x1d,0x5e,0xed,0xfc,0x61,0x39,0x2a,0xae,0x7b,0x3a,0xfd, + 0x47,0x9a,0x8f,0xc3,0x51,0x51,0xc5,0xaa,0xea,0xa1,0x67,0xb5,0x33,0x2a,0xe2,0xc5, + 0x72,0x2a,0xab,0x71,0x9a,0x87,0xfd,0xc1,0x48,0x46,0x97,0xe3,0xaa,0x98,0xe5,0x71, + 0x08,0xed,0x70,0x2d,0x2d,0xbd,0x41,0x54,0x64,0x45,0x65,0xbb,0x40,0x4e,0x6f,0x90, + 0x80,0x8b,0x20,0xdd,0x34,0xcd,0x16,0xb6,0x9b,0x50,0xf6,0x06,0x3b,0x59,0x9a,0xab, + 0xde,0x44,0xb1,0xbd,0x3f,0x0c,0x9e,0x0c,0x7a,0x73,0x35,0xba,0x4c,0x31,0x97,0x16, + 0xe8,0x29,0x38,0x34,0x21,0x14,0x64,0x4e,0x8c,0x4e,0xa5,0x56,0xf1,0x6a,0x27,0x98, + 0x57,0xb2,0x04,0x12,0xd7,0x46,0x0c,0xc2,0xa7,0xcf,0xfb,0xe5,0xf5,0xa0,0x41,0x4a, + 0xc8,0x59,0x5d,0x0c,0x4a,0x19,0xc7,0xb4,0xf0,0xd1,0x51,0x79,0x2d,0x8e,0x30,0x2e, + 0x9e,0x3e,0x2e,0x81,0x3f,0xb9,0x48,0x55,0x2d,0x61,0xed,0xcb,0x8c,0x7c,0x49,0xa6, + 0xae,0x07,0x00,0x3c,0xce,0x7b,0x29,0x68,0xa9,0xc3,0x48,0x11,0xa5,0x07,0x63,0x59, + 0x86,0xcf,0x01,0xd4,0x82,0xc1,0xd9,0xeb,0xba,0x98,0x86,0x87,0x80,0x36,0xd8,0x69, + 0xe9,0x61,0xfa,0x00,0x1b,0x8e,0x34,0x8d,0x85,0x39,0x18,0x09,0x80,0x07,0x24,0x63, + 0x08,0x97,0xc1,0xef,0x19,0x16,0xd9,0x23,0xd2,0xcf,0x5b,0x44,0x83,0x78,0x78,0x03, + 0x0b,0xb4,0x92,0x71,0x0a,0xeb,0x76,0xc8,0xb8,0x1e,0x2e,0x0d,0x19,0xd2,0x7f,0xa8, + 0xf0,0xf0,0x09,0x96,0x72,0x73,0x6e,0x40,0x3d,0xed,0xf7,0xdb,0x33,0x0f,0xe8,0x1c, + 0xe1,0x21,0x76,0xd5,0xb3,0xd1,0xf2,0x36,0xb1,0x89,0xb3,0x96,0x09,0x06,0x18,0xe0, + 0x6f,0x30,0x89,0xa5,0x1a,0x58,0x6b,0xc5,0x4a,0x6a,0x39,0xdc,0x23,0xeb,0x7f,0xf4, + 0x94,0x71,0x39,0x5a,0x6e,0x2d,0xdf,0xc6,0x25,0x53,0xa4,0x73,0x3d,0x12,0x20,0x22, + 0x7c,0xd0,0x7f,0xa1,0xa6,0x03,0x32,0x1a,0xbd,0xba,0x02,0xaf,0x61,0x07,0xa6,0xe1, + 0xac,0x2c,0x55,0x15,0x81,0x8b,0x83,0x9d,0xee,0xee,0xac,0x63,0xde,0x9a,0x83,0x7d, + 0xf1,0x9c,0xf6,0x8c,0xb3,0x8e,0xa4,0x19,0xf2,0x10,0x42,0x77,0x11,0x9c,0x35,0x8e, + 0xa8,0x0e,0x63,0xf6,0x79,0xe6,0x8e,0xb0,0x33,0x09,0x1c,0xb3,0x97,0x79,0xd9,0x88, + 0xc9,0x33,0x92,0x92,0xfb,0xf1,0xb6,0xd9,0x2a,0xae,0x97,0x86,0xea,0xc2,0x8a,0x5c, + 0x87,0x3e,0x8f,0xb6,0xc8,0xbb,0x75,0x40,0x2c,0x8e,0xd7,0x47,0x83,0x97,0xb2,0x82, + 0xdc,0x1f,0xdc,0x83,0x75,0x84,0x26,0x93,0x95,0x4f,0x16,0x56,0xc4,0x80,0x41,0x71, + 0xa5,0x2a,0xb2,0x76,0x3d,0x52,0x0d,0x68,0xcb,0x62,0x3e,0x51,0x95,0xa2,0x7d,0x82, + 0x51,0x3a,0x5e,0x6e,0xa1,0xb6,0xc5,0x3a,0x90,0xad,0xb8,0x5c,0x76,0xf1,0x85,0x5f, + 0x58,0x05,0x64,0x9a,0x36,0x7a,0xa9,0x03,0xfd,0x23,0x19,0x6f,0x74,0x93,0xf8,0x02, + 0xc4,0x48,0x56,0x4b,0x2b,0xe5,0x8f,0xee,0x92,0xf2,0x35,0xed,0x06,0x1d,0xe9,0x22, + 0xa9,0x36,0x87,0x3f,0xec,0xf7,0xbf,0x35,0x60,0x44,0xda,0xb2,0x70,0x94,0x15,0xd1, + 0x65,0xa3,0x3c,0x34,0xe3,0x6e,0xed,0xc1,0xba,0xbc,0xa8,0xd5,0x72,0x8b,0x4c,0xb7, + 0x25,0xbc,0x61,0x01,0xe4,0x0b,0x6c,0x83,0xd0,0x6e,0x98,0x9f,0xc7,0x4f,0x00,0xa8, + 0x2e,0x8a,0x4c,0xdf,0xcb,0x40,0xf0,0x1e,0x9d,0xb3,0x1c,0xb2,0x85,0x19,0xcd,0x20, + 0x3a,0xf9,0xb2,0x2c,0x74,0x4a,0xaa,0x14,0x56,0x2a,0x93,0x35,0xc2,0xdd,0xc1,0x3d, + 0xd5,0xf2,0x96,0x1e,0x3d,0xff,0xa2,0x1e,0x35,0x12,0x0c,0x2d,0x15,0x8c,0x50,0x87, + 0x40,0xbc,0xa0,0x94,0x15,0x50,0xbe,0x4b,0x1c,0x8d,0xac,0xdf,0x6d,0xc0,0xb6,0xec, + 0x11,0x53,0x73,0x56,0x69,0x80,0x28,0x8b,0x94,0x28,0xd0,0x1c,0x34,0x9c,0x90,0xec, + 0x2d,0x6f,0x59,0x7c,0xbb,0xfe,0xb6,0xa0,0xfc,0x71,0xaa,0xe2,0x54,0xba,0x16,0x4c, + 0x18,0x15,0x94,0xa8,0x20,0x1e,0x37,0xd0,0x42,0x99,0xd4,0x0c,0x8e,0xbd,0x7e,0xb8, + 0xbb,0x3b,0x68,0x09,0x29,0x47,0xc0,0x12,0x98,0x0f,0x88,0xd6,0x3d,0xb2,0xce,0x56, + 0x4b,0xf9,0x77,0xa6,0xe0,0x72,0x7b,0xe0,0xc0,0x80,0x35,0x82,0x7f,0xae,0xc0,0xce, + 0xa9,0x1e,0xff,0x2b,0x06,0x91,0xd6,0x23,0xa0,0xe9,0xdd,0xa5,0x20,0x76,0x48,0x55, + 0xd5,0x1d,0xfa,0xf0,0x30,0x2a,0x89,0x69,0x5b,0x92,0xbc,0x16,0xf5,0x56,0xa8,0x3b, + 0xce,0xcb,0x48,0x51,0xbf,0x63,0x97,0xe8,0x64,0x95,0x62,0x14,0x41,0xe5,0x3a,0x45, + 0xa0,0x03,0x1b,0xb5,0xad,0x02,0x1c,0x34,0xdc,0xe1,0x75,0x7f,0x27,0x83,0x89,0x1c, + 0x4c,0x89,0x03,0x52,0x05,0xd1,0xa1,0xd4,0x6a,0xe7,0xd3,0x24,0x8d,0x63,0x95,0xff, + 0xdc,0x9e,0x27,0x2f,0x72,0xf5,0x20,0x9d,0x52,0x90,0x06,0xe7,0x0c,0x42,0x11,0x91, + 0x92,0x84,0x14,0x4a,0xc0,0xba,0x15,0x24,0xbe,0x48,0xee,0x02,0xa8,0x54,0x02,0x2f, + 0x0e,0x9e,0x76,0xce,0xf8,0xa8,0x63,0x7b,0xcd,0xa1,0x59,0x72,0xbf,0x6c,0xe3,0x11, + 0x2d,0xfc,0x0b,0x8c,0x94,0xcb,0xdb,0x1a,0xb0,0xda,0x41,0xf4,0xc9,0xc1,0xe0,0xcb, + 0x03,0x93,0x3b,0x53,0x78,0x73,0xfc,0x32,0x4e,0xaf,0x44,0x94,0x49,0xad,0x11,0x75, + 0xc2,0x9e,0x36,0xa9,0xb5,0xaa,0x90,0x14,0x83,0xd6,0xcd,0x18,0x3c,0x3c,0xe7,0xcb, + 0xe8,0x3a,0x7e,0x39,0x39,0x3c,0x6e,0xe2,0x4c,0xfc,0xdc,0x98,0x07,0x9f,0xbc,0x2b, + 0xd2,0x18,0xb9,0x38,0x40,0xf5,0x9a,0x05,0x07,0x16,0xe4,0xce,0xce,0xcb,0x07,0xbd, + 0x9e,0xf8,0x38,0x49,0xb5,0x28,0xe5,0x58,0x09,0xfc,0x9d,0x4f,0x64,0x2d,0x90,0x0d, + 0x6b,0x81,0xe0,0x58,0xa9,0x5c,0x4f,0x20,0xe9,0x2a,0xf6,0x45,0x51,0x61,0x8e,0xe6, + 0x9f,0xb0,0xf0,0x39,0x68,0x34,0x55,0x84,0xb3,0x90,0xfa,0x52,0x0b,0xd8,0x86,0x1d, + 0x41,0xff,0x9b,0xa8,0xac,0x0c,0x05,0x82,0x7e,0x61,0xd4,0x0a,0x51,0x74,0x99,0x2a, + 0x2d,0x14,0x84,0x68,0x21,0x92,0x54,0x65,0xb1,0xe8,0x64,0xeb,0x62,0x22,0xb5,0x2f, + 0x60,0x41,0xc5,0x2f,0x33,0x5d,0xf3,0x3a,0xf8,0x51,0x1d,0x88,0x5e,0x0f,0x47,0xef, + 0x50,0x83,0x6d,0x23,0xce,0x6c,0xa1,0x52,0x31,0x62,0xb8,0x6b,0x1a,0xe6,0x84,0xd8, + 0x67,0xb1,0x7b,0xfc,0x0a,0xff,0x0a,0x28,0xf9,0x18,0xf1,0x24,0x64,0x16,0x24,0x31, + 0x73,0x36,0xc9,0x02,0xf5,0xb1,0x8b,0x4a,0xfa,0xd9,0x52,0xf2,0x00,0x1b,0x52,0x24, + 0x0f,0xed,0x81,0xe9,0x92,0x76,0x0a,0x35,0x77,0x05,0xda,0x71,0x91,0x67,0x0b,0x61, + 0xc4,0x51,0xc8,0x2a,0x95,0xbd,0x4c,0x8e,0x54,0x36,0xdc,0x7d,0xbd,0xde,0x90,0x80, + 0x35,0xeb,0x89,0xc0,0x36,0xac,0x01,0x97,0x8e,0x8e,0x4d,0x0a,0x08,0xf2,0x1f,0x81, + 0xd1,0xd9,0x31,0x08,0xd6,0x3d,0x21,0xce,0x8d,0xc5,0x71,0x7d,0xfc,0x36,0xad,0xa6, + 0x70,0x78,0x98,0x88,0xc6,0xcb,0x38,0x66,0x34,0x92,0x39,0x33,0x30,0x8e,0x1b,0x2c, + 0x3f,0xb3,0xf8,0x8c,0x52,0xbe,0x8d,0x95,0xa3,0x2c,0xbe,0xdf,0xd2,0x77,0x60,0x2f, + 0x8e,0x09,0xc6,0x54,0xf5,0x06,0x84,0xaa,0xba,0xef,0xde,0x7a,0xb1,0xb9,0x35,0x3a, + 0xee,0xb7,0xf4,0x3d,0xd9,0x11,0x41,0x7a,0xb4,0x01,0xa0,0xcc,0xb7,0x96,0xe3,0x4f, + 0x46,0xec,0xb2,0x54,0xdd,0x22,0xf0,0x07,0x95,0x43,0xae,0xbf,0x4a,0x60,0xde,0x6c, + 0x73,0x9f,0xf9,0x3d,0x8f,0xf8,0xb6,0x42,0x92,0x28,0x2a,0x59,0xaf,0xf1,0xb4,0x53, + 0x10,0xec,0x18,0x99,0x4a,0x4a,0xfd,0x7b,0x60,0x51,0x4e,0xbd,0xc9,0xe9,0xfa,0x7e, + 0xcb,0x4f,0x23,0xf2,0xed,0x50,0x4a,0x4e,0x38,0x37,0x40,0xc8,0xa8,0xbe,0x8b,0x6e, + 0x3b,0x2f,0xcb,0x06,0x0a,0xc5,0x2c,0xbb,0xcd,0xd1,0xb9,0x81,0x99,0xe5,0xe7,0x29, + 0x6b,0xd3,0x5a,0xfd,0x55,0xda,0x7e,0x2c,0x6a,0xb9,0x45,0xdb,0xfa,0xfe,0x92,0x5b, + 0x0b,0x98,0xcc,0x02,0x66,0xe2,0x0e,0x0d,0x28,0xcb,0x7b,0x8a,0x71,0x01,0xe3,0x16, + 0x8b,0xa4,0x2a,0xa6,0x42,0xd7,0x45,0x05,0xc3,0xb6,0x09,0x67,0xfa,0x3b,0xd4,0x21, + 0x03,0x30,0xa1,0xa8,0x44,0xb1,0x01,0x23,0x53,0xf7,0xa6,0x2f,0x41,0xb8,0x07,0x81, + 0xcf,0xd5,0xb4,0xa8,0x16,0x5f,0x25,0x2f,0x67,0xe3,0xb9,0xcc,0x60,0x67,0x65,0x89, + 0x33,0xaa,0xcd,0xb3,0x4d,0xd2,0xfb,0x9e,0xad,0x1a,0x43,0xd1,0x19,0x80,0xe0,0xf8, + 0x60,0x13,0x4c,0x76,0x4f,0xad,0xbd,0xf8,0x70,0x7a,0x7e,0x07,0x16,0xe5,0x7d,0xa8, + 0xb3,0x75,0x16,0xf8,0x1e,0xb2,0xfe,0x3a,0x92,0x55,0xa4,0x04,0x5c,0xbc,0xe8,0x71, + 0xc7,0xbb,0x37,0xaf,0x05,0x47,0x19,0x8e,0x16,0xaf,0xcf,0x4f,0xe1,0x56,0x92,0x44, + 0x55,0x7a,0x87,0xea,0xb8,0x34,0xa1,0x52,0x81,0x38,0x6d,0xb4,0x40,0x4c,0x8b,0x18, + 0xfe,0x9a,0xd9,0xa6,0x05,0x9b,0xeb,0x34,0x11,0x29,0xf8,0xa8,0xe4,0x15,0xbc,0x90, + 0xf1,0x02,0x6c,0xd3,0xf4,0xd5,0xee,0xf1,0xd1,0x63,0xf1,0xfd,0x99,0x35,0xfe,0x3b, + 0x05,0x4f,0xa4,0xc3,0x0c,0x44,0x35,0xcb,0x01,0x63,0xae,0x2a,0x21,0xf3,0x98,0xd1, + 0xc8,0x55,0x3d,0x2f,0xaa,0xcb,0x06,0x4d,0xae,0x5b,0x0a,0xae,0x10,0x08,0x08,0xec, + 0xb8,0x20,0xaf,0x98,0x66,0x8c,0xd1,0x0e,0xa3,0x2b,0x2e,0x95,0x2a,0xb5,0x88,0x2b, + 0x39,0xc7,0xa4,0xe0,0x8b,0xcc,0xbf,0x68,0xc4,0xf4,0xab,0x86,0xcb,0x9c,0xf2,0x2e, + 0xb1,0x4e,0xee,0x69,0x75,0xec,0x5e,0x77,0x08,0x75,0xa2,0xef,0x94,0xea,0x2e,0x14, + 0xa4,0x48,0x80,0x92,0xda,0xe9,0xd4,0x12,0x1c,0xc5,0xd8,0x1a,0x59,0xd8,0x27,0x29, + 0x4f,0xed,0xf2,0x8d,0xe3,0x72,0xa0,0x31,0x2f,0x04,0xa2,0xb4,0x4b,0x22,0x21,0xa2, + 0x8c,0x08,0xac,0x18,0x29,0x31,0x2b,0x05,0x1a,0x45,0x1e,0xa9,0x70,0x83,0xd2,0xf4, + 0xdb,0x50,0xf2,0x17,0x04,0xec,0xd0,0x68,0x97,0x68,0xcb,0x41,0x06,0xc5,0x0e,0x80, + 0xeb,0xb5,0xdc,0x69,0x04,0xa4,0x98,0xe7,0x02,0x01,0x8b,0x2e,0x8b,0x3a,0x10,0x54, + 0xfc,0x33,0xa1,0x4c,0x5a,0x9b,0xa1,0x4e,0x68,0xd1,0x25,0xff,0x0f,0x66,0xc7,0xaf, + 0x92,0xbf,0x9d,0xd7,0x21,0x9a,0xbe,0x2f,0xd9,0x91,0xc7,0x6d,0x19,0xc6,0xea,0x9e, + 0x4b,0x4f,0xe3,0x18,0x5e,0x59,0x6f,0xac,0x4d,0xef,0x69,0x0c,0x7f,0x80,0x8b,0xd9, + 0x5c,0x99,0xdf,0xd3,0xfc,0xfd,0xd9,0x90,0x71,0x53,0xb1,0xf5,0xef,0x5a,0x4b,0xda, + 0xb8,0x65,0x19,0xf4,0xef,0xdb,0x1d,0xfd,0x29,0x92,0x31,0xbd,0x05,0x24,0xba,0x1f, + 0x90,0x0f,0x0a,0x12,0x00,0xc1,0xb9,0x4a,0xe5,0x06,0x00,0xb4,0xbf,0x1a,0x56,0x98, + 0xcc,0xe1,0xf8,0xa5,0x14,0x93,0x4a,0x25,0xc3,0xdd,0x83,0x5d,0xe0,0x44,0x9e,0x5a, + 0x1e,0x8b,0xbd,0x29,0x42,0xc0,0xa2,0x1e,0x88,0xf5,0xa8,0xb5,0x3f,0xa0,0xcf,0xda, + 0x53,0x4a,0x40,0xb4,0x60,0x76,0x9a,0x7d,0x9a,0x8a,0x73,0x5b,0xc0,0xfe,0xc6,0x4d, + 0xdb,0xaa,0x79,0x5c,0x44,0xb3,0x29,0x8e,0x1b,0x40,0x68,0xdf,0x64,0x8a,0x7e,0x9e, + 0x2d,0xbe,0x8b,0x31,0x63,0xb5,0x5e,0x70,0x39,0x72,0x47,0xed,0x8a,0xd1,0xf1,0xf0, + 0xb0,0xff,0xf8,0xf9,0x93,0x67,0x4f,0x4f,0xdc,0xd1,0x81,0xfd,0xe9,0x05,0x75,0xf1, + 0x36,0xbd,0x56,0xb1,0x7b,0xe4,0xed,0x3b,0xe2,0xfc,0xcc,0x09,0xf9,0x4e,0x82,0xf3, + 0x36,0x9e,0x76,0xf4,0x98,0x06,0xbe,0x3f,0x73,0x3a,0x70,0xe3,0x59,0x45,0x55,0x74, + 0xd0,0x92,0x6e,0x1b,0x62,0x73,0x8d,0x91,0x64,0x45,0x81,0xee,0x83,0xe7,0x4f,0x1f, + 0xf7,0xfb,0x9e,0x3f,0xd9,0xe8,0xfd,0x96,0x7b,0x0f,0x1e,0x3d,0xa5,0x21,0xa3,0x98, + 0x62,0xba,0x39,0x83,0xc6,0x0e,0x9e,0x62,0xf8,0x7a,0xa8,0xbf,0x7d,0xda,0x1f,0x60, + 0x56,0x9a,0xb8,0xb1,0xd7,0x1c,0x78,0xdf,0x89,0x85,0xb3,0x3f,0xd9,0x77,0x26,0xce, + 0x00,0x03,0x93,0x66,0x80,0x7a,0x30,0x30,0xdd,0x77,0xa6,0x8e,0x5d,0x34,0x6d,0xc6, + 0xa8,0x13,0x63,0xd7,0xfb,0x8e,0x76,0x9a,0xdb,0x02,0xd3,0xd8,0xe9,0x1c,0x27,0xca, + 0xb4,0xab,0x32,0x3f,0xf2,0x96,0x2a,0x0b,0x58,0x30,0x48,0x15,0x86,0x0e,0xc2,0x34, + 0x2c,0x8e,0x3a,0x33,0xcb,0x59,0xed,0xa6,0xb1,0x5f,0xd3,0x5c,0x3a,0xba,0x1a,0x82, + 0x27,0xb1,0x37,0xd8,0xba,0xd8,0x43,0x7b,0x0d,0x26,0xba,0xb9,0x71,0x40,0xbc,0x83, + 0x03,0xa1,0x74,0xf9,0x77,0x68,0xa7,0xaa,0xf1,0xaf,0xe4,0xeb,0x09,0x24,0x44,0x8d, + 0x8b,0xe0,0x20,0x86,0xef,0x15,0xc8,0x06,0xf9,0x08,0x6e,0x04,0x0c,0x48,0x0c,0x1b, + 0xf4,0x71,0xc2,0x3e,0x4e,0x0b,0x09,0x87,0x36,0x26,0x40,0x91,0x9c,0x69,0x6a,0x63, + 0x41,0xa5,0x62,0x98,0x42,0x18,0xcb,0x0a,0x19,0x56,0x41,0xb6,0x0b,0xc6,0x51,0xcc, + 0x81,0x86,0x98,0xe5,0xf0,0x9d,0x48,0x22,0x62,0x58,0x4e,0x5e,0x61,0xe2,0x9c,0x8e, + 0xdd,0x24,0x58,0x52,0x44,0x95,0xd4,0x13,0xb8,0xb0,0x02,0xc6,0x56,0x5d,0x23,0x1e, + 0x84,0x27,0xcc,0xd2,0x4b,0x85,0xa1,0x09,0x19,0x4d,0x4a,0xa1,0xc8,0x6e,0x66,0xb3, + 0x71,0xb0,0x43,0xa7,0xfe,0xf0,0x61,0xb8,0x2c,0xc9,0xe3,0x15,0x79,0xe8,0xf0,0x0f, + 0xcc,0x71,0x7c,0x50,0x20,0x74,0xf8,0x7c,0xa2,0x4c,0xd1,0xd6,0xf3,0xd0,0xa1,0x4a, + 0xda,0xdc,0xe0,0xc9,0xb9,0x82,0xe3,0xc3,0xfc,0xa6,0x51,0xe8,0x98,0x4d,0x5d,0x6e, + 0x79,0x0e,0x89,0x04,0xa2,0xb9,0xbf,0xcf,0x63,0x80,0xe0,0xfb,0xb3,0xb8,0x18,0x0b, + 0x97,0x6b,0x2f,0xd5,0xac,0xac,0x31,0xa3,0x46,0xee,0xb8,0x3d,0x4e,0x7d,0x18,0xda, + 0xe8,0x65,0x58,0x31,0xf9,0xd3,0x0c,0xff,0x50,0xff,0xa5,0x3d,0x38,0x75,0x0a,0xee, + 0x75,0xfc,0x11,0x14,0x3f,0x2f,0x66,0x58,0xd7,0xfc,0x02,0x2e,0x7c,0x92,0x38,0x2d, + 0x3d,0x67,0x35,0xe0,0x83,0x9e,0xfd,0x78,0xf1,0x1f,0x74,0x19,0xcc,0x5e,0x10,0xa0, + 0xaa,0xb4,0x26,0x27,0xde,0x28,0xb0,0xe3,0xcf,0xca,0x98,0x2f,0x8a,0x2b,0x15,0xa9, + 0xf4,0x8a,0xc6,0x1a,0x4e,0x12,0x08,0x86,0x01,0x39,0xa8,0x87,0x74,0x19,0x3a,0xe8, + 0x88,0x11,0x65,0xfb,0x74,0x7b,0x0a,0x54,0x79,0x3c,0x26,0xc1,0xfd,0xc6,0x75,0x66, + 0xa5,0xe3,0x6d,0x88,0x12,0x7a,0x20,0x82,0xa4,0x6f,0x71,0x00,0x3a,0x20,0xfe,0xf7, + 0x68,0x2a,0xf8,0x76,0x9f,0x4c,0xfc,0x33,0xd9,0x37,0x27,0xde,0x3e,0xfa,0x0d,0x24, + 0x44,0x32,0x13,0x92,0xc3,0x42,0x68,0xb9,0xa0,0xd0,0x04,0xae,0xb0,0x15,0x48,0xf6, + 0xbe,0x6b,0xa0,0xb4,0x59,0x91,0x04,0xe2,0x55,0x81,0x31,0x4c,0xd7,0x94,0x8f,0x53, + 0x64,0x63,0x40,0x01,0xf3,0x4a,0x8a,0xdf,0x7e,0xfd,0x27,0x01,0xb5,0xe1,0x4f,0xac, + 0x12,0x39,0x43,0xb4,0xde,0x46,0x47,0x45,0xaa,0x19,0x69,0xca,0xe4,0x21,0x85,0xf5, + 0x04,0x93,0x03,0xac,0x27,0xe5,0x72,0x92,0xb9,0xe3,0xc7,0xc1,0x15,0xa2,0x36,0x10, + 0x6a,0xdf,0xc5,0xb1,0xaf,0x28,0x7b,0xce,0xeb,0xbd,0xbd,0xf6,0xe7,0x83,0xe1,0xd0, + 0x89,0x80,0x80,0xe3,0x9d,0x38,0xe2,0x7f,0xff,0x87,0x08,0xd4,0x0c,0xed,0xaf,0x17, + 0xfc,0xbb,0x81,0x81,0x29,0x9d,0x71,0xdb,0x19,0x3a,0x8e,0x47,0xff,0x67,0x62,0xf2, + 0xbe,0xc8,0x7c,0x69,0x63,0xbe,0x05,0x85,0xc2,0xf6,0x9c,0xf5,0x50,0x55,0x39,0xfe, + 0x87,0x0f,0x9f,0xe2,0x80,0x65,0xfb,0x03,0xab,0xee,0xcf,0x37,0x37,0x1b,0x6d,0x5e, + 0xe2,0x1f,0xb0,0x38,0xdf,0x40,0x1e,0x6f,0x1a,0xb1,0x3a,0x00,0x3b,0x35,0x78,0xdd, + 0x9d,0x0c,0xb4,0x47,0x32,0x76,0x08,0x81,0xf5,0xfe,0x48,0x7f,0x0d,0x02,0x7a,0x71, + 0xe2,0x92,0xe0,0x7d,0x32,0x0d,0xde,0x89,0x7e,0x00,0x61,0xba,0x20,0xb3,0x53,0x1c, + 0x2a,0x73,0x6f,0x82,0x28,0x73,0x1a,0xe5,0xf0,0xa6,0x39,0x81,0xe1,0x0a,0x76,0x8d, + 0x7b,0xa6,0x1e,0x81,0x10,0x78,0x60,0x62,0xe7,0x79,0x0a,0x7d,0x11,0x14,0x19,0xeb, + 0x46,0xc3,0xed,0x5b,0x0b,0x44,0x45,0x53,0xc5,0xa2,0x15,0x88,0x33,0x70,0x57,0xc4, + 0x12,0xa6,0x82,0x21,0xf1,0xbb,0x08,0x63,0x78,0x0c,0xe7,0x60,0xaf,0xc0,0x7c,0xbf, + 0xb5,0x30,0xcc,0x60,0x7e,0x10,0xa1,0xd1,0x47,0x17,0x9b,0xe8,0x99,0xb2,0x71,0x81, + 0x3b,0x85,0x44,0x53,0x6d,0xd0,0xca,0x4a,0x4a,0x11,0xb9,0x98,0x43,0xe6,0x48,0x60, + 0x48,0x3c,0x24,0x6b,0xa7,0x29,0x1f,0x01,0x5a,0xec,0x1b,0xe3,0xd8,0xe6,0xd1,0x24, + 0x35,0xac,0xca,0x23,0x95,0x14,0x95,0x09,0xa0,0x19,0x96,0x89,0xfd,0xd8,0xea,0x11, + 0xb2,0x01,0xf7,0x9a,0xa1,0x53,0x82,0x4b,0xd1,0x77,0x15,0x1b,0xf4,0x8d,0x48,0x5b, + 0xf9,0xa3,0xd9,0x14,0x52,0xca,0x2c,0x33,0x11,0x65,0xc5,0x65,0x01,0x42,0xb8,0x64, + 0xad,0xa0,0x4b,0xee,0x52,0xc5,0x06,0x56,0x54,0xd0,0x69,0x05,0x3b,0x47,0xe1,0xd2, + 0x9c,0x0b,0x64,0xb0,0x54,0x49,0x25,0x6d,0xea,0x44,0xa1,0x50,0x95,0xb9,0x09,0x25, + 0x49,0xbd,0xed,0x09,0xe8,0x84,0x5c,0xda,0x32,0xc0,0x0c,0x99,0x2c,0x95,0xb6,0x0e, + 0x06,0xed,0xb1,0x1b,0x07,0xe2,0x9d,0x4a,0x08,0x3f,0xca,0x72,0xea,0x56,0xe7,0xb5, + 0x4c,0x2d,0x4e,0xbb,0x92,0x2c,0x9c,0x2f,0x9e,0xf6,0x05,0x55,0x13,0x84,0x1c,0x91, + 0x31,0x93,0x04,0xee,0x1f,0x2a,0x37,0xf8,0xf8,0x0d,0x87,0x3b,0x8a,0x5c,0xaa,0xa2, + 0x04,0xaf,0x35,0xce,0xcb,0x66,0xc2,0x40,0xc3,0x24,0xb6,0x13,0xad,0x99,0x08,0xc4, + 0x77,0xb0,0x2c,0x05,0x98,0xce,0x69,0x4c,0xb8,0x05,0xb9,0xa1,0xa3,0xe1,0x9c,0xcf, + 0x86,0x88,0xce,0xd8,0x42,0x13,0x49,0x26,0xc7,0x74,0x1e,0x8a,0xe0,0x73,0xc4,0xf4, + 0x45,0x3e,0xc6,0xa9,0xf9,0x75,0x06,0x1b,0x90,0xc0,0x46,0x11,0x06,0xea,0xf0,0xc1, + 0x83,0x38,0xe8,0x10,0x76,0xd0,0xc4,0x18,0xe0,0xd3,0x30,0x0e,0x78,0x17,0x52,0x08, + 0x2b,0xaa,0xef,0x59,0x4c,0x6e,0x6e,0xcc,0xe2,0xb5,0x26,0xcc,0xa1,0xb2,0xa6,0xef, + 0xc4,0x61,0xf4,0x72,0x0d,0xdb,0xce,0xf6,0x19,0xe7,0x32,0x06,0xb5,0x13,0xe7,0x98, + 0x2d,0x4f,0xc7,0xea,0x5c,0x23,0xe2,0x41,0x88,0xe2,0x85,0x1c,0xa3,0xd8,0x0d,0x4f, + 0x1c,0x73,0x3c,0x27,0xdc,0xda,0xf8,0xc4,0x31,0x72,0xca,0x50,0xad,0x9b,0x80,0x4e, + 0x32,0x47,0xd6,0x08,0xb0,0xae,0xd3,0x01,0xb6,0x94,0x96,0x8d,0xf8,0x5a,0xeb,0xd6, + 0xc4,0x29,0x8d,0x7b,0x37,0xe2,0x64,0xb2,0x38,0xda,0x60,0xa4,0xe8,0x00,0x45,0xa9, + 0xf2,0xd0,0x8e,0xcd,0xd8,0x90,0x1a,0x50,0x20,0xef,0xac,0xe4,0x12,0xc4,0x88,0x3c, + 0x21,0x29,0x91,0xa4,0x65,0x53,0x48,0x37,0xbd,0x4d,0xe2,0xec,0x73,0x0d,0x11,0x09, + 0xd0,0x94,0xcb,0x08,0x36,0x17,0x05,0x87,0xc8,0xbd,0x29,0xe6,0xc7,0x37,0x4c,0x42, + 0xca,0xbc,0xb7,0xdc,0x91,0x39,0x12,0xd3,0xe6,0xc4,0xf9,0x78,0x87,0xba,0x10,0xb1, + 0x23,0x32,0x1f,0x46,0x47,0x7a,0x08,0xeb,0xa9,0x32,0x82,0x68,0x02,0xd6,0x87,0xb6, + 0x97,0x1b,0x95,0xa8,0x46,0x7c,0x4c,0x68,0xc6,0xfc,0x21,0x2d,0x90,0x46,0x63,0x06, + 0x02,0x46,0x6a,0xc2,0xcb,0x38,0x12,0x40,0xd4,0x82,0xb8,0x17,0xb9,0x99,0x15,0x27, + 0x99,0x41,0xe9,0x46,0x14,0xa7,0x24,0x4a,0x02,0x00,0x8c,0x92,0x34,0x1e,0x1a,0xe2, + 0x2e,0xd9,0xe8,0xcc,0x58,0x26,0xc9,0xa3,0xb1,0xc4,0xc1,0x28,0x2d,0x48,0x2d,0x55, + 0x96,0x30,0xec,0x94,0xe6,0x41,0xa7,0x67,0x30,0x0b,0x20,0x18,0x82,0x22,0xc8,0x7b, + 0xc2,0x3b,0x92,0xaf,0x25,0x65,0x02,0xb2,0x26,0x3b,0x1f,0xc3,0xa2,0xb3,0xad,0xbb, + 0xcc,0x8b,0x91,0x66,0x6d,0x6e,0x4e,0x61,0x8c,0x29,0x6d,0x4c,0x96,0x31,0x86,0xb9, + 0x45,0xf0,0x55,0xa7,0x59,0xc7,0x22,0xd8,0xf8,0x47,0x0f,0x60,0x8d,0x29,0x50,0x6b, + 0x34,0xf2,0x6f,0x69,0xef,0x6d,0x6a,0x75,0x1a,0x2b,0x32,0xb2,0xab,0x97,0x9a,0xb5, + 0x8e,0xe8,0x08,0x84,0x65,0x74,0xd9,0xaa,0xaf,0x9c,0xb6,0xa4,0x63,0x8b,0x53,0xa6, + 0x18,0x94,0xe4,0x82,0xd9,0x92,0x74,0x6d,0x07,0x30,0x1c,0x23,0xcc,0x08,0x1c,0xe6, + 0x56,0x68,0x45,0xd9,0xb2,0xce,0x94,0x50,0x8b,0x24,0xf1,0xed,0xae,0x76,0x03,0x1b, + 0x17,0x06,0xb0,0x9a,0x0b,0x3e,0x27,0x51,0xc5,0x96,0xd1,0x49,0xa8,0x4d,0x68,0x0a, + 0x3e,0xc7,0x85,0x15,0x50,0xeb,0x47,0x3e,0xeb,0x41,0xda,0xed,0xb7,0x14,0xa7,0x49, + 0x7d,0x98,0x62,0x46,0xb2,0x49,0xfe,0x20,0x65,0x1c,0x88,0x18,0x5d,0x1a,0x6c,0xe8, + 0x00,0x48,0x38,0x23,0x8a,0x70,0x50,0xd3,0x18,0xf1,0x18,0x76,0x31,0x20,0x95,0x62, + 0x67,0xc7,0x86,0xa4,0xd4,0xb0,0x14,0x2c,0x40,0x3f,0xea,0x93,0x43,0xf5,0xf4,0xa0, + 0x6d,0x85,0x7d,0x1b,0x67,0x61,0xce,0x96,0x64,0xaf,0x57,0xb8,0x46,0x55,0xe1,0x93, + 0x9c,0x10,0xf3,0xda,0x04,0xe9,0x90,0xf2,0x20,0x5e,0x18,0xc2,0xb9,0x12,0x20,0xca, + 0x1b,0x1a,0x60,0xbe,0x59,0xc5,0x4b,0x8e,0x87,0x8f,0x9f,0x9c,0x38,0xc5,0xa5,0x6d, + 0x1c,0x3d,0x69,0x15,0x9f,0x4c,0x81,0xd7,0xe0,0x50,0x7f,0x0e,0x85,0xbd,0xbd,0x07, + 0x0c,0xcd,0x6d,0x7b,0x8c,0x55,0xda,0xcc,0xd5,0xa6,0xda,0x69,0x10,0x01,0x34,0xe8, + 0xd5,0x2d,0x70,0x64,0x0d,0x40,0x58,0x8a,0x9c,0x8c,0xda,0x0d,0x11,0x38,0xf5,0x9c, + 0x13,0xc7,0x0e,0xfc,0xf6,0xeb,0x7f,0x3b,0x8d,0xa1,0x33,0x13,0x28,0x7c,0x32,0xbf, + 0xbe,0xd3,0xe7,0x5c,0x1e,0x43,0xf8,0x24,0x5c,0x53,0x29,0xf3,0xb0,0x1f,0x1a,0x23, + 0x5b,0x7b,0xf5,0x6c,0x80,0x41,0xc6,0xe2,0xf6,0xde,0x4d,0x8c,0x3c,0xa0,0xe1,0xf2, + 0xf6,0x30,0x87,0x43,0xda,0x22,0x5f,0x4e,0x6f,0x4d,0x30,0x5b,0xea,0xd6,0xa4,0x67, + 0xf4,0x10,0x93,0x4f,0xc4,0x8f,0xc0,0x10,0xe0,0xd0,0xd5,0xde,0x46,0x5f,0x37,0xaa, + 0xb2,0xb2,0x90,0xc5,0x96,0x0c,0xfc,0x76,0xce,0x24,0x87,0x59,0x0c,0x3a,0xc2,0xbb, + 0x70,0xf4,0xcd,0xdb,0x37,0x65,0xd7,0xed,0xd8,0x9b,0xab,0xba,0x9d,0x2a,0x61,0x08, + 0xcf,0xd1,0x2e,0xdf,0x62,0x09,0xa4,0xc3,0xa6,0xb5,0xc4,0x19,0x21,0x86,0x82,0x6f, + 0x81,0xec,0x0a,0xfa,0xf9,0x85,0x05,0xfb,0x98,0x9b,0x15,0x11,0x05,0x27,0xcd,0x7c, + 0xd3,0xfc,0xe2,0x1a,0x10,0x90,0xb3,0x03,0x5a,0xc0,0xbf,0x6f,0xcf,0x66,0x29,0xa1, + 0x73,0xaf,0x84,0xca,0xe0,0x2c,0xbe,0x7a,0xe4,0x1f,0x8a,0xee,0x81,0xe1,0xd9,0x74, + 0x4a,0xf9,0x24,0x59,0xac,0xc0,0x00,0xda,0x59,0x7b,0x2e,0x9e,0x52,0xd9,0x52,0x1a, + 0xa7,0xf1,0x8d,0xa8,0x98,0x0a,0x35,0xc7,0xbc,0xd5,0x95,0xf2,0x9a,0x54,0x80,0xaf, + 0x5d,0x59,0x73,0xa7,0x2c,0x1e,0xda,0xb7,0x61,0x65,0x61,0xad,0x47,0x56,0xcc,0x2a, + 0x4a,0x51,0x32,0x8e,0x89,0x6a,0x0a,0xfa,0xc8,0xb2,0x61,0x09,0x19,0xea,0x84,0x8e, + 0x65,0xe3,0xa6,0x90,0x32,0x5f,0x2e,0x06,0x56,0xb0,0xb5,0x71,0x13,0x71,0xb0,0x79, + 0xe2,0xb8,0xca,0x9a,0xf7,0xe7,0xe2,0xfb,0x33,0xc1,0xe5,0xda,0x26,0xd0,0xa8,0xf4, + 0xd5,0xd0,0x6d,0x04,0xcc,0x20,0x4a,0x1a,0x72,0x27,0xe2,0x37,0x37,0x47,0x8f,0x9f, + 0x3c,0x7b,0x6a,0xc5,0x14,0x2b,0xb7,0xe8,0x75,0x39,0x72,0xd1,0xb9,0x0e,0xc1,0x27, + 0xa9,0xe3,0xa3,0x2f,0x0e,0xa8,0x4c,0xdd,0xd4,0xac,0x61,0x1b,0x36,0xda,0xc7,0x58, + 0xb2,0xff,0xfc,0xf0,0xc5,0x91,0xb1,0x11,0xb7,0x07,0xef,0x34,0x17,0x93,0xec,0xf6, + 0xde,0x66,0xa9,0x2d,0xd0,0xb7,0x13,0xcb,0xcf,0x4d,0x7c,0xaf,0x2b,0x7a,0xf6,0xb6, + 0x63,0x2b,0x2b,0x41,0x02,0x1d,0x87,0x9f,0x52,0xb1,0x55,0x05,0xa2,0x0e,0x59,0x67, + 0xb2,0x44,0xfa,0x47,0x8a,0xbf,0x21,0xe9,0xdc,0xe0,0x9b,0x9a,0x01,0x4f,0x32,0x59, + 0x9a,0xe6,0x73,0x62,0xb8,0x47,0x0b,0xd8,0x32,0xd2,0xf5,0x00,0x08,0xed,0xec,0x9b, + 0x01,0x93,0x5b,0x19,0x4b,0x47,0xd5,0x5f,0x20,0xc5,0xe5,0xdf,0xc0,0xbc,0x90,0xa4, + 0x29,0x27,0xb4,0xf6,0x00,0x3f,0xfe,0x00,0xc1,0x0d,0xfb,0x80,0xf2,0xed,0x86,0xb0, + 0xae,0xf7,0xe2,0x20,0x6e,0x6a,0x90,0x45,0xcb,0xd0,0xe5,0x73,0xa0,0x9d,0xbe,0x33, + 0x80,0x9c,0xb6,0xa8,0xb2,0x95,0x48,0xf4,0xb6,0xdd,0xb0,0x3d,0x5b,0x56,0x03,0x32, + 0x74,0x61,0x2a,0xc6,0x26,0xe0,0x0d,0xe0,0xa1,0x29,0xcf,0x29,0xda,0x5a,0xb3,0xad, + 0x30,0xc3,0x2b,0xb2,0xd0,0xda,0x12,0x32,0x95,0xa6,0x5d,0x13,0x69,0x40,0xd8,0x11, + 0xfe,0x37,0x11,0x70,0x27,0x4b,0xaf,0x6c,0x8d,0x91,0x9e,0xb9,0x7a,0x7e,0xeb,0xc2, + 0xa4,0xd5,0x64,0xce,0x4e,0xcc,0x05,0x37,0x64,0x9d,0x72,0xe6,0x46,0x64,0x27,0xe4, + 0xd1,0xec,0x46,0x8d,0x25,0xe3,0xc3,0x69,0x4d,0x27,0x99,0xa7,0x49,0x7a,0x02,0x1b, + 0xa0,0xd3,0x18,0xd2,0x74,0x95,0xca,0x3f,0x37,0x53,0x27,0x3a,0x98,0x21,0x7c,0x6d, + 0x70,0xa4,0x1c,0x10,0x47,0xb5,0x0f,0x20,0x1c,0xdf,0xda,0x7e,0x5e,0x6f,0x65,0xf1, + 0x8e,0xd5,0x77,0x08,0x63,0x75,0xdb,0x7d,0x5a,0x24,0x2a,0x60,0x01,0x69,0x88,0xcf, + 0xa6,0x5d,0xef,0x94,0x96,0x9f,0x9b,0x9f,0x96,0xcd,0x34,0x6b,0x54,0x46,0x85,0xac, + 0x62,0x6b,0x53,0xf8,0x2d,0x65,0x93,0x78,0xac,0x2b,0x0a,0x94,0x0a,0x81,0x21,0x26, + 0xea,0x31,0xe1,0x95,0x09,0xf0,0xa9,0x88,0xd1,0xd0,0x2c,0x9f,0x0e,0x3f,0xfd,0xdc, + 0x14,0x11,0x41,0x3c,0x5d,0x9f,0x12,0x34,0x2f,0x9f,0x9a,0xd7,0xdc,0x9d,0xbe,0x7d, + 0x27,0x20,0x73,0x9b,0x99,0xe3,0xb5,0xf3,0xc9,0x2a,0xd0,0xdf,0x07,0xc3,0xe1,0xe7, + 0x01,0x6c,0xae,0xc5,0x51,0xf3,0x6d,0x5f,0x86,0xf9,0x24,0x32,0xae,0x29,0x55,0x78, + 0x9c,0x9a,0xaf,0x0d,0x06,0x18,0x68,0xc9,0x0c,0x89,0xe3,0x5a,0x06,0x9a,0xc4,0xc9, + 0x7d,0x17,0x3f,0xd2,0xb2,0xa9,0x70,0x70,0xc3,0x94,0x2d,0xc0,0xbe,0xf5,0xa2,0xcb, + 0x36,0x9d,0x20,0xd5,0xd7,0xdb,0x9b,0x63,0x1a,0x15,0xd6,0xdb,0x4d,0x79,0x52,0x74, + 0x7b,0x12,0x60,0xe1,0x5f,0x5b,0x41,0xef,0x30,0x0e,0xe2,0x70,0x8b,0x73,0x6b,0x11, + 0x39,0x71,0xda,0x00,0xf7,0xb7,0x5f,0xff,0x4b,0x37,0xba,0x00,0x94,0x3a,0xc1,0xa4, + 0xd3,0xe8,0x15,0x7c,0x4b,0x13,0xa7,0xd3,0xe5,0x8a,0x30,0xf7,0x71,0xeb,0x3a,0xd6, + 0x04,0xee,0x12,0x76,0xfe,0xfc,0xdf,0x3e,0x7e,0x64,0x4f,0xe1,0x23,0xc8,0x2f,0xe5, + 0x28,0xcd,0xd2,0x7a,0xd1,0x1b,0x4b,0xaa,0x67,0xf5,0x6c,0xae,0x23,0xa8,0xfc,0x63, + 0x56,0x55,0x8a,0x1e,0xf7,0x98,0xa0,0x5f,0x35,0x1b,0x38,0xda,0xe6,0xfa,0x08,0xf7, + 0xa9,0xa4,0x4a,0xd7,0x7e,0x5b,0x43,0x06,0x52,0xa7,0xfe,0x01,0x65,0x5c,0x4f,0x21, + 0xe9,0x23,0xf0,0x3e,0x43,0x05,0x5d,0x10,0x67,0xb7,0xd5,0x53,0xf3,0x24,0x26,0x10, + 0x2e,0xbd,0x20,0x31,0x70,0x3a,0xcf,0x48,0x6c,0x44,0x1d,0xc9,0xaa,0x4a,0x95,0x6e, + 0x33,0x62,0xf3,0x9e,0xc5,0x96,0x7b,0xcd,0x3e,0x50,0x7d,0xa4,0xf8,0x79,0x1c,0x78, + 0x3b,0x70,0xab,0x80,0xf2,0xdb,0x3f,0x7f,0xc5,0x7f,0x62,0xfb,0x61,0x8a,0xed,0xff, + 0xff,0xfa,0x8f,0x30,0x7b,0xd3,0x79,0x92,0x53,0x98,0x04,0x29,0xa3,0x52,0x50,0x2d, + 0xec,0xdb,0x1c,0xe8,0xe9,0xee,0xa5,0x5a,0x84,0xd0,0xbd,0x6c,0xa6,0x76,0x99,0x81, + 0xf4,0x5a,0x47,0xd1,0x33,0x20,0x51,0xf0,0x77,0x31,0x9a,0x20,0xc5,0xf6,0x8d,0x10, + 0x7c,0xbe,0x34,0xaf,0x84,0xcc,0x3d,0x3e,0xdd,0xb5,0x6a,0x3d,0x53,0x26,0x71,0x8d, + 0xb8,0xec,0x48,0xf5,0x45,0xae,0x98,0x98,0xcb,0x53,0xd9,0xa9,0x5f,0x98,0x32,0x38, + 0x55,0xb8,0x83,0x75,0x5d,0x35,0x01,0x4e,0x6e,0xe1,0x97,0x7e,0x31,0xab,0xd9,0xa7, + 0x21,0x21,0x72,0xc9,0x18,0x5c,0x52,0x39,0xbd,0xe8,0xb8,0xb9,0xab,0x61,0xf1,0xe9, + 0xf2,0x67,0x3f,0x1f,0x96,0x27,0x25,0x54,0x18,0x4e,0x2b,0xbc,0x34,0xfe,0x0a,0xea, + 0x7f,0xb5,0xb7,0x47,0x6f,0x86,0x70,0xce,0x2b,0x8a,0x98,0x0d,0xf2,0x0e,0xa2,0xf2, + 0xd3,0xaa,0x92,0x8b,0x20,0xd5,0xfc,0xd7,0xbd,0xf2,0x3c,0xde,0xf0,0xca,0xcf,0x79, + 0x43,0xb3,0x9e,0x5d,0x17,0x9a,0xc6,0x48,0xe4,0xfb,0x0e,0x47,0x8d,0xdb,0x2b,0x4f, + 0xae,0x82,0xa9,0x2c,0xd7,0x9f,0x1b,0x5c,0xb7,0xd7,0x34,0xd7,0xed,0xe6,0xd7,0x9d, + 0xcd,0x4f,0xfe,0x72,0xf1,0xd7,0x1f,0xe0,0xe5,0xa8,0x86,0x91,0x26,0x0b,0xcc,0x0f, + 0xaf,0x57,0x9e,0x35,0x2c,0x3e,0xcc,0x4a,0x78,0x65,0x1c,0x2e,0xf9,0xbd,0xe6,0xb3, + 0x88,0x59,0x4d,0xd7,0x1c,0x20,0xd4,0xa4,0xae,0xcb,0x90,0xa2,0x23,0x5b,0xac,0x81, + 0xc3,0x89,0x48,0x0d,0xf9,0xfd,0xe2,0x75,0xcd,0xcc,0xc8,0xe5,0x55,0x0a,0x25,0x2b, + 0x2a,0xb2,0x02,0x25,0x5b,0x61,0x9a,0x3f,0xd3,0x33,0x99,0x21,0x77,0x9f,0x82,0x39, + 0x54,0xe6,0x05,0x14,0x4e,0xae,0x9b,0x5b,0x54,0x69,0x2e,0x1c,0x07,0x42,0x5d,0xab, + 0xe8,0x55,0x31,0x9d,0x22,0x2f,0x76,0x1d,0x7a,0x60,0xe5,0x78,0x6b,0x8d,0x31,0x0a, + 0xd4,0x49,0x70,0xc1,0x0f,0xbe,0x11,0x61,0xb1,0xa2,0x5a,0xa9,0x36,0x61,0x5e,0x1b, + 0x19,0x72,0x1e,0x3e,0xa2,0xea,0x63,0x22,0x6d,0x22,0x4d,0xa8,0x12,0x4a,0x54,0x5c, + 0xe3,0x2c,0x91,0x76,0x21,0x0d,0xa7,0x5b,0x8a,0x8e,0x14,0x64,0x6a,0x2c,0xa3,0x05, + 0x29,0x91,0x5b,0xb7,0x97,0x54,0x72,0xd8,0xde,0x98,0xd1,0xeb,0xff,0x5a,0xd9,0x4b, + 0x33,0xd7,0x69,0x5e,0x60,0x21,0x87,0x2b,0x2e,0x87,0x89,0xcc,0xe8,0xbd,0xaa,0x10, + 0x32,0x60,0x49,0x1e,0xd6,0x03,0x49,0xd1,0xf5,0x69,0x0d,0xda,0xc3,0xe7,0x2b,0x38, + 0x3f,0xfb,0xba,0x0b,0x81,0x88,0xb1,0xbc,0xd2,0x86,0x1f,0x91,0xd6,0x1f,0x01,0x6c, + 0xe8,0xb4,0x8f,0x41,0x39,0x62,0xe5,0x97,0xa0,0x7d,0xf3,0xf4,0xb3,0x3f,0x68,0x9e, + 0x1c,0xf6,0xd9,0xc4,0xb6,0x48,0x51,0x81,0x3e,0x30,0xe6,0xe0,0xd5,0x24,0xcd,0x62, + 0x57,0x36,0xa0,0x55,0x06,0x01,0x70,0x3d,0x83,0xc6,0x05,0xb7,0x00,0xfa,0x83,0xcc, + 0xc7,0xca,0xed,0xfb,0x75,0x90,0xa9,0x7c,0x5c,0x4f,0x78,0x76,0x5d,0x2d,0x96,0x38, + 0x43,0x0b,0xf4,0x0e,0x9e,0xac,0x22,0xaa,0x0e,0xd1,0x27,0x5a,0xab,0x5b,0xdb,0x57, + 0x08,0x7e,0xaf,0x54,0x77,0x7b,0xfa,0xfe,0x66,0xeb,0xeb,0x1b,0x86,0x03,0x11,0x4b, + 0x28,0x78,0xa3,0xf3,0x77,0x2f,0xd2,0x30,0x44,0x14,0xb0,0x74,0xc7,0x6a,0xfb,0x49, + 0x5e,0xaa,0x2f,0x58,0xe2,0x5e,0x19,0x81,0xdb,0xdb,0xbb,0x43,0xda,0xee,0xec,0x0c, + 0xb8,0x28,0x45,0x30,0x3d,0xd6,0xae,0xe6,0xf3,0xa7,0x2f,0xcd,0xc4,0xee,0xe6,0x4b, + 0x22,0xfe,0x04,0xad,0xf3,0xc9,0xd2,0x86,0x60,0xac,0xf8,0x80,0x5f,0xfc,0x9c,0x6a, + 0x6b,0xc5,0x60,0xe1,0xf2,0xaa,0xd5,0xce,0x37,0x0d,0x35,0x03,0x88,0x3f,0x4c,0x63, + 0x5e,0xbf,0x4b,0x61,0xc8,0x72,0x45,0xe4,0x21,0x51,0x77,0xba,0xdb,0x5a,0x09,0x9c, + 0x0e,0x69,0x19,0x3d,0x22,0x84,0xa4,0x49,0xd3,0x20,0x62,0xb4,0x51,0xc7,0x03,0xaa, + 0x15,0x7b,0xcb,0xf7,0x6f,0x03,0x18,0x3e,0x04,0x12,0x45,0x63,0x5b,0x17,0xaa,0x86, + 0xac,0x29,0xba,0x87,0x98,0x36,0x1f,0x3d,0xad,0x2c,0x54,0x48,0x5b,0xf7,0x33,0x22, + 0xbb,0x82,0x43,0x06,0x3a,0xd4,0x6b,0x48,0xba,0x4b,0x29,0xe1,0x77,0x17,0x7f,0xbd, + 0x60,0xeb,0xe1,0x52,0xb0,0xce,0xe3,0x9c,0x62,0x02,0x45,0x1b,0xc7,0xfc,0x67,0xee, + 0xec,0xb3,0x3d,0x23,0x34,0x20,0xda,0xfe,0xa7,0x9f,0x1b,0x03,0x83,0x21,0x46,0xb2, + 0xc3,0xdf,0xad,0x6f,0xb5,0xbc,0xa5,0x0c,0xcc,0x63,0xc7,0x61,0x5d,0xcd,0xd4,0xa0, + 0x39,0x04,0xbf,0xe8,0xa4,0x90,0x1d,0x81,0x0a,0x90,0x5f,0x6d,0xd1,0x65,0x53,0xcd, + 0x2c,0x00,0xa3,0x82,0x32,0x48,0x20,0x9c,0xda,0x0a,0xbe,0xd5,0x01,0x5e,0xd2,0xc0, + 0x36,0xbd,0x6b,0xd3,0x60,0x2a,0xe9,0x54,0x97,0x63,0x11,0x4d,0x89,0x68,0x69,0x9e, + 0x14,0x4c,0x35,0x32,0x8d,0xc4,0x3b,0x8a,0xf2,0x01,0xa0,0xf3,0x8d,0x9e,0x7b,0x44, + 0x59,0xf3,0x80,0x7a,0xed,0xf7,0x79,0xa5,0x79,0x84,0xbf,0xb3,0xfe,0xc8,0xf4,0xc0, + 0xbc,0xb8,0x3d,0x30,0x9f,0xb4,0xfe,0x1f,0xad,0x8a,0x95,0x31,0xe3,0x3a,0x00,0x00, }; constexpr size_t STATUS_INDEX_HTML_GZ_LEN = sizeof(STATUS_INDEX_HTML_GZ); // diff --git a/firmware/patternflow/src/webserver/Parsing.cpp b/firmware/patternflow/src/webserver/Parsing.cpp index cdbdfb39..ae69b22e 100644 --- a/firmware/patternflow/src/webserver/Parsing.cpp +++ b/firmware/patternflow/src/webserver/Parsing.cpp @@ -92,6 +92,9 @@ static char* readBytesWithTimeout(WiFiClient& client, size_t maxLength, size_t& // body on timeout (which still aborts the request). readBytesWithTimeout() // and _uploadReadByte() already delay between checks and are left as they are. +// PATTERNFLOW FIX (Fix 5): the longest multipart boundary a request may name. +static const unsigned PF_MAX_BOUNDARY = 70; + // True once a byte is waiting. False when the stream timeout passes with // nothing arriving, or the client has disconnected. static bool waitForByte(WiFiClient& client) @@ -111,13 +114,20 @@ static bool waitForByte(WiFiClient& client) // readStringUntil('\n') pair it replaces: read up to the CR, then discard // through the LF. A line that times out part-way comes back short, and the // LF is only waited for once a CR has actually been seen. -static String readLine(WiFiClient& client) +// +// PATTERNFLOW FIX (Fix 4): `whole`, when asked for, says whether the line +// actually ended. An empty String means two different things here - a blank +// line, or nothing arriving at all - and a caller that loops until it sees a +// particular line has to know which, or it loops for ever on a peer that left. +static String readLine(WiFiClient& client, bool* whole = nullptr) { String line; + if (whole) *whole = false; while (waitForByte(client)) { int c = client.read(); if (c < 0) break; if (c == '\r') { + if (whole) *whole = true; while (waitForByte(client)) { c = client.read(); if (c < 0 || c == '\n') break; @@ -243,8 +253,22 @@ bool WebServer::_parseRequest(WiFiClient& client) { } } - if (!isForm && _currentHandler && _currentHandler->canRaw(_currentUri)){ + // PATTERNFLOW FIX (Fix 5): the raw path is for routes whose body callback + // was written for a raw body. canRaw() is true for ANY route that has a + // body callback and is not a GET, so a POST that is not multipart, sent to + // a multipart upload route, was handed to that route's upload callback as + // a raw body - and the callback's first line, server().upload(), is then + // a reference through a null pointer. `curl -X POST /api/patterns` (or + // /update), with no body at all, panicked the board. A route registered + // for POST with a body callback is a multipart route here; its plain + // POSTs go to the plain path below and reach the completion handler. + if (!isForm && _currentHandler && _currentHandler->canRaw(_currentUri) && + !_currentHandler->canUpload(_currentUri)){ log_v("Parse raw"); + // PATTERNFLOW FIX (Fix 5): a raw request has a query string too. Stock + // never parsed it on this path, so `PUT /update?size=N` found no size - + // and arg() answered with whatever the previous request had carried. + _parseArguments(searchStr); _currentRaw.reset(new HTTPRaw()); _currentRaw->status = RAW_START; _currentRaw->totalSize = 0; @@ -306,6 +330,17 @@ bool WebServer::_parseRequest(WiFiClient& client) { // it IS a form _parseArguments(searchStr); if (!_parseForm(client, boundaryStr, _clientContentLength)) { + // PATTERNFLOW FIX (Fix 4): a form given up on must not leave its + // finished fields behind. arg() and hasArg() look in _postArgs + // first, and nothing cleared it until the next multipart request - + // so a field from the abandoned form answered for the arguments of + // every plain request after it (an `index=3` picking the pattern + // for a later ?index=7, a `last=0` holding every later upload open). + if (_postArgs) { + delete[] _postArgs; + _postArgs = nullptr; + } + _postArgsLen = 0; return false; } } @@ -452,6 +487,12 @@ bool WebServer::_parseForm(WiFiClient& client, String boundary, uint32_t len){ ++retry; } while (line.length() == 0 && retry < 3); + // PATTERNFLOW FIX (Fix 5): a boundary is at most 70 characters (RFC 2046). + // Stock sized a stack array from whatever the Content-Type header said, on + // a task with an 8 KB stack: a 9,000-character boundary was a stack + // overflow and a reboot, from one request. + if (boundary.length() > PF_MAX_BOUNDARY) return false; + //start reading the form if (line == ("--"+boundary)){ if(_postArgs) delete[] _postArgs; @@ -464,7 +505,21 @@ bool WebServer::_parseForm(WiFiClient& client, String boundary, uint32_t len){ String argFilename; bool argIsFile = false; - line = readLine(client); + // PATTERNFLOW FIX (Fix 4): this loop had no way out but a well-formed + // part. A body that stopped after its first boundary - the sender + // closed, or went quiet - came back as an empty line every time, the + // test below failed, and control fell to the top again: with the peer + // gone waitForByte() returns at once, so it was a spin on pf-net with + // no delay in it, and the Core-0 watchdog rebooted the board 5 s later. + // One request, to any route. A line that never ended means the form is + // truncated and no later line is coming either. If a file part of this + // form was already delivered, its handler is told the upload was + // aborted - which is what the stock code does one line after a file + // ends when the peer has gone - because a handler that latched state + // at the file's start releases it only on completion or abort. + bool whole = false; + line = readLine(client, &whole); + if (!whole) return _currentUpload ? _parseFormUploadAborted() : false; if (line.length() > 19 && line.substring(0, 19).equalsIgnoreCase(F("Content-Disposition"))){ int nameStart = line.indexOf('='); if (nameStart != -1){ @@ -493,8 +548,14 @@ bool WebServer::_parseForm(WiFiClient& client, String boundary, uint32_t len){ log_v("PostArg Type: %s", argType.c_str()); if (!argIsFile){ while(1){ - line = readLine(client); + // PATTERNFLOW FIX (Fix 4): same shape, and worse - each pass + // also grew argValue by a byte. The boundary is tested first so + // a closing boundary sent without its CR/LF still ends the form + // as it did; anything else that did not end is a truncation. + bool whole = false; + line = readLine(client, &whole); if (line.startsWith("--"+boundary)) break; + if (!whole) return _currentUpload ? _parseFormUploadAborted() : false; if (argValue.length() > 0) argValue += "\n"; argValue += line; } @@ -525,7 +586,7 @@ bool WebServer::_parseForm(WiFiClient& client, String boundary, uint32_t len){ _currentUpload->status = UPLOAD_FILE_WRITE; int fastBoundaryLen = 4 /* \r\n-- */ + boundary.length() + 1 /* \0 */; - char fastBoundary[ fastBoundaryLen ]; + char fastBoundary[ 4 + PF_MAX_BOUNDARY + 1 ]; // PATTERNFLOW FIX (Fix 5): bounded above, no longer sized by the request snprintf(fastBoundary, fastBoundaryLen, "\r\n--%s", boundary.c_str()); int boundaryPtr = 0; while ( true ) { diff --git a/firmware/patternflow/src/webserver/VENDORED.md b/firmware/patternflow/src/webserver/VENDORED.md index b95b5ff7..0eb569e4 100644 --- a/firmware/patternflow/src/webserver/VENDORED.md +++ b/firmware/patternflow/src/webserver/VENDORED.md @@ -1,12 +1,96 @@ # Vendored: WebServer (arduino-esp32 core 2.0.17) Copied verbatim from the Arduino core's bundled library -(`libraries/WebServer/src`, core 2.0.17 / IDF 4.4.7), plus **three Patternflow +(`libraries/WebServer/src`, core 2.0.17 / IDF 4.4.7), plus **five Patternflow fixes**. Same arrangement as `src/hub75` and `src/pubsubclient`: every firmware include points at this copy (`#include "webserver/WebServer.h"`), so the Library Manager / core-bundled version is never compiled and its version does not matter. +## Fix 5 (2026-10-02): three requests that took the board down + +Found by `firmware/toolchain/check_parser.py`, which replays requests through +these files on a PC, and then reproduced on a panel: each of the first two was +one request and a `panic` reset. + +**A POST that is not multipart, to a multipart route.** `canRaw()` is true for +any route that has a body callback and is not a GET, whatever that callback was +written for. So `curl -X POST http://panel/api/patterns` - or `/update` - with +no body, or any body that is not a form, went down the raw path and called the +route's *upload* callback, whose first line is `server().upload()`: a reference +through a null `_currentUpload`. The raw path now also requires +`!canUpload()`. In this firmware a route registered for POST with a body +callback is a multipart route and one registered for PUT is a raw one; a plain +POST to the former goes down the plain path and reaches the completion +handler, which answers it. + +**A boundary of any length.** `_parseForm()` sized a stack array from the +boundary the `Content-Type` header named. `pf-net` has an 8 KB stack; a +9,000-character boundary overflowed it. A boundary is at most 70 characters +(RFC 2046): a longer one is refused before anything is read, and the array is +a fixed 75 bytes. (That declaration was also the one line MSVC could not +compile; the host check no longer has to rewrite it.) + +**A raw request has a query string too.** The raw path never called +`_parseArguments()`, so `PUT /update?size=N` found no `size`, and `arg()` +answered with whatever the previous request had carried. It is parsed now, +before the body callback is first called. + +Still stock, and pinned as KNOWN in the host check rather than fixed: +`_uploadReadByte()` waits without a deadline, so an uploader that vanishes +mid-file without closing holds the one connection until TCP keepalive notices. +It sleeps while it waits; it is not a watchdog case. + +Marked `PATTERNFLOW FIX (Fix 5)` at the five sites. + +## Fix 4 (2026-10-01): the multipart parser can stop + +`_parseForm()` has two `while(1)` loops whose only exit is a well-formed part: +the outer one reads lines until it finds a `Content-Disposition`, and the one +that collects a plain field's value reads lines until it finds the boundary. +`readLine()` returns an empty String both for a blank line and for nothing +arriving at all, so a body that stopped early - `--boundary`, then the sender +closed - came back as an empty line for ever. With the peer gone +`waitForByte()` returns at once, so that was a spin on `pf-net` with no delay +in it, and the Core-0 task watchdog (5 s, panic) rebooted the board. One +request, to any route: multipart parsing does not depend on the handler. With +the peer still connected but silent the same loops held the single connection +indefinitely, five seconds a pass, and the field-value loop grew its String by +a byte each time. + +Reproduced on hardware before the fix (reset reason `task_wdt` about five +seconds after the request) and not after. + +`readLine()` takes an optional `bool* whole`, set only when the line actually +ended in a CR. Both loops ask for it and return `false` on a line that did +not end: a form that stops mid-way is truncated and no later line is coming. +The field-value loop tests for the boundary first, so a closing boundary sent +without its CR/LF still ends the form as it did before. Every other caller +passes no pointer and sees what it saw. + +Giving a form up has two consequences the spin used to hide by rebooting, and +both are handled where the form is abandoned. `arg()` and `hasArg()` look in +`_postArgs` first and nothing cleared it until the next multipart request, so +a field from the abandoned form would have answered for the arguments of every +plain request after it (`index=3` choosing the pattern for a later +`?index=7`); the caller now clears it on a false return, which also closes the +stock path that already had this (a field, then a file cut mid-body). And a +form can be abandoned after one of its file parts was delivered in full: the +two exits then report the upload aborted, as the stock code does one line +after a file ends when the peer has gone, because a handler that latched +state at the file's start releases it only on completion or abort +(`/api/patterns` would otherwise have stayed "storage busy" with the panel +paused). + +One well-formed shape is now refused that was not: a body that stalls for the +whole stream timeout exactly on a line boundary at the top of the part loop +or inside a field value. Stock waited it out (and, inside a value, quietly +inserted an empty line). It needs a segment to end on that line break and a +five-second gap after it; a stall inside a file body or between the headers +and the body is handled as before. + +Marked `PATTERNFLOW FIX (Fix 4)` at the four sites. + ## Fix 3 (2026-09-07): cooperative network maintenance while reading The wait loops in `waitForByte()`, `readBytesWithTimeout()` and @@ -123,8 +207,9 @@ ever matters. ## Updating this copy Diff against the core's `libraries/WebServer/src` before replacing wholesale; -both fixes above must survive (grep `PATTERNFLOW FIX`, `readLine`, -`readBody`). If the project ever moves to core 3.x, this directory can be +the fixes above must survive (grep `PATTERNFLOW FIX`, `readLine`, +`readBody`, `whole`), and `python firmware/toolchain/check_parser.py` must +still pass - it fails on stock. If the project ever moves to core 3.x, this directory can be deleted and the includes pointed back at `` — but check first that its parser yields between bytes; the Core-0 watchdog does not care which version is spinning. diff --git a/firmware/toolchain/README.md b/firmware/toolchain/README.md index b1f8b88f..45d1bcb9 100644 --- a/firmware/toolchain/README.md +++ b/firmware/toolchain/README.md @@ -13,9 +13,10 @@ The repository-level scripts around the firmware: what builds a module or a pack - the preset twins — `check_presets.py`: `web/src/lib/presets/` and `../patternflow/presets/` stay in step, with the deliberate exceptions listed inside it; - versions — `check_versions.py`: the version the firmware reports, the editions, `AGENTS.md`, the flasher manifest and the shelf all agree; - images — `check_footprint.py` (each edition's size against its baseline); -- host-side unit tests of pure engine code — `check_math.py`, `check_blit.py`, `check_oe.py`, `check_send.py`, `check_thumbs.py`, `check_runtime.py`, `check_network.py`, `check_midi.py`, each compiling and running its twin in `tests/` (`*_test.cpp`) natively, with `--sanitize` for ASan/UBSan; +- host-side unit tests of pure engine code — `check_math.py`, `check_blit.py`, `check_oe.py`, `check_send.py`, `check_thumbs.py`, `check_runtime.py`, `check_crash.py`, `check_network.py`, `check_midi.py`, each compiling and running its twin in `tests/` (`*_test.cpp`) natively, with `--sanitize` for ASan/UBSan; +- the vendored web server — `check_parser.py`: the real `src/webserver/Parsing.cpp` against a scripted socket and a fake clock, failing on a wait that does not sleep or does not end. Its `KNOWN:` lines are what the parser does today that the test would otherwise fail, pinned so that fixing one turns the check red until the pin is removed; - `check_sources.py`, a fast pre-compile sanity pass. -`tests/` also holds `modules/_ctor_probe/`, a module that exists only to exercise the loader's `.init_array` path; build it with `build_module.py` and inspect the ELF as its header comment says. +`tests/` also holds `modules/_ctor_probe/`, a module that exists only to exercise the loader's `.init_array` path; build it with `build_module.py` and inspect the ELF as its header comment says. Beside it, `modules/_crash_probe/` writes through a null pointer from `draw()` when a knob moves: the bench case for the crash record in `/api/status`. And `modules/_hang_probe/` is a module whose `draw()` stops returning eight seconds after it is picked - the bench tool for "the console outlives a hung render loop" (`src/core_loop_sync.h`), which no well-behaved pattern can show. With K1 or K2 turned while it counts down the stop ends after thirty or twenty seconds, for the other half: what the loop does when it comes back to requests that gave up on it. Its header comment has the three commands. `module.ld` is the linker script every `.pfm` is linked with. diff --git a/firmware/toolchain/check_crash.py b/firmware/toolchain/check_crash.py new file mode 100644 index 00000000..0cb955fc --- /dev/null +++ b/firmware/toolchain/check_crash.py @@ -0,0 +1,53 @@ +"""Walk the crash record through the boots it has to tell apart. + +python firmware/toolchain/check_crash.py [--sanitize] + +src/core_crash.h decides, once per boot, what a reset left behind: a +breadcrumb in RAM that only some resets preserve, and a core dump in flash +that only some resets write. Getting that wrong does not crash anything - it +reports last week's backtrace as today's, or pins a death on a pattern that +was not running - and a panel cannot be made to take every reset on demand. +So the production header is compiled here against a scripted reset reason, +coredump partition and dump parser, and booted through reset x breadcrumb x +dump. +""" +import argparse +from pathlib import Path +import subprocess +import tempfile + +from check_module_elf import ROOT, compiler_environment + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--sanitize', action='store_true') + args = parser.parse_args() + compiler, env = compiler_environment() + with tempfile.TemporaryDirectory(prefix='patternflow-crash-') as directory: + work = Path(directory) + # core_module_elf.h is the real one: it is where ELF_MAGIC comes from, + # and it has no device dependencies. + for name in ('core_crash.h', 'core_module_elf.h'): + (work / name).write_bytes((ROOT / 'firmware/patternflow/src' / name).read_bytes()) + for name in ('Arduino.h', 'esp_attr.h', 'esp_core_dump.h', 'esp_partition.h', + 'esp_system.h', 'core_mem.h'): + (work / name).write_text('// Hardware supplied by crash_test.cpp\n') + source = ROOT / 'firmware/toolchain/tests/crash_test.cpp' + exe = work / ('crash_test.exe' if Path(compiler).suffix.lower() == '.exe' else 'crash_test') + if Path(compiler).stem.lower() == 'cl': + if args.sanitize: + raise SystemExit('--sanitize requires GCC or Clang') + command = [compiler, '/nologo', '/std:c++20', '/EHsc', '/O2', '/W4', '/WX', '/utf-8', + f'/I{work}', str(source), f'/Fe:{exe}'] + else: + command = [compiler, '-std=c++20', '-O2', '-Wall', '-Wextra', '-Werror', + f'-I{work}', str(source), '-o', str(exe)] + if args.sanitize: + command += ['-fsanitize=address,undefined', '-fno-omit-frame-pointer'] + subprocess.run(command, cwd=work, env=env, check=True) + subprocess.run([str(exe)], cwd=work, env=env, check=True) + + +if __name__ == '__main__': + main() diff --git a/firmware/toolchain/check_footprint.py b/firmware/toolchain/check_footprint.py index 1757dcab..503d34ac 100644 --- a/firmware/toolchain/check_footprint.py +++ b/firmware/toolchain/check_footprint.py @@ -6,13 +6,16 @@ the .bin in ~/pf-build-editions. WHY. CI builds five compositions and then prints `ls -l *.bin`, which is flash — and -flash is not the scarce resource on this board. Internal DRAM is. A loadable -pattern's code has to fit in one contiguous internal executable block, and that -block is the residual of the internal pool after .data, .bss and IRAM have taken -theirs. So a three-kilobyte static regression — one more inline buffer, one -feature's tables, an unrolled loop in an IRAM_ATTR function — eats a large share -of the budget that decides whether somebody's pattern loads, and nothing in CI -would notice. There are 24 KB of canvas in .bss alone before anything else. +flash is not the scarce resource on this board. Internal DRAM is. It is what the +console, lwIP and every feature allocate from, and what is left for them is the +residual of the internal pool after .data, .bss and IRAM have taken theirs. So a +three-kilobyte static regression — one more inline buffer, one feature's tables, +an unrolled loop in an IRAM_ATTR function — comes straight out of the margin +between a console that answers and one that stops, and nothing in CI would +notice. There are 24 KB of canvas in .bss alone before anything else. (Until +2026-10 the same residual also decided whether a pattern loaded at all: a +module's code had to fit in it. Code runs from PSRAM now, and falls back to +this pool only when PSRAM cannot take it.) WHAT IT MEASURES. Not a sum of section names. `.dram0.heap_start` is a zero-length marker the linker places at the first byte of DRAM the heap may use, so its @@ -54,12 +57,18 @@ # Set 2026-09-10, commit adding this file, PlatformIO espressif32@7.0.1 -> # Arduino core 2.0.17 -> xtensa-esp32s3-elf-gcc 8.4.0 at -Os. A toolchain bump # moves every row; re-pin it in the same commit as the bump. +# +# Re-pinned 2026-10-02: IRAM -996 in every edition (the blit kernel became three +# short passes and its generic loop left IRAM), static DRAM -136..-152. +# Again the same day: static DRAM +304 in every edition - the crash record's 296 B +# (224 of them the SDK's own strings behind esp_core_dump_get_summary, 64 the +# breadcrumb in .noinit) and 8 B of loop-sync state. PINS = { - "default": (141424, 72467), - "audio": (160960, 72947), - "performance": (157848, 72467), - "clock": (141704, 72467), - "midi": (154000, 71667), + "default": (141576, 71471), + "audio": (161104, 71951), + "performance": (158080, 71471), + "clock": (141856, 71471), + "midi": (154168, 70671), } # Enough that an intentional, well-understood adjustment does not fire the check diff --git a/firmware/toolchain/check_parser.py b/firmware/toolchain/check_parser.py new file mode 100644 index 00000000..91b5d6d3 --- /dev/null +++ b/firmware/toolchain/check_parser.py @@ -0,0 +1,75 @@ +"""Replay truncated, stalled and trickled HTTP requests through the vendored +request parser, and fail on a wait that does not sleep or does not end. + +python firmware/toolchain/check_parser.py [--sanitize] + +The real src/webserver/Parsing.cpp and WebServer.cpp are compiled against a +scripted socket and a fake clock (tests/parser_test.cpp says what the two +properties are and why they are the ones that reboot a board). No board, no +PlatformIO. +""" +import argparse +from pathlib import Path +import shutil +import subprocess +import tempfile + +from check_module_elf import ROOT, compiler_environment + +VENDORED = ROOT / 'firmware/patternflow/src/webserver' +# Everything the server includes that is not in its own directory. The test +# supplies what they declare; these only have to exist. +SUPPLIED = ('Arduino.h', 'esp32-hal-log.h', 'WiFi.h', 'WiFiServer.h', 'WiFiClient.h', 'WString.h', + 'pgmspace.h', 'FS.h', 'MD5Builder.h', 'esp_random.h', 'http_parser.h', 'libb64/cencode.h', + 'core_net_maintenance.h') +# Every compiler builds the vendored files byte for byte. (Stock _parseForm +# declared a variable-length array, which cl does not have and which this +# script once had to rewrite; Fix 5 bounded it, for the board's sake.) +# The subject of this test is code that does not return. A replay that neither +# looks at the socket nor sleeps is outside what the fakes can throw out of, +# and must not become a CI job that runs until the runner kills it. +RUN_SECONDS = 120 + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--sanitize', action='store_true') + args = parser.parse_args() + compiler, env = compiler_environment() + msvc = Path(compiler).stem.lower() == 'cl' + with tempfile.TemporaryDirectory(prefix='patternflow-parser-') as directory: + work = Path(directory) + # Laid out like the sketch, so Parsing.cpp's "../core_net_maintenance.h" resolves. + src = work / 'src' + shutil.copytree(VENDORED, src / 'webserver') + for name in SUPPLIED: + (src / name).parent.mkdir(parents=True, exist_ok=True) + (src / name).write_text('// Supplied by parser_test.cpp\n') + source = ROOT / 'firmware/toolchain/tests/parser_test.cpp' + exe = work / ('parser_test.exe' if Path(compiler).suffix.lower() == '.exe' else 'parser_test') + # C++17, not the C++20 the other checks use: the server is C++11 code + # and C++20's reversed comparison candidates make its String == String + # expressions ambiguous. Plain char is unsigned, as it is on Xtensa. + # The vendored directory goes on the system include path (the test + # turns warnings off around it for MSVC): it is not ours to make + # -Wextra clean, and the test itself still is. + if msvc: + if args.sanitize: + raise SystemExit('--sanitize requires GCC or Clang') + command = [compiler, '/nologo', '/std:c++17', '/EHsc', '/O2', '/W4', '/WX', '/utf-8', '/J', + f'/I{src}', str(source), f'/Fe:{exe}'] + else: + command = [compiler, '-std=c++17', '-O2', '-Wall', '-Wextra', '-Werror', '-funsigned-char', + '-isystem', str(src), str(source), '-o', str(exe)] + if args.sanitize: + command += ['-fsanitize=address,undefined', '-fno-omit-frame-pointer'] + subprocess.run(command, cwd=work, env=env, check=True) + try: + subprocess.run([str(exe)], cwd=work, env=env, check=True, timeout=RUN_SECONDS) + except subprocess.TimeoutExpired: + raise SystemExit(f'parser_test did not finish in {RUN_SECONDS} s: a loop in the parser neither ' + 'looks at the socket nor sleeps, so nothing could stop it.') + + +if __name__ == '__main__': + main() diff --git a/firmware/toolchain/check_runtime.py b/firmware/toolchain/check_runtime.py index 99b1c91d..4de32649 100644 --- a/firmware/toolchain/check_runtime.py +++ b/firmware/toolchain/check_runtime.py @@ -26,6 +26,9 @@ def main(): start = registry.index('inline uint32_t activatedAtMs = 0;') end = registry.index('// \u2500\u2500 Naming a pattern', start) (work / 'registry_lifecycle.h').write_text(registry[start:end], encoding='utf-8') + start = registry.index('inline bool jsonStringValue(') + end = registry.index('// Deliberately a substring scan', start) + (work / 'sidecar_name.h').write_text(registry[start:end], encoding='utf-8') manager = (ROOT / 'firmware/patternflow/src/core_patterns_http.h').read_text(encoding='utf-8') start = manager.index('inline char restorePath[') end = manager.index('// The rescan-and-reload', start) diff --git a/firmware/toolchain/tests/blit_test.cpp b/firmware/toolchain/tests/blit_test.cpp index 5def0480..b3a42ba9 100644 --- a/firmware/toolchain/tests/blit_test.cpp +++ b/firmware/toolchain/tests/blit_test.cpp @@ -68,7 +68,9 @@ int main() { std::cout << "depths 11 and 12 refused, buffer untouched" << std::endl; std::mt19937 rng(20260906); uint64_t checked = 0; - for (int width : {32, 64, 128, 256, 127}) { + // Even widths that are not a multiple of the 32-column chunk the passes walk + // (2, 34, 98, 130) run a short last chunk; 127 takes the generic loop. + for (int width : {32, 64, 128, 256, 2, 34, 98, 130, 127}) { if constexpr (PF_TEST_SWAP) { if (width & 1) continue; // FIFO swaps require even rows } @@ -79,7 +81,12 @@ int main() { for (int depth = 2; depth <= 10; ++depth) { for (int sat : {0, 128, 256, 320, 512}) { - for (int trial = 0; trial < 5; ++trial) { + // Trials 0-3: identity LUTs (the raw pass). 4: all three random. 5-7: + // one channel random, the other two identity. 8-10: one entry of one + // channel off by one, at 0, 128 and 255. The last six exist for the + // decision "are all three tables the identity": a version that asks + // one table, or stops short of 256 entries, passes 0-4. + for (int trial = 0; trial < 11; ++trial) { MatrixPanel_I2S_DMA panel(static_cast(width), 32, static_cast(depth)); for (auto& v : panel.data) v = static_cast(rng()); auto expected = panel.data; @@ -88,7 +95,8 @@ int main() { uint8_t lut[3][256]; for (int c = 0; c < 3; ++c) for (int v = 0; v < 256; ++v) - lut[c][v] = trial == 4 ? static_cast(rng()) : static_cast(v); + lut[c][v] = (trial == 4 || trial - 5 == c) ? static_cast(rng()) : static_cast(v); + if (trial >= 8) lut[trial - 8][(trial - 8) * 128 - (trial == 10)] ^= 1; uint64_t onTime = 0; for (int y = 0; y < 64; ++y) { for (int x = 0; x < width; ++x) { diff --git a/firmware/toolchain/tests/crash_test.cpp b/firmware/toolchain/tests/crash_test.cpp new file mode 100644 index 00000000..0a35dccc --- /dev/null +++ b/firmware/toolchain/tests/crash_test.cpp @@ -0,0 +1,594 @@ +// The crash record's boot logic (src/core_crash.h) with the chip replaced at +// its boundary: a reset reason, a coredump partition and the SDK's dump parser +// that the test scripts. Memory that survives a reset is the test not touching +// PFCrash::trail between two boots; a power-on is the test filling it with +// noise. +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// ── The SDK, as far as core_crash.h reaches into it ───────────────────────── +#define __NOINIT_ATTR +using esp_err_t = int; +constexpr esp_err_t ESP_OK = 0, ESP_FAIL = -1; +enum esp_reset_reason_t { + ESP_RST_UNKNOWN, ESP_RST_POWERON, ESP_RST_EXT, ESP_RST_SW, ESP_RST_PANIC, + ESP_RST_INT_WDT, ESP_RST_TASK_WDT, ESP_RST_WDT, ESP_RST_DEEPSLEEP, + ESP_RST_BROWNOUT, ESP_RST_SDIO, +}; +static esp_reset_reason_t resetReason = ESP_RST_POWERON; +static esp_reset_reason_t esp_reset_reason() { return resetReason; } + +// The coredump partition: 64 KB in partitions/app3M_fat9M_16MB.csv. +constexpr uint32_t PARTITION_BYTES = 0x10000; +constexpr int ESP_PARTITION_TYPE_DATA = 1, ESP_PARTITION_SUBTYPE_DATA_COREDUMP = 3; +struct esp_partition_t { uint32_t size; }; +static const esp_partition_t partition{PARTITION_BYTES}; +static std::vector flash(PARTITION_BYTES, 0xFF); +static bool partitionMissing = false, readFails = false; +static const esp_partition_t* esp_partition_find_first(int type, int subtype, const char* label) { + assert(type == ESP_PARTITION_TYPE_DATA && subtype == ESP_PARTITION_SUBTYPE_DATA_COREDUMP && !label); + return partitionMissing ? nullptr : &partition; +} +static esp_err_t esp_partition_read(const esp_partition_t* part, size_t offset, void* out, size_t bytes) { + assert(part == &partition); + // The size word of a dump is whatever flash holds. Nothing in core_crash.h + // may turn it into a read past the partition, whatever it says. + assert(offset <= PARTITION_BYTES && bytes <= PARTITION_BYTES - offset); + if (readFails) return ESP_FAIL; + memcpy(out, flash.data() + offset, bytes); + return ESP_OK; +} + +// esp_core_dump.h, IDF 4.4.7, Xtensa. +constexpr int APP_ELF_SHA256_SZ = 17; +struct esp_core_dump_bt_info_t { uint32_t bt[16]; uint32_t depth; bool corrupted; }; +struct esp_core_dump_summary_extra_info_t { + uint32_t exc_cause, exc_vaddr, exc_a[16], epcx[6], epcx_reg_bits; +}; +struct esp_core_dump_summary_t { + uint32_t exc_tcb; + char exc_task[16]; + uint32_t exc_pc; + esp_core_dump_bt_info_t exc_bt_info; + uint32_t core_dump_version; + uint8_t app_elf_sha256[APP_ELF_SHA256_SZ]; + esp_core_dump_summary_extra_info_t ex_info; +}; +static esp_err_t esp_core_dump_image_check(); +static esp_err_t esp_core_dump_get_summary(esp_core_dump_summary_t* summary); +static esp_err_t esp_core_dump_image_erase(); + +static uint32_t clockMs = 0; +static uint32_t millis() { return clockMs += 7; } + +struct Logger { + std::string text; + void print(const char* s) { text += s; } + void println(const char* s = "") { text += s; text += '\n'; } +#if defined(__GNUC__) + __attribute__((format(printf, 2, 3))) // the production format strings, checked here too +#endif + void printf(const char* format, ...) { + char line[512]; + va_list args; + va_start(args, format); + vsnprintf(line, sizeof(line), format, args); + va_end(args); + text += line; + } + bool said(const char* s) const { return text.find(s) != std::string::npos; } +} Serial; + +static bool allocFails = false; +namespace PFMem { +inline void* alloc(size_t bytes) { return allocFails ? nullptr : calloc(1, bytes); } +} + +#include "core_crash.h" + +// ── The dump parser, scripted ─────────────────────────────────────────────── +static unsigned imageChecks = 0, parses = 0, erases = 0; +static bool crcHolds = true, parserFails = false; +static esp_core_dump_summary_t inFlash; // what the dump says, when it parses +static PFCrash::Trail duringParse; // the breadcrumb as a panic in the parser would leave it + +static esp_err_t esp_core_dump_image_check() { + ++imageChecks; + duringParse = PFCrash::trail; + return crcHolds ? ESP_OK : ESP_FAIL; +} +static esp_err_t esp_core_dump_get_summary(esp_core_dump_summary_t* summary) { + ++parses; + duringParse = PFCrash::trail; + if (parserFails) return ESP_FAIL; + *summary = inFlash; + return ESP_OK; +} +static esp_err_t esp_core_dump_image_erase() { + ++erases; + std::fill(flash.begin(), flash.end(), uint8_t{0xFF}); + return ESP_OK; +} + +// What the SDK leaves in the partition: its header with the image's total +// size in the first word, the ELF 20 bytes in, and a CRC in the last word. +// `checkWord` stands for the CRC - the one word two different dumps do not +// share. +static void writeDumpRaw(uint32_t bytes, uint32_t elfMagic, uint32_t checkWord) { + std::fill(flash.begin(), flash.end(), uint8_t{0xFF}); + const uint32_t head[6] = {bytes, 0x0102, 3, 164, 0, elfMagic}; + memcpy(flash.data(), head, sizeof(head)); + if (bytes >= 4 && bytes <= PARTITION_BYTES) memcpy(flash.data() + bytes - 4, &checkWord, 4); +} +static void writeDump(uint32_t checkWord) { + writeDumpRaw(23108, PFModuleLoader::ELF_MAGIC, checkWord); +} +static void eraseFlash() { std::fill(flash.begin(), flash.end(), uint8_t{0xFF}); } + +static unsigned boots = 0; + +// A reset: everything in ordinary RAM is lost, PFCrash::trail is whatever the +// test left in it. +static void boot(esp_reset_reason_t why) { + free(PFCrash::record); + PFCrash::record = nullptr; + PFCrash::owner = nullptr; + Serial.text.clear(); + imageChecks = parses = 0; + resetReason = why; + ++boots; + PFCrash::begin(); +} + +// The plug: .noinit comes up as whatever the RAM settled to. +static void powerOn() { + memset(&PFCrash::trail, 0xA5, sizeof(PFCrash::trail)); + boot(ESP_RST_POWERON); +} + +constexpr const char* PROBE = "/patterns/_crash_probe.pfm"; +// Internal executable RAM, and PSRAM as the CPU fetches from it: the heap's +// 0x3dc81000 plus the 0x06000000 between the data bus and the instruction bus +// (core_module_loader.h, execAddress). +constexpr uint32_t INTERNAL_CODE = 0x40381c40; +constexpr uint32_t PSRAM_DATA = 0x3dc81000; +constexpr uint32_t PSRAM_CODE = 0x43c81000; +constexpr uint32_t CODE_BYTES = 0x124; + +// The calls the loader makes for a module that loads and then runs, in its +// order, stopping inside draw(). +static void liveIntoDraw(uint32_t codeBase) { + PFCrash::running(PROBE, PROBE); + PFCrash::enter(PFCrash::LOADING); + PFCrash::code(codeBase, CODE_BYTES); + PFCrash::enter(PFCrash::CONSTRUCTORS); + PFCrash::enter(PFCrash::LOADING); + PFCrash::enter(PFCrash::SETUP); + PFCrash::enter(PFCrash::IDLE); + PFCrash::enter(PFCrash::UPDATE); + PFCrash::enter(PFCrash::IDLE); + PFCrash::enter(PFCrash::DRAW); +} + +// A store through null from fall(), called from draw(), in a module at +// `codeBase`: two frames inside the module, the rest in the firmware. +static void dumpOfProbe(uint32_t codeBase) { + inFlash = {}; + snprintf(inFlash.exc_task, sizeof(inFlash.exc_task), "loopTask"); + inFlash.exc_pc = codeBase + 0x4e; + inFlash.exc_bt_info.bt[0] = codeBase + 0x4e; + inFlash.exc_bt_info.bt[1] = codeBase + 0x9c; + inFlash.exc_bt_info.bt[2] = 0x4200f1a3; + inFlash.exc_bt_info.bt[3] = 0x42010b52; + inFlash.exc_bt_info.depth = 4; + inFlash.ex_info.exc_cause = 29; + inFlash.ex_info.exc_vaddr = 0; + memcpy(inFlash.app_elf_sha256, "3f9a0c1e5d7b2a40", 17); +} + +// ── A power-on, with and without something in flash ───────────────────────── +static void testPowerOn() { + eraseFlash(); + powerOn(); + assert(!PFCrash::record && parses == 0 && imageChecks == 0); + assert(PFCrash::trail.magic == PFCrash::TRAIL_MAGIC); + assert(PFCrash::trail.check == PFCrash::seal()); + assert(PFCrash::trail.phase == PFCrash::IDLE && PFCrash::trail.dumpSeen == 0); + assert(PFCrash::trail.slug[0] == '\0' && PFCrash::trail.codeSize == 0); + assert(Serial.text.empty()); // a healthy boot prints nothing + + // A dump from some earlier life - this firmware's, or any image before it: + // the SDK has written them all along. It is reported, as what it is, on + // every boot until somebody clears it. + for (int again = 0; again < 2; ++again) { + writeDump(0xAAAA0001); + dumpOfProbe(PSRAM_CODE); + powerOn(); + const PFCrash::Record* r = PFCrash::record; + assert(r && !r->trailed && r->dumped && !r->dumpFromThisReset); + assert(r->cause == 29 && r->vaddr == 0 && r->pc == PSRAM_CODE + 0x4e); + assert(r->depth == 4 && r->bytes == 23108 && !r->corrupted); + assert(strcmp(r->task, "loopTask") == 0 && strcmp(r->build, "3f9a0c1e5d7b2a40") == 0); + assert(PFCrash::trail.dumpSeen == 0xAAAA0001); + assert(Serial.said("core dump (from an earlier reset): task loopTask, cause 29")); + // Nothing recorded where that module had been, so its frames stay raw. + assert(Serial.said("backtrace: 0x43c8104e 0x43c8109c 0x4200f1a3 0x42010b52\n")); + uint32_t offset; + assert(!PFCrash::inModule(*r, PSRAM_CODE + 0x4e, offset)); + } +} + +// ── The case the record exists for ────────────────────────────────────────── +static void testPanicInModule(uint32_t codeBase, const char* rawFrames) { + writeDump(0xAAAA0001); + powerOn(); // this boot saw the old dump ... + liveIntoDraw(codeBase); + assert(PFCrash::isRunning(PROBE)); + writeDump(0xBBBB0002); // ... and the panic replaced it + dumpOfProbe(codeBase); + boot(ESP_RST_PANIC); + + const PFCrash::Record* r = PFCrash::record; + assert(r && r->trailed && r->dumped && r->dumpFromThisReset); + assert(r->phase == PFCrash::DRAW && strcmp(r->slug, "_crash_probe") == 0); + assert(r->codeBase == codeBase && r->codeSize == CODE_BYTES); + assert(Serial.said("the reset came during draw of \"_crash_probe\", module code at 0x")); + assert(Serial.said("[CRASH] core dump: task loopTask")); + assert(Serial.said("backtrace: _crash_probe+0x4e _crash_probe+0x9c 0x4200f1a3 0x42010b52\n")); + uint32_t offset = 0; + assert(PFCrash::inModule(*r, r->pc, offset) && offset == 0x4e); + assert(!PFCrash::inModule(*r, 0x4200f1a3, offset)); + // This boot's own breadcrumb names nothing until a pattern is loaded, and + // remembers the dump it has now seen. + assert(PFCrash::trail.slug[0] == '\0' && PFCrash::trail.phase == PFCrash::IDLE); + assert(PFCrash::trail.codeSize == 0 && PFCrash::trail.dumpSeen == 0xBBBB0002); + assert(!PFCrash::isRunning(PROBE)); + + // The same death when the SDK could not write its dump (too large for the + // partition: refused before the old one is erased). The breadcrumb is + // today's and the backtrace is not, and an old address that happens to fall + // in today's range must not be dressed up as an offset. + liveIntoDraw(codeBase); + boot(ESP_RST_PANIC); + r = PFCrash::record; + assert(r && r->trailed && r->dumped && !r->dumpFromThisReset); + assert(r->codeBase == codeBase && !PFCrash::inModule(*r, r->pc, offset)); + assert(Serial.said("core dump (from an earlier reset)")); + assert(Serial.said(rawFrames)); +} + +// ── Reset reason x breadcrumb x dump ──────────────────────────────────────── +enum TrailState { INTACT, NOISE, BROKEN_SEAL, BAD_MAGIC, BAD_PHASE, TRAIL_STATES }; +enum DumpState { + NO_DUMP, // none before, none after + OLD_DUMP, // the one the last boot saw, untouched + NEW_DUMP, // the reset replaced the one the last boot saw + FIRST_DUMP, // the reset wrote one where there was none + TORN_DUMP, // a new one whose CRC does not hold: power lost mid-write + DUMP_STATES +}; + +static void testMatrix() { + const esp_reset_reason_t reasons[] = { + ESP_RST_UNKNOWN, ESP_RST_POWERON, ESP_RST_EXT, ESP_RST_SW, ESP_RST_PANIC, + ESP_RST_INT_WDT, ESP_RST_TASK_WDT, ESP_RST_WDT, ESP_RST_DEEPSLEEP, + ESP_RST_BROWNOUT, ESP_RST_SDIO, + }; + for (esp_reset_reason_t why : reasons) { + const bool died = why == ESP_RST_PANIC || why == ESP_RST_INT_WDT || + why == ESP_RST_TASK_WDT || why == ESP_RST_WDT; + for (int trailState = 0; trailState < TRAIL_STATES; ++trailState) { + for (int dumpState = 0; dumpState < DUMP_STATES; ++dumpState) { + // The life before the reset. + crcHolds = true; + if (dumpState == NO_DUMP || dumpState == FIRST_DUMP) eraseFlash(); + else writeDump(0xAAAA0001); + dumpOfProbe(PSRAM_CODE); + powerOn(); + liveIntoDraw(PSRAM_CODE); + + // What the reset left in flash ... + if (dumpState == NEW_DUMP || dumpState == FIRST_DUMP || dumpState == TORN_DUMP) { + writeDump(0xBBBB0002); + } + crcHolds = dumpState != TORN_DUMP; + // ... and in RAM. + switch (trailState) { + case NOISE: memset(&PFCrash::trail, 0x5A, sizeof(PFCrash::trail)); break; + case BROKEN_SEAL: PFCrash::trail.slug[3] ^= 0x20; break; + case BAD_MAGIC: PFCrash::trail.magic ^= 1; break; + case BAD_PHASE: PFCrash::trail.phase = PFCrash::PHASE_COUNT; break; + default: break; + } + boot(why); + + const bool trailed = died && trailState == INTACT; + const bool dumped = dumpState == OLD_DUMP || dumpState == NEW_DUMP || + dumpState == FIRST_DUMP; + const bool fresh = trailed && (dumpState == NEW_DUMP || dumpState == FIRST_DUMP); + const PFCrash::Record* r = PFCrash::record; + assert((r != nullptr) == (trailed || dumped)); + if (r) { + assert(r->trailed == trailed && r->dumped == dumped); + assert(r->dumpFromThisReset == fresh); + if (trailed) { + assert(r->phase == PFCrash::DRAW && strcmp(r->slug, "_crash_probe") == 0); + assert(r->codeBase == PSRAM_CODE && r->codeSize == CODE_BYTES); + } else { + assert(r->slug[0] == '\0' && r->codeSize == 0); + } + uint32_t offset; + assert(PFCrash::inModule(*r, PSRAM_CODE + 0x4e, offset) == fresh); + } + // Whatever it found, this boot's breadcrumb is one the next boot will + // accept, and it knows which dump is in flash now. + assert(PFCrash::trail.magic == PFCrash::TRAIL_MAGIC); + assert(PFCrash::trail.check == PFCrash::seal()); + assert(PFCrash::trail.phase == PFCrash::IDLE); + const uint32_t inFlashNow = dumpState == NO_DUMP ? 0u + : dumpState == OLD_DUMP ? 0xAAAA0001u : 0xBBBB0002u; + assert(PFCrash::trail.dumpSeen == inFlashNow); + } + } + } + crcHolds = true; +} + +// ── A dump that kills the parser must not keep the board from starting ────── +static void testParserChoke() { + writeDump(0xAAAA0001); + dumpOfProbe(PSRAM_CODE); + powerOn(); + assert(parses == 1 && imageChecks == 1); + // The parser ran between two marks, on a breadcrumb that was already + // sealed: had it panicked there, this is what the next boot would find. + assert(duringParse.phase == PFCrash::READING_DUMP); + assert(duringParse.magic == PFCrash::TRAIL_MAGIC); + PFCrash::trail = duringParse; + boot(ESP_RST_PANIC); + assert(parses == 0 && imageChecks == 0); + assert(Serial.said("died reading the core dump")); + const PFCrash::Record* r = PFCrash::record; + assert(r && r->trailed && r->phase == PFCrash::READING_DUMP && !r->dumped); + assert(strcmp(PFCrash::phaseName(r->phase), "dump-read") == 0); + // It still noted which dump is there, so a later one is told apart from it. + assert(PFCrash::trail.dumpSeen == 0xAAAA0001); + + // Only a death skips the parse. A reboot from the same mark for any other + // reason is not evidence against the dump. + PFCrash::trail = duringParse; + boot(ESP_RST_SW); + assert(parses == 1 && PFCrash::record && PFCrash::record->dumped); + + // The parser refusing a dump is an answer, not a death. + parserFails = true; + powerOn(); + assert(parses == 1 && !PFCrash::record); + parserFails = false; +} + +// ── What is in flash is not trusted to be a dump this parser can read ─────── +static void testDumpGuards() { + dumpOfProbe(PSRAM_CODE); + struct Case { uint32_t bytes; uint32_t magic; }; + const Case refused[] = { + {0xFFFFFFFFu, 0xFFFFFFFFu}, // erased + {PARTITION_BYTES + 4, PFModuleLoader::ELF_MAGIC}, // a size the partition cannot hold + {0x7FFFFFFFu, PFModuleLoader::ELF_MAGIC}, + {24, PFModuleLoader::ELF_MAGIC}, // no room for the header and a CRC + {0, PFModuleLoader::ELF_MAGIC}, + {23108, 0x0102}, // another SDK generation's header: no ELF 20 bytes in + }; + for (const Case& c : refused) { + writeDumpRaw(c.bytes, c.magic, 0xCCCC0003); + powerOn(); + assert(parses == 0 && imageChecks == 0 && !PFCrash::record); + assert(PFCrash::trail.dumpSeen == 0); + } + // The largest size that is still inside the partition is read to its end + // and no further (esp_partition_read above asserts the bound). + writeDumpRaw(PARTITION_BYTES, PFModuleLoader::ELF_MAGIC, 0xCCCC0003); + powerOn(); + assert(parses == 1 && PFCrash::record && PFCrash::record->bytes == PARTITION_BYTES); + assert(PFCrash::trail.dumpSeen == 0xCCCC0003); + + writeDump(0xAAAA0001); + readFails = true; + powerOn(); + assert(parses == 0 && !PFCrash::record); + assert(!PFCrash::clear()); // unreadable flash is not "something to clear" + readFails = false; + + partitionMissing = true; // a partition table without the line + powerOn(); + assert(parses == 0 && !PFCrash::record && !PFCrash::clear()); + partitionMissing = false; + + // A parse that fills in less than it promises: no terminator on the task + // name or the hash, more frames than the array holds. + writeDump(0xAAAA0001); + memset(inFlash.exc_task, 'T', sizeof(inFlash.exc_task)); + memset(inFlash.app_elf_sha256, 'h', sizeof(inFlash.app_elf_sha256)); + inFlash.exc_bt_info.depth = 40; + inFlash.exc_bt_info.corrupted = true; + powerOn(); + const PFCrash::Record* r = PFCrash::record; + assert(r && strlen(r->task) == 15 && strlen(r->build) == 16 && r->depth == 16); + assert(r->corrupted && Serial.said(" |<-CORRUPTED\n")); + + // No memory for the record: nothing reported, nothing dereferenced. + allocFails = true; + powerOn(); + assert(!PFCrash::record); + allocFails = false; +} + +// ── Naming ────────────────────────────────────────────────────────────────── +static void slugIs(const char* pathOrSlug, const char* expected) { + static const int who = 0; + PFCrash::running(&who, pathOrSlug); + assert(strcmp(PFCrash::trail.slug, expected) == 0); + // Everything after the slug is zero: the seal is over the whole field, and + // the next boot copies all of it. + for (size_t i = strlen(expected); i < PFCrash::SLUG_BYTES; ++i) assert(PFCrash::trail.slug[i] == '\0'); + assert(PFCrash::trail.check == PFCrash::seal()); + assert(PFCrash::isRunning(&who)); +} + +static void testNaming() { + eraseFlash(); + powerOn(); + slugIs("/patterns/cell_ripple.pfm", "cell_ripple"); + slugIs("cell_ripple.pfm", "cell_ripple"); + slugIs("origin", "origin"); // a preset's slug is itself + slugIs("/patterns/v1.2_rings.pfm", "v1.2_rings"); // only the last dot is the extension + slugIs("/patterns/", ""); + slugIs("", ""); + slugIs(nullptr, ""); + // Longer than the field: cut, terminated, and never a bare ".p" at the end. + slugIs("/patterns/a_name_that_is_longer_than_the_breadcrumb_has_room_for.pfm", + "a_name_that_is_longer_than_the_breadcru"); + slugIs("/patterns/abcdefghijklmnopqrstuvwxyz_0123456789.pfm", + "abcdefghijklmnopqrstuvwxyz_0123456789"); + + // Naming a pattern drops the last one's code range; a new range reseals. + PFCrash::running(PROBE, PROBE); + PFCrash::code(PSRAM_CODE, CODE_BYTES); + assert(PFCrash::trail.check == PFCrash::seal()); + static const char preset[] = "Origin"; + PFCrash::running(preset, "origin"); + assert(PFCrash::trail.codeBase == 0 && PFCrash::trail.codeSize == 0); + assert(PFCrash::isRunning(preset) && !PFCrash::isRunning(PROBE)); + + // A preset that dies in setup() at boot: named, and no code range, because + // its addresses are the firmware's own. + PFCrash::enter(PFCrash::SETUP); + eraseFlash(); + boot(ESP_RST_TASK_WDT); + const PFCrash::Record* r = PFCrash::record; + assert(r && r->trailed && !r->dumped && r->phase == PFCrash::SETUP); + assert(strcmp(r->slug, "origin") == 0 && r->codeSize == 0); + assert(Serial.said("the reset came during setup of \"origin\"\n")); + + // A load that failed, or a module that left: nothing is resident, and a + // death afterwards is not pinned on it. + liveIntoDraw(PSRAM_CODE); + PFCrash::forget(); + assert(!PFCrash::isRunning(PROBE)); + boot(ESP_RST_INT_WDT); + r = PFCrash::record; + assert(r && r->trailed && r->slug[0] == '\0' && r->phase == PFCrash::IDLE && r->codeSize == 0); + assert(Serial.said("the reset came during idle, no pattern resident\n")); + + // A death before any pattern was named at all. + boot(ESP_RST_PANIC); + r = PFCrash::record; + assert(r && r->trailed && r->slug[0] == '\0' && r->phase == PFCrash::IDLE); + + const char* names[] = {"idle", "loading", "constructors", "setup", "update", "draw", "dump-read"}; + for (uint32_t phase = 0; phase < PFCrash::PHASE_COUNT; ++phase) { + assert(strcmp(PFCrash::phaseName(phase), names[phase]) == 0); + } + assert(strcmp(PFCrash::phaseName(PFCrash::PHASE_COUNT), "idle") == 0); +} + +// ── An address inside the module, or not ──────────────────────────────────── +static void testInModule() { + PFCrash::Record r = {}; + r.trailed = r.dumpFromThisReset = true; + r.codeSize = CODE_BYTES; + uint32_t offset = 0xdead; + for (uint32_t base : {INTERNAL_CODE, PSRAM_CODE}) { + r.codeBase = base; + assert(PFCrash::inModule(r, base, offset) && offset == 0); + assert(PFCrash::inModule(r, base + CODE_BYTES - 1, offset) && offset == CODE_BYTES - 1); + assert(!PFCrash::inModule(r, base + CODE_BYTES, offset)); + assert(!PFCrash::inModule(r, base - 1, offset)); + assert(!PFCrash::inModule(r, 0, offset)); + assert(!PFCrash::inModule(r, 0x4200f1a3, offset)); // firmware code in flash + } + // Code in PSRAM is one set of bytes at two addresses, and a PC only ever + // carries the instruction-bus one. A breadcrumb that recorded the address + // the loader WROTE the code through matches no frame of any crash - which + // is what this record did before it was put on a panel. + r.codeBase = PSRAM_CODE; + assert(PFCrash::inModule(r, PSRAM_CODE + 0x4e, offset) && offset == 0x4e); + assert(!PFCrash::inModule(r, PSRAM_DATA + 0x4e, offset)); + r.codeBase = PSRAM_DATA; + assert(!PFCrash::inModule(r, PSRAM_CODE + 0x4e, offset)); + + // A range that ends at the top of the address space does not wrap round to + // claim address zero. + r.codeBase = 0xFFFFF000u; + r.codeSize = 0x1000; + assert(PFCrash::inModule(r, 0xFFFFFFFFu, offset) && offset == 0xFFF); + assert(!PFCrash::inModule(r, 0, offset)); + + // Each of the three conditions is necessary. + r.codeBase = PSRAM_CODE; + r.codeSize = CODE_BYTES; + r.trailed = false; + assert(!PFCrash::inModule(r, PSRAM_CODE, offset)); + r.trailed = true; + r.dumpFromThisReset = false; + assert(!PFCrash::inModule(r, PSRAM_CODE, offset)); + r.dumpFromThisReset = true; + r.codeSize = 0; // a preset + assert(!PFCrash::inModule(r, PSRAM_CODE, offset)); +} + +// ── DELETE /api/crash ─────────────────────────────────────────────────────── +static void testClear() { + eraseFlash(); + powerOn(); + erases = 0; + assert(!PFCrash::clear() && erases == 0); // nothing recorded: the 404 + + writeDump(0xAAAA0001); + dumpOfProbe(PSRAM_CODE); + powerOn(); + assert(PFCrash::record); + assert(PFCrash::clear() && erases == 1 && !PFCrash::record); + assert(!PFCrash::clear() && erases == 1); // and it is gone + powerOn(); + assert(!PFCrash::record); // from flash too + + // A breadcrumb with no dump (a watchdog that wrote none) is a record to + // drop and not a partition to erase. + liveIntoDraw(PSRAM_CODE); + boot(ESP_RST_WDT); + assert(PFCrash::record && PFCrash::record->trailed && !PFCrash::record->dumped); + assert(PFCrash::clear() && erases == 1 && !PFCrash::record); + + // A dump the parser would not read still occupies the partition, and + // clearing is how it leaves. + writeDump(0xAAAA0001); + crcHolds = false; + powerOn(); + assert(!PFCrash::record); + assert(PFCrash::clear() && erases == 2); + crcHolds = true; +} + +int main() { + testPowerOn(); + testPanicInModule(PSRAM_CODE, "backtrace: 0x43c8104e 0x43c8109c 0x4200f1a3 0x42010b52\n"); + testPanicInModule(INTERNAL_CODE, "backtrace: 0x40381c8e 0x40381cdc 0x4200f1a3 0x42010b52\n"); + testMatrix(); + testParserChoke(); + testDumpGuards(); + testNaming(); + testInModule(); + testClear(); + free(PFCrash::record); + PFCrash::record = nullptr; + printf("PASS crash: %u boots walked\n", boots); + return 0; +} diff --git a/firmware/toolchain/tests/modules/_crash_probe/pattern.cpp b/firmware/toolchain/tests/modules/_crash_probe/pattern.cpp new file mode 100644 index 00000000..4b8f0b3f --- /dev/null +++ b/firmware/toolchain/tests/modules/_crash_probe/pattern.cpp @@ -0,0 +1,62 @@ +// Bench probe for the crash record (src/core_crash.h), not a showable +// pattern. It is a dim green field until a knob moves, and then it writes +// through a null pointer from inside draw() - one panic, on demand, at a +// known place, which is what it takes to check that /api/status afterwards +// names this module, says "draw", and gives the PC as an offset that lands in +// fall() below. +// +// python firmware/toolchain/build_module.py --out firmware/toolchain/tests/modules/_crash_probe +// xtensa-esp32s3-elf-nm -n /_crash_probe.pfm # where fall() and draw() are +// +// Triggered rather than timed, on purpose. A probe that dies N seconds after +// it starts dies again after the reboot restores it, and whether that ends is +// then up to the boot latch - a second thing under test in an experiment +// about the first. This one comes back from the reboot resident and quiet, +// so the record can be read at leisure. Turn K1, K2 or K3, or from the desk: +// +// curl -X POST "http:///api/params?d1=1" +#include "pf_module.h" + +namespace CrashProbe { + +const char* NAME = "Crash Probe"; +const char* const KNOB_LABELS[4] = {"DIE", "DIE", "DIE", "-"}; + +bool armed = false; + +// Read at run time so the compiler cannot see the null. A store through a +// constant null is undefined behaviour it is entitled to delete, or to +// replace with a trap of its own - and then the exception under test is not +// the one that happens. +int* volatile nowhere = nullptr; + +void setup() {} + +void update(float dt, const InputFrame& input) { + (void)dt; + for (int i = 0; i < 3; ++i) { + if (input.knobDeltas[i] != 0) armed = true; + } +} + +// Its own frame, so the backtrace has two addresses inside the module - the +// fault here and the return into draw() - and both have to come out as +// offsets. +__attribute__((noinline)) void fall() { + *nowhere = 1; +} + +void draw() { + PFCanvas::clear(); + for (int y = 0; y < PANEL_RES_H; ++y) { + for (int x = 0; x < PANEL_RES_W; ++x) { + PFCanvas::setPixel(x, y, 0, 24, 0); + } + } + if (armed) fall(); + PFCanvas::present(); +} + +} // namespace CrashProbe + +PF_REGISTER_PATTERN(CrashProbe) diff --git a/firmware/toolchain/tests/modules/_ctor_probe/pattern.cpp b/firmware/toolchain/tests/modules/_ctor_probe/pattern.cpp index a1137655..dad9bdb3 100644 --- a/firmware/toolchain/tests/modules/_ctor_probe/pattern.cpp +++ b/firmware/toolchain/tests/modules/_ctor_probe/pattern.cpp @@ -7,7 +7,12 @@ // xtensa-esp32s3-elf-readelf -S .../_ctor_probe.pfm # expect .init_array // // On device it draws a red vertical ramp when the constructor ran and stays -// black when it did not. +// black when it did not. The ramp turns yellow when the constructor could also +// reach the host: it allocates and logs, which is what a community pattern's +// namespace-scope initialiser does (`float* trail = PFMem::allocFloats(n);`). +// Constructors once ran before the module had its host API pointer, and this +// probe then took the board down at load instead of drawing anything; the log +// line below is how a bench reads the result without looking at the panel. #include "pf_module.h" namespace CtorProbe { @@ -23,11 +28,14 @@ volatile int probeSeed = 4; struct Probe { uint8_t ramp[PANEL_RES_H]; bool ready; + float* fromHost; Probe() { for (int y = 0; y < PANEL_RES_H; ++y) { ramp[y] = (uint8_t)((y * probeSeed) & 0xff); } + fromHost = PFMem::allocFloats(PANEL_RES_H); + Serial.printf("[CTOR] constructor ran, host alloc %s\n", fromHost ? "ok" : "refused"); ready = true; } }; @@ -46,7 +54,7 @@ void draw() { if (probe.ready) { for (int y = 0; y < PANEL_RES_H; ++y) { for (int x = 0; x < PANEL_RES_W; ++x) { - PFCanvas::setPixel(x, y, probe.ramp[y], 0, 0); + PFCanvas::setPixel(x, y, probe.ramp[y], probe.fromHost ? probe.ramp[y] : 0, 0); } } } diff --git a/firmware/toolchain/tests/modules/_hang_probe/pattern.cpp b/firmware/toolchain/tests/modules/_hang_probe/pattern.cpp new file mode 100644 index 00000000..ce66506b --- /dev/null +++ b/firmware/toolchain/tests/modules/_hang_probe/pattern.cpp @@ -0,0 +1,148 @@ +// Bench probe for a render loop that stops, not a showable pattern: a module +// whose draw() stops returning. Nothing in the catalog does that on purpose, +// and the thing under test - the console outliving a hung loop +// (src/core_loop_sync.h) - cannot be seen on a panel whose patterns all +// behave. +// +// python firmware/toolchain/build_module.py --out firmware/toolchain/tests/modules/_hang_probe +// curl -X PUT --data-binary @/_hang_probe.pfm -H "X-PF-Name: _hang_probe.pfm" http:///api/patterns +// curl "http:///api/patterns/select?index=" +// +// N is its index in GET /api/patterns, the entry whose "module" is +// "_hang_probe" - by index because it has no sidecar, so the list knows it +// only by its slug until it has been loaded once. +// +// It runs a green bar down for eight seconds, shows one red frame, and never +// comes back from the draw() after that. The panel stays red; /api/status +// keeps answering and its loopAgeMs climbs. +// +// A hang that never ends shows only half of it. What the loop does when it +// COMES BACK to requests that gave up on it is the other half, and the half +// with the dead stack frame in it, so a knob moved while the fuse burns turns +// the stop into one that ends: +// +// K1 (or curl -X POST http:///api/params -d d1=3): the bar turns +// blue and draw() holds the loop for thirty seconds. A request waiting +// on the loop is taken back at twenty; ten seconds later the loop wakes +// to find nothing posted, and must not run what was withdrawn. +// K2 (d2=3): the bar turns amber and the hold is twenty seconds and half +// a slice - the host's PF_LOOP_STALL_MS, which a module cannot see, so +// the two are kept in step by hand, plus half of the 25 ms a waiting +// caller sleeps between looks. The loop comes round inside the slice in +// which that caller gives up, so over many rounds each of them gets +// there first sometimes. That checks both ORDERS on a real board - +// loop first, caller first - and not the tie itself: the loop stamps +// before it takes the request, so the window in which both exchanges +// are live is a few instructions wide. The tie is the host test's +// (runtime_test.cpp) and the generated code's. Either may win. +// Neither may crash. +// +// A hold that ends starts the fuse again in the same colour, so a script can +// keep asking for as many rounds as it likes without a hand on the panel. +// +// Picked within twenty seconds of boot it stays green and never hangs. That +// is what a boot restore of the remembered pattern looks like from in here, +// and a probe that hung there would hang again on every restart: the panel +// would be trapped in the very loop this exists to test the way out of. +// (The boot latch would catch it one restart later - it is armed for the +// first fifteen seconds - but a bench tool should not lean on that.) So a +// restart always ends the hang, and picking another pattern and then this +// one again starts a new countdown. +#include "pf_module.h" + +namespace HangProbe { + +const char* NAME = "Hang Probe"; +const char* const KNOB_LABELS[4] = {"30 s", "20 s", "-", "-"}; + +constexpr uint32_t BOOT_RESTORE_MS = 20000; +constexpr uint32_t FUSE_MS = 8000; +constexpr uint32_t HOLD_MS = 30000; +constexpr uint32_t EDGE_MS = 20000 + 12; // PF_LOOP_STALL_MS in src/core_loop_sync.h, and half a slice + +enum Stop : uint8_t { FOR_EVER, HOLD, EDGE }; + +uint32_t startedAtMs = 0; +uint32_t nowMs = 0; +bool started = false; +bool armed = false; +bool stopShown = false; +Stop stop = FOR_EVER; +// volatile so the loops below have a side effect: an empty for(;;) is one the +// compiler is allowed to delete, and a probe that did not hang would pass +// every check for the wrong reason. +volatile uint32_t spins = 0; + +void setup() { + started = false; + armed = false; + stopShown = false; + stop = FOR_EVER; +} + +void update(float dt, const InputFrame& input) { + (void)dt; + nowMs = input.now; + if (!started) { + started = true; + startedAtMs = input.now; + armed = input.now >= BOOT_RESTORE_MS; + } + if (armed && !stopShown) { + if (input.knobDeltas[0] != 0) stop = HOLD; + else if (input.knobDeltas[1] != 0) stop = EDGE; + } +} + +// Green for the stop that never ends, blue for thirty seconds, amber for +// twenty: the fuse says which is coming and the stopped panel says which +// came (red, when it is for ever). +void colour(bool stopped, uint8_t level, uint8_t& r, uint8_t& g, uint8_t& b) { + r = g = b = 0; + if (stop == HOLD) { g = level / 3; b = level; } + else if (stop == EDGE) { r = level; g = level / 2; } + else if (stopped) r = level; + else g = level; +} + +void draw() { + const uint32_t ranMs = nowMs - startedAtMs; + uint8_t r, g, b; + if (armed && ranMs >= FUSE_MS) { + if (stopShown) { + if (stop == FOR_EVER) { + for (;;) spins = spins + 1; + } + const uint32_t holdMs = stop == HOLD ? HOLD_MS : EDGE_MS; + const uint32_t from = millis(); + while ((uint32_t)(millis() - from) < holdMs) spins = spins + 1; + // Back. The next update() lights a new fuse, in the same colour. + stopShown = false; + started = false; + return; + } + // One whole frame of the colour first, so the frozen panel says it froze + // here and which way. + colour(true, 160, r, g, b); + for (int y = 0; y < PANEL_RES_H; ++y) { + for (int x = 0; x < PANEL_RES_W; ++x) PFCanvas::setPixel(x, y, r, g, b); + } + PFCanvas::present(); + stopShown = true; + return; + } + PFCanvas::clear(); + // The fuse: full width when picked, gone when it hangs. Unarmed it stays + // full and dim, which is how to tell a boot restore from a countdown. + const int width = armed ? (int)((uint32_t)PANEL_RES_W * (FUSE_MS - ranMs) / FUSE_MS) + : PANEL_RES_W; + colour(false, armed ? 160 : 40, r, g, b); + for (int y = PANEL_RES_H / 2 - 4; y < PANEL_RES_H / 2 + 4; ++y) { + for (int x = 0; x < width; ++x) PFCanvas::setPixel(x, y, r, g, b); + } + PFCanvas::present(); +} + +} // namespace HangProbe + +PF_REGISTER_PATTERN(HangProbe) diff --git a/firmware/toolchain/tests/parser_test.cpp b/firmware/toolchain/tests/parser_test.cpp new file mode 100644 index 00000000..735857d1 --- /dev/null +++ b/firmware/toolchain/tests/parser_test.cpp @@ -0,0 +1,1016 @@ +// The vendored HTTP request parser - the real src/webserver/Parsing.cpp and +// WebServer.cpp - against a socket that replays a script and a clock that only +// moves when somebody sleeps. +// +// The parser runs on pf-net, pinned to Core 0 at priority 1, and this sdkconfig +// panics the board when IDLE0 has not run for 5 s. A loop in it that waits for +// the peer without sleeping is therefore a reboot, and it has been one three +// times (src/webserver/VENDORED.md): a request trickling in over a slow link, +// an upload that went quiet, a multipart body that stopped after its first +// boundary. None of them needed a handler, and nothing on a PC stood between +// any of them and a board. +// +// So every request replayed here is held to two things, whatever else its +// case is about: +// +// it does not SPIN the parser never looks at the socket SPIN_LOOKS times in +// a row and comes away empty-handed each time - no byte +// read, no delay() in between. That is the loop IDLE0 +// starves behind. +// it is not HELD the request is over within REQUEST_BOUND_MS of fake +// time. This server takes one connection; a wait with no +// deadline is a console that does not answer. +// +// Both are thrown out of the fakes from inside the parser, because a loop that +// would never end cannot be asked to return. +// +// What a host cannot say: anything about lwIP, the heap or the stack. The +// socket here is a model (WiFiClient below says what it copies and from +// where), and so is String: the real one is not in this repository, and CI has +// no Arduino core to take it from. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef _MSC_VER +#include // _alloca: check_parser.py says which one line needs it +#define __attribute__(x) +#endif + +namespace bench { +// A request cut off on its way out asks a dead socket a handful of times - for +// the line that was cut, the line after it, and the multipart opener's three +// tries, two looks each - and that count has a ceiling the fuzz below measures +// (10). A spin has none: the watchdog bites after 5 s of it. 64 keeps the two +// apart without pinning the parser's shape. +constexpr unsigned SPIN_LOOKS = 64; +// The stream timeout is 5 s and a request silenced in the worst place waits it +// out five times over (a header line, the line after it, the multipart +// opener's three tries): 25 s, also measured below. Past 30 s nothing in the +// parser is counting. +constexpr uint32_t REQUEST_BOUND_MS = 30000; +constexpr uint32_t EPOCH_MS = 1000; + +struct Spin {}; +struct Held {}; +static uint32_t nowMs = EPOCH_MS, beganMs = EPOCH_MS; +static unsigned idleLooks = 0, worstLooks = 0, sleeps = 0, polls = 0; + +static void consumed() { idleLooks = 0; } +static void looked() { + worstLooks = std::max(worstLooks, ++idleLooks); + if (idleLooks >= SPIN_LOOKS) throw Spin{}; +} +} // namespace bench + +static unsigned long millis() { return bench::nowMs; } +static void delay(uint32_t ms) { + // delay(0) is a yield, and a yield lets equal priorities run: IDLE0 is below + // pf-net and starves through it just the same. + if (!ms) return; + bench::nowMs += ms; + ++bench::sleeps; + bench::idleLooks = 0; + if (bench::nowMs - bench::beganMs > bench::REQUEST_BOUND_MS) throw bench::Held{}; +} +static void yield() {} +namespace PFNetMaintenance { +static void poll() { ++bench::polls; } +} + +// ---- Arduino, as much of it as the server touches -------------------------- + +class __FlashStringHelper; +#define PROGMEM +#define PSTR(s) (s) +#define F(s) (reinterpret_cast(s)) +#define FPSTR(p) (reinterpret_cast(p)) +#define strlen_P strlen +#define strcpy_P strcpy +#define memccpy_P memccpy +#define log_v(...) ((void)0) +#define log_d(...) ((void)0) +#define log_i(...) ((void)0) +#define log_w(...) ((void)0) +#define log_e(...) ((void)0) +typedef const char* PGM_P; +typedef const void* PGM_VOID_P; +typedef bool boolean; + +// Arduino's String where the parser leans on it. The parts that matter are the +// ones std::string does differently: substring() clamps and swaps its bounds +// instead of throwing (the parser hands it -1 + 2 and length() - 1 of an empty +// line without looking), indexOf() past the end is -1, comparisons stop at a +// NUL, and toInt() is a 32-bit long. +// +// Compared with the real cores/esp32/WString.cpp (2.0.17) when it was written, +// both built on a PC and run through every one of these operations over 40 +// strings - request lines, boundaries, empty, padded, with a NUL, with high +// bytes - and every in- and out-of-range index: the transcripts were the same +// but for replace() on a string that holds a NUL, where the real class loses +// track of its own length. The server calls replace() once, to take the +// quotes off a boundary. (toInt() was only comparable inside 32 bits: a PC's +// long is wider than the device's, which is why this one clamps.) +class String { + std::string s; + static int low(char c) { return std::tolower(static_cast(c)); } + static bool space(char c) { return std::isspace(static_cast(c)) != 0; } + +public: + String(const char* c = "") : s(c ? c : "") {} + String(const __FlashStringHelper* f) : s(reinterpret_cast(f)) {} + explicit String(char c) : s(1, c) {} + explicit String(unsigned char v) : s(std::to_string(v)) {} + explicit String(int v) : s(std::to_string(v)) {} + explicit String(unsigned int v) : s(std::to_string(v)) {} + explicit String(long v) : s(std::to_string(v)) {} + explicit String(unsigned long v) : s(std::to_string(v)) {} + explicit String(long long v) : s(std::to_string(v)) {} + explicit String(unsigned long long v) : s(std::to_string(v)) {} + + unsigned int length() const { return static_cast(s.size()); } + const char* c_str() const { return s.c_str(); } + + String& operator+=(const String& r) { s += r.s; return *this; } + String& operator+=(const char* c) { if (c) s += c; return *this; } + String& operator+=(char c) { s += c; return *this; } + + bool equals(const String& o) const { return s.size() == o.s.size() && std::strcmp(c_str(), o.c_str()) == 0; } + bool equals(const char* c) const { + if (s.empty()) return !c || !*c; + return c ? std::strcmp(c_str(), c) == 0 : s[0] == 0; + } + bool operator==(const String& o) const { return equals(o); } + bool operator==(const char* c) const { return equals(c); } + bool operator!=(const String& o) const { return !equals(o); } + bool operator!=(const char* c) const { return !equals(c); } + bool equalsConstantTime(const String& o) const { return equals(o); } + bool equalsIgnoreCase(const String& o) const { + if (s.size() != o.s.size()) return false; + const char* a = c_str(); + const char* b = o.c_str(); + while (*a) if (low(*a++) != low(*b++)) return false; + return true; + } + bool startsWith(const String& p) const { + return s.size() >= p.s.size() && std::strncmp(c_str(), p.c_str(), p.s.size()) == 0; + } + bool endsWith(const String& p) const { + return s.size() >= p.s.size() && std::strcmp(c_str() + (s.size() - p.s.size()), p.c_str()) == 0; + } + + char charAt(unsigned int i) const { return (*this)[i]; } + char operator[](unsigned int i) const { return i < s.size() ? s[i] : 0; } + char& operator[](unsigned int i) { + static char outside; + return i < s.size() ? s[i] : (outside = 0); + } + + int indexOf(char c, unsigned int from = 0) const { + if (from >= s.size()) return -1; + const char* at = std::strchr(c_str() + from, c); + return at ? static_cast(at - c_str()) : -1; + } + int indexOf(const String& n, unsigned int from = 0) const { + if (from >= s.size()) return -1; + const char* at = std::strstr(c_str() + from, n.c_str()); + return at ? static_cast(at - c_str()) : -1; + } + String substring(unsigned int left, unsigned int right = ~0u) const { + if (left > right) std::swap(left, right); + String out; + if (left >= s.size()) return out; + out.s = s.substr(left, std::min(right, s.size()) - left); + return out; + } + + void trim() { + size_t b = 0, e = s.size(); + while (b < e && space(s[b])) ++b; + while (e > b && space(s[e - 1])) --e; + s = s.substr(b, e - b); + } + void replace(const String& find, const String& with) { + if (find.s.empty()) return; + for (size_t at = 0; (at = s.find(find.s, at)) != std::string::npos; at += with.s.size()) + s.replace(at, find.s.size(), with.s); + } + long toInt() const { + const long long v = std::strtoll(c_str(), nullptr, 10); + return static_cast(std::max(INT32_MIN, std::min(INT32_MAX, v))); + } +}; + +// The server builds strings as String(a) + b + c and passes the result to +// functions that take a String& - which only binds because Arduino's operator+ +// returns a reference to the temporary on its left. Same trick here. +class StringSumHelper : public String { +public: + StringSumHelper(const String& v) : String(v) {} + StringSumHelper(const char* c) : String(c) {} +}; +static StringSumHelper& operator+(const StringSumHelper& l, const String& r) { + auto& a = const_cast(l); a += r; return a; +} +static StringSumHelper& operator+(const StringSumHelper& l, const char* r) { + auto& a = const_cast(l); a += r; return a; +} +static StringSumHelper& operator+(const StringSumHelper& l, char r) { + auto& a = const_cast(l); a += r; return a; +} + +// cores/esp32/Stream.cpp, the three calls stock Parsing.cpp was built on. +// timedRead() is copied as it is upstream - read() until the timeout, nothing +// between - so a parser put back on readStringUntil()/readBytes() spins here +// exactly as it did on Core 0 (VENDORED.md, Fix 2). +class Stream { +protected: + unsigned long _timeout = 1000, _startMillis = 0; + int timedRead() { + _startMillis = millis(); + do { + const int c = read(); + if (c >= 0) return c; + } while (millis() - _startMillis < _timeout); + return -1; + } + +public: + virtual ~Stream() {} + virtual int available() = 0; + virtual int read() = 0; + virtual int peek() = 0; + void setTimeout(unsigned long ms) { _timeout = ms; } + unsigned long getTimeout() { return _timeout; } + size_t readBytes(char* buffer, size_t length) { + size_t count = 0; + for (int c; count < length && (c = timedRead()) >= 0; ++count) *buffer++ = static_cast(c); + return count; + } + size_t readBytes(uint8_t* buffer, size_t length) { return readBytes(reinterpret_cast(buffer), length); } + String readStringUntil(char terminator) { + String out; + for (int c = timedRead(); c >= 0 && c != terminator; c = timedRead()) out += static_cast(c); + return out; + } +}; + +namespace bench { +// One scripted connection: every byte the peer will ever send, when each run +// of them lands on the device, and whether a FIN follows the last. The peer +// keeps its own schedule - it does not wait for the parser to catch up. +struct Wire { + std::string rx, tx; + std::vector> lands; // (fake ms, bytes of rx here by then), in time order + bool closes = false; + uint32_t closesAt = 0; + size_t pos = 0, landed = 0, next = 0; + + size_t waiting() { + while (next < lands.size() && lands[next].first <= nowMs) landed = lands[next++].second; + return landed - pos; + } + bool gone() { return closes && closesAt <= nowMs && waiting() == 0; } +}; +static std::shared_ptr knocking; // what WiFiServer::available() hands out next +} // namespace bench + +// libraries/WiFi/src/WiFiClient.cpp (core 2.0.17), the receive side. What is +// copied: read() is non-blocking and returns what is there; available() is the +// count; flush() discards what has arrived; connected() stays true while the +// peer is merely quiet and goes false once its FIN has been read up to (lwIP's +// recv() reports ENOTCONN then - it is what the Fix 4 reproduction on hardware +// depended on); copies share the socket; operator= does not carry Stream's +// timeout and setTimeout() takes seconds (WebServer.cpp's handleClient comment +// is about exactly that pair). Not copied, because the source does not settle +// it: what connected() answers when the FIN is in and bytes are still unread. +// Here it is true. Only _parseForm's check after a file part asks in that +// state. +class WiFiClient : public Stream { + std::shared_ptr wire; + +public: + WiFiClient() {} + WiFiClient(const WiFiClient&) = default; + explicit WiFiClient(std::shared_ptr w) : wire(std::move(w)) {} + WiFiClient& operator=(const WiFiClient& other) { wire = other.wire; return *this; } + + int available() override { + const size_t n = wire ? wire->waiting() : 0; + if (!n) bench::looked(); + return static_cast(n); + } + int read(uint8_t* buf, size_t size) { + const size_t n = wire ? std::min(size, wire->waiting()) : 0; + if (!n) { bench::looked(); return 0; } + std::memcpy(buf, wire->rx.data() + wire->pos, n); + wire->pos += n; + bench::consumed(); + return static_cast(n); + } + int read() override { + uint8_t byte = 0; + return read(&byte, 1) == 1 ? byte : -1; + } + int peek() override { + if (wire && wire->waiting()) return static_cast(wire->rx[wire->pos]); + bench::looked(); + return -1; + } + void flush() { + if (!wire || !wire->waiting()) return; + wire->pos = wire->landed; + bench::consumed(); + } + void stop() { wire.reset(); } + uint8_t connected() { + bench::looked(); + return wire && !wire->gone(); + } + operator bool() { return connected(); } + int setTimeout(uint32_t seconds) { Stream::setTimeout(seconds * 1000ul); return 0; } + + size_t write(const char* b, size_t n) { if (wire) wire->tx.append(b, n); return wire ? n : 0; } + size_t write(const uint8_t* b, size_t n) { return write(reinterpret_cast(b), n); } + size_t write_P(PGM_P b, size_t n) { return write(b, n); } + size_t write(Stream&) { return 0; } +}; + +struct IPAddress {}; +class WiFiServer { +public: + WiFiServer(uint16_t = 80) {} + WiFiServer(const IPAddress&, uint16_t = 80) {} + void begin(uint16_t = 0) {} + void close() {} + void setNoDelay(bool) {} + WiFiClient available() { return WiFiClient(std::exchange(bench::knocking, nullptr)); } +}; + +// Compiled because WebServer.cpp is taken whole, never reached: no case here +// authenticates or serves a file. +namespace fs { +struct File : Stream { + operator bool() { return false; } + bool isDirectory() { return false; } + int available() override { return 0; } + int read() override { return -1; } + int peek() override { return -1; } + size_t size() { return 0; } + const char* name() { return ""; } +}; +struct FS { + File open(const String&, const char* = "r") { return File(); } + bool exists(const String&) { return false; } +}; +} // namespace fs +using fs::File; +using fs::FS; +struct MD5Builder { + void begin() {} + void add(const String&) {} + void calculate() {} + String toString() { return String(); } +}; +static int base64_encode_expected_len(int n) { return (n + 2) / 3 * 4; } +static int base64_encode_chars(const char*, int, char* out) { *out = 0; return 0; } +static uint32_t esp_random() { return 0x5eed5eedu; } + +// http_parser.h (nodejs/http-parser, MIT, as IDF 4.4 ships it): the method +// list Parsing.cpp turns into both the HTTPMethod enum and the strings it +// matches the request line against. +#define HTTP_METHOD_MAP(XX) \ + XX(0, DELETE, DELETE) \ + XX(1, GET, GET) \ + XX(2, HEAD, HEAD) \ + XX(3, POST, POST) \ + XX(4, PUT, PUT) \ + XX(5, CONNECT, CONNECT) \ + XX(6, OPTIONS, OPTIONS) \ + XX(7, TRACE, TRACE) \ + XX(8, COPY, COPY) \ + XX(9, LOCK, LOCK) \ + XX(10, MKCOL, MKCOL) \ + XX(11, MOVE, MOVE) \ + XX(12, PROPFIND, PROPFIND) \ + XX(13, PROPPATCH, PROPPATCH) \ + XX(14, SEARCH, SEARCH) \ + XX(15, UNLOCK, UNLOCK) \ + XX(16, BIND, BIND) \ + XX(17, REBIND, REBIND) \ + XX(18, UNBIND, UNBIND) \ + XX(19, ACL, ACL) \ + XX(20, REPORT, REPORT) \ + XX(21, MKACTIVITY, MKACTIVITY) \ + XX(22, CHECKOUT, CHECKOUT) \ + XX(23, MERGE, MERGE) \ + XX(24, MSEARCH, M-SEARCH) \ + XX(25, NOTIFY, NOTIFY) \ + XX(26, SUBSCRIBE, SUBSCRIBE) \ + XX(27, UNSUBSCRIBE, UNSUBSCRIBE) \ + XX(28, PATCH, PATCH) \ + XX(29, PURGE, PURGE) \ + XX(30, MKCALENDAR, MKCALENDAR) \ + XX(31, LINK, LINK) \ + XX(32, UNLINK, UNLINK) +enum http_method { +#define XX(num, name, string) HTTP_##name = num, + HTTP_METHOD_MAP(XX) +#undef XX +}; + +// ---- The code under test, as the firmware compiles it ---------------------- +// check_parser.py lays the vendored directory out beside empty stand-ins for +// the headers above and puts it on the system include path: the server is not +// ours to make -Wextra clean (sign compares, a size_t printed with %x). +#ifdef _MSC_VER +#pragma warning(push, 0) +#endif +#include "webserver/WebServer.cpp" +#include "webserver/Parsing.cpp" +#include "webserver/detail/mimetable.cpp" +#ifdef _MSC_VER +#pragma warning(pop) +#endif + +// ---- The bench -------------------------------------------------------------- + +namespace bench { + +static std::string text(const String& v) { return std::string(v.c_str(), v.length()); } + +// What the server handed its handlers, in the order it handed it, and what +// went back down the wire. +struct Seen { + std::string log, upload, raw, reply; + bool handled = false, uploading = false; + bool operator==(const Seen& o) const { + return log == o.log && upload == o.upload && raw == o.raw && reply == o.reply; + } +}; + +struct Panel : WebServer { + Panel() : WebServer(80) {} + // ~WebServer() frees neither list. On the device the server is never + // destroyed; here there is one per replay and LeakSanitizer is counting. + ~Panel() override { delete[] _currentArgs; delete[] _postArgs; } + bool hasUpload() const { return static_cast(_currentUpload); } + bool hasRaw() const { return static_cast(_currentRaw); } +}; + +struct Script { + Wire wire; + uint32_t at = EPOCH_MS; + Script& send(const std::string& bytes) { + wire.rx += bytes; + wire.lands.push_back({at, wire.rx.size()}); + return *this; + } + Script& wait(uint32_t ms) { at += ms; return *this; } + Script& close() { wire.closes = true; wire.closesAt = at; return *this; } +}; + +struct Result { + enum End { Returned, Spun, Held } end = Returned; + uint32_t ms = 0; // fake time from accept to return + unsigned looks = 0; // most looks at the socket in a row that found nothing + bool unpolled = false; // slept more often than it polled network maintenance + Seen seen; +}; + +struct Bench { + Panel server; + Seen seen; + const char* probe = nullptr; // an argument asked for by name, as handlers ask + + Bench() { + static const char* collected[] = {"X-PF-Name"}; + server.collectHeaders(collected, 1); + // The firmware's own shapes: a multipart route and a raw route, each with + // a body callback (core_web_update.h, core_patterns_http.h), and routes + // with none. + server.on("/form", HTTP_POST, [this] { handle("/form"); }, [this] { body(); }); + server.on("/raw", HTTP_PUT, [this] { handle("/raw"); }, [this] { body(); }); + server.on("/plain", HTTP_POST, [this] { handle("/plain"); }); + server.on("/plain", HTTP_DELETE, [this] { handle("/plain"); }); + server.on("/page", HTTP_GET, [this] { handle("/page"); }); + server.begin(); + } + Bench(const Bench&) = delete; + + void line(const std::string& entry) { seen.log += entry + "\n"; } + + void handle(const char* route) { + seen.handled = true; + line(std::string("handle ") + route); + for (int i = 0; i < server.args(); ++i) line("arg " + text(server.argName(i)) + "=" + text(server.arg(i))); + if (probe) line(std::string("named ") + probe + "=" + (server.hasArg(probe) ? text(server.arg(probe)) : "(absent)")); + if (server.hasHeader("X-PF-Name")) line("header X-PF-Name=" + text(server.header("X-PF-Name"))); + line("host " + text(server.hostHeader())); + server.send(200, "text/plain", "ok"); + } + + // One callback per route, which is all FunctionRequestHandler has: the + // server calls it for a multipart file's events and for a raw body's, and + // says which only by which of upload() and raw() exists. + void body() { + if (server.hasUpload()) { + HTTPUpload& up = server.upload(); + switch (up.status) { + case UPLOAD_FILE_START: + seen.uploading = true; + line("upload start " + text(up.name) + " " + text(up.filename) + " " + text(up.type)); + break; + case UPLOAD_FILE_WRITE: + seen.upload.append(reinterpret_cast(up.buf), up.currentSize); + line("upload write " + std::to_string(up.currentSize)); + break; + case UPLOAD_FILE_END: + seen.uploading = false; + line("upload end " + std::to_string(up.totalSize)); + break; + case UPLOAD_FILE_ABORTED: + seen.uploading = false; + line("upload aborted"); + break; + } + } else if (server.hasRaw()) { + // Chunk sizes follow the segments the body arrived in, so only the + // bytes and the total are kept. + HTTPRaw& in = server.raw(); + switch (in.status) { + case RAW_START: line("raw start"); break; + case RAW_WRITE: seen.raw.append(reinterpret_cast(in.buf), in.currentSize); break; + case RAW_END: line("raw end " + std::to_string(in.totalSize)); break; + case RAW_ABORTED: line("raw aborted " + std::to_string(in.totalSize)); break; + } + } + } + + // One connection through handleClient(), the way pf-net drives it: accepted + // with its first bytes already here, parsed, handled, dropped. + Result play(Script script) { + seen = Seen(); + nowMs = beganMs = EPOCH_MS; + idleLooks = worstLooks = sleeps = polls = 0; + auto wire = std::make_shared(std::move(script.wire)); + knocking = wire; + Result r; + try { + server.handleClient(); + } catch (const Spin&) { + r.end = Result::Spun; + } catch (const Held&) { + r.end = Result::Held; + } + r.ms = nowMs - beganMs; + r.looks = worstLooks; + r.unpolled = polls < sleeps; + seen.reply = wire->tx; + r.seen = seen; + return r; + } +}; + +static int failures = 0; +static void fail(const std::string& what) { + ++failures; + std::printf("FAIL: %s\n", what.c_str()); +} + +// What the vendored parser does today that this file would otherwise fail. +// Pinned in both directions: the day one stops being true the check goes red +// until the pin is deleted and the behaviour becomes a requirement. +static void known(bool stillTrue, const std::string& what) { + if (stillTrue) std::printf("KNOWN: %s\n", what.c_str()); + else fail("no longer true - delete this pin and require the fixed behaviour: " + what); +} + +static std::string printable(const std::string& bytes) { + std::string out; + for (unsigned char c : bytes) { + if (c == '\n') out += "\\n"; + else if (c == '\r') out += "\\r"; + else if (c < 0x20 || c > 0x7e) { char hex[8]; std::snprintf(hex, sizeof(hex), "\\x%02x", c); out += hex; } + else out += static_cast(c); + } + return out; +} +static std::string shown(const Seen& s) { + return "log \"" + printable(s.log) + "\", " + std::to_string(s.upload.size()) + " upload bytes, " + + std::to_string(s.raw.size()) + " raw bytes, reply \"" + printable(s.reply) + "\""; +} +static std::string outcome(const Result& r) { + if (r.end == Result::Spun) + return "SPUN - looked at the socket " + std::to_string(SPIN_LOOKS) + + " times in a row with no byte read and no delay() between (on pf-net: the Core-0 watchdog)"; + if (r.end == Result::Held) + return "HELD - still parsing " + std::to_string(REQUEST_BOUND_MS) + " ms of fake time after the request began"; + return "returned after " + std::to_string(r.ms) + " ms, " + (r.seen.handled ? "handled" : "not handled"); +} + +// The two properties, and Fix 3's: a wait that sleeps also lets Wi-Fi and +// name maintenance run. Empty when all three hold. +static std::string unsound(const Result& r) { + if (r.end != Result::Returned) return outcome(r); + if (r.unpolled) return "a wait slept without PFNetMaintenance::poll() (VENDORED.md, Fix 3)"; + return ""; +} +static bool ends(const std::string& name, const Result& r) { + const std::string why = unsound(r); + if (!why.empty()) fail(name + ": " + why); + return why.empty(); +} + +// ---- Requests --------------------------------------------------------------- + +static const std::string BOUNDARY = "----PFBoundary7MA4YWxk"; +static const std::string OK_REPLY = + "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok"; + +// Every byte value, and what a byte-at-a-time boundary matcher gets wrong +// first: the boundary's own opening inside the file, once across the +// 1436-byte upload buffer. +static std::string binary(size_t n) { + std::string b(n, '\0'); + uint32_t x = 0x2545f491u; + for (auto& c : b) { x = x * 1664525u + 1013904223u; c = static_cast(x >> 24); } + auto plant = [&b](size_t at, const std::string& lookalike) { + if (at + lookalike.size() <= b.size()) b.replace(at, lookalike.size(), lookalike); + }; + plant(40, "\r\n--"); + plant(200, "\r\n--" + BOUNDARY.substr(0, BOUNDARY.size() - 1) + "X"); + plant(600, "\r\r\n--\r\n-"); + plant(1430, "\r\n--" + BOUNDARY.substr(0, 8)); + return b; +} + +struct Request { + std::string name, bytes; + size_t body = 0; // offset of the first body byte; bytes.size() when there is none + Seen whole; // what a complete delivery hands the handlers +}; + +static Request request(const std::string& name, const std::string& line, const std::string& headers, + const std::string& body, const std::string& log, + const std::string& upload = "", const std::string& raw = "") { + Request q; + q.name = name; + q.bytes = line + "\r\n" + headers; + if (!body.empty()) q.bytes += "Content-Length: " + std::to_string(body.size()) + "\r\n"; + q.bytes += "\r\n"; + q.body = q.bytes.size(); + q.bytes += body; + q.whole.log = log; + q.whole.upload = upload; + q.whole.raw = raw; + q.whole.reply = OK_REPLY; + q.whole.handled = true; + return q; +} + +static const std::string MULTIPART = "Content-Type: multipart/form-data; boundary=" + BOUNDARY + "\r\n"; +static std::string field(const std::string& name, const std::string& value) { + return "--" + BOUNDARY + "\r\nContent-Disposition: form-data; name=\"" + name + "\"\r\n\r\n" + value + "\r\n"; +} +static std::string file(const std::string& name, const std::string& called, const std::string& type, + const std::string& bytes) { + return "--" + BOUNDARY + "\r\nContent-Disposition: form-data; name=\"" + name + "\"; filename=\"" + called + + "\"\r\n" + (type.empty() ? "" : "Content-Type: " + type + "\r\n") + "\r\n" + bytes + "\r\n"; +} +static const std::string CLOSING = "--" + BOUNDARY + "--\r\n"; + +struct Corpus { + Request upload, blob, fields, urlencoded, json, put, get, del; + std::vector all() const { return {&upload, &blob, &fields, &urlencoded, &json, &put, &get, &del}; } +}; + +static Corpus corpus() { + const std::string pattern = binary(1500), small = binary(40); + Corpus c; + c.upload = request( + "multipart, a field then a file", "POST /form?src=query HTTP/1.1", "Host: 192.168.4.1\r\n" + MULTIPART, + field("note", "hello panel") + file("file", "wave.pfm", "application/octet-stream", pattern) + CLOSING, + "upload start file wave.pfm application/octet-stream\nupload write 1436\nupload write 64\nupload end 1500\n" + "handle /form\narg note=hello panel\narg src=query\nhost 192.168.4.1\n", + pattern); + // A file with no Content-Type of its own, named "blob" (what FormData calls + // a Blob), followed by another part: the parser's other way out of a file. + c.blob = request( + "multipart, a file then a field", "POST /form?filename=late.pfm HTTP/1.1", "Host: 192.168.4.1\r\n" + MULTIPART, + file("file", "blob", "", small) + field("note", "after the file") + CLOSING, + "upload start file late.pfm text/plain\nupload write 40\nupload end 40\n" + "handle /form\narg note=after the file\narg filename=late.pfm\nhost 192.168.4.1\n", + small); + c.fields = request( + "multipart, fields only", "POST /plain HTTP/1.1", "Host: patternflow.local\r\n" + MULTIPART, + field("ssid", "home net") + field("memo", "line one\r\nline two") + CLOSING, + "handle /plain\narg ssid=home net\narg memo=line one\nline two\nhost patternflow.local\n"); + c.urlencoded = request( + "urlencoded POST", "POST /plain?src=query HTTP/1.1", + "Host: 192.168.4.1\r\nContent-Type: application/x-www-form-urlencoded\r\n", "a=1&b=two+words&c=%41%2f", + "handle /plain\narg src=query\narg a=1\narg b=two words\narg c=A/\nhost 192.168.4.1\n"); + c.json = request( + "JSON POST", "POST /plain HTTP/1.1", "Host: 192.168.4.1\r\nContent-Type: application/json\r\n", + "{\"index\":3}", "handle /plain\narg plain={\"index\":3}\nhost 192.168.4.1\n"); + // 1500 is one full HTTP_RAW_BUFLEN and a short tail: the tail is what stock + // waited 5 s for (VENDORED.md, Fix 1). "arg size=1500" is in the log since + // Fix 5: stock never parsed a raw request's query string - see carriesOver(). + c.put = request( + "raw PUT", "PUT /raw?size=1500 HTTP/1.1", + "Host: 192.168.4.1\r\nX-PF-Name: wave.pfm\r\nContent-Type: application/octet-stream\r\n", pattern, + "raw start\nraw end 1500\nhandle /raw\narg size=1500\nheader X-PF-Name=wave.pfm\nhost 192.168.4.1\n", "", pattern); + c.get = request( + "GET", "GET /page?x=1&y=two HTTP/1.1", "Host: patternflow.local\r\nX-PF-Name: probe\r\nAccept: */*\r\n", "", + "handle /page\narg x=1\narg y=two\nheader X-PF-Name=probe\nhost patternflow.local\n"); + c.del = request( + "DELETE, no body", "DELETE /plain?name=old HTTP/1.1", "Host: 192.168.4.1\r\n", "", + "handle /plain\narg name=old\nhost 192.168.4.1\n"); + return c; +} + +enum class Peer { Closes, Waits }; // after its last byte: a FIN, or connected and saying nothing +static const char* said(Peer p) { return p == Peer::Closes ? "then the peer closes" : "then silence, peer connected"; } + +static Script cut(const std::string& bytes, size_t n, Peer then) { + Script s; + s.send(bytes.substr(0, n)); + if (then == Peer::Closes) s.close(); + return s; +} +static Script split(const std::string& bytes, size_t n, uint32_t gap) { + Script s; + s.send(bytes.substr(0, n)).wait(gap).send(bytes.substr(n)); + return s; +} + +// The request arrives in full, however slowly: it parses to exactly what a +// single segment parses to, and costs the time its last byte took to arrive +// and no more (2 ms is the upload wait's step). Empty when it does. +static std::string undelivered(const Request& q, const Script& script, uint32_t lastByteMs) { + Bench b; + const Result r = b.play(script); + const std::string why = unsound(r); + if (!why.empty()) return why; + if (!(r.seen == q.whole)) return "parsed to " + shown(r.seen) + "; expected " + shown(q.whole); + if (r.ms < lastByteMs || r.ms > lastByteMs + 2) + return "took " + std::to_string(r.ms) + " ms; its last byte arrived at " + std::to_string(lastByteMs); + return ""; +} +static void parses(const std::string& name, const Request& q, const Script& script, uint32_t lastByteMs) { + const std::string why = undelivered(q, script, lastByteMs); + if (!why.empty()) fail(name + ": " + why); +} + +// The request stops short: the parser gives up, in time, without handing the +// handler a body it did not get and without a reply. +static void refuses(const std::string& name, const Script& script, const std::string& log = "") { + Bench b; + const Result r = b.play(script); + if (!ends(name, r)) return; + if (r.seen.handled || !r.seen.reply.empty() || r.seen.log != log) + fail(name + ": expected no handler, no reply and log \"" + printable(log) + "\"; got " + shown(r.seen)); +} + +// A body callback told that a body began, and never told how it ended. The +// firmware's callbacks latch at the start - storage busy and the panel +// paused, the UPDATE card up - and let go only on an end or an abort. +static bool dropped(const Seen& s) { + const bool began = s.log.find("upload start") != std::string::npos || s.log.find("raw start") != std::string::npos; + return began && !s.handled && s.log.find(" aborted") == std::string::npos; +} + +static Script whole(const Request& q) { return cut(q.bytes, q.bytes.size(), Peer::Waits); } + +static void complete(const Corpus& c) { + // The peer stays connected and waits for its answer, as a browser does. A + // parser that waits for more than was promised shows up as fake time. + for (const Request* q : c.all()) parses(q->name, *q, whole(*q), 0); + if (!failures) + std::printf("PASS: %u well-formed requests reach their handlers whole and cost no fake time\n", + static_cast(c.all().size())); +} + +static void truncated(const Corpus& c) { + const int before = failures; + const std::string head = "POST /form HTTP/1.1\r\nHost: 192.168.4.1\r\n" + MULTIPART + "\r\n"; + const std::string opened = head + "--" + BOUNDARY + "\r\n"; + const std::string disposition = "Content-Disposition: form-data; name=\"note\"\r\n"; + for (Peer then : {Peer::Closes, Peer::Waits}) { + const std::string tail = std::string(", ") + said(then); + // Fix 4's request: the part-header loop had no way out but a part. + refuses("multipart: the first boundary" + tail, cut(opened, opened.size(), then)); + // ...and its other loop, which also grew the value by a byte a pass. + const std::string value = opened + disposition + "\r\nhello pan"; + refuses("multipart: a field value that never reaches its boundary" + tail, cut(value, value.size(), then)); + const std::string midHeader = opened + disposition.substr(0, 31); + refuses("multipart: a part header cut mid-line" + tail, cut(midHeader, midHeader.size(), then)); + const std::string midLine = head.substr(0, head.find("boundary=") + 4); + refuses("multipart: the request's header block cut mid-line" + tail, cut(midLine, midLine.size(), then)); + refuses("multipart: headers and no body" + tail, cut(head, head.size(), then)); + // A file delivered in full, and then the form stops: the callback that + // was told the file ended is told the upload did not. + refuses("multipart: a field cut short after a whole file" + tail, + cut(c.blob.bytes, c.blob.bytes.find("after the file") + 5, then), + "upload start file late.pfm text/plain\nupload write 40\nupload end 40\nupload aborted\n"); + + // A GET whose head stops is still handled (stock: nothing arriving ends + // the header block). Only the two properties are asked of it. + Bench b; + ends("GET: the header block cut mid-line" + tail, b.play(cut(c.get.bytes, c.get.bytes.find("X-PF-Name") + 6, then))); + + refuses("raw PUT: a body shorter than its Content-Length" + tail, cut(c.put.bytes, c.put.body + 700, then), + "raw start\nraw aborted 700\n"); + refuses("urlencoded POST: a body shorter than its Content-Length" + tail, + cut(c.urlencoded.bytes, c.urlencoded.bytes.size() - 5, then)); + + // A boundary far past the 70 characters a boundary may have. Stock sized + // a stack array from it - 9,005 bytes on an 8 KB task stack, a reboot on + // the board and a stack overflow here under the sanitizers (Fix 5). + const std::string wide(9000, 'B'); + const std::string oversized = "POST /form HTTP/1.1\r\nHost: 192.168.4.1\r\nContent-Type: multipart/form-data; boundary=" + + wide + "\r\n\r\n--" + wide + + "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.pfm\"\r\n\r\nxx"; + refuses("multipart: a 9,000-character boundary" + tail, cut(oversized, oversized.size(), then)); + } + + // A closing boundary without its CR/LF still closes the form. With the peer + // gone that is immediate; with the peer waiting the parser sits out one + // stream timeout for a line ending that is not coming, then answers. (After + // a file the parser asks connected() with the "--" still unread, the one + // state the socket model does not vouch for, so the form that ends in a + // file is only asked about with the peer still there.) + parses(c.fields.name + ": closing boundary without CR/LF, then the peer closes", c.fields, + cut(c.fields.bytes, c.fields.bytes.size() - 2, Peer::Closes), 0); + parses(c.fields.name + ": closing boundary without CR/LF, then silence", c.fields, + cut(c.fields.bytes, c.fields.bytes.size() - 2, Peer::Waits), 5000); + parses(c.upload.name + ": closing boundary without CR/LF, then silence", c.upload, + cut(c.upload.bytes, c.upload.bytes.size() - 2, Peer::Waits), 5000); + if (failures == before) + std::puts("PASS: a multipart, raw or urlencoded body that stops short is refused without a spin, closed or silent; " + "a closing boundary needs no CR/LF"); +} + +static void slow(const Corpus& c) { + const int before = failures; + // An upload that goes quiet mid-file and comes back. The wait for it is + // _uploadReadByte()'s, the loop that lost its braces (VENDORED.md, + // 2026-09-10) and spun for as long as the peer said nothing. + parses("multipart: 300 ms of silence inside the file", c.upload, split(c.upload.bytes, c.upload.body + 900, 300), 300); + parses("raw PUT: 300 ms of silence inside the body", c.put, split(c.put.bytes, c.put.body + 900, 300), 300); + // A link that delivers one byte every 3 ms: Fix 2's case, where the stream + // timeout never fires because the gap between two bytes is always short. + for (const Request* q : c.all()) { + Script s; + for (char byte : q->bytes) s.send(std::string(1, byte)).wait(3); + parses(q->name + ": one byte every 3 ms", *q, s, static_cast(3 * (q->bytes.size() - 1))); + } + if (failures == before) + std::puts("PASS: stalled and trickled requests parse to the same thing, every wait sleeping and polling"); +} + +// Every prefix of every request, ended both ways, and every two-segment +// delivery of it. Deterministic: the corpus is fixed and the cut points are +// all of them. +static void fuzz(const Corpus& c) { + unsigned replays = 0, mostLooks = 0, quietUploads = 0; + uint32_t longestMs = 0; + for (const Request* each : c.all()) { + const Request& q = *each; + for (Peer then : {Peer::Closes, Peer::Waits}) { + unsigned bad = 0; + std::string first; + for (size_t n = 1; n < q.bytes.size(); ++n, ++replays) { + Bench b; + const Result r = b.play(cut(q.bytes, n, then)); + std::string why; + if (r.end == Result::Held && r.seen.uploading) { + ++quietUploads; // pinned below + } else if (!unsound(r).empty()) { + why = unsound(r); + } else if (n >= q.body && r.seen.handled && !(r.seen == q.whole)) { + // A head that stops is a request without a body (stock). A body + // that stops must not reach the handler looking finished. + why = "handled a body that stopped short: " + shown(r.seen); + } else if (dropped(r.seen)) { + why = "gave the request up without telling the body callback: " + shown(r.seen); + } + if (r.end == Result::Returned) { + mostLooks = std::max(mostLooks, r.looks); + longestMs = std::max(longestMs, r.ms); + } + if (!why.empty() && !bad++) first = "cut after byte " + std::to_string(n) + ", " + said(then) + ": " + why; + } + if (bad) + fail("fuzz, " + q.name + ": " + std::to_string(bad) + " of " + std::to_string(q.bytes.size() - 1) + + " truncations fail; the first is " + first); + } + unsigned bad = 0; + std::string first; + for (size_t n = 1; n < q.bytes.size(); ++n, ++replays) { + const std::string why = undelivered(q, split(q.bytes, n, 40), 40); + if (!why.empty() && !bad++) first = "split after byte " + std::to_string(n) + ": " + why; + } + if (bad) + fail("fuzz, " + q.name + ": " + std::to_string(bad) + " of " + std::to_string(q.bytes.size() - 1) + + " two-segment deliveries (40 ms apart) fail; the first is " + first); + } + // The absence of a deadline in _uploadReadByte() is stock and was left alone + // on purpose when its braces were fixed ("a separate decision from + // yielding", Parsing.cpp). It sleeps every pass, so it is not the watchdog; + // it is the single connection held by an uploader that vanished without a + // FIN, until TCP keepalive notices - by lwIP's defaults (7,200 s idle, then + // nine probes 75 s apart) a little over two hours. Derived, not measured. + known(quietUploads > 0, + std::to_string(quietUploads) + " truncations leave a file part open with the peer silent, and the upload wait " + "has no deadline: HELD past " + std::to_string(REQUEST_BOUND_MS) + " ms (sleeping, not spinning)"); + if (mostLooks >= SPIN_LOOKS / 2 || longestMs > REQUEST_BOUND_MS - 5000) + fail("the bounds have lost their margin: a truncated request now looks at a dead socket " + + std::to_string(mostLooks) + " times in a row, or waits " + std::to_string(longestMs) + " ms"); + std::printf("%s: %u replays - every truncation point of every request, closed and silent, and every two-segment " + "delivery; the longest wait that ended was %u ms, the most looks at a dead socket %u\n", + failures ? "DONE" : "PASS", replays, static_cast(longestMs), mostLooks); +} + +// One server, several requests: what a request leaves behind for the next. +static void carriesOver(const Corpus& c) { + { + Bench b; + b.play(whole(c.get)); + b.probe = "x"; + const Result r = b.play(whole(c.fields)); + if (ends("a form after a GET", r) && r.seen.log.find("named x=(absent)") == std::string::npos) + fail("a form after a GET still sees the GET's arguments: " + shown(r.seen)); + } + { + // _parseForm() gives its fields to the request only when the form + // completes, and arg() and hasArg() look in _postArgs first. A form + // refused after a field used to leave it there, answering for the + // arguments of every plain request until the next multipart one (Fix 4). + Bench b; + const std::string aborted = "POST /form HTTP/1.1\r\nHost: 192.168.4.1\r\n" + MULTIPART + "\r\n" + + field("note", "left behind") + file("file", "wave.pfm", "", "stops here"); + b.play(cut(aborted, aborted.size(), Peer::Closes)); + b.probe = "note"; + const Result r = b.play(whole(c.get)); + if (ends("a GET after a refused form", r) && r.seen.log.find("named note=(absent)") == std::string::npos) + fail("a GET after a refused form is answered with the form's field: " + shown(r.seen)); + } + { + // The raw path used not to call _parseArguments(), so a raw request had + // no arguments of its own and args()/arg() answered with the previous + // request's. core_web_update.h asks a raw PUT for "size" (Fix 5). + Bench b; + b.play(whole(c.get)); + b.probe = "size"; + const Result r = b.play(whole(c.put)); + if (ends("a raw PUT after a GET", r) && + (r.seen.log.find("arg x=1\n") != std::string::npos || r.seen.log.find("named size=1500") == std::string::npos)) + fail("a raw PUT after a GET does not see its own query string, or still sees the GET's: " + shown(r.seen)); + } + { + // FunctionRequestHandler::canRaw() is true for any route with a body + // callback that is not a GET, whatever the callback was written for. A + // POST that is not multipart, sent to a multipart route, used to be + // delivered to the upload callback as a raw body, where upload() is a + // null reference: core_web_update.h and core_patterns_http.h read + // .status through it, and `curl -X POST /api/patterns` panicked a board. + // It is a plain request now: handled, and no body callback (Fix 5). + Bench b; + const Request post = request("", "POST /form HTTP/1.1", "Host: 192.168.4.1\r\nContent-Type: text/plain\r\n", "x=1", ""); + const Result r = b.play(whole(post)); + if (ends("a POST that is not multipart, to a multipart route", r) && + (!r.seen.handled || r.seen.log.find("raw start") != std::string::npos || + r.seen.log.find("upload ") != std::string::npos)) + fail("a POST that is not multipart, to a route with an upload callback, reached that callback or was not " + "handled: " + shown(r.seen)); + } +} + +} // namespace bench + +int main() { + using namespace bench; + const Corpus requests = corpus(); + complete(requests); + truncated(requests); + slow(requests); + fuzz(requests); + carriesOver(requests); + std::fflush(stdout); + if (failures) { + std::printf("%d failure(s)\n", failures); + std::fflush(stdout); + // A replay that tripped was thrown out of the parser mid-allocation. The + // leak report that would follow says nothing the lines above have not. + std::_Exit(1); + } + return 0; +} diff --git a/firmware/toolchain/tests/runtime_test.cpp b/firmware/toolchain/tests/runtime_test.cpp index 7e2e5036..879dc718 100644 --- a/firmware/toolchain/tests/runtime_test.cpp +++ b/firmware/toolchain/tests/runtime_test.cpp @@ -14,10 +14,14 @@ #ifdef _MSC_VER #define __ATOMIC_ACQUIRE 0 #define __ATOMIC_RELEASE 0 +#define __ATOMIC_ACQ_REL 0 template T __atomic_load_n(T* p, int) { return std::atomic_ref(*p).load(); } template void __atomic_store_n(T* p, V v, int) { std::atomic_ref(*p).store(static_cast(v)); } +template T __atomic_exchange_n(T* p, V v, int) { + return std::atomic_ref(*p).exchange(static_cast(v)); +} #endif constexpr unsigned MALLOC_CAP_INTERNAL=1, MALLOC_CAP_8BIT=2, MALLOC_CAP_SPIRAM=4; constexpr unsigned MALLOC_CAP_EXEC=8, MALLOC_CAP_32BIT=16; @@ -25,9 +29,14 @@ static size_t internalFree=100000, largest=100000; static bool externalFails=false; static std::map internalOwned; static size_t heap_caps_get_free_size(unsigned) { return internalFree; } -static size_t heap_caps_get_largest_free_block(unsigned) { return largest; } +static size_t externalLargest=1u<<20; +static size_t heap_caps_get_largest_free_block(unsigned caps) { + if (caps & MALLOC_CAP_SPIRAM) return externalFails ? 0 : externalLargest; + return largest; +} +static size_t externalAsked=0; static void* heap_caps_malloc(size_t n, unsigned caps) { - if (caps & MALLOC_CAP_SPIRAM) return externalFails ? nullptr : malloc(n); + if (caps & MALLOC_CAP_SPIRAM) { externalAsked=n; return externalFails ? nullptr : malloc(n); } if (n>internalFree || n>largest) return nullptr; void* p=malloc(n); assert(p); internalOwned[p]=n; internalFree-=n; return p; @@ -41,6 +50,7 @@ static void heap_caps_free(void* p) { free(p); } #include "core_module_memory.h" +#include "sidecar_name.h" using TaskHandle_t = void*; static thread_local TaskHandle_t task=reinterpret_cast(1); @@ -49,9 +59,19 @@ static uint32_t micros() { return static_cast(std::chrono::duration_cast( std::chrono::steady_clock::now().time_since_epoch()).count()); } -static uint32_t millis() { return micros()/1000; } +// A stalled loop is told by the clock, and nobody waits ten real seconds for +// one: real time supplies the 25 ms slices, leapMs the stall. Taken from the +// 64-bit count rather than micros()/1000, which steps back to zero every 71 +// minutes - a step the loop's age would read as 49 days without a beat. +static std::atomic leapMs{0}; +static uint32_t millis() { + return static_cast(std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count())+leapMs; +} constexpr unsigned portMAX_DELAY=0xffffffffu, pdTRUE=1, pdPASS=1; #define pdMS_TO_TICKS(x) (x) +// One tick. Shortened where a case wants its caller to look thousands of times. +static std::atomic tickUs{1000}; struct Semaphore { std::mutex m; std::condition_variable cv; bool ready=false; }; using SemaphoreHandle_t=Semaphore*; static Semaphore* xSemaphoreCreateBinary() { return new Semaphore; } @@ -59,7 +79,7 @@ static Semaphore* xSemaphoreCreateMutex() { auto p=new Semaphore; p->ready=true; static unsigned xSemaphoreTake(Semaphore* s,unsigned ms) { std::unique_lock lock(s->m); if(ms==portMAX_DELAY) s->cv.wait(lock,[&]{return s->ready;}); - else if(!s->cv.wait_for(lock,std::chrono::milliseconds(ms),[&]{return s->ready;})) return 0; + else if(!s->cv.wait_for(lock,std::chrono::microseconds(uint64_t(ms)*tickUs),[&]{return s->ready;})) return 0; s->ready=false; return pdTRUE; } static void xSemaphoreGive(Semaphore* s) { @@ -68,6 +88,42 @@ static void xSemaphoreGive(Semaphore* s) { struct Logger { void println(const char*) {} template void printf(const char*,T...) {} } Serial; namespace PFRuntime { inline void noteSync(uint32_t) {} } #include "core_loop_sync.h" +// A thread standing in for the network task. Its maintenance hook runs once a +// slice, just before the caller decides whether to give up on the loop - the +// one moment these cases need to meet - and `looks` counts them. With +// `rendezvous` set the caller is held at that point until the loop says go, +// and for `callerLag` spins more, so the two can be let at the request in the +// same instant whichever of them is the quicker off the mark. +static std::atomic looks{0}; +static std::atomic rendezvous{false}, atDoor{false}, go{false}; +static std::atomic callerLag{0}; +// A few nanoseconds a spin, in work the optimiser has to leave in: a loop of +// loads whose value nobody used compiled to nothing, and the aim never moved. +static void spin(int spins) { + static thread_local std::atomic turns{0}; + for(; spins>0; --spins) ++turns; +} +static void becomeCaller() { + task=reinterpret_cast(2); + PFNetMaintenance::attach([]{ + ++looks; + if(!rendezvous) return; + atDoor=true; + while(!go) {} + spin(callerLag); + atDoor=false; + }); +} +static bool callerLooked(unsigned times) { + const unsigned from=looks; + const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(10); + while(looks-fromlimit) return false; + std::this_thread::yield(); + } + return true; +} +static bool posted() { return __atomic_load_n(&PFLoopSync::pendingFn,__ATOMIC_ACQUIRE)!=nullptr; } constexpr int MODULE_PATH_BYTES=96, MODULE_NAME_BYTES=64, NUM_PRESETS=1, PF_CUSTOM_SLOT_COUNT=0; struct PatternEntry { const char* name; const char* modulePath; }; @@ -117,7 +173,8 @@ int main() { externalFails=true; internalFree=PF_MODULE_INTERNAL_RESERVE+99; assert(!PFModuleMemory::data(100,true,true)); // PSRAM failure cannot bypass reserve internalFree=100000; largest=50; - assert(!PFModuleMemory::code(100)); // fragmentation despite ample total heap + void* held=nullptr; + assert(!PFModuleMemory::code(100,&held) && !held); // fragmentation despite ample total heap // Admission is on the SUM of the module's executable sections, and refuses // without allocating anything - no allocate-then-roll-back. largest=100000; externalFails=false; @@ -143,6 +200,89 @@ int main() { for(int i=0;i<128;++i) assert(static_cast(p)[i]==0); heap_caps_free(p); + // Code has a second home. With the fallback policy, what fits internally + // still goes there and is charged; what does not fit moves to PSRAM, is + // charged nothing, and leaves the whole budget to the module's data. + internalFree=PF_MODULE_INTERNAL_RESERVE+4000; largest=100000; + PFModuleMemory::codePolicy=PF_MODULE_CODE_PSRAM_FALLBACK; + assert(PFModuleMemory::admitCode(2500) && !PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==1500); + assert(PFModuleMemory::admitCode(5000) && PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==4000); + void* far=PFModuleMemory::code(5000,&held); + assert(far && internalOwned.empty() && internalFree==PF_MODULE_INTERNAL_RESERVE+4000); + // ...in a block that owns whole cache lines, so the write-back never + // touches a line the allocator or a neighbour is using. + // One line of slack, the code on a line boundary inside the allocation: + // every line it occupies is the allocation's own. + assert(externalAsked==5056+64 && reinterpret_cast(far)%64==0); + assert(static_cast(far)>=static_cast(held) && + static_cast(far)+5056<=static_cast(held)+externalAsked); + heap_caps_free(held); + // Room in total is not a block: code refused for fragmentation is exactly + // what the second home is for. + largest=2000; + assert(PFModuleMemory::admitCode(2500) && PFModuleMemory::codeExternal); + largest=100000; + // Services under the reserve: budget is zero and code still has somewhere. + internalFree=PF_MODULE_INTERNAL_RESERVE-1; + assert(PFModuleMemory::admitCode(1) && PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==0); + // PSRAM-first: code never takes internal RAM while PSRAM can hold it. + internalFree=PF_MODULE_INTERNAL_RESERVE+4000; + PFModuleMemory::codePolicy=PF_MODULE_CODE_PSRAM_FIRST; + assert(PFModuleMemory::admitCode(2500) && PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==4000); + // No PSRAM, or no block that large: both policies fall back to the internal + // rule exactly - same verdict, same budget, nothing allocated to find out. + externalLargest=2000; + assert(PFModuleMemory::admitCode(2500) && !PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==1500); + externalFails=true; externalLargest=1u<<20; + assert(!PFModuleMemory::admitCode(5000) && !PFModuleMemory::codeExternal); + assert(PFModuleMemory::dataBudget==0 && internalOwned.empty()); + PFModuleMemory::codePolicy=PF_MODULE_CODE_PSRAM_FALLBACK; + assert(!PFModuleMemory::admitCode(5000) && !PFModuleMemory::codeExternal); + // The old rule is still there to be chosen. + externalFails=false; + PFModuleMemory::codePolicy=PF_MODULE_CODE_INTERNAL; + assert(!PFModuleMemory::admitCode(5000) && !PFModuleMemory::codeExternal); + // ...and is what a unit falls back to once a PSRAM placement has failed to + // verify: the same pick then lands internally instead of failing again. + PFModuleMemory::codePolicy=PF_MODULE_CODE_PSRAM_FIRST; + PFModuleMemory::codeDemoted=true; + assert(PFModuleMemory::codeRule()==PF_MODULE_CODE_INTERNAL); + assert(PFModuleMemory::admitCode(2500) && !PFModuleMemory::codeExternal); + assert(!PFModuleMemory::admitCode(5000)); + PFModuleMemory::codeDemoted=false; + PFModuleMemory::codePolicy=PF_MODULE_CODE_POLICY; PFModuleMemory::endLoad(); + internalFree=100; + + // A sidecar's name, as the build writes it and as a person might. + { + char name[64]; + assert(jsonStringValue("Wave Saw\", \"abi\": 2}", name, sizeof(name)) && !strcmp(name,"Wave Saw")); + // An escaped quote is part of the name, not the end of it. + assert(jsonStringValue("Say \\\"hi\\\" \\\\ there\"", name, sizeof(name))); + assert(!strcmp(name,"Say \"hi\" \\ there")); + // The build spells non-ASCII as \uXXXX; the stored name is UTF-8. + assert(jsonStringValue("Dynamic Moir\\u00e9\"", name, sizeof(name))); + assert(!strcmp(name,"Dynamic Moir\xC3\xA9")); + assert(jsonStringValue("\\ud328\\ud134\"", name, sizeof(name)) && !strcmp(name,"\xED\x8C\xA8\xED\x84\xB4")); + assert(jsonStringValue("\\ud83c\\udf0a\"", name, sizeof(name)) && !strcmp(name,"\xF0\x9F\x8C\x8A")); + // A character that does not fit is left out whole, and nothing after it. + char tight[6]; + assert(jsonStringValue("abcd\\u00e9z\"", tight, sizeof(tight)) && !strcmp(tight,"abcd")); + assert(jsonStringValue("ab\xED\x8C\xA8z\"", tight, sizeof(tight)) && !strcmp(tight,"ab\xED\x8C\xA8")); + // Control escapes vanish; what never closes, or holds nothing, is no name. + assert(jsonStringValue("a\\nb\\tc\"", name, sizeof(name)) && !strcmp(name,"abc")); + assert(!jsonStringValue("never closed", name, sizeof(name))); + assert(!jsonStringValue("ends in a backslash\\", name, sizeof(name))); + assert(!jsonStringValue("bad \\u12 escape\"", name, sizeof(name))); + assert(!jsonStringValue("\"", name, sizeof(name))); + assert(!jsonStringValue("\\n\\t\"", name, sizeof(name))); + } + PFLoopSync::attach(); int attempts=0, commits=0; assert(!PFLoopSync::runWhen([&]{ ++attempts; return false; })); @@ -156,6 +296,159 @@ int main() { while(!finished) { PFLoopSync::service(); ++frames; std::this_thread::yield(); } caller.join(); assert(commits==1 && frames>=4); + // The loop stops. Nothing services, the stamp turns PF_LOOP_STALL_MS old, + // and the caller takes its request back and is told it did not run. What it + // took back must never run - not when the loop wakes, not ever: the lambda + // it pointed at went with the caller's stack frame. + { + int ran=0; bool answered=true; + std::thread stuck([&]{ becomeCaller(); answered=PFLoopSync::run([&]{ ++ran; }); }); + while(!posted()) std::this_thread::yield(); + assert(!PFLoopSync::stalled() && PFLoopSync::gaveUp==0); + leapMs+=PF_LOOP_STALL_MS; + stuck.join(); + assert(!answered && ran==0 && PFLoopSync::gaveUp==1 && PFLoopSync::stalled()); + PFLoopSync::service(); + assert(ran==0 && !PFLoopSync::stalled()); + } + // How long a caller has waited is not evidence. A transaction the loop + // keeps testing outlasts the limit - a module's setup() is seconds - for as + // long as the loop keeps arriving. When the loop then stops, it is taken + // back like any other, and the attempt is not tried again. + { + int tries=0; bool committed=true; + std::thread waiting([&]{ becomeCaller(); committed=PFLoopSync::runWhen([&]{ ++tries; return false; }); }); + while(!posted()) std::this_thread::yield(); + for(int frame=1; frame<=3; ++frame) { + PFLoopSync::service(); + leapMs+=PF_LOOP_STALL_MS/2; + assert(callerLooked(2) && posted() && tries==frame); + } + leapMs+=PF_LOOP_STALL_MS; + waiting.join(); + assert(!committed && tries==3 && PFLoopSync::gaveUp==2); + PFLoopSync::service(); + assert(tries==3); + } + // A call the loop has taken is waited out, however dead the loop looks from + // outside while it is in there: the body is running on the caller's stack + // frame. And the loop stamps on its way out, or the next request would find + // a stalled loop one frame before it answered. + { + int ran=0; std::atomic returned=false; + std::thread waiter([&]{ + becomeCaller(); + const bool ok=PFLoopSync::run([&]{ + leapMs+=2*PF_LOOP_STALL_MS; + assert(callerLooked(2) && !returned); + ++ran; + }); + assert(ok); returned=true; + }); + while(!ran) PFLoopSync::service(); + assert(!PFLoopSync::stalled()); + waiter.join(); assert(ran==1 && PFLoopSync::gaveUp==2); + } + // The age is never read from the future: with the loop stamping flat out, + // a reader that took the clock before the stamp would see 49 days. Only a + // millisecond turning over between its two reads can show that, so this + // runs across three hundred of them. + { + std::atomic read=false; + std::thread reader([&]{ + const auto until=std::chrono::steady_clock::now()+std::chrono::milliseconds(300); + while(std::chrono::steady_clock::now() live{false}; std::atomic ran{0}; }; + static Call calls[4000]; + tickUs=4; + std::atomic over=false; + unsigned completed=0; std::atomic withdrawn=0; + std::thread racer([&]{ + becomeCaller(); + const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(3); + for(Call& call : calls) { + if(std::chrono::steady_clock::now()>limit) break; + call.live=true; + const bool ok=PFLoopSync::runRaw([](void* p){ + Call* mine=static_cast(p); assert(mine->live); ++mine->ran; + }, &call); + call.live=false; + // Told the truth, and nothing of this call left on the table. + assert(call.ran==(ok?1u:0u) && !posted()); + if(ok) ++completed; else ++withdrawn; + } + over=true; + }); + uint32_t dice=1; unsigned lost=0; + int lead=0; // spins the loop gives the caller; negative, the caller gives the loop + int step=16; bool callerWon=false; + rendezvous=true; + while(!over) { + if(!posted()) continue; + // Go quiet for a stall's worth, hold the caller at the point of looking, + // let both go at once. How much start one needs over the other to make + // a tie is the machine's business, so aim: the loop a little later + // after a call it got to first, a little sooner after one the caller + // took back, in steps that halve each time the winner changes. The two + // exchanges then keep meeting, which a fixed delay does only by luck. + const bool won=withdrawn!=lost; lost=withdrawn; + if(won!=callerWon && step>1) step/=2; + callerWon=won; + if(won ? lead>-100000 : lead<100000) lead+=won ? -step : step; + dice=dice*1664525u+1013904223u; + const int jitter=int(dice>>30); + callerLag=lead<0 ? jitter-lead : 0; + leapMs+=PF_LOOP_STALL_MS; + while(!atDoor && !over) {} + go=true; + spin(lead>0 ? lead+jitter : 0); + PFLoopSync::service(); + while(atDoor) {} + go=false; + } + racer.join(); rendezvous=false; tickUs=1000; + PFLoopSync::service(); + unsigned bodies=0; + for(Call& call : calls) bodies+=call.ran; + assert(bodies==completed && PFLoopSync::gaveUp==2+withdrawn && !posted()); + printf("loop hand-off race: %u ran, %u withdrawn\n", completed, withdrawn.load()); + // How hard this case tries depends on the machine. Under g++ on Linux it + // gets about 4,000 rounds and a non-atomic take on either side fails it + // every run; under MSVC, whose waits are coarse, it gets about 200 and the + // same mutation has passed. CI (g++, sanitizers) is the gate for the race, + // a local run on Windows is not. + // Both ways, or the two never met and everything above passed for want of + // a race. The aim swings back each time one side wins, so with a core each + // both must turn up - they did with a third thread spinning on the same + // two cores. On a single core the threads take turns instead of tying + // (the loop won 366 of 366 pinned to one), and a machine that has only + // the one is let off; pinned to one core of many with taskset this still + // fails, which is the honest answer for a run that raced nothing. + if(std::thread::hardware_concurrency()>1) assert(completed && withdrawn); + } + createFails=true; assert(!activatePatternAsync(2)); assert(activePatternIdx==1 && PFModuleLoader::active && PFModuleLoader::unloads==0); @@ -195,5 +488,5 @@ int main() { loadPatternJob(); assert(!loadResult && PFModuleLoader::loads-beforeLoads==1 && retryPauseMs==0); delete PFLoopSync::doneSignal; delete PFLoopSync::callerLock; - puts("PASS: reserve/fallback/fragmentation/race, deferred loop requests, task failure, mutation hold, rescan and deleted selection"); + puts("PASS: reserve/fallback/fragmentation/race, deferred loop requests, stalled-loop hand-off, task failure, mutation hold, rescan and deleted selection"); } diff --git a/firmware/toolchain/tests/thumbs_test.cpp b/firmware/toolchain/tests/thumbs_test.cpp index 045ff029..c0e2a56e 100644 --- a/firmware/toolchain/tests/thumbs_test.cpp +++ b/firmware/toolchain/tests/thumbs_test.cpp @@ -34,7 +34,11 @@ static void* heap_caps_calloc(size_t n, size_t s, unsigned caps) { } static uint32_t micros() { static uint32_t now = 0; return ++now; } namespace PFCanvas { constexpr int W = 128, H = 64; uint8_t buffer[W * H * 3]; } -namespace PFLoopSync { template void run(F&& f) { f(); } } +// The loop runs the body, or has stopped coming round and never will. +static bool loopGone = false; +namespace PFLoopSync { +template bool run(F&& f) { if (loopGone) return false; f(); return true; } +} constexpr bool FILE_READ = false, FILE_WRITE = true; static unsigned fileOpens = 0; @@ -143,6 +147,20 @@ int main() { failWrites = false; capture("alloc", true); service(); finish(); assert(find("alloc")->savedThisBoot); + // A delete while the loop is not answering cannot touch the cache, so it + // leaves a note. A save queued before the stall must not put the picture + // back on the volume, and the loop empties the cache when it returns. + colour(255, 0, 0); assert(capture("stuck", true)); service(); + assert(ioState == IO_PENDING); + loopGone = true; forget("stuck"); loopGone = false; + assert(dropAllDeferred && find("stuck")->px); + serviceDisk(); assert(!FFat.exists("/patterns/stuck.thumb")); + service(); assert(!dropAllDeferred && slotCount == 0 && ioState == IO_IDLE); + assert(get("stuck") == nullptr); finish(); + loopGone = true; forgetAll(); loopGone = false; + assert(dropAllDeferred && slotCount == 1); + service(); assert(!dropAllDeferred && slotCount == 0); + forgetAll(); free(slots); slots = nullptr; - puts("thumbnail mailbox: read/capture, immutable save, deletion, format, short I/O, PSRAM OOM passed"); + puts("thumbnail mailbox: read/capture, immutable save, deletion, format, short I/O, PSRAM OOM, stalled loop passed"); }