ESP32 integration suites exceed #806 900s cap on Windows; zccache persist_failed (tempdir nondeterminism) makes every framework rebuild cold

[zackees]

Summary

The #[ignore]d ESP32 integration suites (esp32_build, eh_frame_strip_esp32) cannot complete a single clean esp32dev framework build within #806's 900 s cap on this Windows machine — and the compile backend's zccache layer gives them no relief, for two independent reasons discovered while repairing the #800 backend-install gap (phase-8b PR, #1346):

1. Sequential framework builds blow the 900 s cap

The esp32dev framework stage compiles sequentially at roughly 4 s/file on this machine (Windows + xtensa-gcc). The Arduino-ESP32 core is ~1500 files → ~100 minutes per cold build, vs the 900 s budget both eh_frame_strip_drops_firmware_at_least_150kb (two builds!) and the four esp32_build blink/fixture tests need. Observed twice: timeout panic at exactly the 900 s cap, 938 s and 1055 s wall respectively.

2. zccache never hits for these runs — stores are refused

Embedded-zccache journals (~/.fbuild/prod/zccache/v1.13.1/logs/) show the esp32 framework compiles are effectively never cached:

journal window	hit	miss	error
rotation ending 2026-08-20 12:58	1	1761	91
rotation ending 2026-08-20 23:45	0	1838	2
current (2026-08-22 runs)	0	1558+	—

Two mechanisms in daemon-lifecycle.log:

• daemon_cache_root_contended — a test process's embedded backend does not own the cache-root writer slot whenever any other fbuild process touched it recently; its stores then can't be committed locally.
• persist_failed: "same cache key produced different staged output bytes" — the decisive one. Test builds run in a fresh tempfile::TempDir each invocation; the same content-keyed compile produces different output bytes across runs (almost certainly DWARF DW_AT_comp_dir/debug paths embedding the per-run tempdir), so zccache correctly refuses to persist them. Every subsequent run recompiles everything.

Real-daemon builds (stable project dirs) don't hit mechanism 2 — this is specific to fresh-tempdir test harnesses.

Impact

• All seven repaired-from-perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800 integration files now install a test backend; six go green (avr ×5, cache_survives ×3, clangd parity, teensy/nxplpc pending re-measure). The two ESP32 suites time out at the cap on Windows regardless of warm toolchain/cargo state.
• CI (Linux) may still fit the cap; these suites are #[ignore]d there anyway — so the gap is invisible until someone runs --include-ignored locally.

Suggested fixes (in dependency order)

1. Deterministic debug paths for cached framework compiles — -fdebug-prefix-map-style normalization (or drop -g for framework objects) so identical keys produce identical bytes and persist_failed stops eating every store.
2. Test-process cache isolation — route test-launched CompileBackends at a scratch/dev cache root (the way FBUILD_DEV_MODE=1 isolates), so tests neither contend for nor pollute the prod writer slot.
3. Parallelize the ESP32 framework stage (or revisit the sequential pipeline wall on Windows) — even fully cached-first-run builds will otherwise stay near the cap.
4. Optionally: have the orchestrator emit per-stage progress into the test-visible log so future timeouts are diagnosable without the deleted-tempdir problem noted in compile_many_stage2_perf oracle: unrunnable since #800; with backend installed, stage-2 wall ~6.5s fails 50%-of-stage-1 bound — #337 seed regression or stale baseline #1346.

Repro

soldr cargo test -p fbuild-build --test eh_frame_strip_esp32 -- --include-ignored
# → panicked: real-toolchain test exceeded 900s budget — see FastLED/fbuild#806

Found during phase-8b verification; cross-referenced from #1346.

[zackees]

Investigation note on suggested fix 1 (deterministic debug paths), since the obvious implementation breaks a core board.

The flag choice is forced, and it is not the modern one

-ffile-prefix-map is the flag you would reach for — it normalizes __FILE__, __builtin_FILE() and debug paths in one go. It was added in GCC 8. fbuild's pinned toolchains:

toolchain	version	-fdebug-prefix-map	-ffile-prefix-map
avr	7.3.0-atmel3.6.1-arduino7	yes (GCC 4.3+)	no — predates it
arm	15.2.Rel1	yes	yes
arm_gcc8	9.2.1	yes	yes
esp32	14.2.0_20241119	yes	yes
esp8266	3.2.0-gcc10.3	yes	yes
riscv	14.2.0-3	yes	yes

So -ffile-prefix-map would break every AVR build. That matters more than it did last week: since #1396, uno is one of three core boards that run on every shared-code PR, so an AVR break is now maximally visible rather than caught by a nightly.

-fdebug-prefix-map is the portable choice — supported by all six, including avr-gcc 7.3.0 — and it is sufficient for the failure in this issue. The persist_failed cause named above is DW_AT_comp_dir / debug line paths, which is exactly what -fdebug-prefix-map rewrites. __FILE__ is not implicated, because a tempdir-varying __FILE__ would show up as differing source text rather than differing staged output for an identical key.

Consequence worth planning for

Adding any flag changes the compile command, which changes the zccache key for every object. The first build after it lands is a full cold rebuild on every platform — including the CI caches. Worth landing deliberately (and probably not alongside anything else) rather than discovering it as an unexplained slow day.

What I did not do

I have not implemented this. Two things should be settled first by someone who can run the suites:

1. Confirm the mechanism empirically. The issue infers DW_AT_comp_dir "almost certainly". That is very likely right, but the fix is a global compile-flag change and the confirmation is cheap: run the same compile in two different tempdirs and objdump --dwarf=info | grep comp_dir the two objects, or just cmp them and bisect the differing bytes. If something other than debug paths differs, -fdebug-prefix-map will not fix persist_failed and the cold rebuild will have been spent for nothing.

2. Decide what to map to. The map target has to be stable across runs and not collide between concurrently-building projects. -fdebug-prefix-map=<abs build root>=/fbuild is the obvious shape, but the build root already varies per env/profile, so the substitution needs to be chosen where the layout is known (BuildLayout), not per call site.

Suggested fix 2 (test-process cache isolation) is independent of all of this and does not carry the cold-rebuild cost — it may be the better first move.

[stevenmunn312-tech]

A data point from outside the integration suites, in case it is useful: the same wall
shows up in an ordinary hand-run build.

fbuild 2.5.21, Windows 11, a throwaway project with no libraries at all and this sketch,
built for esp32-s3-devkitc-1 (opi PSRAM, 16MB):

#include <Arduino.h>
void setup() { pinMode(2, OUTPUT); }
void loop() { digitalWrite(2, HIGH); delay(500); digitalWrite(2, LOW); delay(500); }

build succeeded in 736.2s (flash: 370527 bytes, ram: 74507 bytes)

Twelve minutes to cold-build a blink. That is consistent with the ~4s-per-file figure you
give, and it is not a test harness — just a fresh project directory. I did not capture the
full log, so I cannot confirm whether persist_failed appeared in it; the timing is the
only claim I am making here.

For contrast on the same project, the immediately following no-op build took 161.2s while
writing nothing. I have filed that separately as #1411, since a fingerprint hit that
still costs three minutes looks like a different problem from the cold-rebuild one you
describe here.