Skip to content

DO NOT MERGE: MEMFS stress on 2 CPUs with a shared-log writer - #6758

Closed
Fedr wants to merge 2 commits into
masterfrom
repro/memfs-2cpu
Closed

DO NOT MERGE: MEMFS stress on 2 CPUs with a shared-log writer#6758
Fedr wants to merge 2 commits into
masterfrom
repro/memfs-2cpu

Conversation

@Fedr

@Fedr Fedr commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Throwaway experiment -- do not merge, do not review. Deleted once it has answered.

Fourth attempt at a reproducer for the wasm import stall, aimed at the two differences between the failing environment and every attempt that came back clean.

Two CPUs. Every stall has been on MIC's private ubuntu-latest, which is 2 cores. #6659's 49,600 loads and #6701's ~890,000 filesystem rounds all ran on 4-core runners, so nothing has been tested at the contention the app actually sees. emrun now runs under taskset -c 0,1.

A shared file, held open, flushed per line. The app pairs one thread writing zip entries with another holding a log file open and flushing every message -- setupLoggerByDefault installs a rotating_file_sink_mt and calls flush_on( minLevel ). #6701 gave every thread its own files, so it never had that. There is now an appender thread doing exactly that against one shared file while the workers write their own.

Workers leave a pool slot for the appender: asking for more threads than PTHREAD_POOL_SIZE deadlocks the test itself, as the first run of #6701 demonstrated (threads past the pool need a Worker spawned on demand, which needs the main thread at the browser event loop, which the watchdog never allows).

480 s per shard, 4 shards on each multithreaded config. The watchdog exits 3 if the round counter stops for 60 s, printing every thread's tally and the log-line count.

Expectation: maybe one in four. If it comes back clean, the upstream report goes in with the two bracketed traces and an explicit "no minimal reproducer", listing what has been ruled out.

Refs MeshInspector/MeshInspectorCode#7724.

Every stall has been on a 2-core runner and every failed repro ran on a
4-core one, so emrun goes under taskset -c 0,1. The other untested
difference is that the app has ONE thread writing zip entries next to
ANOTHER holding a log file open and flushing every line, which the
previous stress lacked: each thread owned its own files.

Workers now leave a pool slot for that appender, since asking for more
threads than PTHREAD_POOL_SIZE deadlocks the test itself.
My copy of MRTest.vcxproj was made from an older master and dropped the
entries for MRMeshToPointCloudTests.cpp and MRRayBoxIntersectionTests.cpp,
which the vcxproj-missing-files hook caught in prepare-config, skipping
every build in the run.
@Fedr

Fedr commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Answered: no reproduction. Per shard 123,212 file rounds alongside 9,293,346 flushed writes to one shared file, on 2 CPUs under taskset -- ~985k rounds and ~74M log writes across 8 shards, all green.

Caveat on fidelity: hardware_concurrency() still returns 4 under taskset, so the pool stayed 4 while only 2 CPUs ran. Matching the failing environment's pool of 2 would need a dom.maxHardwareConcurrency pref in a custom emrun profile.

Fourth failed attempt (after 49,600 serial loads, 200 app-level opens of the exact archive, and ~890k concurrent rounds on 4 cores). Closing; the stall is not reproducible synthetically and only the real app hits it, at ~1/1200.

@Fedr Fedr closed this Sep 2, 2026
@Fedr
Fedr deleted the repro/memfs-2cpu branch September 2, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant