DO NOT MERGE: MEMFS stress on 2 CPUs with a shared-log writer - #6758
Closed
Fedr wants to merge 2 commits into
Closed
DO NOT MERGE: MEMFS stress on 2 CPUs with a shared-log writer#6758Fedr wants to merge 2 commits into
Fedr wants to merge 2 commits into
Conversation
Every stall has been on a 2-core runner and every failed repro ran on a 4-core one, so emrun goes under taskset -c 0,1. The other untested difference is that the app has ONE thread writing zip entries next to ANOTHER holding a log file open and flushing every line, which the previous stress lacked: each thread owned its own files. Workers now leave a pool slot for that appender, since asking for more threads than PTHREAD_POOL_SIZE deadlocks the test itself.
My copy of MRTest.vcxproj was made from an older master and dropped the entries for MRMeshToPointCloudTests.cpp and MRRayBoxIntersectionTests.cpp, which the vcxproj-missing-files hook caught in prepare-config, skipping every build in the run.
Contributor
Author
|
Answered: no reproduction. Per shard 123,212 file rounds alongside 9,293,346 flushed writes to one shared file, on 2 CPUs under Caveat on fidelity: Fourth failed attempt (after 49,600 serial loads, 200 app-level opens of the exact archive, and ~890k concurrent rounds on 4 cores). Closing; the stall is not reproducible synthetically and only the real app hits it, at ~1/1200. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway experiment -- do not merge, do not review. Deleted once it has answered.
Fourth attempt at a reproducer for the wasm import stall, aimed at the two differences between the failing environment and every attempt that came back clean.
Two CPUs. Every stall has been on MIC's private
ubuntu-latest, which is 2 cores. #6659's 49,600 loads and #6701's ~890,000 filesystem rounds all ran on 4-core runners, so nothing has been tested at the contention the app actually sees. emrun now runs undertaskset -c 0,1.A shared file, held open, flushed per line. The app pairs one thread writing zip entries with another holding a log file open and flushing every message --
setupLoggerByDefaultinstalls arotating_file_sink_mtand callsflush_on( minLevel ). #6701 gave every thread its own files, so it never had that. There is now an appender thread doing exactly that against one shared file while the workers write their own.Workers leave a pool slot for the appender: asking for more threads than
PTHREAD_POOL_SIZEdeadlocks the test itself, as the first run of #6701 demonstrated (threads past the pool need a Worker spawned on demand, which needs the main thread at the browser event loop, which the watchdog never allows).480 s per shard, 4 shards on each multithreaded config. The watchdog exits 3 if the round counter stops for 60 s, printing every thread's tally and the log-line count.
Expectation: maybe one in four. If it comes back clean, the upstream report goes in with the two bracketed traces and an explicit "no minimal reproducer", listing what has been ruled out.
Refs MeshInspector/MeshInspectorCode#7724.