fix for issue1232 - #1233
Conversation
bnaras
left a comment
There was a problem hiding this comment.
Thank you @pascoec!
I built the issue1232 branch (46732b2) on macOS arm64 (16 cores, Homebrew libomp) and ran it against the harness from #1224, extended with your worker-thread scenario.
Compared three builds: stock dev, the #1224 live-ICV clamp, and this PR. The probe spawns the exact hot-region clause — num_threads(OpenFHEParallelControls.GetThreadLimit(1 << 15)) — and counts the team size; the corroborating workload is 50 top-level CKKS EvalMults (depth 20, ring dim 2^16) under SetNumThreads(2).
| scenario | stock dev | #1224 clamp | this PR |
|---|---|---|---|
bare omp_set_num_threads(2), region on main thread |
16 | 2 (same-thread only) | 16 — as your usage note says |
SetNumThreads(2), region on main thread |
16 | 2 | 2 |
SetNumThreads(2) on main, region entered from a std::thread worker |
16 | 16 | 2 |
EvalMult workload CPU/wall under SetNumThreads(2) |
4.26 | 1.89 | 1.89 |
You're right about the thread-local ICV — reproduced against my own patch: after capping on the main thread, omp_get_max_threads() from a fresh std::thread reports 16 and the hot region spawns a full 16-thread team. The process-global atomic in this PR closes exactly that hole: the same worker-thread entry spawns 2.
Also read through the rest of the diff: every library-side parallel region now carries the GetThreadLimit clause (the only remaining bare #pragma omp parallel sites are in unittests, examples/parallel.cpp, and commented-out code), and the Strassen NUM_THREADS latch now agrees with the capped team size its schedule chunk math assumes.
Confirmed: this resolves my use case. The R package needs an in-process cap (CRAN's check farm limits packages to 2 cores, and a package cannot set the environment before its own process starts), and SetNumThreads(2) under this PR caps the hot path process-wide, with the workload ratio matching what the #1224 clamp achieved on the main thread. I'll switch the R wrapper from bare omp_set_num_threads() to OpenFHEParallelControls::SetNumThreads() and drop our carried patch when we next rebase onto a release containing this.
Noted on workflow as well — future PRs will come from an in-repo branch (or with "Allow edits by maintainers" checked).
Fixes #1232.
Problem
OpenFHEParallelControls::SetNumThreads(n)is silently ignored by the library'sparallel hot regions. Every hot region is emitted as
#pragma omp parallel for num_threads(GetThreadLimit(size)), and per the OpenMPspec an explicit
num_threads(...)clause overrides the nthreads ICV thatomp_set_num_threads()sets. SinceGetThreadLimit()compares only against athread count latched once at library load, there is no in-process way to cap the
library. Related defects in
parallel.h:thread-local, so a cap set on one thread never applies to regions entered from
another (e.g. FHE work on a worker pool). This is also why clamping by a live
omp_get_max_threads()insideGetThreadLimit()would be insufficient.UnitTestStart()halves the wrong base — it reads the current mutable ICV, sorepeated calls compound (16 → 8 → 4) and any prior cap skews it; its throttle
is itself bypassed by the
GetThreadLimitregions.UnitTestStop()unconditionally restoresmachineThreads, stomping any capthe application had set.
omp_set_num_threads(0)is reachable (viaSetNumThreads(0), orUnitTestStart()with one thread available); the spec requires a positiveargument.
switching) have no
num_threadsclause at all, so they follow only thethread-local ICV.
Changes
src/core/include/utils/parallel.h:std::atomic<int> threadLimitholds the process-global cap;GetThreadLimit()mins against it (relaxed load, ~0.3 ns — measured). A cachedmember is preferred over reading the live ICV because the ICV is per-thread:
after
omp_set_num_threads(2)on one thread, another thread still reads themachine default.
SetNumThreads()clamps to[1, machineThreads], stores the cap, and stillsets the ICV for clause-less regions.
Enable()/Disable()route throughSetNumThreads()so they actually governthe hot regions.
UnitTestStart()usesGetNumProcs()/2(stable hardware base, immune toOMP_NUM_THREADSover/undersubscription) and saves the current cap;UnitTestStop()restores it.Enable(); constructing aParallelControlshas no side effects on global OpenMP state.
GetNumThreads()is now a relaxed atomic load.Added
num_threads(OpenFHEParallelControls.GetThreadLimit(<trip count>))to the27 clause-less regions in
matrix.h,matrix-impl.h,matrixstrassen.h,matrixstrassen-impl.h,dgsampling-impl.h, andckksrns-schemeswitching.cpp, so all regions obey the same process-global cap.The Strassen
NUM_THREADSlatch now readsOpenFHEParallelControls.GetNumThreads()so itsschedulechunk math matchesthe capped team size.