gh-75876: Limit the address space of a bigmem test

A test run with -M now runs in a subprocess whose address space is limited
to what it declares plus 512 MiB.  A test which uses much more memory than
it declares fails with a MemoryError instead of making the machine swap.

Pass limit_address_space=False for a test which reserves much more address
space than it uses.  The two threaded tests in test_interpreters do: glibc
reserves an arena per thread, up to 8 per CPU, and this alone exceeds the
declared size on a machine with many cores.

Author: serhiy-storchaka

Doc/library/test.rst

@@ -835,7 +835,7 @@ The :mod:`!test.support` module defines the following functions:
    the trace function.
 
 
-.. decorator:: bigmemtest(size, memuse, dry_run=True)
+.. decorator:: bigmemtest(size, memuse, dry_run=True, *, limit_address_space=True)
 
    Decorator for bigmem tests.
 
@@ -849,6 +849,12 @@ The :mod:`!test.support` module defines the following functions:
    method may be less than the requested value.  If *dry_run* is ``False``, it
    means the test doesn't support dummy runs when ``-M`` is not specified.
 
+   A test which really allocates the memory it asks for runs in a separate
+   process, whose address space is limited to what the test declares plus a
+   margin.
+   Set *limit_address_space* to ``False`` for a test which reserves much more
+   address space than it uses, for example one which starts many threads.
+
 
 .. decorator:: bigaddrspacetest
 

Lib/test/support/__init__.py

@@ -1270,6 +1270,27 @@ def set_memlimit(limit: str) -> None:
     max_memuse = memlimit
 
 
+def _limit_address_space(nbytes):
+    """Limit the address space of this process to *nbytes* plus a margin.
+
+    A test which uses much more memory than it declares then fails with a
+    MemoryError instead of making the machine swap.
+    """
+    try:
+        import resource
+        rlimit = resource.RLIMIT_AS
+    except (ImportError, AttributeError):
+        return
+    # The margin does not grow with the declared size: the interpreter itself
+    # reserves about 250 MiB, whatever the test asks for.
+    limit = int(nbytes) + 512 * _1M
+    soft, hard = resource.getrlimit(rlimit)
+    for current in soft, hard:
+        if current != resource.RLIM_INFINITY:
+            limit = min(limit, current)
+    resource.setrlimit(rlimit, (limit, hard))
+
+
 def _memory_watchdog(pid):
     """Return a function printing the memory usage of process *pid*."""
     # Imported here: test.support does not depend on test.libregrtest.
@@ -1283,7 +1304,7 @@ def watch():
     return watch
 
 
-def bigmemtest(size, memuse, dry_run=True):
+def bigmemtest(size, memuse, dry_run=True, *, limit_address_space=True):
     """Decorator for bigmem tests.
 
     'size' is a requested size for the test (in arbitrary, test-interpreted
@@ -1299,6 +1320,12 @@ def bigmemtest(size, memuse, dry_run=True):
     A test that actually allocates the requested memory (that is, one run with
     -M) runs in a subprocess, so that the memory it uses and the address space
     it fragments are released when it ends.  A dummy run stays in the process.
+
+    The address space of that subprocess is limited to what the test declares
+    plus a margin, so that a test which uses much more memory than it declares
+    fails instead of making the machine swap.  Pass 'limit_address_space' as
+    false for a test which reserves much more address space than it uses, for
+    example one which starts many threads.
     """
     def decorator(f):
         from test.support import isolation
@@ -1336,6 +1363,11 @@ def wrapper(self):
                 isolation._replay_test(self, *proc.wait(tick=watchdog))
                 return
 
+            if (real_max_memuse and limit_address_space
+                    and isolation.runningInSubprocess):
+                # Only in the subprocess: the limit is never lifted.
+                _limit_address_space(size * memuse)
+
             return f(self, maxsize)
 
         wrapper.size = size

Lib/test/test_interpreters/test_stress.py

@@ -26,7 +26,8 @@ def test_create_many_sequential(self):
         support.gc_collect()
 
     @threading_helper.requires_working_threading()
-    @support.bigmemtest(size=200, memuse=32*2**20, dry_run=False)
+    @support.bigmemtest(size=200, memuse=32*2**20, dry_run=False,
+                        limit_address_space=False)
     def test_create_many_threaded(self, size):
         alive = []
         start = threading.Event()
@@ -43,7 +44,8 @@ def task():
         support.gc_collect()
 
     @threading_helper.requires_working_threading()
-    @support.bigmemtest(size=200, memuse=34*2**20, dry_run=False)
+    @support.bigmemtest(size=200, memuse=34*2**20, dry_run=False,
+                        limit_address_space=False)
     def test_many_threads_running_interp_in_other_interp(self, size):
         start = threading.Event()
         interp = interpreters.create()

Misc/NEWS.d/next/Tests/2026-08-18-19-36-04.gh-issue-75876.5AhLHL.rst

@@ -0,0 +1,5 @@
+A test decorated with :func:`~test.support.bigmemtest` now runs with the
+address space limited to what it declares plus 512 MiB, so that a test which
+uses much more memory than it declares fails instead of making the machine
+swap.  Pass ``limit_address_space=False`` for a test which reserves much more
+address space than it uses.