gh-75876: Limit the address space of a bigmem test - #156023
Open
serhiy-storchaka wants to merge 5 commits into
Open
gh-75876: Limit the address space of a bigmem test#156023serhiy-storchaka wants to merge 5 commits into
serhiy-storchaka wants to merge 5 commits into
Conversation
A test run with -M now runs in a subprocess whose address space is limited to what it declares plus 512 MiB. A test which uses much more memory than it declares fails with a MemoryError instead of making the machine swap. Pass limit_address_space=False for a test which reserves much more address space than it uses. The two threaded tests in test_interpreters do: glibc reserves an arena per thread, up to 8 per CPU, and this alone exceeds the declared size on a machine with many cores.
Documentation build overview
4 files changed± library/tarfile.html± library/test.html± tutorial/stdlib.html± whatsnew/changelog.html |
AddressSanitizer reserves terabytes of address space for its shadow memory, so any limit stops the interpreter from starting. macOS reserves much more address space than Linux does, so 512 MiB is not enough of a margin there.
Member
Author
|
!buildbot bigmem |
|
🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 8bc2c01 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F156023%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
repr() of a list or a tuple of 2**31 False takes 15.3 bytes per item, not 15, and str.split() takes 10.7 bytes per item and more for a larger string, not 10. The difference is small, but it grows with the size of the test: 614 MiB for test_repr_large.
…lares The address space cannot be limited everywhere: not under AddressSanitizer, which reserves terabytes for its shadow memory, and not on macOS, which reserves much more than it uses. The parent process already watches the memory of the subprocess running the test, so let it kill the test which uses more than it declares plus the same margin. This also catches a test which fills the memory without reserving that much address space.
The memory watchdog now both reports the peak (pythonGH-156024) and kills a test which uses more memory than it declares.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A test run with
-Mnow runs with the address space limited to what it declares plus 512 MiB, so that a test which uses much more memory than it declares fails with aMemoryErrorinstead of making the machine swap.The margin does not grow with the declared size: the interpreter itself reserves about 250 MiB, whatever the test asks for.
limit_address_space=Falseexempts a test which reserves much more address space than it uses. The two threaded tests intest_interpretersneed it: glibc reserves an arena per thread, up to 8 per CPU, and this alone exceeds the declared size on a machine with many cores.