Memory errors during parsing may trigger a debug abort

[stestagg]

Crash report

What happened?

import os
import sys
import _testcapi

source = b'''class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})'''

for counter in range(200, 1200):
    print("Trying:", counter)
    sys.stdout.flush()
    _testcapi.set_nomemory(counter, counter + 1)
    try:
        _testcapi.run_compilestringflags(source, b"/missing.py", _testcapi.Py_file_input)
    except Exception:
        pass
    _testcapi.remove_mem_hooks()

For me, I get (ymmv):

Trying: 200
Trying: 201
Trying: 202
...
Trying: 340
Trying: 341
python3: Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.

This is related to these issues/prs:

• gh-113602
• gh-100050
• gh-112388

A fix was put in for this type of issue already in _PyPegen_raise_error_known_location:

cpython/Parser/pegen_errors.c

Lines 276 to 279 in e8158d1

	// Bail out if we already have an error set.
	if (p->error_indicator && PyErr_Occurred()) {
	return NULL;
	}

and _PyPegen_raise_error:

cpython/Parser/pegen_errors.c

Lines 185 to 187 in e8158d1

	if (p->error_indicator && PyErr_Occurred()) {
	return NULL;
	}

But in this case, PyErr_Occurred() == True but p->error_indicator=0, because the allocation failure happened in an area of the code that could backtrack, it seems like the error_indicator isn't set.

Two code patterns that hit both sites are:

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
f(*E(), **{&a": 3})

and

class E:
    def __iter__(self):
        yield 1
        yield from ()
def f(a, b=0, *args, **kwargs): pass
with f(*E())
    pass

(The with here is the key)

So, to my mind, there are two questions:

• Is it valid to get to either of these places with PyErr_Occurred() but not p->error_indicator?
• If it is, then does the check just become checking PyErr_Occurred()?
• If not, then the places where the parser checks for memory error and returns should probably always set the parser error_indicator. I guess technically a mis-predicted parsing branch could cause a MemoryError, and then backtrack and find the correct branch without triggering another MemoryError?? but istm that if you hit an allocation failure on a backtrackable branch, the only valid thing to do is to bail out.

LLDB info

LLDB

(lldb) thread list
Process 2661 stopped
* thread #1: tid = 2661, 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)

(lldb) bt all
* thread #1, name = 'fuzz_python', stop reason = SIGABRT: sent by tkill system call (sender pid=2661, uid=0)
  * frame #0: 0x0000fffff692a848 libc.so.6`___lldb_unnamed_symbol_8a720 + 296
    frame #1: 0x0000fffff68d7af8 libc.so.6`raise + 24
    frame #2: 0x0000fffff68c2630 libc.so.6`abort + 40
    frame #3: 0x0000fffff691dd84 libc.so.6`___lldb_unnamed_symbol_7dbe0 + 420
    frame #4: 0x0000fffff68d13ac libc.so.6`__assert_fail + 204
    frame #5: 0x0000aaaaaaee0834 fuzz_python`_PyObject_Call(tstate=0x0000aaaaab776358, callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:342:5
    frame #6: 0x0000aaaaaad35d70 fuzz_python`PyObject_Call(callable=0x0000aaaaab6f4100, args=0x0000fffff4eb8f50, kwargs=0x0000000000000000) at call.c:373:12 [inlined]
    frame #7: 0x0000aaaaaad35d5c fuzz_python`PyErr_SetFromErrnoWithFilenameObjects(exc=0x0000aaaaab6f4100, filenameObject=0x0000fffff507eb60, filenameObject2=0x0000000000000000) at errors.c:900:13
    frame #8: 0x0000aaaaab0f5a7c fuzz_python`PyErr_SetFromErrnoWithFilenameObject(exc=<unavailable>, filenameObject=0x0000fffff507eb60) at errors.c:811:12 [inlined]
    frame #9: 0x0000aaaaab0f5a78 fuzz_python`Py_fopen(path=0x0000fffff507eb60, mode="rb") at fileutils.c:1817:9
    frame #10: 0x0000aaaaaad3ce30 fuzz_python`_PyErr_ProgramDecodedTextObject(filename=0x0000fffff507eb60, lineno=7, encoding=0x0000000000000000) at errors.c:2064:16
    frame #11: 0x0000aaaaab23e734 fuzz_python`_PyPegen_raise_error_known_location(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, lineno=7, col_offset=12, end_lineno=7, end_col_offset=13, errmsg="Invalid star expression", va=<unavailable>) at pegen_errors.c:302:22
    frame #12: 0x0000aaaaab23e360 fuzz_python`_PyPegen_raise_error(p=0x0000fffff516a740, errtype=0x0000aaaaab6f6bd8, use_mark=0, errmsg="Invalid star expression") at pegen_errors.c:221:5
    frame #13: 0x0000aaaaab2fad34 fuzz_python`invalid_starred_expression_rule(p=0x0000fffff516a740) at parser.c:27408:20
    frame #14: 0x0000aaaaab2fa614 fuzz_python`starred_expression_rule(p=0x0000fffff516a740) at parser.c:18983:47
    frame #15: 0x0000aaaaab2d59a8 fuzz_python`_tmp_89_rule(p=0x0000fffff516a740) at parser.c:34278:39
    frame #16: 0x0000aaaaab2d58a0 fuzz_python`genexp_rule(p=0x0000fffff516a740) at parser.c:18464:18
    frame #17: 0x0000aaaaab2f3e68 fuzz_python`t_primary_raw(p=0x0000fffff516a740) at parser.c:20208:18
    frame #18: 0x0000aaaaab2f3400 fuzz_python`t_primary_rule(p=0x0000fffff516a740) at parser.c:20077:22
    frame #19: 0x0000aaaaab34a674 fuzz_python`single_subscript_attribute_target_rule(p=0x0000fffff516a740) at parser.c:19968:18
    frame #20: 0x0000aaaaab32773c fuzz_python`_tmp_11_rule(p=0x0000fffff516a740) at parser.c:29309:54
    frame #21: 0x0000aaaaab2b9878 fuzz_python`assignment_rule(p=0x0000fffff516a740) at parser.c:2196:18
    frame #22: 0x0000aaaaab2b97cc fuzz_python`simple_stmt_rule(p=0x0000fffff516a740) at parser.c:1620:31
    frame #23: 0x0000aaaaab2ae8c0 fuzz_python`simple_stmts_rule(p=0x0000fffff516a740) at parser.c:1515:18
    frame #24: 0x0000aaaaab2aafc0 fuzz_python`statement_rule(p=0x0000fffff516a740) at parser.c:1297:34
    frame #25: 0x0000aaaaab2aad28 fuzz_python`_loop1_2_rule(p=0x0000fffff516a740) at parser.c:28730:30
    frame #26: 0x0000aaaaab2aab60 fuzz_python`statements_rule(p=0x0000fffff516a740) at parser.c:1230:18
    frame #27: 0x0000aaaaab2a7578 fuzz_python`file_rule(p=0x0000fffff516a740) at parser.c:1032:18
    frame #28: 0x0000aaaaab2a7510 fuzz_python`_PyPegen_parse(p=0x0000fffff516a740) at parser.c:39618:18
    frame #29: 0x0000aaaaab22d3cc fuzz_python`_PyPegen_run_parser(p=0x0000fffff516a740) at pegen.c:1019:9
    frame #30: 0x0000aaaaab22f308 fuzz_python`_PyPegen_run_parser_from_string(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", start_rule=257, filename_ob=0x0000fffff507eb60, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=<unavailable>) at pegen.c:1145:14
    frame #31: 0x0000aaaaab0f1c14 fuzz_python`_PyParser_ASTFromString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, mode=257, flags=0x0000000000000000, arena=0x0000fffff51772e0, module=0x0000000000000000) at peg_api.c:14:21 [inlined]
    frame #32: 0x0000aaaaab0f1bd8 fuzz_python`_Py_CompileString(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1, module=0x0000000000000000) at pythonrun.c:1578:11
    frame #33: 0x0000aaaaab0f1ef0 fuzz_python`Py_CompileStringObject(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename=0x0000fffff507eb60, start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1561:12 [inlined]
    frame #34: 0x0000aaaaab0f1ed4 fuzz_python`Py_CompileStringExFlags(str="class E:\n    def __iter__(self):\n        yield 1\n        yield from ()\ndef f(a, b=0, *urnc, **d):\n    return a, b, c, d\nf(*E(), **{&a\": 3})", filename_str="<fuzz>", start=257, flags=0x0000000000000000, optimize=-1) at pythonrun.c:1608:10
    frame #35: 0x0000aaaaaad15054 fuzz_python`main(argc=<unavailable>, argv=<unavailable>) at fuzz_python.c:576:30
    frame #36: 0x0000fffff68c2d94 libc.so.6`___lldb_unnamed_symbol_22d20 + 116
    frame #37: 0x0000fffff68c2ef8 libc.so.6`__libc_start_main + 152
    frame #38: 0x0000aaaaaad06370 fuzz_python`_start + 48

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x0000000000000a65
        x2 = 0x0000000000000006
        x3 = 0x0000000000000022
        x4 = 0xffffffffffffffff
        x5 = 0x0000000000000000
        x6 = 0x0000aaaaaab9e165  
        x7 = 0x0000fffff69f9750  
        x8 = 0x0000000000000083
        x9 = 0x5210160bf182cd92
       x10 = 0x0000000000000021
       x11 = 0x0000fffff6a66618  
       x12 = 0x0000000000000480
       x13 = 0x0000fffff7f03740
       x14 = 0x0000fffff51036f0
       x15 = 0x0000000000000001
       x16 = 0x0000000000000001
       x17 = 0x0000fffff693a36c  libc.so.6`__libc_free
       x18 = 0x0000000000000002
       x19 = 0x0000000000000a65
       x20 = 0x0000fffff7f03020
       x21 = 0x000000000000000b
       x22 = 0x0000000000000006
       x23 = 0x0000000000000000
       x24 = 0x000000000000009d
       x25 = 0x0000ffffffff7a38
       x26 = 0x0000fffff69f9740  
       x27 = 0x0000000000000002
       x28 = 0x0000fffff51a0000
        fp = 0x0000ffffffff7900
        lr = 0x0000fffff692a834  libc.so.6`___lldb_unnamed_symbol_8a720 + 276
        sp = 0x0000ffffffff78f0
        pc = 0x0000fffff692a848  libc.so.6`___lldb_unnamed_symbol_8a720 + 296
      cpsr = 0x60001000


(lldb) memory region $sp
[0x0000fffffffdd000-0x0001000000000000) rw-

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main-dirty:20e6c2f, Aug 19 2026, 15:22:30) [Clang 22.1.8 ]

[picnixz]

It's very unlikely you run out of memory while parsing, and if this is the case, I think it's the least of your problems, but it's worth propogating the error if possible.

We could change the way we read the error indicator so that error_indicator = 1 means a faliure outside backtracking and error_indicator = 2 means a failure that is unrecoverable (and thus backtracking must not continue) but I'm not sure if it's the best design approach.

I guess technically a mis-predicted parsing branch could cause a MemoryError, and then backtrack and find the correct branch without triggering another MemoryError?? but istm that if you hit an allocation failure on a backtrackable branch, the only valid thing to do is to bail out.

If a mis-predicated parsing branch causes a MemoryError, you can have the same with the correctly predicted one if someone is allocating just some memory elsewhere at the same time. So it doesn't matter here. If you hit a MemoryError, you should stop doing what you're doing.

[stestagg]

@picnixz agreed, this is definitely a corner case! but the similar issue was already addressed.

Is there a reason why we can't just set error_indicator to 1 on any parser allocation failure? as you say, it's not really a case where trying to recover inside the parser seems sensible.

[picnixz]

I don't know. I would expect error_indicator to be synchronized. But there could be a few places that don't do it properly. For instance, there are also various places that ignored the error value, e.g.,

cpython/Parser/pegen.c

Line 652 in e8158d1

PyBytes_AsStringAndSize(t->bytes, &the_token, &size);

(but PyBytes_AsStringAndSize doesn't raise a MemoryError, or maybe it can raise one when setting another kind of exception?)

Can you check if by any chance we are missing a p->error_indicator=1 somewhere? since parser.c (I think it's parser) is autogenerated, you'll need to look at the generator. And otherwise the bug could be in a helper function.

cc @pablogsal Is there a reason why the condition p->error_indicator == 0 && PyErr_Occurred() is valid? (or vice-versa, p->error_indicator == 1 && !PyErr_Occurred()).

[stestagg]

The original crash came from here:
invalid_starred_expression - which doesn't really know about the allocation failure at this level, it's just any parse error.

But the actual memory error comes from _PyPegen_insert_memo which handles the allocation failure by returning -1 but doesn't set the parser error flag (question is whether it should or not):

cpython/Parser/pegen.c

Lines 91 to 97 in e8158d1

	_PyPegen_insert_memo(Parser *p, int mark, int type, void *node)
	{
	// Insert in front
	Memo *m = _PyArena_Malloc(p->arena, sizeof(Memo));
	if (m == NULL) {
	return -1;
	}

---

LLM Output

---

I'm going to be really lazy and just paste some loosely edited chatgpt output here (sorry!) but it seems to be concise and accurate as far as my knowledge goes here. (I think other people with more familiarity with the parser would know more). And it can read the codebase faster than I can. It found a couple of other areas it thinks might also have issues:

I found two additional concrete parser primitives:

_PyPegen_name_from_token() and _PyPegen_number_token()
Both _PyAST_Name() returns can fail while allocating the AST node:
return _PyAST_Name(..., p->arena) / _PyAST_Constant(c, NULL, ..., p->arena)
A failed arena allocation returns NULL with MemoryError, but does not set p->error_indicator.
These should probably use CHECK: return CHECK(expr_ty, _PyAST_Constant(...));

The Memoization path has more than the _PyPegen_update_memo() case
There are three generated cases:

• Left-recursive rules check _PyPegen_update_memo() and return _res on failure, but do not set error_indicator.
• Ordinary (memo) rules ignore _PyPegen_insert_memo()’s return value entirely.
• The generator also ignores insertion failure for memoized loop rules, although the current Python grammar does not appear to generate one of these.
  The ordinary memo case is used by 20 rules in the current grammar, including simple_stmt, expression, factor, arguments, and strings.
  For example, generated rules end with:
  _PyPegen_insert_memo(p, _mark, simple_stmt_type, _res);
  return _res;
  That can even return a non-NULL result with MemoryError pending.
  The cleanest memoization fix is probably inside _PyPegen_insert_memo() itself:

if (m == NULL) {
    p->error_indicator = 1;
    return -1;
}

That automatically covers _PyPegen_update_memo() too. The generator should arguably still stop immediately when direct insertion fails.

[pablogsal]

Will look into this later this night