Skip to content

Observed memory leak in ssl library: Python 3.14 GC issue #142516

Closed
Closed
Observed memory leak in ssl library: Python 3.14 GC issue#142516
Labels
extension-modulesC modules in the Modules dirtopic-SSLtype-bugAn unexpected behavior, bug, or error

Description

@HenriBlacksmith
HenriBlacksmith
opened on Dec 10, 2025

Bug report

Bug description:

I already opened a bug on urllib3 for this issue, it looks that may application using MSAL for Python library (which relies on requests , which relies on urllib3, which relies on ssl) is leaking memory when being upgraded from Python 3.13 to Python 3.14 (I tested 3.14.{0, 1, 2})

The details are explained in the urllib3 ticket including outputs from memray and it points to ssl package (precisely ssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

Activity

  1. HenriBlacksmith commented on Dec 17, 2025

    @HenriBlacksmith
    HenriBlacksmith
    on Dec 17, 2025
    Author

    I tested the same program under Python 3.13.11+debug and here is the memory graph:

    Image

    The pattern shows something like GC is happening

    Image

    I updated the MRE repo to reflect my tests, no need to use Docker, just running those two programs on my system shows differences

  2. vstinner commented on Dec 17, 2025

    @vstinner
    vstinner
    on Dec 17, 2025
    Member

    Do you also see the leak on Python 3.14 if you call gc.collect() time to time?

  3. HenriBlacksmith commented on Dec 17, 2025

    @HenriBlacksmith
    HenriBlacksmith
    on Dec 17, 2025
    Author

    Here is the same code with 3.14.2+debug running the same number of iterations (the other graph was on a shorter time frame):

    Image

    I will see to run with a call to gc.collect() every iteration

    import httpx
import asyncio
import gc


async def _main() -> None:
    for i in range(500):
        async with httpx.AsyncClient() as client:
            await client.request("GET", "https://example.com/")
        await asyncio.sleep(1)
        gc.collect()


if __name__ == "__main__":
    asyncio.run(_main())

  4. HenriBlacksmith commented on Dec 17, 2025

    @HenriBlacksmith
    HenriBlacksmith
    on Dec 17, 2025
    Author

    Here are the results for the code above:

    Image Image

  5. vstinner commented on Dec 17, 2025

    @vstinner
    vstinner
    on Dec 17, 2025
    Member

    So if you call explicitly gc.collect(), there is no more leak?

    It seems like Python 3.14 GC behaves differently.

    cc @nascheme @markshannon

  6. sergey-miryanov commented on Dec 17, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 17, 2025
    Contributor

    I would like to look into this if no one mind.

  7. vstinner commented on Dec 17, 2025

    @vstinner
    vstinner
    on Dec 17, 2025
    Member

    Reproducer modified to use www.google.com and display the RSS memory usage at each iteration:

    import httpx
import asyncio
import gc
import os


def get_rss():
    with open(f"/proc/{os.getpid()}/status") as fp:
        for line in fp:
            if line.startswith("VmRSS"):
                rss = line.split(":", 1)[1].rstrip()
                rss = rss.removesuffix(" kB")
                return int(rss)
    return None


async def _main() -> None:
    for i in range(500):
        async with httpx.AsyncClient() as client:
            await client.request("GET", "https://www.google.com/")
        await asyncio.sleep(0.2)
        #gc.collect()
        rss = get_rss()
        print(f"Loop #{i}: {rss:,} kB")


if __name__ == "__main__":
    asyncio.run(_main())

    Output, the script leaks around 1 MiB per iteration:

    Loop #0: 40,272 kB
Loop #1: 41,140 kB
Loop #2: 42,080 kB
Loop #3: 43,008 kB
Loop #4: 44,076 kB
Loop #5: 44,932 kB
(...)
Loop #48: 85,000 kB
Loop #49: 85,912 kB
Loop #50: 86,840 kB
(...)

    If I uncomment gc.collect() line, the memory usage is basically stable:

    Loop #0: 40,208 kB
Loop #1: 40,992 kB
Loop #2: 41,136 kB
Loop #3: 41,140 kB
Loop #4: 41,148 kB
Loop #5: 41,152 kB
(...)
Loop #48: 41,668 kB
Loop #49: 41,668 kB
Loop #50: 41,668 kB
(...)
  8. vstinner
    changed the title [-]Observed memory leak in ssl library[/-] [+]Observed memory leak in ssl library: Python 3.14 GC issue[/+] on Dec 17, 2025
  9. pradyunsg
    added
    3.14bugs and security fixes
    on Dec 17, 2025

  10. zweizeichen commented on Dec 17, 2025

    @zweizeichen
    zweizeichen
    on Dec 17, 2025

    Thanks for putting in the time and effort! Unfortunately, I did not really have the time to look into this sooner.

    I can confirm sprinkling gc.collect() in strategic locations (non-critical not that hot code paths that get called regularly in long-running processes, e.g. liveness probe endpoint) in the codebase alleviates the issue. It's not pretty but at least a workaround until a potential fix makes it into a 3.14.x release - still better than downgrading to 3.13 :D

  11. nascheme commented on Dec 17, 2025

    @nascheme
    nascheme
    on Dec 17, 2025
    Member

    Some additional info: 3.13 memory use stable, 3.14.1 grows, 3.14.2 grows, 3.14.2t grows. I initially suspected this would be fixed by #142001 but since it leaks for both the 3.14.1 release and for 3.14.2t, that's not it. It's also should not be the "incremental GC" added to 3.14. Needs additional investigation to find why adding gc.collect() fixes the leak.

  12. sergey-miryanov commented on Dec 17, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 17, 2025
    Contributor

    @nascheme On main (1eddef8) I see that rss only grows, on my gc-fix branch it looks like a saw https://gist.github.com/sergey-miryanov/8e8d11311d154ab2c7656b8e546e64aa

  13. sergey-miryanov commented on Dec 17, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 17, 2025
    Contributor

    Following doesn't leak on main (92243dc):

    import httpx
import psutil
import asyncio
import gc
import os
import urllib.parse


def get_rss():
    p = psutil.Process()
    mem_info = p.memory_info()
    vms = mem_info.vms
    rss = mem_info.rss
    return rss / 1024.0


async def _main() -> None:
    for i in range(120):
        url = urllib.parse.urlsplit('https://www.google.com/')
        if url.scheme == 'https':
            reader, writer = await asyncio.open_connection(
                url.hostname, 443, ssl=True)
        else:
            reader, writer = await asyncio.open_connection(
                url.hostname, 80)

        query = (
            f"HEAD {url.path or '/'} HTTP/1.0\r\n"
            f"Host: {url.hostname}\r\n"
            f"\r\n"
        )

        writer.write(query.encode('latin-1'))
        while True:
            line = await reader.readline()
            if not line:
                break

        # Ignore the body, close the socket
        writer.close()
        await writer.wait_closed()

        await asyncio.sleep(0.1)
        #gc.collect()
        rss = get_rss()
        print(f"Loop #{i}: {rss:,} kB")


if __name__ == "__main__":
    asyncio.run(_main())


Loop #100: 40,100.0 kB
Loop #101: 40,592.0 kB
Loop #102: 40,604.0 kB
Loop #103: 40,612.0 kB
Loop #104: 40,616.0 kB
Loop #105: 40,616.0 kB
Loop #106: 40,088.0 kB
Loop #107: 40,520.0 kB
Loop #108: 40,532.0 kB
Loop #109: 40,596.0 kB
Loop #110: 40,068.0 kB
Loop #111: 40,544.0 kB
Loop #112: 40,128.0 kB
Loop #113: 40,608.0 kB
Loop #114: 40,644.0 kB
Loop #115: 40,088.0 kB
Loop #116: 40,536.0 kB
Loop #117: 40,564.0 kB
Loop #118: 40,072.0 kB
Loop #119: 40,528.0 kB

  14. sergey-miryanov commented on Dec 18, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 18, 2025
    Contributor

    @nascheme Memory grows on 3.13.1 and 3.13.9, but it stabilizes at some point:

    Details 
    
(.venv) ➜  3.13.1 python ../../issues/b142516.py  
Loop #0: 34,360.0 kB
Loop #1: 35,152.0 kB
Loop #2: 35,944.0 kB
Loop #3: 36,736.0 kB
Loop #4: 37,792.0 kB
Loop #5: 38,584.0 kB
Loop #6: 39,376.0 kB
Loop #7: 40,168.0 kB
Loop #8: 40,960.0 kB
Loop #9: 41,752.0 kB
Loop #10: 42,808.0 kB
Loop #11: 43,600.0 kB
Loop #12: 44,392.0 kB
Loop #13: 45,184.0 kB
Loop #14: 45,976.0 kB
Loop #15: 46,768.0 kB
Loop #16: 47,824.0 kB
Loop #17: 47,824.0 kB
Loop #18: 47,824.0 kB
Loop #19: 47,824.0 kB
Loop #20: 47,824.0 kB
Loop #21: 47,824.0 kB
Loop #22: 47,824.0 kB
Loop #23: 47,824.0 kB
Loop #24: 47,824.0 kB
Loop #25: 47,824.0 kB
Loop #26: 47,824.0 kB
Loop #27: 47,824.0 kB
Loop #28: 47,824.0 kB
Loop #29: 47,824.0 kB
Loop #30: 47,824.0 kB
Loop #31: 47,824.0 kB
Loop #32: 47,824.0 kB
Loop #33: 48,676.0 kB
Loop #34: 49,468.0 kB
Loop #35: 50,260.0 kB
Loop #36: 51,052.0 kB
Loop #37: 51,844.0 kB
Loop #38: 52,636.0 kB
Loop #39: 53,692.0 kB
Loop #40: 54,484.0 kB
Loop #41: 55,276.0 kB
Loop #42: 56,068.0 kB
Loop #43: 56,860.0 kB
Loop #44: 56,860.0 kB
Loop #45: 56,860.0 kB
Loop #46: 56,860.0 kB
Loop #47: 56,860.0 kB
Loop #48: 56,860.0 kB
Loop #49: 56,860.0 kB
Loop #50: 56,860.0 kB
Loop #51: 56,860.0 kB
Loop #52: 56,860.0 kB
Loop #53: 56,860.0 kB
Loop #54: 56,860.0 kB
Loop #55: 56,860.0 kB
Loop #56: 56,860.0 kB
Loop #57: 56,860.0 kB
Loop #58: 56,860.0 kB
Loop #59: 56,860.0 kB
Loop #60: 56,860.0 kB
Loop #61: 56,860.0 kB
Loop #62: 56,860.0 kB
Loop #63: 56,860.0 kB
Loop #64: 56,860.0 kB
Loop #65: 56,860.0 kB
Loop #66: 56,860.0 kB
Loop #67: 56,860.0 kB
Loop #68: 56,860.0 kB
Loop #69: 56,860.0 kB
Loop #70: 56,860.0 kB
Loop #71: 57,652.0 kB
Loop #72: 58,444.0 kB
Loop #73: 59,236.0 kB
Loop #74: 59,236.0 kB
Loop #75: 59,236.0 kB
Loop #76: 59,236.0 kB
Loop #77: 59,236.0 kB
Loop #78: 59,236.0 kB
Loop #79: 59,236.0 kB
Loop #80: 59,236.0 kB
Loop #81: 59,236.0 kB
Loop #82: 59,236.0 kB
Loop #83: 59,236.0 kB
Loop #84: 59,236.0 kB
Loop #85: 59,236.0 kB
Loop #86: 59,236.0 kB
Loop #87: 59,500.0 kB
Loop #88: 59,500.0 kB
Loop #89: 59,500.0 kB
Loop #90: 59,500.0 kB
Loop #91: 59,500.0 kB
Loop #92: 59,500.0 kB
Loop #93: 59,500.0 kB
Loop #94: 59,500.0 kB
Loop #95: 59,500.0 kB
Loop #96: 59,500.0 kB
Loop #97: 59,500.0 kB
Loop #98: 59,500.0 kB
Loop #99: 59,500.0 kB
Loop #100: 59,500.0 kB
Loop #101: 59,500.0 kB
Loop #102: 59,500.0 kB
Loop #103: 59,500.0 kB
Loop #104: 59,500.0 kB
Loop #105: 59,500.0 kB
Loop #106: 59,500.0 kB
Loop #107: 59,500.0 kB
Loop #108: 59,500.0 kB
Loop #109: 59,500.0 kB
Loop #110: 59,500.0 kB
Loop #111: 59,500.0 kB
Loop #112: 59,500.0 kB
Loop #113: 59,500.0 kB
Loop #114: 59,500.0 kB
Loop #115: 59,500.0 kB
Loop #116: 59,500.0 kB
Loop #117: 59,500.0 kB
Loop #118: 59,500.0 kB
Loop #119: 59,500.0 kB

  15. nascheme commented on Dec 18, 2025

    @nascheme
    nascheme
    on Dec 18, 2025
    Member

    I tried to narrow down the issue a bit better. It seems SSL is not required. I'm running the script below. It loads from a local web server (port 8001 without SSL) and the memory still goes up quite high. With 3.14.2, it stabilizes at about 420 MB. Version 3.14.0 stabilizes at about 80 MB (sometimes 60 MB). Bisecting between those two versions shows this commit as the cause for the increased memory use: d1a434f (backport of GH-140262).

    I'm still not sure what's going on. With the free-threaded GC (which is quite different), the memory usage is also very high. I think "httpx" is creating reference cycles and the GC is not going a good job to free them. If you run full GC collections frequently enough then memory usage stays low. But if you defer collection too long, memory usage goes much higher. That behavior makes me suspect weak references are involved or maybe finalizers that are resurrecting objects.

    # /// script
# requires-python = ">=3.13"
# dependencies = [
#   "httpx",
# ]
# ///
import httpx
import asyncio
import gc
import os
import sys


def get_rss():
    with open(f"/proc/{os.getpid()}/status") as fp:
        for line in fp:
            if line.startswith("VmRSS"):
                rss = line.split(":", 1)[1].rstrip()
                rss = rss.removesuffix(" kB")
                return int(rss)
    return None


URL = 'http://localhost:8001/'


async def _main() -> None:
    for i in range(3000):
        async with httpx.AsyncClient() as client:
            await client.get(URL)
        rss = get_rss()
        print(f"Loop #{i}: {rss:,} kB")


if __name__ == "__main__":

    def gc_callback(phase, info):
        if phase == 'stop':
            print(
                f'gen={info.get("generation")} collected={info.get("collected")}',
                file=sys.stderr,
            )

    if False:
        gc.callbacks.append(gc_callback)
    if False:
        gc.set_threshold(200, 2)
    asyncio.run(_main())

  16. sergey-miryanov commented on Dec 18, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 18, 2025
    Contributor

    Yeah, if we set first gc threshold quite low then memory usage is stable (I tried set_threshold(100)).
    Yes, httpx (httpx -> httpcore -> h11) creates cycles for example for sentinel values CLIENT, SERVER and others.
    Also, if I call gc_collect_full instead of gc_collect_increment then memory usage grows but not so fast.

    For 120 loops gc called about 12 times. First six gc runs do nothing (because of work_to_do is low). Maybe we can't break all cycles with one pass?

  17. sergey-miryanov commented on Dec 18, 2025

    @sergey-miryanov
    sergey-miryanov
    on Dec 18, 2025
    Contributor

    One example of the cycle:

    Response -> BoundAsyncStream -> Response -> ... 
    gc: collectable <Response 0x7fd8e3c4cc20>
  --> <bytes 0x561a014e9218 rc=3221225472> <b''>
  --> <IdentityDecoder 0x7fd8e3c4d010 rc=1> <<httpx._decoders.IdentityDecoder object at 0x7fd8e3c4d010>>
    --> <type 0x561a238f7ad0 rc=17> <<class 'httpx._decoders.IdentityDecoder'>>
  --> <datetime.timedelta 0x7fd8e551a490 rc=1> <datetime.timedelta(microseconds=329171)>
  --> <int 0x561a014e71f8 rc=3221225472> <0>
  --> <Request 0x7fd8e3cf4ec0 rc=1> <<Request('HEAD', 'http://www.google.com/')>>
    --> <bytes 0x561a014e9218 rc=3221225472> <b''>
    --> <dict 0x7fd8e3bd1200 rc=2> <{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}>
    --> <Headers 0x7fd8e3d68d10 rc=1> <Headers({'host': 'www.google.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.28.1'})>
      --> <str 0x7fd8e54e26d0 rc=3221225472> <'ascii'>
      --> <list 0x7fd8e52edb80 rc=1> <[(b'Host', b'host', b'www.google.com'), (b'Accept', b'accept', b'*/*'), (b'Accept-Encoding', b'accept-encoding', b'gzip, deflate'), (b'Connection', b'connection', b'keep-alive'), (b'User-Agent', b'user-agent', b'python-httpx/0.28.1')]>
      --> <ABCMeta 0x561a23906090 rc=35> <<class 'httpx.Headers'>>
        --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
        --> <dict 0x7fd8e44e8280 rc=1> <{'__module__': 'httpx', '__firstlineno__': 139, '__doc__': '\nHTTP headers, as a case-insensitive multi-dict.\n', '__init__': <function Headers.__init__ at 0x7fd8e44ef320>, 'encoding': <property object at 0x7fd8e44b98f0>, 'raw': <property object at 0x7fd8e44fdc60>, 'keys': <function Headers.keys at 0x7fd8e44ef8a0>, 'values': <function Headers.values at 0x7fd8e44efa00>, 'items': <function Headers.items at 0x7fd8e44efb60>, 'multi_items': <function Headers.multi_items at 0x7fd8e44efcc0>, 'get': <function Headers.get at 0x7fd8e44efe20>, 'get_list': <function Headers.get_list at 0x7fd8e44f0040>, 'update': <function Headers.update at 0x7fd8e44f01a0>, 'copy': <function Headers.copy at 0x7fd8e44f0300>, '__getitem__': <function Headers.__getitem__ at 0x7fd8e44f0460>, '__setitem__': <function Headers.__setitem__ at 0x7fd8e44f05c0>, '__delitem__': <function Headers.__delitem__ at 0x7fd8e44f0720>, '__contains__': <function Headers.__contains__ at 0x7fd8e44f0880>, '__iter__': <function Headers.__iter__ at 0x7fd8e44f09e0>, '__len__': <function Headers.__len__ at 0x7fd8e44f0b40>, '__eq__': <function Headers.__eq__ at 0x7fd8e44f0ca0>, '__repr__': <function Headers.__repr__ at 0x7fd8e44f0e00>, '__static_attributes__': ('_encoding', '_list'), '__orig_bases__': (typing.MutableMapping[str, str],), '__dict__': <attribute '__dict__' of 'Headers' objects>, '__weakref__': <attribute '__weakref__' of 'Headers' objects>, '__hash__': None, '__parameters__': (), '__abstractmethods__': frozenset(), '_abc_impl': <_abc._abc_data object at 0x7fd8e44e9d80>}>
        --> <tuple 0x7fd8e448ca50 rc=1> <(<class 'httpx.Headers'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'typing.Generic'>, <class 'object'>)>
        --> <tuple 0x7fd8e44b7940 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'typing.Generic'>)>
        --> <ABCMeta 0x561a2365ddd0 rc=27> <<class 'collections.abc.MutableMapping'>>
          --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
          --> <dict 0x7fd8e535ee80 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 908, '__doc__': 'A MutableMapping is a generic container for associating\nkey/value pairs.\n\nThis class provides concrete generic implementations of all\nmethods except for __getitem__, __setitem__, __delitem__,\n__iter__, and __len__.\n', '__slots__': (), '__setitem__': <function MutableMapping.__setitem__ at 0x7fd8e5364670>, '__delitem__': <function MutableMapping.__delitem__ at 0x7fd8e5364720>, '_MutableMapping__marker': <object object at 0x7fd8e54c01b0>, 'pop': <function MutableMapping.pop at 0x7fd8e53647d0>, 'popitem': <function MutableMapping.popitem at 0x7fd8e5364880>, 'clear': <function MutableMapping.clear at 0x7fd8e5364930>, 'update': <function MutableMapping.update at 0x7fd8e53649e0>, 'setdefault': <function MutableMapping.setdefault at 0x7fd8e5364a90>, '__static_attributes__': (), '__abstractmethods__': frozenset({'__setitem__', '__delitem__', '__getitem__', '__len__', '__iter__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535ef40>}>
          --> <tuple 0x7fd8e551f880 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
          --> <tuple 0x7fd8e5351480 rc=1> <(<class 'collections.abc.Mapping'>,)>
          --> <ABCMeta 0x561a2365ce10 rc=34> <<class 'collections.abc.Mapping'>>
            --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
            --> <dict 0x7fd8e535e840 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 775, '__doc__': 'A Mapping is a generic container for associating key/value\npairs.\n\nThis class provides concrete generic implementations of all\nmethods except for __getitem__, __iter__, and __len__.\n', '__slots__': (), '__getitem__': <function Mapping.__getitem__ at 0x7fd8e531f950>, 'get': <function Mapping.get at 0x7fd8e531fa00>, '__contains__': <function Mapping.__contains__ at 0x7fd8e531fab0>, 'keys': <function Mapping.keys at 0x7fd8e531fb60>, 'items': <function Mapping.items at 0x7fd8e531fc10>, 'values': <function Mapping.values at 0x7fd8e531fcc0>, '__eq__': <function Mapping.__eq__ at 0x7fd8e531fd70>, '__reversed__': None, '__static_attributes__': (), '__hash__': None, '__abstractmethods__': frozenset({'__iter__', '__len__', '__getitem__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535e8c0>}>
            --> <tuple 0x7fd8e551f580 rc=1> <(<class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
            --> <tuple 0x7fd8e53511b0 rc=1> <(<class 'collections.abc.Collection'>,)>
            --> <ABCMeta 0x561a2365b360 rc=36> <<class 'collections.abc.Collection'>>
    --> <str 0x7fd8e40b2e20 rc=1> <'HEAD'>
    --> <ByteStream 0x7fd8e3cf57f0 rc=2> <<httpx.ByteStream object at 0x7fd8e3cf57f0>>
      --> <bytes 0x561a014e9218 rc=3221225472> <b''>
      --> <type 0x561a238ede30 rc=18> <<class 'httpx.ByteStream'>>
    --> <URL 0x7fd8e3ce7250 rc=1> <URL('http://www.google.com/')>
      --> <ParseResult 0x7fd8e3bbc430 rc=1> <ParseResult(scheme='http', userinfo='', host='www.google.com', port=None, path='/', query=None, fragment=None)>
        --> <type 0x561a2390fdb0 rc=17> <<class 'httpx._urlparse.ParseResult'>>
        --> <NoneType 0x561a014be300 rc=3221225472> <None>
        --> <NoneType 0x561a014be300 rc=3221225472> <None>
        --> <str 0x561a014f6748 rc=3221225472> <'/'>
        --> <NoneType 0x561a014be300 rc=3221225472> <None>
        --> <str 0x7fd8e3f37070 rc=1> <'www.google.com'>
        --> <str 0x561a014ec530 rc=3221225472> <''>
        --> <str 0x7fd8e3ffbcc0 rc=1> <'http'>
      --> <type 0x561a2390f9c0 rc=25> <<class 'httpx.URL'>>
    --> <type 0x561a23905ca0 rc=30> <<class 'httpx.Request'>>
  --> <str 0x7fd8e52c9650 rc=17> <'utf-8'>
  --> <dict 0x7fd8e3c2c540 rc=1> <{'http_version': b'HTTP/1.1', 'reason_phrase': b'OK', 'network_stream': <httpcore._backends.anyio.AnyIOStream object at 0x7fd8e3c4c590>}>
  --> <Headers 0x7fd8e3d6ae00 rc=1> <Headers([('content-type', 'text/html; charset=ISO-8859-1'), ('content-security-policy-report-only', "object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), ('p3p', 'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), ('date', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('server', 'gws'), ('x-xss-protection', '0'), ('x-frame-options', 'SAMEORIGIN'), ('expires', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('cache-control', 'private'), ('set-cookie', 'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), ('set-cookie', 'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), ('transfer-encoding', 'chunked')])>
    --> <str 0x7fd8e54e26d0 rc=3221225472> <'ascii'>
    --> <list 0x7fd8e3c9eb80 rc=1> <[(b'Content-Type', b'content-type', b'text/html; charset=ISO-8859-1'), (b'Content-Security-Policy-Report-Only', b'content-security-policy-report-only', b"object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), (b'P3P', b'p3p', b'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), (b'Date', b'date', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Server', b'server', b'gws'), (b'X-XSS-Protection', b'x-xss-protection', b'0'), (b'X-Frame-Options', b'x-frame-options', b'SAMEORIGIN'), (b'Expires', b'expires', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Cache-Control', b'cache-control', b'private'), (b'Set-Cookie', b'set-cookie', b'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), (b'Set-Cookie', b'set-cookie', b'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), (b'Transfer-Encoding', b'transfer-encoding', b'chunked')]>
    --> <ABCMeta 0x561a23906090 rc=35> <<class 'httpx.Headers'>>
      --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
      --> <dict 0x7fd8e44e8280 rc=1> <{'__module__': 'httpx', '__firstlineno__': 139, '__doc__': '\nHTTP headers, as a case-insensitive multi-dict.\n', '__init__': <function Headers.__init__ at 0x7fd8e44ef320>, 'encoding': <property object at 0x7fd8e44b98f0>, 'raw': <property object at 0x7fd8e44fdc60>, 'keys': <function Headers.keys at 0x7fd8e44ef8a0>, 'values': <function Headers.values at 0x7fd8e44efa00>, 'items': <function Headers.items at 0x7fd8e44efb60>, 'multi_items': <function Headers.multi_items at 0x7fd8e44efcc0>, 'get': <function Headers.get at 0x7fd8e44efe20>, 'get_list': <function Headers.get_list at 0x7fd8e44f0040>, 'update': <function Headers.update at 0x7fd8e44f01a0>, 'copy': <function Headers.copy at 0x7fd8e44f0300>, '__getitem__': <function Headers.__getitem__ at 0x7fd8e44f0460>, '__setitem__': <function Headers.__setitem__ at 0x7fd8e44f05c0>, '__delitem__': <function Headers.__delitem__ at 0x7fd8e44f0720>, '__contains__': <function Headers.__contains__ at 0x7fd8e44f0880>, '__iter__': <function Headers.__iter__ at 0x7fd8e44f09e0>, '__len__': <function Headers.__len__ at 0x7fd8e44f0b40>, '__eq__': <function Headers.__eq__ at 0x7fd8e44f0ca0>, '__repr__': <function Headers.__repr__ at 0x7fd8e44f0e00>, '__static_attributes__': ('_encoding', '_list'), '__orig_bases__': (typing.MutableMapping[str, str],), '__dict__': <attribute '__dict__' of 'Headers' objects>, '__weakref__': <attribute '__weakref__' of 'Headers' objects>, '__hash__': None, '__parameters__': (), '__abstractmethods__': frozenset(), '_abc_impl': <_abc._abc_data object at 0x7fd8e44e9d80>}>
      --> <tuple 0x7fd8e448ca50 rc=1> <(<class 'httpx.Headers'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'typing.Generic'>, <class 'object'>)>
      --> <tuple 0x7fd8e44b7940 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'typing.Generic'>)>
      --> <ABCMeta 0x561a2365ddd0 rc=27> <<class 'collections.abc.MutableMapping'>>
        --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
        --> <dict 0x7fd8e535ee80 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 908, '__doc__': 'A MutableMapping is a generic container for associating\nkey/value pairs.\n\nThis class provides concrete generic implementations of all\nmethods except for __getitem__, __setitem__, __delitem__,\n__iter__, and __len__.\n', '__slots__': (), '__setitem__': <function MutableMapping.__setitem__ at 0x7fd8e5364670>, '__delitem__': <function MutableMapping.__delitem__ at 0x7fd8e5364720>, '_MutableMapping__marker': <object object at 0x7fd8e54c01b0>, 'pop': <function MutableMapping.pop at 0x7fd8e53647d0>, 'popitem': <function MutableMapping.popitem at 0x7fd8e5364880>, 'clear': <function MutableMapping.clear at 0x7fd8e5364930>, 'update': <function MutableMapping.update at 0x7fd8e53649e0>, 'setdefault': <function MutableMapping.setdefault at 0x7fd8e5364a90>, '__static_attributes__': (), '__abstractmethods__': frozenset({'__setitem__', '__delitem__', '__getitem__', '__len__', '__iter__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535ef40>}>
        --> <tuple 0x7fd8e551f880 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
        --> <tuple 0x7fd8e5351480 rc=1> <(<class 'collections.abc.Mapping'>,)>
        --> <ABCMeta 0x561a2365ce10 rc=34> <<class 'collections.abc.Mapping'>>
          --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
          --> <dict 0x7fd8e535e840 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 775, '__doc__': 'A Mapping is a generic container for associating key/value\npairs.\n\nThis class provides concrete generic implementations of all\nmethods except for __getitem__, __iter__, and __len__.\n', '__slots__': (), '__getitem__': <function Mapping.__getitem__ at 0x7fd8e531f950>, 'get': <function Mapping.get at 0x7fd8e531fa00>, '__contains__': <function Mapping.__contains__ at 0x7fd8e531fab0>, 'keys': <function Mapping.keys at 0x7fd8e531fb60>, 'items': <function Mapping.items at 0x7fd8e531fc10>, 'values': <function Mapping.values at 0x7fd8e531fcc0>, '__eq__': <function Mapping.__eq__ at 0x7fd8e531fd70>, '__reversed__': None, '__static_attributes__': (), '__hash__': None, '__abstractmethods__': frozenset({'__iter__', '__len__', '__getitem__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535e8c0>}>
          --> <tuple 0x7fd8e551f580 rc=1> <(<class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
          --> <tuple 0x7fd8e53511b0 rc=1> <(<class 'collections.abc.Collection'>,)>
          --> <ABCMeta 0x561a2365b360 rc=36> <<class 'collections.abc.Collection'>>
            --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
            --> <dict 0x7fd8e535dcc0 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 435, '__slots__': (), '__subclasshook__': <classmethod(<function Collection.__subclasshook__ at 0x7fd8e531e2a0>)>, '__static_attributes__': (), '__doc__': None, '__abstractmethods__': frozenset({'__iter__', '__len__', '__contains__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535de40>}>
            --> <tuple 0x7fd8e533a160 rc=1> <(<class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
            --> <tuple 0x7fd8e533d200 rc=1> <(<class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>)>
            --> <ABCMeta 0x561a2365ab80 rc=34> <<class 'collections.abc.Sized'>>
  --> <list 0x7fd8e3c2c900 rc=1> <[]>
  --> <bool 0x561a014a73c0 rc=3221225472> <True>
  --> <bool 0x561a014a73c0 rc=3221225472> <True>
  --> <NoneType 0x561a014be300 rc=3221225472> <None>
  --> <int 0x561a014e8af8 rc=3221225472> <200>
  --> <BoundAsyncStream 0x7fd8e3c4cec0 rc=1> <<httpx._client.BoundAsyncStream object at 0x7fd8e3c4cec0>>
    --> <Response 0x7fd8e3c4cc20 rc=1> <<Response [200 OK]>>
      --> <bytes 0x561a014e9218 rc=3221225472> <b''>
      --> <IdentityDecoder 0x7fd8e3c4d010 rc=1> <<httpx._decoders.IdentityDecoder object at 0x7fd8e3c4d010>>
        --> <type 0x561a238f7ad0 rc=17> <<class 'httpx._decoders.IdentityDecoder'>>
      --> <datetime.timedelta 0x7fd8e551a490 rc=1> <datetime.timedelta(microseconds=329171)>
      --> <int 0x561a014e71f8 rc=3221225472> <0>
      --> <Request 0x7fd8e3cf4ec0 rc=1> <<Request('HEAD', 'http://www.google.com/')>>
        --> <bytes 0x561a014e9218 rc=3221225472> <b''>
        --> <dict 0x7fd8e3bd1200 rc=2> <{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}>
        --> <Headers 0x7fd8e3d68d10 rc=1> <Headers({'host': 'www.google.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.28.1'})>
          --> <str 0x7fd8e54e26d0 rc=3221225472> <'ascii'>
          --> <list 0x7fd8e52edb80 rc=1> <[(b'Host', b'host', b'www.google.com'), (b'Accept', b'accept', b'*/*'), (b'Accept-Encoding', b'accept-encoding', b'gzip, deflate'), (b'Connection', b'connection', b'keep-alive'), (b'User-Agent', b'user-agent', b'python-httpx/0.28.1')]>
          --> <ABCMeta 0x561a23906090 rc=35> <<class 'httpx.Headers'>>
            --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
            --> <dict 0x7fd8e44e8280 rc=1> <{'__module__': 'httpx', '__firstlineno__': 139, '__doc__': '\nHTTP headers, as a case-insensitive multi-dict.\n', '__init__': <function Headers.__init__ at 0x7fd8e44ef320>, 'encoding': <property object at 0x7fd8e44b98f0>, 'raw': <property object at 0x7fd8e44fdc60>, 'keys': <function Headers.keys at 0x7fd8e44ef8a0>, 'values': <function Headers.values at 0x7fd8e44efa00>, 'items': <function Headers.items at 0x7fd8e44efb60>, 'multi_items': <function Headers.multi_items at 0x7fd8e44efcc0>, 'get': <function Headers.get at 0x7fd8e44efe20>, 'get_list': <function Headers.get_list at 0x7fd8e44f0040>, 'update': <function Headers.update at 0x7fd8e44f01a0>, 'copy': <function Headers.copy at 0x7fd8e44f0300>, '__getitem__': <function Headers.__getitem__ at 0x7fd8e44f0460>, '__setitem__': <function Headers.__setitem__ at 0x7fd8e44f05c0>, '__delitem__': <function Headers.__delitem__ at 0x7fd8e44f0720>, '__contains__': <function Headers.__contains__ at 0x7fd8e44f0880>, '__iter__': <function Headers.__iter__ at 0x7fd8e44f09e0>, '__len__': <function Headers.__len__ at 0x7fd8e44f0b40>, '__eq__': <function Headers.__eq__ at 0x7fd8e44f0ca0>, '__repr__': <function Headers.__repr__ at 0x7fd8e44f0e00>, '__static_attributes__': ('_encoding', '_list'), '__orig_bases__': (typing.MutableMapping[str, str],), '__dict__': <attribute '__dict__' of 'Headers' objects>, '__weakref__': <attribute '__weakref__' of 'Headers' objects>, '__hash__': None, '__parameters__': (), '__abstractmethods__': frozenset(), '_abc_impl': <_abc._abc_data object at 0x7fd8e44e9d80>}>
            --> <tuple 0x7fd8e448ca50 rc=1> <(<class 'httpx.Headers'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'typing.Generic'>, <class 'object'>)>
            --> <tuple 0x7fd8e44b7940 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'typing.Generic'>)>
            --> <ABCMeta 0x561a2365ddd0 rc=27> <<class 'collections.abc.MutableMapping'>>
        --> <str 0x7fd8e40b2e20 rc=1> <'HEAD'>
        --> <ByteStream 0x7fd8e3cf57f0 rc=2> <<httpx.ByteStream object at 0x7fd8e3cf57f0>>
          --> <bytes 0x561a014e9218 rc=3221225472> <b''>
          --> <type 0x561a238ede30 rc=18> <<class 'httpx.ByteStream'>>
        --> <URL 0x7fd8e3ce7250 rc=1> <URL('http://www.google.com/')>
          --> <ParseResult 0x7fd8e3bbc430 rc=1> <ParseResult(scheme='http', userinfo='', host='www.google.com', port=None, path='/', query=None, fragment=None)>
            --> <type 0x561a2390fdb0 rc=17> <<class 'httpx._urlparse.ParseResult'>>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <str 0x561a014f6748 rc=3221225472> <'/'>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <str 0x7fd8e3f37070 rc=1> <'www.google.com'>
            --> <str 0x561a014ec530 rc=3221225472> <''>
            --> <str 0x7fd8e3ffbcc0 rc=1> <'http'>
          --> <type 0x561a2390f9c0 rc=25> <<class 'httpx.URL'>>
        --> <type 0x561a23905ca0 rc=30> <<class 'httpx.Request'>>
      --> <str 0x7fd8e52c9650 rc=17> <'utf-8'>
      --> <dict 0x7fd8e3c2c540 rc=1> <{'http_version': b'HTTP/1.1', 'reason_phrase': b'OK', 'network_stream': <httpcore._backends.anyio.AnyIOStream object at 0x7fd8e3c4c590>}>
      --> <Headers 0x7fd8e3d6ae00 rc=1> <Headers([('content-type', 'text/html; charset=ISO-8859-1'), ('content-security-policy-report-only', "object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), ('p3p', 'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), ('date', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('server', 'gws'), ('x-xss-protection', '0'), ('x-frame-options', 'SAMEORIGIN'), ('expires', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('cache-control', 'private'), ('set-cookie', 'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), ('set-cookie', 'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), ('transfer-encoding', 'chunked')])>
        --> <str 0x7fd8e54e26d0 rc=3221225472> <'ascii'>
        --> <list 0x7fd8e3c9eb80 rc=1> <[(b'Content-Type', b'content-type', b'text/html; charset=ISO-8859-1'), (b'Content-Security-Policy-Report-Only', b'content-security-policy-report-only', b"object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), (b'P3P', b'p3p', b'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), (b'Date', b'date', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Server', b'server', b'gws'), (b'X-XSS-Protection', b'x-xss-protection', b'0'), (b'X-Frame-Options', b'x-frame-options', b'SAMEORIGIN'), (b'Expires', b'expires', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Cache-Control', b'cache-control', b'private'), (b'Set-Cookie', b'set-cookie', b'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), (b'Set-Cookie', b'set-cookie', b'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), (b'Transfer-Encoding', b'transfer-encoding', b'chunked')]>
        --> <ABCMeta 0x561a23906090 rc=35> <<class 'httpx.Headers'>>
          --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
          --> <dict 0x7fd8e44e8280 rc=1> <{'__module__': 'httpx', '__firstlineno__': 139, '__doc__': '\nHTTP headers, as a case-insensitive multi-dict.\n', '__init__': <function Headers.__init__ at 0x7fd8e44ef320>, 'encoding': <property object at 0x7fd8e44b98f0>, 'raw': <property object at 0x7fd8e44fdc60>, 'keys': <function Headers.keys at 0x7fd8e44ef8a0>, 'values': <function Headers.values at 0x7fd8e44efa00>, 'items': <function Headers.items at 0x7fd8e44efb60>, 'multi_items': <function Headers.multi_items at 0x7fd8e44efcc0>, 'get': <function Headers.get at 0x7fd8e44efe20>, 'get_list': <function Headers.get_list at 0x7fd8e44f0040>, 'update': <function Headers.update at 0x7fd8e44f01a0>, 'copy': <function Headers.copy at 0x7fd8e44f0300>, '__getitem__': <function Headers.__getitem__ at 0x7fd8e44f0460>, '__setitem__': <function Headers.__setitem__ at 0x7fd8e44f05c0>, '__delitem__': <function Headers.__delitem__ at 0x7fd8e44f0720>, '__contains__': <function Headers.__contains__ at 0x7fd8e44f0880>, '__iter__': <function Headers.__iter__ at 0x7fd8e44f09e0>, '__len__': <function Headers.__len__ at 0x7fd8e44f0b40>, '__eq__': <function Headers.__eq__ at 0x7fd8e44f0ca0>, '__repr__': <function Headers.__repr__ at 0x7fd8e44f0e00>, '__static_attributes__': ('_encoding', '_list'), '__orig_bases__': (typing.MutableMapping[str, str],), '__dict__': <attribute '__dict__' of 'Headers' objects>, '__weakref__': <attribute '__weakref__' of 'Headers' objects>, '__hash__': None, '__parameters__': (), '__abstractmethods__': frozenset(), '_abc_impl': <_abc._abc_data object at 0x7fd8e44e9d80>}>
          --> <tuple 0x7fd8e448ca50 rc=1> <(<class 'httpx.Headers'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'typing.Generic'>, <class 'object'>)>
          --> <tuple 0x7fd8e44b7940 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'typing.Generic'>)>
          --> <ABCMeta 0x561a2365ddd0 rc=27> <<class 'collections.abc.MutableMapping'>>
            --> <type 0x561a23638a00 rc=158> <<class 'abc.ABCMeta'>>
            --> <dict 0x7fd8e535ee80 rc=1> <{'__module__': 'collections.abc', '__firstlineno__': 908, '__doc__': 'A MutableMapping is a generic container for associating\nkey/value pairs.\n\nThis class provides concrete generic implementations of all\nmethods except for __getitem__, __setitem__, __delitem__,\n__iter__, and __len__.\n', '__slots__': (), '__setitem__': <function MutableMapping.__setitem__ at 0x7fd8e5364670>, '__delitem__': <function MutableMapping.__delitem__ at 0x7fd8e5364720>, '_MutableMapping__marker': <object object at 0x7fd8e54c01b0>, 'pop': <function MutableMapping.pop at 0x7fd8e53647d0>, 'popitem': <function MutableMapping.popitem at 0x7fd8e5364880>, 'clear': <function MutableMapping.clear at 0x7fd8e5364930>, 'update': <function MutableMapping.update at 0x7fd8e53649e0>, 'setdefault': <function MutableMapping.setdefault at 0x7fd8e5364a90>, '__static_attributes__': (), '__abstractmethods__': frozenset({'__setitem__', '__delitem__', '__getitem__', '__len__', '__iter__'}), '_abc_impl': <_abc._abc_data object at 0x7fd8e535ef40>}>
            --> <tuple 0x7fd8e551f880 rc=1> <(<class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)>
            --> <tuple 0x7fd8e5351480 rc=1> <(<class 'collections.abc.Mapping'>,)>
            --> <ABCMeta 0x561a2365ce10 rc=34> <<class 'collections.abc.Mapping'>>
      --> <list 0x7fd8e3c2c900 rc=1> <[]>
      --> <bool 0x561a014a73c0 rc=3221225472> <True>
      --> <bool 0x561a014a73c0 rc=3221225472> <True>
      --> <NoneType 0x561a014be300 rc=3221225472> <None>
      --> <int 0x561a014e8af8 rc=3221225472> <200>
      --> <BoundAsyncStream 0x7fd8e3c4cec0 rc=1> <<httpx._client.BoundAsyncStream object at 0x7fd8e3c4cec0>>
        --> <Response 0x7fd8e3c4cc20 rc=1> <<Response [200 OK]>>
          --> <bytes 0x561a014e9218 rc=3221225472> <b''>
          --> <IdentityDecoder 0x7fd8e3c4d010 rc=1> <<httpx._decoders.IdentityDecoder object at 0x7fd8e3c4d010>>
            --> <type 0x561a238f7ad0 rc=17> <<class 'httpx._decoders.IdentityDecoder'>>
          --> <datetime.timedelta 0x7fd8e551a490 rc=1> <datetime.timedelta(microseconds=329171)>
          --> <int 0x561a014e71f8 rc=3221225472> <0>
          --> <Request 0x7fd8e3cf4ec0 rc=1> <<Request('HEAD', 'http://www.google.com/')>>
            --> <bytes 0x561a014e9218 rc=3221225472> <b''>
            --> <dict 0x7fd8e3bd1200 rc=2> <{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}>
            --> <Headers 0x7fd8e3d68d10 rc=1> <Headers({'host': 'www.google.com', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.28.1'})>
            --> <str 0x7fd8e40b2e20 rc=1> <'HEAD'>
            --> <ByteStream 0x7fd8e3cf57f0 rc=2> <<httpx.ByteStream object at 0x7fd8e3cf57f0>>
            --> <URL 0x7fd8e3ce7250 rc=1> <URL('http://www.google.com/')>
            --> <type 0x561a23905ca0 rc=30> <<class 'httpx.Request'>>
          --> <str 0x7fd8e52c9650 rc=17> <'utf-8'>
          --> <dict 0x7fd8e3c2c540 rc=1> <{'http_version': b'HTTP/1.1', 'reason_phrase': b'OK', 'network_stream': <httpcore._backends.anyio.AnyIOStream object at 0x7fd8e3c4c590>}>
          --> <Headers 0x7fd8e3d6ae00 rc=1> <Headers([('content-type', 'text/html; charset=ISO-8859-1'), ('content-security-policy-report-only', "object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), ('p3p', 'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), ('date', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('server', 'gws'), ('x-xss-protection', '0'), ('x-frame-options', 'SAMEORIGIN'), ('expires', 'Thu, 18 Dec 2025 19:58:31 GMT'), ('cache-control', 'private'), ('set-cookie', 'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), ('set-cookie', 'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), ('transfer-encoding', 'chunked')])>
            --> <str 0x7fd8e54e26d0 rc=3221225472> <'ascii'>
            --> <list 0x7fd8e3c9eb80 rc=1> <[(b'Content-Type', b'content-type', b'text/html; charset=ISO-8859-1'), (b'Content-Security-Policy-Report-Only', b'content-security-policy-report-only', b"object-src 'none';base-uri 'self';script-src 'nonce-wYhXOYJqh4UnZEs4DxhhSQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp"), (b'P3P', b'p3p', b'CP="This is not a P3P policy! See g.co/p3phelp for more info."'), (b'Date', b'date', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Server', b'server', b'gws'), (b'X-XSS-Protection', b'x-xss-protection', b'0'), (b'X-Frame-Options', b'x-frame-options', b'SAMEORIGIN'), (b'Expires', b'expires', b'Thu, 18 Dec 2025 19:58:31 GMT'), (b'Cache-Control', b'cache-control', b'private'), (b'Set-Cookie', b'set-cookie', b'AEC=AaJma5s1zLZQ5IPrCVHEYXZ36YmVgcr5-AEKPVg2jUl76XZ8_WnBc1Hi1w; expires=Tue, 16-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax'), (b'Set-Cookie', b'set-cookie', b'NID=527=3iGMZI4Y7equ8Mm7al5iqN6B8scsLZWhqvIoNekMv-XogyGDxHpXAHp8u2R7PENdMzUoboTQ7K13pz2v4F0T8rlk6STI8Svq1M9C7y6OLyXFuBVwhfA_oueOg7wWzo6cpBGVLZ9BpJe_FZHsEQ4jhdiw2YpY-QI93PWn8l50q_Dc0WuOh4JESyPMws1qZbp1efSUN5aVp_zlV4Gn8Cu-Kf48hx8H; expires=Fri, 19-Jun-2026 19:58:31 GMT; path=/; domain=.google.com; HttpOnly'), (b'Transfer-Encoding', b'transfer-encoding', b'chunked')]>
            --> <ABCMeta 0x561a23906090 rc=35> <<class 'httpx.Headers'>>
          --> <list 0x7fd8e3c2c900 rc=1> <[]>
          --> <bool 0x561a014a73c0 rc=3221225472> <True>
          --> <bool 0x561a014a73c0 rc=3221225472> <True>
          --> <NoneType 0x561a014be300 rc=3221225472> <None>
          --> <int 0x561a014e8af8 rc=3221225472> <200>
          --> <BoundAsyncStream 0x7fd8e3c4cec0 rc=1> <<httpx._client.BoundAsyncStream object at 0x7fd8e3c4cec0>>
            --> <Response 0x7fd8e3c4cc20 rc=1> <<Response [200 OK]>>
            --> <float 0x7fd8e3cdd3b0 rc=1> <39037.104077293>
            --> <AsyncResponseStream 0x7fd8e3cf6120 rc=1> <<httpx._transports.default.AsyncResponseStream object at 0x7fd8e3cf6120>>
            --> <type 0x561a23924160 rc=15> <<class 'httpx._client.BoundAsyncStream'>>
          --> <type 0x561a239058b0 rc=30> <<class 'httpx.Response'>>
        --> <float 0x7fd8e3cdd3b0 rc=1> <39037.104077293>
        --> <AsyncResponseStream 0x7fd8e3cf6120 rc=1> <<httpx._transports.default.AsyncResponseStream object at 0x7fd8e3cf6120>>
          --> <PoolByteStream 0x7fd8e3c4cd70 rc=1> <<httpcore._async.connection_pool.PoolByteStream object at 0x7fd8e3c4cd70>>
            --> <bool 0x561a014a73c0 rc=3221225472> <True>
            --> <AsyncConnectionPool 0x7fd8e3cf4c20 rc=1> <<AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>>
            --> <AsyncPoolRequest 0x7fd8e3cf6270 rc=1> <<httpcore._async.connection_pool.AsyncPoolRequest object at 0x7fd8e3cf6270>>
            --> <HTTP11ConnectionByteStream 0x7fd8e3cf7770 rc=1> <<httpcore._async.http11.HTTP11ConnectionByteStream object at 0x7fd8e3cf7770>>
            --> <type 0x561a23ad0b10 rc=17> <<class 'httpcore._async.connection_pool.PoolByteStream'>>
          --> <type 0x561a2391c5d0 rc=15> <<class 'httpx._transports.default.AsyncResponseStream'>>
        --> <type 0x561a23924160 rc=15> <<class 'httpx._client.BoundAsyncStream'>>
      --> <type 0x561a239058b0 rc=30> <<class 'httpx.Response'>>
    --> <float 0x7fd8e3cdd3b0 rc=1> <39037.104077293>
    --> <AsyncResponseStream 0x7fd8e3cf6120 rc=1> <<httpx._transports.default.AsyncResponseStream object at 0x7fd8e3cf6120>>
      --> <PoolByteStream 0x7fd8e3c4cd70 rc=1> <<httpcore._async.connection_pool.PoolByteStream object at 0x7fd8e3c4cd70>>
        --> <bool 0x561a014a73c0 rc=3221225472> <True>
        --> <AsyncConnectionPool 0x7fd8e3cf4c20 rc=1> <<AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>>
          --> <list 0x7fd8e3ee70c0 rc=1> <[]>
          --> <bool 0x561a014a73c0 rc=3221225472> <True>
          --> <bool 0x561a014a73e0 rc=3221225472> <False>
          --> <float 0x7fd8e48bd710 rc=113> <5.0>
          --> <NoneType 0x561a014be300 rc=3221225472> <None>
          --> <int 0x561a014e7e78 rc=3221225472> <100>
          --> <int 0x561a014e7478 rc=3221225472> <20>
          --> <AutoBackend 0x7fd8e3cf4ad0 rc=2> <<httpcore._backends.auto.AutoBackend object at 0x7fd8e3cf4ad0>>
            --> <AnyIOBackend 0x7fd8e3cf7380 rc=1> <<httpcore.AnyIOBackend object at 0x7fd8e3cf7380>>
            --> <type 0x561a23accb80 rc=18> <<class 'httpcore._backends.auto.AutoBackend'>>
          --> <AsyncThreadLock 0x7fd8e3cf4d70 rc=1> <<httpcore._synchronization.AsyncThreadLock object at 0x7fd8e3cf4d70>>
            --> <type 0x561a23a532c0 rc=19> <<class 'httpcore._synchronization.AsyncThreadLock'>>
          --> <NoneType 0x561a014be300 rc=3221225472> <None>
          --> <list 0x7fd8e40aa540 rc=1> <[]>
          --> <int 0x561a014e71f8 rc=3221225472> <0>
          --> <NoneType 0x561a014be300 rc=3221225472> <None>
          --> <SSLContext 0x7fd8e3caa3c0 rc=2> <<ssl.SSLContext object at 0x7fd8e3caa3c0>>
            --> <type 0x561a23809210 rc=20> <<class 'ssl.SSLContext'>>
          --> <NoneType 0x561a014be300 rc=3221225472> <None>
          --> <type 0x561a23ad0410 rc=22> <<class 'httpcore.AsyncConnectionPool'>>
        --> <AsyncPoolRequest 0x7fd8e3cf6270 rc=1> <<httpcore._async.connection_pool.AsyncPoolRequest object at 0x7fd8e3cf6270>>
          --> <AsyncEvent 0x7fd8e3cf63c0 rc=1> <<httpcore._synchronization.AsyncEvent object at 0x7fd8e3cf63c0>>
            --> <Event 0x7fd8e3cf6900 rc=1> <<anyio._backends._asyncio.Event object at 0x7fd8e3cf6900>>
            --> <str 0x7fd8e5353630 rc=3221225472> <'asyncio'>
            --> <type 0x561a23a97010 rc=19> <<class 'httpcore._synchronization.AsyncEvent'>>
          --> <AsyncHTTPConnection 0x7fd8e3cf6660 rc=1> <<AsyncHTTPConnection ['http://www.google.com:80', HTTP/1.1, CLOSED, Request Count: 1]>>
            --> <bool 0x561a014a73e0 rc=3221225472> <False>
            --> <AsyncHTTP11Connection 0x7fd8e3c4c6e0 rc=2> <<AsyncHTTP11Connection ['http://www.google.com:80', CLOSED, Request Count: 1]>>
            --> <bool 0x561a014a73c0 rc=3221225472> <True>
            --> <bool 0x561a014a73e0 rc=3221225472> <False>
            --> <float 0x7fd8e48bd710 rc=113> <5.0>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <AutoBackend 0x7fd8e3cf4ad0 rc=2> <<httpcore._backends.auto.AutoBackend object at 0x7fd8e3cf4ad0>>
            --> <Origin 0x7fd8e3cf6510 rc=2> <<httpcore.Origin object at 0x7fd8e3cf6510>>
            --> <AsyncLock 0x7fd8e3cf67b0 rc=1> <<httpcore._synchronization.AsyncLock object at 0x7fd8e3cf67b0>>
            --> <int 0x561a014e71f8 rc=3221225472> <0>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <SSLContext 0x7fd8e3caa3c0 rc=2> <<ssl.SSLContext object at 0x7fd8e3caa3c0>>
            --> <NoneType 0x561a014be300 rc=3221225472> <None>
            --> <type 0x561a23acf3a0 rc=20> <<class 'httpcore.AsyncHTTPConnection'>>
          --> <Request 0x7fd8e3cf5d30 rc=2> <<Request [b'HEAD']>>
            --> <dict 0x7fd8e3bd1200 rc=2> <{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}>
            --> <list 0x7fd8e3d534c0 rc=1> <[(b'Host', b'www.google.com'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.28.1')]>
            --> <bytes 0x7fd8e3c967f0 rc=1> <b'HEAD'>
            --> <ByteStream 0x7fd8e3cf57f0 rc=2> <<httpx.ByteStream object at 0x7fd8e3cf57f0>>
            --> <URL 0x7fd8e3cf5fd0 rc=1> <URL(scheme=b'http', host=b'www.google.com', port=None, target=b'/')>
            --> <type 0x561a23a8b5f0 rc=30> <<class 'httpcore.Request'>>
          --> <type 0x561a23ada870 rc=18> <<class 'httpcore._async.connection_pool.AsyncPoolRequest'>>
        --> <HTTP11ConnectionByteStream 0x7fd8e3cf7770 rc=1> <<httpcore._async.http11.HTTP11ConnectionByteStream object at 0x7fd8e3cf7770>>
          --> <bool 0x561a014a73c0 rc=3221225472> <True>
          --> <AsyncHTTP11Connection 0x7fd8e3c4c6e0 rc=2> <<AsyncHTTP11Connection ['http://www.google.com:80', CLOSED, Request Count: 1]>>
            --> <float 0x7fd8e3cdd470 rc=1> <39042.433826994>
            --> <Connection 0x7fd8e3c4c830 rc=1> <<h11._connection.Connection object at 0x7fd8e3c4c830>>
            --> <float 0x7fd8e48bd710 rc=113> <5.0>
            --> <AnyIOStream 0x7fd8e3c4c590 rc=2> <<httpcore._backends.anyio.AnyIOStream object at 0x7fd8e3c4c590>>
            --> <Origin 0x7fd8e3cf6510 rc=2> <<httpcore.Origin object at 0x7fd8e3cf6510>>
            --> <int 0x561a014e7218 rc=3221225472> <1>
            --> <HTTPConnectionState 0x7fd8e3d50c50 rc=16> <<HTTPConnectionState.CLOSED: 3>>
            --> <AsyncLock 0x7fd8e3bd8910 rc=1> <<httpcore._synchronization.AsyncLock object at 0x7fd8e3bd8910>>
            --> <type 0x561a23ace1b0 rc=19> <<class 'httpcore.AsyncHTTP11Connection'>>
          --> <Request 0x7fd8e3cf5d30 rc=2> <<Request [b'HEAD']>>
            --> <dict 0x7fd8e3bd1200 rc=2> <{'timeout': {'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0}}>
            --> <list 0x7fd8e3d534c0 rc=1> <[(b'Host', b'www.google.com'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.28.1')]>
            --> <bytes 0x7fd8e3c967f0 rc=1> <b'HEAD'>
            --> <ByteStream 0x7fd8e3cf57f0 rc=2> <<httpx.ByteStream object at 0x7fd8e3cf57f0>>
            --> <URL 0x7fd8e3cf5fd0 rc=1> <URL(scheme=b'http', host=b'www.google.com', port=None, target=b'/')>
            --> <type 0x561a23a8b5f0 rc=30> <<class 'httpcore.Request'>>
          --> <type 0x561a23acebc0 rc=17> <<class 'httpcore._async.http11.HTTP11ConnectionByteStream'>>
        --> <type 0x561a23ad0b10 rc=17> <<class 'httpcore._async.connection_pool.PoolByteStream'>>
      --> <type 0x561a2391c5d0 rc=15> <<class 'httpx._transports.default.AsyncResponseStream'>>
    --> <type 0x561a23924160 rc=15> <<class 'httpx._client.BoundAsyncStream'>>
  --> <type 0x561a239058b0 rc=30> <<class 'httpx.Response'>>

  18. nascheme commented on Dec 18, 2025

    @nascheme
    nascheme
    on Dec 18, 2025
    Member

    I was wrong about SSL not being involved. Here is a memray flame graph. Nearly all the memory is allocated from load_verify_locations(), which is implemented in the _ssl.c module, it seems. Given that explicitly calling gc.collect() fixes the memory usage, I would suspect it's not memory leak in the _ssl.c code but due to reference cycles keeping the default context alive.

    Edit: passing a context to the verify parameter drastically reduces the memory usage, e.g.

        ssl_ctx = ssl.create_default_context()
    for i in range(2000):
        async with httpx.AsyncClient(verify=ssl_ctx) as client:
            resp = await client.get(URL)
    Image

  19. nascheme commented on Dec 18, 2025

    @nascheme
    nascheme
    on Dec 18, 2025
    Member

    One example of the cycle:

    Response -> BoundAsyncStream -> Response -> ...

    Good detective work! If I break that cycle, the high memory usage goes away. E.g. add this line at the end of the "with" block: resp.stream = None. It would be good if httpx would break that cycle when the "with" block is exited. However, we should also try to fix the GC so that these cycles are cleaned more quickly.

  20. nascheme commented on Dec 18, 2025

    @nascheme
    nascheme
    on Dec 18, 2025
    Member

    I don't know if this is an accurate minimization of the test case but it does use more memory in 3.14 vs 3.13 (and more yet in 3.14.2). It's just reference cycles that contain the value from ssl.create_default_context(). To me this looks like a case of relatively few objects involved in a reference cycle keeping quite a bit of memory in use. A single value from create_default_context() uses 800 kB, at least on my machine. So it doesn't take many Response->Stream cycles to chew up a lot of memory.

    import gc
import ssl


def get_rss():
    with open(f"/proc/self/status") as fp:
        for line in fp:
            if line.startswith("VmRSS"):
                rss = line.split(":", 1)[1].rstrip()
                rss = rss.removesuffix(" kB")
                return int(rss)
    return None


class Cell:
    pass


def main():
    for i in range(100_000):
        c = Cell()
        c.ctx = ssl.create_default_context()
        c.cell = c
        if (i % 100) == 0:
            print(f'{get_rss():,} kB')
            # gc.collect()


if __name__ == '__main__':
    main()

  21. nascheme commented on Dec 19, 2025

    @nascheme
    nascheme
    on Dec 19, 2025
    Member

    Another piece of info: the peak memory used by this script is going to be quite dependant on the malloc being used by the SSL library. On my machine, running with LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 causes the RSS to be quite a bit lower compared to using the default malloc (my guess is the default is GNU libc's malloc). Also measuring RSS is tricky. For Linux it would be more accurate to use the following to compute RSS (more expensive to compute but more accurate of actual resistent-set-size).

    def get_rss():
    with open('/proc/self/smaps_rollup') as fp:
        for line in fp:
            if line.startswith('Rss:'):
                rss = line.split(":", 1)[1].rstrip()
                rss = rss.removesuffix(" kB")
                return int(rss)

  22. HenriBlacksmith commented on Jan 5, 2026

    @HenriBlacksmith
    HenriBlacksmith
    on Jan 5, 2026
    Author

    If understand correctly, the patch referenced above (#142001) should improve the GC situation?

    Is this planned for the next patch release ?

  23. hyperized
    added a commit that references this issue on Jan 8, 2026

    Possible fix for python#142516

    c0827e6
  24. picnixz
    self-assigned this
    on Jan 9, 2026
  25. picnixz
    added 3 commits that reference this issue on Jan 11, 2026

    pythongh-142516: fix reference leaks in `ssl` objects

    2157b87

    pythongh-142516: fix reference leaks on `ssl.SSLContext` objects

    07cc2aa

    pythongh-142516: fix reference leaks in `ssl.SSLContext` objects

    cfca05f
  26. picnixz
    removed
    3.14bugs and security fixes
    on Jan 11, 2026

  27. picnixz commented on Jan 11, 2026

    @picnixz
    picnixz
    on Jan 11, 2026
    Member

    I found some (other) possible leaks. I don't know if they are directly related or not, but some context objects may leak in some cases. I don't know if any of the involved libraries actually set one of the attributes, and possibly create some cycles, but I there were some refleaks.

    With the following patch on main, there would be reference leaks. With my patch, those don't leak anymore:

    diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index ebdf5455163..80df9c25d2c 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -57,6 +57,10 @@
 CAN_GET_SELECTED_OPENSSL_SIGALG = ssl.OPENSSL_VERSION_INFO >= (3, 5)
 PY_SSL_DEFAULT_CIPHERS = sysconfig.get_config_var('PY_SSL_DEFAULT_CIPHERS')

+HAS_KEYLOG = hasattr(ssl.SSLContext, 'keylog_filename')
+requires_keylog = unittest.skipUnless(
+    HAS_KEYLOG, 'test requires OpenSSL 1.1.1 with keylog callback')
+
 PROTOCOL_TO_TLS_VERSION = {}
 for proto, ver in (
     ("PROTOCOL_SSLv3", "SSLv3"),
@@ -265,33 +269,69 @@ def utc_offset(): #NOTE: ignore issues like #1647654
 )


-def test_wrap_socket(sock, *,
-                     cert_reqs=ssl.CERT_NONE, ca_certs=None,
-                     ciphers=None, ciphersuites=None,
-                     min_version=None, max_version=None,
-                     certfile=None, keyfile=None,
-                     **kwargs):
-    if not kwargs.get("server_side"):
-        kwargs["server_hostname"] = SIGNED_CERTFILE_HOSTNAME
-        context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
-    else:
+def make_test_context(
+    *,
+    server_side=False,
+    check_hostname=None,
+    cert_reqs=ssl.CERT_NONE,
+    ca_certs=None, certfile=None, keyfile=None,
+    ciphers=None, ciphersuites=None,
+    min_version=None, max_version=None,
+):
+    if server_side:
         context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
-    if cert_reqs is not None:
+    else:
+        context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+
+    if check_hostname is None:
         if cert_reqs == ssl.CERT_NONE:
             context.check_hostname = False
+    else:
+        context.check_hostname = check_hostname
+
+    if cert_reqs is not None:
         context.verify_mode = cert_reqs
+
     if ca_certs is not None:
         context.load_verify_locations(ca_certs)
     if certfile is not None or keyfile is not None:
         context.load_cert_chain(certfile, keyfile)
+
     if ciphers is not None:
         context.set_ciphers(ciphers)
     if ciphersuites is not None:
         context.set_ciphersuites(ciphersuites)
+
     if min_version is not None:
         context.minimum_version = min_version
     if max_version is not None:
         context.maximum_version = max_version
+
+    return context
+
+
+def test_wrap_socket(
+    sock,
+    *,
+    server_side=False,
+    check_hostname=None,
+    cert_reqs=ssl.CERT_NONE,
+    ca_certs=None, certfile=None, keyfile=None,
+    ciphers=None, ciphersuites=None,
+    min_version=None, max_version=None,
+    **kwargs
+):
+    context = make_test_context(
+        server_side=server_side,
+        check_hostname=check_hostname,
+        cert_reqs=cert_reqs,
+        ca_certs=ca_certs, certfile=certfile, keyfile=keyfile,
+        ciphers=ciphers, ciphersuites=ciphersuites,
+        min_version=min_version, max_version=max_version,
+    )
+    kwargs["server_side"] = server_side
+    if not server_side:
+        kwargs.setdefault("server_hostname", SIGNED_CERTFILE_HOSTNAME)
     return context.wrap_socket(sock, **kwargs)


@@ -1730,6 +1770,39 @@ def test_num_tickest(self):
         with self.assertRaises(ValueError):
             ctx.num_tickets = 1

+    @support.cpython_only
+    def test_refcycle_msg_callback(self):
+        # See https://github.com/python/cpython/issues/142516.
+        ctx = make_test_context()
+        def msg_callback(*args, _=ctx, **kwargs): ...
+        ctx._msg_callback = msg_callback
+
+    @support.cpython_only
+    @requires_keylog
+    def test_refcycle_keylog_filename(self):
+        # See https://github.com/python/cpython/issues/142516.
+        self.addCleanup(os_helper.unlink, os_helper.TESTFN)
+        ctx = make_test_context()
+        class KeylogFilename(str): ...
+        ctx.keylog_filename = KeylogFilename(os_helper.TESTFN)
+        ctx.keylog_filename._ = ctx
+
+    @support.cpython_only
+    @unittest.skipUnless(ssl.HAS_PSK, 'requires TLS-PSK')
+    def test_refcycle_psk_client_callback(self):
+        # See https://github.com/python/cpython/issues/142516.
+        ctx = make_test_context()
+        def psk_client_callback(*args, _=ctx, **kwargs): ...
+        ctx.set_psk_client_callback(psk_client_callback)
+
+    @support.cpython_only
+    @unittest.skipUnless(ssl.HAS_PSK, 'requires TLS-PSK')
+    def test_refcycle_psk_server_callback(self):
+        # See https://github.com/python/cpython/issues/142516.
+        ctx = make_test_context(server_side=True)
+        def psk_server_callback(*args, _=ctx, **kwargs): ...
+        ctx.set_psk_server_callback(psk_server_callback)
+

 class SSLErrorTests(unittest.TestCase):

@@ -5163,10 +5236,6 @@ def test_internal_chain_server(self):
                 self.assertEqual(res, b'\x02\n')


-HAS_KEYLOG = hasattr(ssl.SSLContext, 'keylog_filename')
-requires_keylog = unittest.skipUnless(
-    HAS_KEYLOG, 'test requires OpenSSL 1.1.1 with keylog callback')
-
 class TestSSLDebug(unittest.TestCase):

     def keylog_lines(self, fname=os_helper.TESTFN):

  28. picnixz commented on Jan 11, 2026

    @picnixz
    picnixz
    on Jan 11, 2026
    Member

    @nascheme By the way, I found out that the SSL socket object holds some weak references to some other objects (that is, holds the result of PyWeakref_NewRef). Should these pointers be visited? AFAIK, there is no need to since these are weak references but may it's needed? I wasn't able to come up with a reproducer solely relying on SSL and that would actually call traverse.

  29. picnixz commented on Jan 11, 2026

    @picnixz
    picnixz
    on Jan 11, 2026
    Member

    In addition, with Python 3.14.2 (release build provided by uv), your test script with Cell() has a constant RSS on my side (it stays at 36Mb). Same for Victor's script. OTOH, on a DEBUG build with main, it's not constant and sometimes grows and sometimes not. With Victor's script, even if I uncomment gc.collect(), on a debug build on main, it still leaks!

    EDIT: I think the issue stems from the fact that I was using an ASAN DEBUG build.

  30. picnixz commented on Jan 11, 2026

    @picnixz
    picnixz
    on Jan 11, 2026
    Member

    Ok, so I actually dig into this problem a bit more, especially in the "bad" functions. I found out that when we load certificates, we also create lots of small lists and small tuples and lots of nested tuples. I don't know if this could add some burden to the GC. We have a lot of list->tuple conversions and we kinda construct a tree as well (this is _create_tuple_for_X509_NAME).

    I also found some other leaks at import time but it's a separate bug. To help investigating the issue, I'm also considering rewriting the helper functions (e.g., _decode_certificate is quite hard to follow and maybe I missed something).

  31. picnixz
    removed their assignment
    on Jan 11, 2026

  32. youribonnaffe commented on Jan 14, 2026

    @youribonnaffe
    youribonnaffe
    on Jan 14, 2026

    We hit this issue after upgrading some of our applications to Python 3.14.

    What made it worse for us is that we had long running applications that were frequently creating HTTP clients:

    • using the Prometheus client and push_to_gateway periodically. Which is likely not meant to be used like that as it creates clients, thus SSL contexts for every call.
    • re-creating httpx clients over and over, which was also creating SSL contexts.

    Avoiding creating such HTTP clients solved the memory leaks we were observing.
    While this is not going to help solve the actual issue here, I just thought about sharing this in case it might help others.

    Memray was very useful to troubleshoot this and figure out where the HTTP client creations were coming from.

  33. nascheme commented on Jan 14, 2026

    @nascheme
    nascheme
    on Jan 14, 2026
    Member

    @youribonnaffe Thank you for that information, it is helpful I think for CPython maintainers to help prioritize bugfixes and whether to backport them into older releases.

    This kind of scenario is what concerned me about the GC threshold change in GH-140262, which I strongly suggested should be reverted in a 3.14.x bugfix release by GH-142001. My reasoning is as follows: Python is often used for long running services running in VMs or containers with memory usage limits. If a bugfix release of Python causes some of those services to suddenly start using multiple times more memory, I think that's quite serious. It likely means these services will be killed (e.g. by the OOM mechanism in Linux) and the service will fail. To me, that seems much worse than losing some performance due to more frequent GC collections.

    The GH-142001 only affects programs that consume a lot of memory due to garbage reference cycles. Those are very much a minority of all Python programs and so perhaps this is not too major of a problem. However, this garbage cycle in httpx with the SSL context in one example. We can fix it by avoiding the cycle (or explicitly breaking it). But there are likely many more programs that also create garbage cycles and would be affected by this.

  34. HenriBlacksmith commented on Feb 5, 2026

    @HenriBlacksmith
    HenriBlacksmith
    on Feb 5, 2026
    Author

    @nascheme @sergey-miryanov, is there anything in 3.14.3 that could help with GC of reference cycles?

    (I mean should I try it or nothing specific was merged related to that?)

  35. sergey-miryanov commented on Feb 5, 2026

    @sergey-miryanov
    sergey-miryanov
    on Feb 5, 2026
    Contributor

    @HenriBlacksmith AFAIK, 3.14.3 has no related changes to this problem. We are still waiting some feedback in GH-142001.

  36. picnixz
    added 2 commits that reference this issue on Feb 21, 2026

    gh-142516: fix reference leaks in `ssl.SSLContext` objects (#143685)

    3a2a686

    [3.14] pythongh-142516: fix reference leaks in `ssl.SSLContext` objec…

    4607b00
  37. gpshead
    added a commit that references this issue on Feb 21, 2026

    [3.14] gh-142516: fix reference leaks in `ssl.SSLContext` objects (GH…

    1decc7e

  38. gpshead commented on Feb 21, 2026

    @gpshead
    gpshead
    on Feb 21, 2026
    Member

    With the ssl specific PRs merged above, where do we sit with regards to the need for #142001 ?

  39. nascheme commented on Feb 21, 2026

    @nascheme
    nascheme
    on Feb 21, 2026
    Member

    With the ssl specific PRs merged above, where do we sit with regards to the need for #142001 ?

    The ssl context issue is only an example of where this GC collection threshold change causes the problem. Any time you have reference cycles that hold on to a significant amount of memory without having a large number of container objects, you will see the same issue. I.e. sporadic higher memory usage.

    Summarizing Sergey's` tests, Python <= 3.14.1 will break cycles after about 1000 new container objects. Versions >= 3.14.2 will break cycles after about 4600 new container objects. However, it's not very predicable about how many new container objects it takes before it actually looks for cycles.

    If your program doesn't create cycles or those cycles don't consume a lot of memory relative to the number of container objects in them, the 3.14.2 change doesn't hurt. It even gives a performance boost, since the GC runs less often.

  40. brijkapadia
    added a commit that references this issue on Feb 28, 2026

    pythongh-142516: fix reference leaks in `ssl.SSLContext` objects (pyt…

    ef867b3

  41. sergey-miryanov commented on Mar 10, 2026

    @sergey-miryanov
    sergey-miryanov
    on Mar 10, 2026
    Contributor

    JFYI, reproducer without ssl module:

    import gc

def get_rss():
    import psutil

    p = psutil.Process()
    mem_info = p.memory_info()
    rss = mem_info.rss
    return rss / 1024.0

class Cell:
    def __init__(self, ctx):
        self.ctx = ctx
        self.self = self

def main():
    gc.collect(2)
    rss = get_rss()
    N = 16_000
    for i in range(N):
        Cell(b'bb'*(1024*256))
        if i % 100 == 0:
            rss2 = get_rss()
            size = (rss2 - rss)
            print('idx', i, f'size {size:_} kB')

if __name__ == '__main__':
    main()

    For 3.14 and main memory usage grows to 5_687_720.0 kB.

  42. ymyzk
    added a commit that references this issue on Mar 11, 2026

    Add GC workaround for CPython 3.14 memory leak in CloudFunctionsSandbox

    900e81c

  43. cpoppema commented on Mar 18, 2026

    @cpoppema
    cpoppema
    on Mar 18, 2026

    We hit this issue after upgrading some of our applications to Python 3.14.

    What made it worse for us is that we had long running applications that were frequently creating HTTP clients:

    • using the Prometheus client and push_to_gateway periodically. Which is likely not meant to be used like that as it creates clients, thus SSL contexts for every call.
    • re-creating httpx clients over and over, which was also creating SSL contexts.

    Avoiding creating such HTTP clients solved the memory leaks we were observing. While this is not going to help solve the actual issue here, I just thought about sharing this in case it might help others.

    Memray was very useful to troubleshoot this and figure out where the HTTP client creations were coming from.

    FWIW: I ran into similar issues while using the same prometheus client and also push gateways. However, we don't use ssl for this, so in my case the memory did not leak from ssl contexts. Instead it was leaked from the handlers created by this package.

    pushadd_to_gateway - without a handler arg!

    def pushadd_to_gateway(
        gateway: str,
        job: str,
        registry: Optional[Collector],
        grouping_key: Optional[Dict[str, Any]] = None,
        timeout: Optional[float] = 30,
        handler: Callable = default_handler,
        compression: CompressionType = None,
) -> None:
    _use_gateway('POST', gateway, job, registry, grouping_key, timeout, handler, compression)

    _use_gateway:

    def _use_gateway(
        method: str,
        gateway: str,
        job: str,
        registry: Optional[Collector],
        grouping_key: Optional[Dict[str, Any]],
        timeout: Optional[float],
        handler: Callable,
        compression: CompressionType = None,
) -> None:
    ...

    handler(
        url=url, method=method, timeout=timeout,
        headers=headers, data=data,
    )()

    The default_handler is in fact not a real handler: it creates one on the fly:

    def default_handler(
        url: str,
        method: str,
        timeout: Optional[float],
        headers: List[Tuple[str, str]],
        data: bytes,
) -> Callable[[], None]:
    ...
    return _make_handler(url, method, timeout, headers, data, HTTPHandler)

def _make_handler(
        url: str,
        method: str,
        timeout: Optional[float],
        headers: Sequence[Tuple[str, str]],
        data: bytes,
        base_handler: Union[BaseHandler, type],
) -> Callable[[], None]:
    def handle() -> None:
        ...

    return handle

    As a quick experiment I cached the output of default_handler and the memory issues went away quickly. In the end I chose the least invasive change which is to just simply run gc.collect() right after the call to pushadd_to_gateway(), for now.

    NB: also in python 3.13 there is more memory being used than "strictly needed" - but that was not an issue so far. A machine with 16 GB RAM running multiple background tasks (25+) pushing metrics it ran out of memory within a day. With 3.13 it never got this bad: since this only hit our staging environment I don't have data for more than a week; those jobs are redeployed and start memory usage from scratch, so all I can say is that so far it has never exceeded that 16 GB of RAM in a week.

  44. nascheme commented on Apr 10, 2026

    @nascheme
    nascheme
    on Apr 10, 2026
    Member

    Two suggestions for testing: add explicit gc.collect() calls, if this keeps memory usage in check, it basically confirms it is an issue with cyclic garbage not being freed soon enough. You can also try apply the patch from #142001 which would confirm the same. If you are using httpx, make sure you have this fix in your version: encode/httpx#3733 . That might not be the only reference cycle though and if there are others, 3.14.2+ will likely use more RAM before breaking those cycles.

  45. picnixz
    added 2 commits that reference this issue on Apr 11, 2026

    [3.13] pythongh-142516: fix reference leaks in `ssl.SSLContext` objec…

    5aa7ad2

    [3.14] pythongh-142516: fix reference leaks in `ssl.SSLContext` objec…

    c690e8c

  46. bedevere-app commented on Apr 17, 2026

    @bedevere-app
    bedevere-appBot
    on Apr 17, 2026

    GH-148694 is a backport of this pull request to the 3.14 branch.

  47. ljfp
    added a commit that references this issue on Apr 25, 2026

    pythongh-142516: fix reference leaks in `ssl.SSLContext` objects (pyt…

    bbaf112

  48. vstinner commented on Apr 30, 2026

    @vstinner
    vstinner
    on Apr 30, 2026
    Member

    It was decided to Reverting the incremental GC in Python 3.14 and 3.15: see PR gh-148720 for the 3.14 branch. So can we close this issue?

  49. hugovk commented on May 5, 2026

    @hugovk
    hugovk
    on May 5, 2026
    Member

    This has been reverted and released in Python 3.14.5 release candidate, please test.

    https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1

    We plan a 3.14.5 final for Friday.

  50. HenriBlacksmith commented on May 19, 2026

    @HenriBlacksmith
    HenriBlacksmith
    on May 19, 2026
    Author

    This has been reverted and released in Python 3.14.5 release candidate, please test.

    https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1

    We plan a 3.14.5 final for Friday.

    I have been testing this on two APIs in our dev environment and it looks memory is stable. Will provide more feedback once it will be in prod.

  51. picnixz
    added a commit that references this issue on May 24, 2026

    [3.13] gh-142516: fix reference leaks in `ssl.SSLContext` objects (GH…

    e102378
  52. RonnyPfannschmidt
    added 4 commits that reference this issue on May 28, 2026

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    53167e5

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    5fbde2c

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    9e43244

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    594e4f5

  53. HenriBlacksmith commented on Jun 2, 2026

    @HenriBlacksmith
    HenriBlacksmith
    on Jun 2, 2026
    Author

    This has been reverted and released in Python 3.14.5 release candidate, please test.
    https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1
    We plan a 3.14.5 final for Friday.

    I have been testing this on two APIs in our dev environment and it looks memory is stable. Will provide more feedback once it will be in prod.

    It has been running in prod for a week with similar memory profiles as for 3.13.x.

    The issue is resolved from my point of view

  54. miketheman commented on Jun 2, 2026

    @miketheman
    miketheman
    on Jun 2, 2026
    Member

    I can also confirm that the memory footprint is now "stable" since 3.14.5 - and a side benefit has been that the "floor" is lower from 3.13.x

    Image
  55. HenriBlacksmith
    closed this as completedon Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-SSLtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions