Merge main

.github/workflows/build.yml

@@ -4,7 +4,7 @@ on:
   workflow_dispatch:
   push:
     branches:
-    - 'main'
+    - 'meta/main'
     - '3.*'
   pull_request:
     branches:

Doc/c-api/exceptions.rst

@@ -412,7 +412,7 @@ an error value).
 
 .. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...)
 
-   Function similar to :c:func:`PyErr_WarnEx`, but use
+   Function similar to :c:func:`PyErr_WarnEx`, but uses
    :c:func:`PyUnicode_FromFormat` to format the warning message.  *format* is
    an ASCII-encoded string.
 
@@ -1392,7 +1392,7 @@ Tracebacks
 
    This function will return ``NULL`` on success, or an error message on error.
 
-   This function is meant to debug debug situations such as segfaults, fatal
+   This function is meant to debug situations such as segfaults, fatal
    errors, and similar. It calls :c:func:`PyUnstable_DumpTraceback` for each
    thread. It only writes the tracebacks of the first *max_threads* threads,
    further output is truncated with the line ``...``. If *max_threads* is 0, the

Doc/c-api/import.rst

@@ -393,11 +393,6 @@ Importing Modules
 
       Make all imports lazy by default.
 
-   .. c:enumerator:: PyImport_LAZY_NONE
-
-      Disable lazy imports entirely. Even explicit ``lazy`` statements become
-      eager imports.
-
    .. versionadded:: 3.15
 
 .. c:function:: PyObject* PyImport_CreateModuleFromInitfunc(PyObject *spec, PyObject* (*initfunc)(void))

Doc/c-api/sentinel.rst

@@ -31,12 +31,12 @@ Sentinel objects
 
    .. versionadded:: 3.15
 
-.. c:function:: PyObject* PySentinel_New(const char *name, const char *module_name)
+.. c:function:: PyObject* PySentinel_New(const char *name, const char *module_name, const char *repr)
 
    Return a new :class:`sentinel` object with :attr:`~sentinel.__name__` set to
    *name* and :attr:`~sentinel.__module__` set to *module_name*.
    *name* must not be ``NULL``. If *module_name* is ``NULL``, :attr:`~sentinel.__module__`
-   is set to ``None``.
+   is set to ``None``. If *repr* is ``NULL``, ``repr()`` returns :attr:`~sentinel.__name__`.
    Return ``NULL`` with an exception set on failure.
 
    For pickling to work, *module_name* must be the name of an importable

Doc/c-api/threads.rst

@@ -736,10 +736,10 @@ Low-level APIs
 .. c:function:: PyObject* PyThreadState_GetDict()
 
    Return a dictionary in which extensions can store thread-specific state
-   information.  Each extension should use a unique key to use to store state in
+   information.  Each extension should use a unique key to store a state in
    the dictionary.  It is okay to call this function when no :term:`thread state`
    is :term:`attached <attached thread state>`. If this function returns
-   ``NULL``, no exception has been raised and the caller should assume no
+   ``NULL`` and no exception has been raised, then the caller should assume no
    thread state is attached.
 
 

Doc/c-api/typehints.rst

@@ -31,7 +31,7 @@ two types exist -- :ref:`GenericAlias <types-genericalias>` and
       static PyMethodDef my_obj_methods[] = {
           // Other methods.
           ...
-          {"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "See PEP 585"}
+          {"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "my_obj is generic over its contained type"}
           ...
       }
 

Doc/conf.py

@@ -349,8 +349,6 @@
   \sphinxstrong{Python Software Foundation}\\
   Email: \sphinxemail{docs@python.org}
 }
-\let\Verbatim=\OriginalVerbatim
-\let\endVerbatim=\endOriginalVerbatim
 \setcounter{tocdepth}{2}
 ''',
     # The paper size ('letterpaper' or 'a4paper').

Doc/data/refcounts.dat

@@ -2040,6 +2040,7 @@ PySeqIter_New:PyObject*:seq:0:
 PySentinel_New:PyObject*::+1:
 PySentinel_New:const char*:name::
 PySentinel_New:const char*:module_name::
+PySentinel_New:const char*:repr::
 
 PySequence_Check:int:::
 PySequence_Check:PyObject*:o:0:

Doc/deprecations/index.rst

@@ -13,6 +13,8 @@ Deprecations
 
 .. include:: pending-removal-in-3.20.rst
 
+.. include:: pending-removal-in-3.21.rst
+
 .. include:: pending-removal-in-future.rst
 
 .. include:: soft-deprecations.rst

Doc/deprecations/pending-removal-in-3.21.rst

@@ -0,0 +1,19 @@
+Pending removal in Python 3.21
+------------------------------
+
+* :mod:`abc`
+
+   * Soft-deprecated since Python 3.3 :class:`abc.abstractclassmethod`,
+     :class:`abc.abstractstaticmethod`, and :class:`abc.abstractproperty`
+     now raise a :exc:`DeprecationWarning`.
+     These classes will be removed in Python 3.21, instead
+     use :func:`abc.abstractmethod` with :func:`classmethod`,
+     :func:`staticmethod`, and :class:`property` respectively.
+
+* :mod:`ast`:
+
+  * Classes ``slice``, ``Index``, ``ExtSlice``, ``Suite``, ``Param``,
+    ``AugLoad`` and ``AugStore``, will be removed in Python 3.21. These types
+    are not generated by the parser or accepted by the code generator.
+  * The ``dims`` property of ``ast.Tuple`` will be removed in Python 3.21. Use
+    the ``ast.Tuple.elts`` property instead.

Doc/howto/a-conceptual-overview-of-asyncio.rst

@@ -115,7 +115,7 @@ The terms "coroutine function" and "coroutine object" are often conflated
 as coroutine.
 That can be confusing!
 In this article, coroutine specifically refers to a coroutine object, or more
-precisely, an instance of :data:`types.CoroutineType` (native coroutine).
+precisely, an instance of :class:`types.CoroutineType` (native coroutine).
 Note that coroutines can also exist as instances of
 :class:`collections.abc.Coroutine` -- a distinction that matters for type
 checking.

Doc/howto/free-threading-python.rst

@@ -165,3 +165,132 @@ to false.  If the flag is true then the :class:`warnings.catch_warnings`
 context manager uses a context variable for warning filters.  If the flag is
 false then :class:`~warnings.catch_warnings` modifies the global filters list,
 which is not thread-safe.  See the :mod:`warnings` module for more details.
+
+
+Increased memory usage
+----------------------
+
+The free-threaded build will typically use more memory compared to the default
+build.  There are multiple reasons for this, mostly due to design decisions.
+
+
+All interned strings are immortal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For modern Python versions (since version 2.3), interning a string (e.g. with
+:func:`sys.intern`) does not cause it to become immortal.  Instead, if the last
+reference to that string disappears, it will be removed from the interned
+string table.  This is not the case for the free-threaded build and any interned
+string will become immortal, surviving until interpreter shutdown.
+
+
+Non-GC objects have a larger object header
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The free-threaded build uses a different :c:type:`PyObject` structure.  Instead
+of having the GC related information allocated before the :c:type:`PyObject`
+structure, like in the default build, the GC related info is part of the normal
+object header.  For example, on the AMD64 platform, ``None`` uses 32 bytes on
+the free-threaded build vs 16 bytes for the default build.  GC objects (such as
+dicts and lists) are the same size for both builds since the free-threaded
+build does not use additional space for the GC info.
+
+
+QSBR can delay freeing of memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to safely implement lock-free data structures, a safe memory
+reclamation (SMR) scheme is used, known as quiescent state-based reclamation
+(QSBR).  This means that the memory backing data structures allowing lock-free
+access will use QSBR, which defers the free operation, rather than immediately
+freeing the memory.  Two examples of these data structures are the list object
+and the dictionary keys object.  See ``InternalDocs/qsbr.md`` in the CPython
+source tree for more details on how QSBR is implemented.  Running
+:func:`gc.collect` should cause all memory being held by QSBR to be actually
+freed.  Note that even when QSBR frees the memory, the underlying memory
+allocator may not immediately return that memory to the OS and so the resident
+set size (RSS) of the process might not decrease.
+
+
+mimalloc allocator vs pymalloc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default build will normally use the "pymalloc" memory allocator for small
+allocations (512 bytes or smaller).  The free-threaded build does not use
+pymalloc and allocates all Python objects using the "mimalloc" allocator.  The
+pymalloc allocator has the following properties that help keep memory usage
+low: small per-allocated-block overhead, effective memory fragmentation
+prevention, and quick return of free memory to the operating system.  The
+mimalloc allocator does quite well in these respects as well but can have some
+more overhead.
+
+In the free-threaded build, mimalloc manages memory in a number of separate
+heaps (currently four).  For example, all GC supporting objects are allocated
+from their own heap.  Using separate heaps means that free memory in one heap
+cannot be used for an allocation that uses another heap.  Also, some heaps are
+configured to use QSBR (quiescent-state based reclamation) when freeing the
+memory that backs up the heap (known as "pages" in mimalloc terminology).  The
+use of QSBR creates a delay between all memory blocks for a page being freed
+and the memory page being released, either for new allocations or back to the
+OS.
+
+The mimalloc allocator also defers returning freed memory back to the OS.  You
+can reduce that delay by setting the environment variable
+:envvar:`!MIMALLOC_PURGE_DELAY` to ``0``.  Note that this will likely reduce
+the performance of the allocator.
+
+
+Free-threaded reference counting can cause objects to live longer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the default build, when an object's reference count reaches zero, it is
+normally deallocated.  The free-threaded build uses "biased reference
+counting", with a fast-path for objects "owned" by the current thread and a
+slow path for other objects.  See :pep:`703` for additional details.  Any time
+an object's reference count ends up in a "queued" state, deallocation can be
+deferred.  The queued state is cleared from the "eval breaker" section of the
+bytecode evaluator.
+
+The free-threaded build also allows a different mode of reference counting,
+known as "deferred reference counting".  This mode is enabled by setting a flag
+on a per-object basis.  Deferred reference counting is enabled for the
+following types:
+
+* module objects
+* module top-level functions
+* class methods defined in the class scope
+* descriptor objects
+* thread-local objects, created by :class:`threading.local`
+
+When deferred reference counting is enabled, references from Python function
+stacks are not added to the reference count.  This scheme reduces the overhead
+of reference counting, especially for objects used from multiple threads.
+Because the stack references are not counted, objects with deferred reference
+counting are not immediately freed when their internal reference count goes to
+zero.  Instead, they are examined by the next GC run and, if no stack
+references to them are found, they are freed.  This means these objects are
+freed by the GC and not when their reference count goes to zero, as is typical.
+
+
+Per-thread reference counting can delay freeing objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid contention on the reference count fields of frequently shared
+objects, the free-threaded build also uses "per-thread reference counting"
+for a few selected object types.  Rather than updating a single shared
+reference count, each thread maintains its own local reference count array,
+indexed by a unique id assigned to the object.  The true reference count is
+only computed by summing the per-thread counts when the object's local
+count drops to zero.  Per-thread reference counting is currently used for:
+
+* heap type objects (classes created in Python)
+* code objects
+* the ``__dict__`` of module objects
+
+Because the per-thread counts must be merged back to the object before it
+can be deallocated, objects using per-thread reference counting are
+typically freed later than they would be in the default build.  In
+particular, such an object is usually not freed until the thread that
+referenced it reaches a safe point (for example, in the "eval breaker"
+section of the bytecode evaluator) or exits.  Running :func:`gc.collect`
+will merge the per-thread counts and allow these objects to be freed.