diff --git a/peps/pep-0803.rst b/peps/pep-0803.rst index cb1643e4343..8eb78771d89 100644 --- a/peps/pep-0803.rst +++ b/peps/pep-0803.rst @@ -1,5 +1,5 @@ PEP: 803 -Title: Stable ABI for Free-Threaded Builds +Title: "abi3t": Stable ABI for Free-Threaded Builds Author: Petr Viktorin Discussions-To: https://discuss.python.org/t/103628 Status: Draft @@ -13,13 +13,19 @@ Post-History: `08-Sep-2025 `__ Abstract ======== -Version 3.15 of the Stable ABI will be compatible with both free-threaded and -GIL-enabled builds. -To allow this, the :c:type:`PyObject` internal structure and related APIs -will be removed from version 3.15 of the Limited API, requiring migration to -new API for common tasks like defining modules and most classes. +Add a new variant of the Stable ABI, called “Stable ABI for Free-Threaded +Python” (or ``abi3t`` for short), and a corresponding API limitations. +``abi3t`` will be compatible with both free-threaded and GIL-enabled builds of +CPython. -Binary distributions (wheels) built with Limited API version 3.15 and above +To build for ``abi3t``, users define both :c:macro:`Py_GIL_DISABLED` +and :c:macro:`Py_LIMITED_API` before including ``Python.h``, +and avoid using API that relies on the layout of the :c:type:`PyObject` +structure. +This will require users to migrate to new API for common tasks like defining +modules and most classes. + +Binary distributions (wheels) built with this variant of the Limited API should use the ABI tag ``abi3.abi3t``. @@ -34,8 +40,10 @@ Motivation ========== The Stable ABI is currently not available for free-threaded builds. -Extensions will fail to build when :c:macro:`Py_LIMITED_API` is defined, -and extensions built for GIL-enabled builds of CPython will fail to load +Extensions will fail to build for a both Limited API and +free-threaded Python (that is, when both :c:macro:`Py_LIMITED_API` and +:c:macro:`Py_GIL_DISABLED` preprocessor macros are defined). +Extensions built for GIL-enabled builds of CPython will fail to load (or crash) on free-threaded builds. In its `acceptance post `__ @@ -48,42 +56,44 @@ This PEP proposes the Stable ABI for free-threading. Background ---------- -Python's Stable ABI, as defined in :pep:`384` and :pep:`652`, provides a way to -compile extension modules that can be loaded on multiple minor versions of the -CPython interpreter. +Python's Stable ABI (``abi3`` for short), as defined in :pep:`384` and +:pep:`652`, provides a way to compile extension modules that can be loaded +on multiple minor versions of the CPython interpreter. Several projects use this to limit the number of :ref:`wheels ` (binary artefacts) that need to be built and distributed for each release, and/or to make it easier to test with pre-release versions of Python. With free-threading builds (:pep:`703`) being on track to eventually become -the default (:pep:`779`), we need a way to make the Stable ABI available +the default (:pep:`779`), we need a way to make a Stable ABI available to those builds. -To build against the Stable ABI, the extension must use a *Limited API*, +To build against a Stable ABI, the extension must use a *Limited API*, that is, only a subset of the functions, structures, etc. that CPython exposes. -Both the Limited API and the Stable ABI are versioned, and building against -Stable ABI 3.X requires using only Limited API 3.X, and yields an extension -that is ABI-compatible with CPython 3.X and *any* later version +Both the Limited API and the current Stable ABI are versioned, and building +against Stable ABI 3.X requires using only Limited API 3.X, and yields an +extension that is ABI-compatible with CPython 3.X and *any* later version (though bugs in CPython sometimes cause incompatibilities in practice). The Limited API is not “stable”: newer versions may remove API that were a part of older versions. -This PEP proposes the most significant such removal to date. + +This PEP proposes additional API limitations, as required to be compatible +with both GIL-enabled and free-threaded builds of CPython. Rationale ========= -The design in this PEP makes several assumptions: +The design in this PEP uses several constraints: -One ABI +Unified ABI A single compiled extension module should support both free-threaded and GIL-enabled builds. No backwards compatibility now - The new limited API will not support CPython 3.14 and below. + The new stable ABI variant will not support CPython 3.14 and below. Projects that need this support can build separate extensions specifically for the 3.14 free-threaded interpreter, and for older stable ABI versions. @@ -92,24 +102,51 @@ No backwards compatibility now See a :ref:`rejected idea ` for how this could work. API changes are OK - The new Limited API may require extension authors to make significant - changes to their code. - Projects that cannot do this (yet) can continue using Limited API 3.14, - which will yield extensions compatible with GIL-enabled builds only. + The new Limited API variant may require extension authors to make + significant changes to their code. + Projects that cannot do this (yet) can continue using Limited API, + which will yield separate extensions compatible with GIL-enabled builds + and with specific versions of free-threaded builds. -No extra configuration +No new configuration We do not introduce new “knobs” that influence what API is available and what the ABI is compatible with. + Instead, we reuse the existing ``Py_GIL_DISABLED`` macro. + + +Tag name +-------- + +The tag ``abi3t`` is chosen to reflect the fact that this ABI is similar to +``abi3``, with minimal changes necessary to support free-threading (which +uses the letter ``t`` in existing, version-specific ABI tags like ``cp314t``). Specification ============= +Building for ``abi3`` +--------------------- + +When the :c:macro:`Py_LIMITED_API` macro is set to ``0x030f0000`` (3.15) +or above, Python will newly allow the :c:macro:`Py_GIL_DISABLED` to be defined. +(Currently -- in 3.14 -- it is an error when both are defined.) + +The :c:macro:`Py_GIL_DISABLED` macro can be set in several ways, for example: + +- on non-Windows platforms, by compiling with headers configured for + free-threaded CPython; +- by passing ``Py_GIL_DISABLED`` to the compiler (possibly via a build tool); + or +- using ``#define`` directly before including ``Python.h``. + + Opaque PyObject --------------- -Version 3.15 of the Limited API will: +When both :c:macro:`Py_GIL_DISABLED` and :c:macro:`Py_LIMITED_API` is defined, +the CPython headers will: - make the following structures *opaque* (or in C terminology, *incomplete types*): @@ -127,10 +164,12 @@ Version 3.15 of the Limited API will: - :c:macro:`PyObject_VAR_HEAD` - :c:func:`Py_SET_TYPE` -- export the following as functions in the ABI, rather than macros: +In both the regular stable ABI (``abit3`` 3.15+) and the new +``abi3t``, the following will be exported functions (exposed in the ABI) +rather than macros: - - :c:func:`Py_SIZE` - - :c:func:`Py_SET_SIZE` +- :c:func:`Py_SIZE` +- :c:func:`Py_SET_SIZE` Implications @@ -160,10 +199,10 @@ opaque means: extension modules. Extensions will need to switch to API added in :pep:`793`. -The following functions will become unusable in practice (in the new Limited -API), since an extension cannot create valid, statically allocated, input -for them. To ease the transition for extension developers, -they will not yet be removed from the Limited API: +The following functions will become practically unusable in ``abi3t``, +since an extension cannot create valid, statically allocated, input +for them. +They will, however, not be removed. - :c:func:`PyModuleDef_Init` - :c:func:`PyModule_Create`, :c:func:`PyModule_Create2` @@ -177,7 +216,7 @@ Implementation of this PEP requires :pep:`793` (``PyModExport``): A new entry point for C extension modules), which was accepted for Python 3.15. -Since existing ways of defining modules use API that this PEP removes +Since existing ways of defining modules use API that is removed in ``abi3t`` (namely, :c:type:`PyModuleDef`), extensions will need to migrate to PEP 793's new “export hook” when switching to Limited API 3.15. @@ -210,19 +249,19 @@ This slot will become *mandatory* with the new export hook added in (That PEP currently says “there are no required slots”; it will be updated.) -Check for older ``abi3`` -^^^^^^^^^^^^^^^^^^^^^^^^ +Check for non-free-threaded ABI in free-threading builds +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Additionally, in free-threaded builds, :c:func:`PyModuleDef_Init` will detect -extensions using the pre-free-threading Stable ABI, emit an informative +extensions using the non-free-threading Stable ABI, emit an informative message when one is loaded, *and* raise an exception. (Implementation note: A message will be printed before raising the exception, because extensions that attempt to handle an exception using incompatible ABI will likely crash and lose the exception's message.) -This check for older ``abi3`` relies on internal bit patterns and may be -removed in future CPython versions, if the internal object layout needs -to change. +This check for non-free-threading ``abi3`` relies on internal bit patterns +and may be removed in future CPython versions, +if the internal object layout needs to change. The ``abi3t`` wheel tag @@ -230,9 +269,6 @@ The ``abi3t`` wheel tag Wheels that use a stable ABI compatible with free-threading CPython builds should use a new `ABI tag`_: ``abi3t``. -The name is chosen to reflect the fact that this ABI is similar to ``abi3``, -with limitations necessary to support free-threading (which uses the letter -``t`` in existing, version-specific ABI tags like ``cp314t``). .. _ABI tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#abi-tag @@ -241,9 +277,10 @@ They should allow ``abi3t``-tagged wheels for free-threaded builds wherever they currently allow ``abi3``-tagged ones for (orherwise equal) non-free-threaded builds. -Build tools should generate ``abi3.abi3t`` instead of ``abi3`` when the Python -tag is ``cp315`` and above (or equivalently: when setting ``Py_LIMITED_API`` -to ``3.15`` (``0x030f0000``) or above). +Build tools should generate ``abi3.abi3t`` instead of ``abi3`` for extensions +targetting the Stable ABI for free-threaded Python -- or equivalently: when +both defining ``Py_GIL_DISABLED`` and setting ``Py_LIMITED_API`` to ``3.15`` +(``0x030f0000``) or above. ``abi3.abi3t`` is a `compressed tag set`_ that signals compatibility with both ``abi3`` and ``abi3t``. @@ -277,23 +314,26 @@ Implementing this PEP will make it possible to build extensions that can be successfully loaded on free-threaded Python, but not necessarily ones that are thread-safe without a GIL. -Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, ``PyCriticalSection``, and -similar -- will be added via the C API working group, or in a follow-up PEP. +Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, +``PyCriticalSection``, and similar -- will be added via the C API working group, +or in a follow-up PEP. Backwards and Forwards Compatibility ==================================== -Limited API 3.15 will not be backwards-compatible with older CPython releases, -due to the need to use new ``PyModExport`` API added in :pep:`793`. +Extensions targetting ``abi3t`` will not be backwards-compatible with older +CPython releases, due to the need to use new ``PyModExport`` API added +in :pep:`793`. -Extension authors who cannot switch may continue to use Limited API 3.14 -and below. +Extension authors who cannot switch may continue to use the existing ``abi3``, +that is, build on GIL-enabled Python without defining ``Py_GIL_DISABLED``. For compatibility with free-threaded builds, they can compile using -version-specific ABI -- for example, compile on CPython 3.15 without defining -``Py_LIMITED_API``. +version-specific ABI -- that is, compile ``abi3``-compatible source +on free-threaded CPython 3.15 without defining ``Py_LIMITED_API``. -Limited API 3.15 will be forward-compatible with future versions of CPython 3.x. +Limited API 3.15 for free-threading is a subset of the general Limited API, +and as such, it will be forward-compatible with future versions of CPython 3.x. Older versions of the Limited API (that is, 3.14 and below) will continue to be forward-compatible with GIL-enabled builds of CPython 3.x, starting with the version that introduced the given Limited API. @@ -432,24 +472,24 @@ built with a given interpreter and ``Py_LIMITED_API`` macro: * * ``cp315-abi3`` * 3.15+ (GIL) * ``PY_PACK_VERSION(3, 15)`` - * discontinued + * continued * * ``cp315-abi3t`` * N/A - * N/A + * N/A` * out of spec * * ``cp315-abi3.abi3t`` - * 3.15+ (any) + * 3.15+ (FT) * ``PY_PACK_VERSION(3, 15)`` * new +In the “Compile on” column, *FT* means that the :c:macro:`Py_GIL_DISABLED` +macro must be defined -- either explicitly or, on non-Windows platforms, +by including CPython headers configured with :option:`--disable-gil`. + Values in the *Note* column: * *existing*: The wheel tag is currently in use * *continued*: The wheel tag continues the existing scheme -* *discontinued*: The wheel tag continues the existing scheme, but it will - be discouraged. Older tools may still generate it. Installers will - continue to accept it, but only for GIL-enabled builds, even though the wheel - would be compatible with free-threaded ones. * *new*: Proposed in this PEP. * *reserved*: A mechanism to build a matching extension is not proposed in this PEP, but may be added in the future. @@ -496,23 +536,38 @@ Rejected Ideas ============== -Add an alternative stable ABI for free-threading +Add an separate stable ABI for free-threading ------------------------------------------------ -It would be possible to: +It would be possible to define a new stable ABI compatible *only* with +free-threading builds, not with the existing ``abi3``. +Extensions would need no code changes to target ``abi3t`` and builds would be +compatible with free-threaded CPython (3.14 and above). + +In this scheme, an additional macro (“``Py_OPAQUE_PYOBJECT``”), could make +``PyObject`` opaque as in this PEP. To use it, extensions would need code +changes as in this PEP, and as in this PEP, compiled extensions +(“``abi3.abi3t``”) would be compatible with all builds of CPython 3.15+. -- Add a new stable ABI (“``abi3t``”) specifically for free-threading, - which would be incompatible with the existing ``abi3``. - Extensions would need no code changes to target ``abi3t`` and builds would be - compatible with free-threaded CPython (3.14 and above). -- Define an additional macro (“``Py_OPAQUE_PYOBJECT``”), which would make - ``PyObject`` opaque as in this PEP. Extensions would need code changes as in - this PEP, and as in this PEP, compiled extensions (“``abi3.abi3t``”) would - be compatible with all builds of CPython 3.15+. +This would make the free-threading memory layout of ``PyObject`` part +of a stable ABI, preventing future adjustments. -This scheme was rejected as too complex. -It would also make the free-threading memory layout of ``PyObject`` part -of the stable ABI, preventing future adjustments. + +Make ``PyObject`` opaque in Limited API 3.15 +-------------------------------------------- + +It would be possible to make ``PyObject`` struct opaque in Limited API 3.15 +(that is, the new version of the existing Limited API), +rather than introduce a new variant of the Stable ABI and Limited API. + +This would mean that extension authors would need to adapt their code to the +new limitations, or abandon Limited API altogether, in order to use any C API +introduced in Python 3.15. + +It would also not remove the need for a new wheel tag (``abi3t``), +which would be required to express that an extension +is compatible with both GIL-enabled and free-threaded builds +of CPython 3.14 or lower. .. _pep803-no-shim: @@ -547,29 +602,11 @@ extension to query the running interpreter, and for 3.14, use a ``struct`` definition corresponding to the detected build's ``PyModuleDef``. -.. _pep803-no-avoid-abi3t: - -Using the Python version wheel tag to determine compatibility -------------------------------------------------------------- - -A previous version of this PEP avoided adding a new wheel tag (``abi3t``), -and specified that wheels tagged ``abi3`` would be compatible with -free-threading if the *Python tag* is ``cp315`` or higher. - -Such a scheme would work for this PEP, but it cannot express that an extension -is compatible with both GIL-enabled and free-threaded builds -of CPython 3.14 or lower. -Adding a new explicit tag means that *if* we allow building such wheels in the -future, packaging tools should not need additional changes to support them. -They would be tagged ``cp314-abi3.abi3t``. - - -Adding an ``abi4`` wheel tag ----------------------------- +Naming this ``abi4`` +-------------------- -Instead of ``abi3t``, we could “bump the version” and use ``abi4`` instead -as the wheel ABI tag. -In the wheel tag, the difference is largely cosmetic. +Instead of ``abi3t``, we could “bump the version” and use ``abi4`` instead. +The difference is largely cosmetic. However, one thing this PEP does not propose is changing the *filename* tag: extensions will be named with the extensions like ``.abi3.so``. @@ -579,7 +616,7 @@ an unnecessary technical change. Using ``abi3.abi4`` in wheel tags but only ``.abi3`` in filenames would look more inconsistent than ``abi3.abi3t`` and ``.abi3``. -If we add ``abi4`` tag, the ``Py_LIMITED_API`` value would either need to: +If we added an ``abi4`` tag, the ``Py_LIMITED_API`` value would either need to: * change to start with ``4`` to match ``abi4``, but no longer correspond to ``PY_VERSION_HEX`` (making it harder to generate and check), or