Repository navigation
gh-161184: Use a managed dict and weakref list for the _io types - #161236
Conversation
Replace the explicit dict slot of _io._IOBase and its C subclasses with Py_TPFLAGS_MANAGED_DICT, so instance attributes are stored as inline values with shared keys instead of a per-instance dict, and attribute access can be specialized. The flag is inherited by all C and Python subclasses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replace the explicit weakreflist field of the _io types with Py_TPFLAGS_MANAGED_WEAKREF, inherited from _IOBase, so the weakref list lives in the pre-header like the managed dict. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cmaloney
left a comment
There was a problem hiding this comment.
This exposes a potential double-dealloc in _PyObject_ResurrectEnd on free-threaded builds that should probably fix in a separate issue before landing:
import io, weakref
class Raw(io.RawIOBase): pass
class Attr:
def __del__(self):
self.wr() # deref a weakref to the owner during its dealloc
o = Raw(); o.a = Attr(); o.a.wr = weakref.ref(o)
del oFrom Claude:
- io subclasses always have tp_finalize. On free-threaded builds, the _PyObject_ResurrectEnd fast path then leaves ob_tid set to the current thread, with both refcounts at 0 and the object not marked merged.
- subtype_dealloc now clears the managed dict before the C base's dealloc clears the weakrefs. Previously most _io bases cleared weakrefs first.
- A weakref deref during that window succeeds through _Py_TryIncrefFast, and its decref starts a second dealloc inside the first.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ate__ Check _PyObject_IsInstanceDictEmpty() first, as object.__getstate__() does, instead of creating an empty dict just to find out that it is empty. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
I created #161419 |
|
|
gh-162342 for this particular bot failure. It looks like there's a cache in |
_io._IOBaseand its C subclasses carried an explicitdictandweakreflistfield exposed through__dictoffset__/__weaklistoffset__. This PR switches toPy_TPFLAGS_MANAGED_DICT | Py_TPFLAGS_MANAGED_WEAKREF(inherited by subclasses, as #160616 did forast.AST). The managed dict improves performance, the weakref part is added for consistency (performance-neutral, saves a pointer per instance).Benchmarks
http.client:HTTPResponse(sock).begin()+read()of a 2-byte bodyhttp.client:read()of a chunked body (1000 chunks)http.client:read(100)x 1000http.client:for line in response(1000 lines)zipfile:zf.read(name)for 200 small memberszipfile:for line in zf.open(name)(1000 lines)zipfile:ZipExtFile.read(100)x 1000gzip:GzipFile.read(100)x 1000gzip:GzipFile.write(100 bytes)x 1000tempfile:SpooledTemporaryFile.write(100 bytes)x 1000urllib3:HTTPResponse.read(100)x 1000urllib3:for chunk in response.stream(100)list(range(20000)))Compatibility analysis from Claude
Python subclasses are unaffected:
__dict__,__weakref__,__slots__, pickling and copying behave as before. Two changes are visible to C code and introspection:__dictoffset__and__weakrefoffset__now report the managed pre-header offsets (-1 and -32 on 64-bit builds), and the_iostruct sizes shrink by two pointers. An extension type created withPyType_FromSpecon an_iobase that also declares its own__dictoffset__or__weaklistoffset__member now fails at type creation withTypeError: type X has the Py_TPFLAGS_MANAGED_DICT flag but tp_dictoffset is set, instead of silently getting a second dict slot. StaticPyTypeObjectsubclasses of the_iotypes have been impossible since they became heap types, Cython does not allowcdef class X(io.RawIOBase), and a survey of PyPI projects that build on the_iotypes (python-zstandard, pyzstd, cramjam, pyarrow, rapidgzip, indexed_gzip, pysam, cffi, among others) found no compiled subclass of any_iotype, so no known package is affected.Benchmark script