Skip to content

gh-161184: Use a managed dict and weakref list for the _io types - #161236

Merged
kumaraditya303 merged 7 commits into
python:mainfrom
eendebakpt:io-managed-weakref
Nov 20, 2026
Merged

kumaraditya303 merged 7 commits into
python:mainfrom
eendebakpt:io-managed-weakref

Conversation

@eendebakpt

@eendebakpt eendebakpt commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

_io._IOBase and its C subclasses carried an explicit dict and weakreflist field exposed through __dictoffset__/__weaklistoffset__. This PR switches to Py_TPFLAGS_MANAGED_DICT | Py_TPFLAGS_MANAGED_WEAKREF (inherited by subclasses, as #160616 did for ast.AST). The managed dict improves performance, the weakref part is added for consistency (performance-neutral, saves a pointer per instance).

Benchmarks

Benchmark main PR
http.client: HTTPResponse(sock).begin() + read() of a 2-byte body 22 us 21 us: 1.06x faster
http.client: read() of a chunked body (1000 chunks) 707 us 525 us: 1.35x faster
http.client: read(100) x 1000 293 us 144 us: 2.03x faster
http.client: for line in response (1000 lines) 317 us 166 us: 1.91x faster
zipfile: zf.read(name) for 200 small members 1.23 ms 922 us: 1.33x faster
zipfile: for line in zf.open(name) (1000 lines) 269 us 176 us: 1.53x faster
zipfile: ZipExtFile.read(100) x 1000 266 us 177 us: 1.50x faster
gzip: GzipFile.read(100) x 1000 229 us 155 us: 1.48x faster
gzip: GzipFile.write(100 bytes) x 1000 427 us 349 us: 1.22x faster
tempfile: SpooledTemporaryFile.write(100 bytes) x 1000 133 us 78 us: 1.71x faster
urllib3: HTTPResponse.read(100) x 1000 1.63 ms 1.21 ms: 1.34x faster
urllib3: for chunk in response.stream(100) 2.05 ms 1.61 ms: 1.27x faster
control (list(range(20000))) 958 us 955 us: 1.00x
Geometric mean (ex control) 1.45x faster

Compatibility analysis from Claude

Python subclasses are unaffected: __dict__, __weakref__, __slots__, pickling and copying behave as before. Two changes are visible to C code and introspection: __dictoffset__ and __weakrefoffset__ now report the managed pre-header offsets (-1 and -32 on 64-bit builds), and the _io struct sizes shrink by two pointers. An extension type created with PyType_FromSpec on an _io base that also declares its own __dictoffset__ or __weaklistoffset__ member now fails at type creation with TypeError: type X has the Py_TPFLAGS_MANAGED_DICT flag but tp_dictoffset is set, instead of silently getting a second dict slot. Static PyTypeObject subclasses of the _io types have been impossible since they became heap types, Cython does not allow cdef class X(io.RawIOBase), and a survey of PyPI projects that build on the _io types (python-zstandard, pyzstd, cramjam, pyarrow, rapidgzip, indexed_gzip, pysam, cffi, among others) found no compiled subclass of any _io type, so no known package is affected.

Benchmark script
"""Real-world code paths that run Python methods on io subclasses."""
import gzip, io, sys, tempfile, zipfile, pyperf
import http.client
from urllib3.response import HTTPResponse as U3Response

CHUNK = 100
N = 1000
BODY = bytes(range(256)) * (CHUNK * N // 256)          # ~100 KB
LINES = b"".join(b"%06d: some text on this line, to be read\n" % i for i in range(N))

class FakeSocket:
    def __init__(self, data):
        self.data = data
    def makefile(self, mode):
        return io.BytesIO(self.data)
    def close(self):
        pass

HEADERS = (b"HTTP/1.1 200 OK\r\nServer: bench\r\nDate: Mon, 02 Nov 2026 10:00:00 GMT\r\n"
           b"Content-Type: text/plain\r\nCache-Control: no-cache\r\n"
           b"Connection: keep-alive\r\nX-Request-Id: 0123456789abcdef\r\n")

def chunked_body(data, size):
    out = []
    for i in range(0, len(data), size):
        c = data[i:i + size]
        out.append(b"%x\r\n%s\r\n" % (len(c), c))
    out.append(b"0\r\n\r\n")
    return b"".join(out)

RESP_CL = HEADERS + b"Content-Length: %d\r\n\r\n" % len(BODY) + BODY
RESP_LINES = HEADERS + b"Content-Length: %d\r\n\r\n" % len(LINES) + LINES
RESP_CHUNKED = HEADERS + b"Transfer-Encoding: chunked\r\n\r\n" + chunked_body(BODY, CHUNK)
RESP_SMALL = HEADERS + b"Content-Length: 2\r\n\r\nok"

def _response(raw):
    r = http.client.HTTPResponse(FakeSocket(raw), method="GET")
    r.begin()
    return r

def httpclient_begin(loops):
    """HTTPResponse(sock).begin(): one per HTTP request."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        _response(RESP_SMALL).read()
    return pyperf.perf_counter() - t0

def httpclient_read_chunked(loops):
    """read() of a chunked body with 1000 100-byte chunks."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        _response(RESP_CHUNKED).read()
    return pyperf.perf_counter() - t0

def httpclient_read_small(loops):
    """read(100) x 1000 on a Content-Length body."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        resp = _response(RESP_CL)
        read = resp.read
        while read(CHUNK):
            pass
    return pyperf.perf_counter() - t0

def httpclient_iter_lines(loops):
    """for line in response: (1000 lines)."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        for line in _response(RESP_LINES):
            pass
    return pyperf.perf_counter() - t0

# --- zipfile ---------------------------------------------------------------
_zbuf = io.BytesIO()
with zipfile.ZipFile(_zbuf, "w", zipfile.ZIP_DEFLATED) as zf:
    for i in range(200):
        zf.writestr("member%03d.txt" % i, b"x" * 100)
    zf.writestr("lines.txt", LINES)
    zf.writestr("body.bin", BODY)
ZF = zipfile.ZipFile(_zbuf)
MEMBERS = [n for n in ZF.namelist() if n.startswith("member")]

def zipfile_read_members(loops):
    """zf.read(name) for 200 small members (ZipExtFile created per member)."""
    r = range(loops); read = ZF.read; t0 = pyperf.perf_counter()
    for _ in r:
        for name in MEMBERS:
            read(name)
    return pyperf.perf_counter() - t0

def zipfile_iter_lines(loops):
    """for line in zf.open(name): (1000 lines)."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        with ZF.open("lines.txt") as f:
            for line in f:
                pass
    return pyperf.perf_counter() - t0

def zipfile_read_small(loops):
    """f.read(100) x 1000 from a deflated member."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        with ZF.open("body.bin") as f:
            read = f.read
            while read(CHUNK):
                pass
    return pyperf.perf_counter() - t0

# --- gzip -------------------------------------------------------------------
GZ = gzip.compress(BODY)

def gzip_read_small(loops):
    """GzipFile.read(100) x 1000."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        with gzip.GzipFile(fileobj=io.BytesIO(GZ)) as f:
            read = f.read
            while read(CHUNK):
                pass
    return pyperf.perf_counter() - t0

def gzip_write_small(loops):
    """GzipFile.write(100 bytes) x 1000."""
    r = range(loops); piece = BODY[:CHUNK]; t0 = pyperf.perf_counter()
    for _ in r:
        with gzip.GzipFile(fileobj=io.BytesIO(), mode="wb") as f:
            write = f.write
            for _ in range(N):
                write(piece)
    return pyperf.perf_counter() - t0

# --- tempfile ---------------------------------------------------------------
def spooled_write_small(loops):
    """SpooledTemporaryFile.write(100 bytes) x 1000 (stays in memory)."""
    r = range(loops); piece = BODY[:CHUNK]; t0 = pyperf.perf_counter()
    for _ in r:
        with tempfile.SpooledTemporaryFile(max_size=10**7) as f:
            write = f.write
            for _ in range(N):
                write(piece)
    return pyperf.perf_counter() - t0

# --- urllib3 ----------------------------------------------------------------
def _u3():
    return U3Response(body=io.BytesIO(BODY), headers={"content-length": str(len(BODY))},
                      status=200, preload_content=False, decode_content=True)

def urllib3_read_small(loops):
    """urllib3 HTTPResponse.read(100) x 1000."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        resp = _u3(); read = resp.read
        while read(CHUNK):
            pass
    return pyperf.perf_counter() - t0

def urllib3_stream(loops):
    """for chunk in resp.stream(100): (1000 chunks)."""
    r = range(loops); t0 = pyperf.perf_counter()
    for _ in r:
        for chunk in _u3().stream(CHUNK):
            pass
    return pyperf.perf_counter() - t0

def control_list(loops):
    """Unaffected code: noise floor of the two-binary comparison."""
    r = range(loops); q = range(20000); t0 = pyperf.perf_counter()
    for _ in r:
        list(q); list(q); list(q); list(q); list(q)
    return pyperf.perf_counter() - t0

BENCHES = (httpclient_begin, httpclient_read_chunked, httpclient_read_small, httpclient_iter_lines,
           zipfile_read_members, zipfile_iter_lines, zipfile_read_small,
           gzip_read_small, gzip_write_small, spooled_write_small,
           urllib3_read_small, urllib3_stream, control_list)

if __name__ == "__main__":
    if "--check" in sys.argv:
        for f in BENCHES:
            print(f.__name__, "%.3f ms" % (f(1) * 1e3))
        sys.exit()
    runner = pyperf.Runner()
    for f in BENCHES:
        runner.bench_time_func(f.__name__, f)

eendebakpt and others added 2 commits November 2, 2026 12:10
Replace the explicit dict slot of _io._IOBase and its C subclasses with
Py_TPFLAGS_MANAGED_DICT, so instance attributes are stored as inline
values with shared keys instead of a per-instance dict, and attribute
access can be specialized. The flag is inherited by all C and Python
subclasses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replace the explicit weakreflist field of the _io types with
Py_TPFLAGS_MANAGED_WEAKREF, inherited from _IOBase, so the weakref
list lives in the pre-header like the managed dict.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bedevere-app bedevere-app Bot added awaiting core review type-feature A feature request or enhancement labels Nov 3, 2026
blurb-it Bot and others added 2 commits November 3, 2026 08:21
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@cmaloney cmaloney left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exposes a potential double-dealloc in _PyObject_ResurrectEnd on free-threaded builds that should probably fix in a separate issue before landing:

import io, weakref
class Raw(io.RawIOBase): pass
class Attr:
    def __del__(self):
        self.wr()          # deref a weakref to the owner during its dealloc
o = Raw(); o.a = Attr(); o.a.wr = weakref.ref(o)
del o

From Claude:

  1. io subclasses always have tp_finalize. On free-threaded builds, the _PyObject_ResurrectEnd fast path then leaves ob_tid set to the current thread, with both refcounts at 0 and the object not marked merged.
  2. subtype_dealloc now clears the managed dict before the C base's dealloc clears the weakrefs. Previously most _io bases cleared weakrefs first.
  3. A weakref deref during that window succeeds through _Py_TryIncrefFast, and its decref starts a second dealloc inside the first.

Comment thread Modules/_io/bufferedio.c Resolved
Comment thread Modules/_io/bytesio.c Resolved
eendebakpt and others added 2 commits November 4, 2026 23:51
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ate__

Check _PyObject_IsInstanceDictEmpty() first, as object.__getstate__() does,
instead of creating an empty dict just to find out that it is empty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@eendebakpt

Copy link
Copy Markdown
Contributor Author

This exposes a potential double-dealloc in _PyObject_ResurrectEnd on free-threaded builds that should probably fix in a separate issue before landing:

I created #161419

@kumaraditya303
kumaraditya303 merged commit 6f34060 into python:main Nov 20, 2026
56 checks passed
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 MacOS Intel ASAN NoGIL 3.x (tier-2) has failed when building commit 6f34060.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1366/builds/8609) and take a look at the build logs.
  4. Check if the failure is related to this commit (6f34060) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1366/builds/8609

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3-none-any.whl'>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3.6.egg' mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3.6.egg'>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl' mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl'>
k


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3.6.egg'>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3-none-any.whl' mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example-21.12-py3-none-any.whl'>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl' mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/support/__init__.py", line 1004, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/Users/buildbot/buildarea/3.x.itamaro-macos-intel-aws.macos-with-brew.asan.nogil/build/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl'>
k

@cmaloney

Copy link
Copy Markdown
Contributor

gh-162342 for this particular bot failure. It looks like there's a cache in importlib.metadata.FastPath where the GC timing can make this warning visible. Investigating

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants