Skip to content

gh-129813: Use PyBytesWriter in ctypes - #162260

Merged
vstinner merged 1 commit into
python:mainfrom
maurycy:ctypes-pybyteswriter-slices
Nov 19, 2026
Merged

vstinner merged 1 commit into
python:mainfrom
maurycy:ctypes-pybyteswriter-slices

Conversation

@maurycy

@maurycy maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Write strided slices of c_char arrays and pointers w/ a PyBytesWriter. Spares a temporary allocation, a copy. Similar to #162257

I believe this is mostly a simplification, but, again, I'm surprised that's good for the memory:

Result size Step Baseline peak (bytes) Patched peak (bytes)
4 KiB 2 8,225 4,129
8 MiB 2 16,777,249 8,388,641
8 MiB -1 16,777,249 8,388,641
8 MiB 1 8,388,641 8,388,641

pyperf:

Type Result size Step Baseline 1 Patched 1 Baseline 2 Patched 2
array 32 B 2 36.9 ns 35.2 ns 37.6 ns 35.0 ns
pointer 32 B 2 34.0 ns 32.8 ns 34.2 ns 34.2 ns
array 4 KiB 2 1.18 µs 1.11 µs 1.17 µs 1.11 µs
pointer 4 KiB 2 616.2 ns 566.3 ns 615.9 ns 564.2 ns
array 8 MiB 2 2.36 ms 2.19 ms 2.35 ms 2.21 ms
pointer 8 MiB 2 1.33 ms 1.16 ms 1.30 ms 1.15 ms
array 8 MiB -1 2.40 ms 2.18 ms 2.41 ms 2.18 ms
pointer 8 MiB -1 1.39 ms 1.11 ms 1.25 ms 1.11 ms
array 8 MiB 1 125 µs 126 µs 145 µs 124 µs
pointer 8 MiB 1 136 µs 123 µs 140 µs 134 µs

(PGO+LTO, macOS arm64)

pyperf
import ctypes

import pyperf

runner = pyperf.Runner()

for size, step in ((32, 2), (4096, 2), (8 << 20, 2), (8 << 20, -1), (8 << 20, 1)):
    length = size * abs(step) + (step < 0)
    array = (ctypes.c_char * length)()
    ctypes.memset(array, 65, length)
    pointer = ctypes.cast(ctypes.byref(array), ctypes.POINTER(ctypes.c_char))
    key = slice(size, 0, -1) if step < 0 else slice(None, length, step)
    for kind, obj in (("array", array), ("pointer", pointer)):
        name = f"{kind} {size} B step {step}"
        runner.timeit(name, "obj[key]", globals={"obj": obj, "key": key})
memory
import ctypes
import tracemalloc

for size, step in ((32, 2), (4096, 2), (8 << 20, 2), (8 << 20, -1), (8 << 20, 1)):
    length = size * abs(step) + (step < 0)
    array = (ctypes.c_char * length)()
    ctypes.memset(array, 65, length)
    pointer = ctypes.cast(ctypes.byref(array), ctypes.POINTER(ctypes.c_char))
    key = slice(size, 0, -1) if step < 0 else slice(None, length, step)
    for kind, obj in (("array", array), ("pointer", pointer)):
        name = f"{kind} {size} B step {step}"
        tracemalloc.start()
        result = obj[key]
        _, peak = tracemalloc.get_traced_memory()
        tracemalloc.stop()
        assert len(result) == size
        print(name, peak)
        del result

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Nov 19, 2026
@maurycy

maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

cc @vstinner Another one in the today's series, inspired by PyBytesWriter in the remote_debugging module

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Straightforward usage of the PyBytesWriter API.

It should be faster. The old code uses 2 heap memory allocations (PyMem_Malloc + PyBytes_FromStringAndSize) and had to copy bytes once (buffer => bytes). With the updated code, there is a single heap memory allocation, and bytes are only copied if the bytes object is smaller than 256 (PyBytesWriter uses a "small buffer" in this case). So for large bytes objects, bytes are written once and not copied.

@vstinner

Copy link
Copy Markdown
Member

Reducing the peak memory usage by a factor of 2 is great! Also, the slice operation is up to 1.10x faster than before, which is also nice to have.

@vstinner
vstinner merged commit 3edcc34 into python:main Nov 19, 2026
61 of 64 checks passed
@maurycy
maurycy deleted the ctypes-pybyteswriter-slices branch November 19, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip news type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants