Repository navigation
Conversation
…tesobject Format %d, %i, %u, %ld, %lu, %zd, %zu and %x with _PyLong_FormatBytesWriter(), the code bytes % int uses, which writes straight into the PyBytesWriter instead of going through a stack buffer. Output is unchanged. This leaves the current %p as it has some platform specific variation which changing requires more validation and a NEWS. Each row is 1000 calls made from a C extension module (below), so 1 us is 1 ns per call. Linux x86-64, clang PGO+LTO, pyperf, upstream/main b90572a against this change. The %s and %p rows are not changed. | Benchmark | main | PR | |------------------------------|:-------:|:---------------------:| | run(b'%x', 123456789, 1000) | 36.9 us | 26.9 us: 1.37x faster | | run(b'%d', 5, 1000) | 28.1 us | 20.6 us: 1.37x faster | | run(b'%d.%d.%d', 42, 1000) | 74.7 us | 55.0 us: 1.36x faster | | run(b'%i', 42, 1000) | 34.1 us | 26.0 us: 1.31x faster | | run(b'%d', 42, 1000) | 33.9 us | 26.3 us: 1.29x faster | | run(b'%d', -123456789, 1000) | 40.5 us | 35.1 us: 1.16x faster | | run(b'%d', 123456789, 1000) | 40.0 us | 34.6 us: 1.16x faster | | run(b'%u', 123456789, 1000) | 39.7 us | 34.6 us: 1.15x faster | | run(b'%ld', 10**18, 1000) | 50.5 us | 46.0 us: 1.10x faster | | run(b'%lu', 2**64 - 1, 1000) | 50.1 us | 47.3 us: 1.06x faster | | run(b'%zu', 2**64 - 1, 1000) | 49.2 us | 47.7 us: 1.03x faster | | run(b'%s', 0, 1000) | 19.4 us | 18.8 us: 1.03x faster | | run(b'%p', 0, 1000) | 35.8 us | 34.8 us: 1.03x faster | Not significant: run(b'%p', 2**47 - 4096, 1000), run(b'%zd', 2**62, 1000). Benchmark script (bench.py): # PyBytes_FromFormat() integer formatting, called from C by # fromformat_bench.c (build it first, see its header). Each row is 1000 # calls, so 1 us is 1 ns per call. The %s and %p rows are formats the PR # does not change. # python bench.py -o main.json (main build) # python bench.py -o pr.json (PR build) # python -m pyperf compare_to main.json pr.json --table import pyperf runner = pyperf.Runner() for stmt in [ "run(b'%d', 5, 1000)", "run(b'%d', 42, 1000)", "run(b'%d', 123456789, 1000)", "run(b'%d', -123456789, 1000)", "run(b'%i', 42, 1000)", "run(b'%u', 123456789, 1000)", "run(b'%ld', 10**18, 1000)", "run(b'%lu', 2**64 - 1, 1000)", "run(b'%zd', 2**62, 1000)", "run(b'%zu', 2**64 - 1, 1000)", "run(b'%x', 123456789, 1000)", "run(b'%d.%d.%d', 42, 1000)", "run(b'%p', 2**47 - 4096, 1000)", "run(b'%p', 0, 1000)", "run(b'%s', 0, 1000)", ]: runner.timeit(stmt, stmt, setup="from fromformat_bench import run") C extension module (fromformat_bench.c): // run(format, value, loops): call PyBytes_FromFormat(format, value, value, // value) loops times. The first conversion in format picks the C type of // value; %s gets "abc". // // cc -shared -fPIC -O2 $(python3-config --includes) fromformat_bench.c // -o fromformat_bench.abi3.so #define Py_LIMITED_API 0x030c0000 #include <Python.h> #include <string.h> static PyObject * run(PyObject *self, PyObject *args) { const char *format; unsigned long long value; Py_ssize_t loops; if (!PyArg_ParseTuple(args, "yKn", &format, &value, &loops)) { return NULL; } const char *conv = strchr(format, '%'); if (conv == NULL) { PyErr_SetString(PyExc_ValueError, "no conversion in format"); return NULL; } conv++; for (Py_ssize_t i = 0; i < loops; i++) { PyObject *bytes; if (*conv == 'l') { long v = (long)value; bytes = PyBytes_FromFormat(format, v, v, v); } else if (*conv == 'z') { size_t v = (size_t)value; bytes = PyBytes_FromFormat(format, v, v, v); } else if (*conv == 'p') { void *v = (void *)(uintptr_t)value; bytes = PyBytes_FromFormat(format, v, v, v); } else if (*conv == 's') { bytes = PyBytes_FromFormat(format, "abc", "abc", "abc"); } else { int v = (int)value; bytes = PyBytes_FromFormat(format, v, v, v); } if (bytes == NULL) { return NULL; } Py_DECREF(bytes); } Py_RETURN_NONE; } static PyMethodDef methods[] = { {"run", run, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL}, }; static struct PyModuleDef module = { PyModuleDef_HEAD_INIT, "fromformat_bench", NULL, 0, methods, }; PyMODINIT_FUNC PyInit_fromformat_bench(void) { return PyModule_Create(&module); } Co-Authored-By: Claude Fable 5.2 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handle
%d,%i,%u,%ld,%lu,%zd,%zuand%xwith existing implementation in_PyLong_FormatBytesWriter(). That writes straight into thePyBytesWriterinstead of going through a local stack buffer.This leaves the current
%pas it has some platform specific variation which changing requires more validation and a NEWS. When that case is changed the stack buffer goes away. The temporary buffer could be shrunk by 2 bytes here but I think leaving the existing code is less risky short term.Performance: Slightly faster but not majorly.
Code to help validate output didn't change
Benchmarking results, code, c module
Each row is 1000 calls made from a C extension module (below), so 1 us is 1 ns per call. Linux x86-64, clang PGO+LTO, pyperf, upstream/main b90572a against this change. The %s and %p rows are not changed.
Not significant: run(b'%p', 247 - 4096, 1000), run(b'%zd', 262, 1000).
Benchmark script (bench.py):
C extension module (fromformat_bench.c):
Another option for gh-161994 (does all but
%pwhich would be a separate PR)