Skip to content

gh-85267: Improvements to inspect.signature __text_signature__ handling - #98796

Merged
JelleZijlstra merged 7 commits into
python:mainfrom
hauntsaninja:inspect-signature
Dec 21, 2022
Merged

JelleZijlstra merged 7 commits into
python:mainfrom
hauntsaninja:inspect-signature

Conversation

@hauntsaninja

@hauntsaninja hauntsaninja commented Oct 28, 2022 •

Copy link
Copy Markdown
Contributor

This makes a couple related changes to inspect.signature's behaviour when parsing a signature from __text_signature__.

First, inspect.signature is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing #83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message)

Second, inspect.signature could randomly drop parameters that it didn't understand (corresponding to return None in the p function). This is the core issue in #85267. I think this is very surprising behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow inspect.signature(select.epoll.register) as in #85267), I add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation arbitrary powerful in #68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope)

Fourth, while #85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get tokenize.TokenError.

Finally, the if name is invalid: code path was dead, since parse_name never returned invalid.

…handling

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing python#83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in python#85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in python#68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while python#85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`.
Comment thread Lib/inspect.py Resolved
@hauntsaninja

hauntsaninja commented Oct 28, 2022 •

Copy link
Copy Markdown
Contributor Author

The tokenize.TokenError fix could be made its own PR, but the other changes are somewhat linked.

Comment thread Lib/inspect.py
except NameError:
raise RuntimeError()
raise ValueError

@hauntsaninja hauntsaninja Oct 28, 2022 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These now get caught on L2254 and acquire an error message there

@JelleZijlstra
JelleZijlstra self-requested a review November 3, 2022 02:26

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a few comments.

Can we backport this to the bugfix branches? All of the changes here seem like they fix buggy behavior, so I'm leaning to yes.

Comment thread Lib/inspect.py Resolved
Comment thread Lib/test/test_inspect.py Resolved
Comment thread Lib/test/test_inspect.py Resolved
Comment thread Lib/inspect.py Resolved
@hauntsaninja

hauntsaninja commented Nov 9, 2022 •

Copy link
Copy Markdown
Contributor Author

Thanks! Made the requested changes. I agree that this seems backport-able

@netlify

ghost commented Dec 12, 2022 •

Copy link
Copy Markdown

✅ Deploy Preview for python-cpython-preview canceled.

Name Link
🔨 Latest commit fef9787
🔍 Latest deploy log https://app.netlify.com/sites/python-cpython-preview/deploys/6396ac051af5c300087493ef

@JelleZijlstra

ghost commented Dec 12, 2022

Copy link
Copy Markdown
Member

#21104 did a subset of what this PR is changing, fixed the merge conflict.

@JelleZijlstra JelleZijlstra self-assigned this Dec 12, 2022
@JelleZijlstra
JelleZijlstra merged commit 79311cb into python:main Dec 21, 2022
@miss-islington

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Contributor

Thanks @hauntsaninja for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

@miss-islington

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Contributor

Sorry, @hauntsaninja and @JelleZijlstra, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 79311cbfe718f17c89bab67d7f89da3931bfa2ac 3.11

@miss-islington

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Contributor

Sorry @hauntsaninja and @JelleZijlstra, I had trouble checking out the 3.10 backport branch.
Please retry by removing and re-adding the "needs backport to 3.10" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 79311cbfe718f17c89bab67d7f89da3931bfa2ac 3.10

@JelleZijlstra

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Member

@hauntsaninja do you want to do the manual backports?

@bedevere-bot

ghost commented Dec 21, 2022

Copy link
Copy Markdown

GH-100392 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Dec 21, 2022
@hauntsaninja
hauntsaninja deleted the inspect-signature branch December 21, 2022 03:58
@bedevere-bot

ghost commented Dec 21, 2022

Copy link
Copy Markdown

GH-100393 is a backport of this pull request to the 3.10 branch.

@hauntsaninja

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Contributor Author

@JelleZijlstra

ghost commented Dec 21, 2022

Copy link
Copy Markdown
Member

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants