You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
email: as_bytes() / as_string() raise UnicodeEncodeError for a MIME-Version header containing non-ASCII bytes #162080
A message whose MIME-Version header contains an undecodable (non-ASCII) byte parses without error under policy.default, but cannot be serialized again:
importemailfromemailimportpolicymsg=email.message_from_bytes(b'MIME-Version: 1.\xe9\n\nbody\n', policy=policy.default)
print(msg['MIME-Version'].defects) # (InvalidHeaderDefect(...),)msg.as_bytes() # also as_string()
File ".../email/_header_value_parser.py", line 3126, in _fold_as_ew
encoded_word = _ew.encode(to_encode_word, charset=encode_as)
File ".../email/_encoded_words.py", line 224, in encode
bstring = string.encode(charset)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 0: surrogates not allowed
Other structured headers with such bytes are written with the unknown-8bit charset, for example Content-Type: foo\xa7/bar becomes Content-Type: =?unknown-8bit?q?foo=A7?=/bar.
parse_mime_version() appends the invalid text as a bare ValueTerminal(..., 'xtext') and never calls _validate_xtext(), so no UndecodableBytesDefect is registered, and _refold_parse_tree() then chooses utf-8
instead of unknown-8bit for the encoded word.
Expected: MIME-Version: 1.=?unknown-8bit?q?=E9?=, as for the other headers, and an UndecodableBytesDefect on
the header.
Suggested fix: run _validate_xtext() on the four invalid-text terminals created in parse_mime_version(). I
have a patch with tests (test.test_email passes with it) and can open a PR.
Reproduced on 3.12, 3.15.2 and main (60eca2df). This is the failure mode of gh-143712 in a different header.
Activity