Skip to content

email: as_bytes() / as_string() raise UnicodeEncodeError for a MIME-Version header containing non-ASCII bytes #162080

Open

Description

@jimbou

Bug report

A message whose MIME-Version header contains an undecodable (non-ASCII) byte parses without error under
policy.default, but cannot be serialized again:

import email
from email import policy

msg = email.message_from_bytes(b'MIME-Version: 1.\xe9\n\nbody\n', policy=policy.default)
print(msg['MIME-Version'].defects)   # (InvalidHeaderDefect(...),)
msg.as_bytes()                       # also as_string()
  File ".../email/_header_value_parser.py", line 3126, in _fold_as_ew
    encoded_word = _ew.encode(to_encode_word, charset=encode_as)
  File ".../email/_encoded_words.py", line 224, in encode
    bstring = string.encode(charset)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 0: surrogates not allowed

Other structured headers with such bytes are written with the unknown-8bit charset, for example
Content-Type: foo\xa7/bar becomes Content-Type: =?unknown-8bit?q?foo=A7?=/bar.

parse_mime_version() appends the invalid text as a bare ValueTerminal(..., 'xtext') and never calls
_validate_xtext(), so no UndecodableBytesDefect is registered, and _refold_parse_tree() then chooses utf-8
instead of unknown-8bit for the encoded word.

Expected: MIME-Version: 1.=?unknown-8bit?q?=E9?=, as for the other headers, and an UndecodableBytesDefect on
the header.

Suggested fix: run _validate_xtext() on the four invalid-text terminals created in parse_mime_version(). I
have a patch with tests (test.test_email passes with it) and can open a PR.

Reproduced on 3.12, 3.15.2 and main (60eca2df). This is the failure mode of gh-143712 in a different header.

CPython versions tested on:

3.12, 3.15, CPython main branch

Operating systems tested on:

Linux

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    on Nov 16, 2026
  2. added
    stdlibStandard Library Python modules in the Lib/ directory
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-emailtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions