C12.2: read and write MIME by the tree

The reader walks the MIME tree with its container context: alternative
picks versions, related finds its root by start, attached multiparts
stay whole, digest children are messages. Nesting deeper than 64
levels, counting enclosed messages, is refused, and over-deep parse
results are taken apart iteratively so dropping them cannot overflow
the stack. Composite parts with a Content-ID are cid targets.

The writer keeps attachment parameters and Content-IDs, writes
multipart and message attachments unencoded, and makes its own
Message-ID. SMTP adds BODY=8BITMIME for 8-bit bodies and SMTPUTF8
when any header block at any depth is 8-bit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ZacharyZhang-NYandClaude Opus 5.5 committed 2026-10-19 17:35:05 -04:00
1 parent 9f47b94714
commit fe90221c11
23 files changed
+1672 -136

No files matched your search

+1 -1
View File
@@ -135,7 +135,7 @@
- [x] C12.0 选型修订:PRD §1.1、§12.1、§17.3、§21.2、§23 与 `agents.md` 改为上述选型;删去 MailCore 2 与 libetpan 的取码、构建与探测脚本;`oauth.env` 不入库;`references.sh` 随之(并修上游钉与工作区不一致的组件库钉) — R1 ok
- [x] C12.1 `crates/ely-mail-engine` 与连通(MAIL-01 前半):IMAP、POP3、SMTP 经 TLS(rustls,系统根证书)或 STARTTLS 连接,密码与 XOAUTH2 认证,失败带服务器原话;替身服务器测三协议的登录、拒绝、列邮箱、取信、发信。已做:`Imap`、`Pop`、`Smtp` 三个会话,`Server`(主机、端口、`Security::Tls` 或 `StartTls`,无明文)、`Login`(密码或 OAuth 令牌);系统信任库经 rustls-platform-verifier,测试换入只信测试证书的连接器;每步 60 秒超时;`Error` 分连接、TLS、凭据被拒(服务器原话)、命令被拒、读不懂、超时、发不出(信封地址或登录带换行等)。IMAP 由引擎自写,以 imap-proto 解析(`wire.rs` 收一个个完整回复,含字面量),换下 async-imap(它的 LIST、FETCH 不看结尾的 NO 与 BAD,拒绝成了空结果与“无此 UID”;登录与命令中途的 `* BYE` 被丢掉,只剩断线;trace 日志写出密码与邮件):每条命令看结尾的状态,NO 与 BAD 带原话报出(登录为凭据被拒,其余为命令被拒);问候或任何时候收到 `* BYE`,此后断线即带其原话;非 ASCII 或带换行的参数以字面量发出,等服务器的 `+`;STARTTLS 之后若已有明文字节即拒绝;LIST 里带引号的邮箱名去掉转义(imap-proto 照原样留着),字面量照原样,列出的名字可直接打开;日志只记标签与命令名。POP3 由引擎自写(RFC 1939:问候、STLS、USER/PASS、AUTH XOAUTH2、UIDL、RETR、QUIT),换下 async-pop(它留着点填充、丢末尾 CRLF、吞掉登录后 CAPA 的连接错误、不答 XOAUTH2 的错误挑战、在 trace 日志里写出密码与邮件);STLS 之后若已有明文字节即拒绝;用户名与密码带换行即拒绝。SMTP 的 XOAUTH2 与事务由引擎自走(mail-send 遇错误挑战会重发令牌;它的 DATA 不给首行的点加点,且自加末尾 CRLF):信封地址照 RFC 5321 的 Mailbox 核对(点串或带引号的本地部分,域名或地址字面量;地址或信头有 UTF-8 时,服务器报 SMTPUTF8 才发出并带此参数,否则拒绝(RFC 6531)),不合即拒绝、不发任何命令;DATA 每行开头的点(含首行)加一个点,裸 CR 或 LF 即拒绝。服务器拒绝事务中的一步后发 RSET,会话可再发信。三协议的 XOAUTH2 被拒都以空行答错误挑战、留下服务器最后的原话。34 个测试(`cargo test -p ely-mail-engine`,含以真日志器在 trace 下跑三协议、日志里无密码、令牌与正文),真服务器未连。审查三轮:第 3 轮的两处(带引号的邮箱名、信头的 UTF-8)已修并以测试核对,未再审查(每条至多三轮)
- [ ] C12.2 MIME(MAIL-04、05 的基础):mail-parser 读出头、纯文本、HTML、cid 内嵌图片、附件与原始头;mail-builder 写出多部分邮件、附件、Message-ID、In-Reply-To 与 References;样本 EML 往返测试
- [x] C12.2 MIME(MAIL-04、05 的基础):mail-parser 读出头、纯文本、HTML、cid 内嵌图片、附件与原始头;mail-builder 写出多部分邮件、附件、Message-ID、In-Reply-To 与 References;样本 EML 往返测试。已做:`read` 得 `Mail`(主题、发件人、收件人、抄送(分组展开)、日期、Message-ID、In-Reply-To、References、纯文本、HTML、`inline`、`attachments`、原始头块);按 MIME 树与所在容器读(不用 mail-parser 摊平的列表):正文只取发件人写的部分(mail-parser 会把 HTML 转成文字或反之顶替缺的一种,不取);alternative 里纯文本与 HTML 各取最后一个(RFC 2046 §5.1.4),其余版本不并入,没有正文的版本(如日历)为附件;related 的根照 `start`,无 `start` 取首个(RFC 2387),其余带 Content-ID 的即 cid 目标,不论处置;mixed 等的部分依次并成正文;处置为 attachment 的实体整个是附件,多部分实体连同其头(RFC 2183 §2.9),子孙不进正文;附件字节只解传输编码,文字附件不按字符集转换;不在 related 里的部分,带 Content-ID 且不是 attachment 的为 cid 目标;无 Content-Type 照 RFC 2045 为 text/plain,digest 的子部分为 message/rfc822;解码有问题的部分与无地址的收件人记警告。`build` 由 `Draft` 写出(密送不进信头,留在 SMTP 信封):Message-ID 为 16 字节随机数加发件人的域(mail-builder 默认写本机主机名,不用),交回供发件记录;地址照 SMTP 的 Mailbox 核对,主题、姓名、Message-ID、文件名、类型带换行即拒绝;附件须有文件名,Content-ID 照写(带换行即拒绝);正文与附件自己组成 mixed 与 alternative。`tests/fixtures/mail/sample.eml` 读全;由它回复写出再读回,各字段与附件(名、类型、Content-ID、含 0 至 255 每个字节)相同;第 1 轮审查后另 7 个结构测试(`tests/mime_tree.rs`:Latin-1 文字附件、`start` 在图后、related 里标为附件的图、附件化的多部分、同型 alternative、digest),去掉各自的修复即失败;第 2 轮审查后:嵌套深于 64 层即报错(先以迭代算出深度,原递归在 2048 层时栈溢出、进程退出);mail-parser 开 `full_encoding`(原 GB2312 等多字节字符集读成替换字符);`Part::params` 带 Content-Type 的参数(不含 name),写出时照写,转发的 Latin-1 附件收件方读出 Café;多部分与 message 附件照原样写出(不编码,7bit 或 8bit,多部分带原分界,RFC 2045 §6.4),SMTP 遇 8 位正文即带 `BODY=8BITMIME`,服务器不支持即在发出前拒绝;related 的根不看处置,其余每个资源整个是一个实体(多部分资源带其 Content-ID 与文件名);带 Content-ID 的正文文字部分另为 cid 目标(related 的根除外);未知处置照附件(RFC 2183 §2.8)。另 9 个测试(`tests/mime_edges.rs`、`tests/mime_forward.rs`,SMTP 一个),去掉各自的修复即失败;第 3 轮审查(最后一轮,修完不再审查)后:深度也数内嵌的邮件(mail-parser 以迭代读出层层相套的 message/rfc822,丢弃时却递归,8192 层即栈溢出),超深的解析结果以迭代逐层拆开再丢;带 Content-ID 的多部分实体(related 的根除外)连同其头另为 cid 目标,子孙照常读;SMTP 按每一层的头块判定是否需 SMTPUTF8(转发的 message/global 里的中文主题),只深入未经传输编码的内嵌邮件;每次发送都先解析邮件,不是邮件即拒绝。另 5 个测试(`tests/mime_deep.rs` 四个、`tests/smtp_utf8.rs` 一个;SMTP 的编码测试与辅助函数移入 `tests/smtp_utf8.rs` 与 `tests/fake/smtp.rs`),去掉各自的修复即失败或栈溢出;原往返测试按子串查 bcc,随机的 Message-ID 约四十次一次含 bcc 而误报,改为查 `Bcc:` 头行;引擎共 61 个测试
- [ ] C12.3 本地存储(§12.3、§18.1、§18.3):SQLite 索引(账号、邮箱、消息身份为账号加邮箱加 UIDVALIDITY 加 UID、标记、头字段)、原文按内容存进内容目录、FTS5 全文索引(可重建)、结构版本与升级前备份、升级失败只读;各类大小可查
- [ ] C12.4 凭据与 OAuth(MAIL-02,§18.2):密码与刷新令牌只进系统凭据库(不可用即只留到本次退出并写明);Gmail 与 Microsoft 的授权码加 PKCE、本机回环、换令牌与刷新;客户端 ID 编进构建,缺 ID 即写明不可用;刷新被拒转为“需重新登录”;替身授权端点测试
- [ ] C12.5 账号(MAIL-01、02):ElyMail 主窗口与添加账号对话框:Gmail、Microsoft、iCloud(应用专用密码向导)、其他 IMAP/SMTP、POP3/SMTP;服务器、端口、TLS 方式、认证方式;连通性检查逐项报结果;多账号;删除账号先写明删去的本地缓存、草稿与凭据,不动服务端
+15
View File
@@ -0,0 +1,15 @@
FAIL
- crates/ely-mail-engine/src/mime.rs:198 — [P1] Text attachments lose their original file bytes. `MessagePart::contents()` returns charset-decoded UTF-8 for `text/*`, rather than just transfer-decoded octets. A `text/plain; charset=iso-8859-1` attachment containing quoted-printable `Caf=E9` becomes `[67,97,102,195,169]` instead of `[67,97,102,233]`. Building and reading a `text/plain` attachment with those original bytes instead produces `[67,97,102,239,191,189]`. Saving or forwarding therefore changes the file silently. Preserve attachment octets after transfer decoding; apply charset conversion separately to displayed bodies. Add non-UTF-8 text attachment tests; the existing every-byte test only covers `application/octet-stream`. Both failures are recorded in `.work/C12.2-r1/probes.log`.
- crates/ely-mail-engine/src/mime.rs:104 — [P2] HTML body selection ignores `multipart/related`'s `start` parameter. With `type="text/html"; start="<root@example.com>"`, an image first and the HTML part with that Content-ID second, `read` returns `html: None` and puts both parts in `inline`. The actual message body is thus unavailable despite its explicit root declaration. Resolve the related root from `start` before choosing body parts, using the first child only when `start` is absent, as specified by [RFC 2387 §3.2](https://www.rfc-editor.org/rfc/rfc2387.html#section-3.2). Add this ordering to the MIME tests. Reproduced by `a_related_html_root_can_come_after_its_image` in the probe log.
- crates/ely-mail-engine/src/mime.rs:81 — [P2] An attached multipart entity disappears as an attachment and its contents become the parent message's body. For an outer `multipart/mixed` containing plain `body` followed by `multipart/alternative` with `Content-Disposition: attachment; filename=alternatives.mime`, whose text child contains `private attachment`, `read` returns `text: Some("body\nprivate attachment")` and zero attachments. The parent's disposition applies to the whole multipart entity ([RFC 2183 §2.9](https://www.rfc-editor.org/rfc/rfc2183.html#section-2.9)). Respect that disposition while traversing the MIME tree, retain the attached entity, and exclude its descendants from the main bodies. Add a regression for this structure; `multipart_attachment_is_not_displayed_as_parent_body` reproduces both incorrect outputs.
- crates/ely-mail-engine/src/mime.rs:228 — [P2] Joining every body part concatenates mutually exclusive alternatives. A `multipart/alternative` with an ASCII plain-text version `Cafe` followed by its UTF-8 plain-text version `Café` yields `Some("Cafe\nCafé")`. The reader cannot subsequently select a representation because the alternatives and their container relationship have already been discarded. Select the supported alternative within each alternative container instead of joining its siblings; concatenation can remain for independent mixed body segments. [RFC 2046 §5.1.4](https://www.rfc-editor.org/rfc/rfc2046.html#section-5.1.4) specifies alternative selection. Add a same-type alternative regression; `alternatives_of_the_same_type_are_not_concatenated` fails in the probe log.
- crates/ely-mail-engine/src/mime.rs:86 — [P2] A related image's `attachment` disposition prevents it from being exposed as a CID target. With an HTML root referencing `cid:image@example.com` and a related image carrying that Content-ID plus `Content-Disposition: attachment`, `read` returns an empty `inline` list and places the image only in `attachments`. Within a related object, disposition does not override the object's relationships ([RFC 2387 §4](https://www.rfc-editor.org/rfc/rfc2387.html#section-4)). Determine CID resources using their MIME container context so this image remains available to the HTML body. Add coverage for a related resource with this disposition; `related_cid_image_is_available_even_when_disposition_is_attachment` reproduces the failure.
- crates/ely-mail-engine/src/mime.rs:154 — [P2] Building attachments silently discards the supplied `Part.content_id`. An image attachment with `content_id: Some("image@example.com")` is written without any Content-ID header and reads back as `None`; HTML referencing that ID cannot resolve it. The supplied sample's PDF Content-ID is likewise lost, while the roundtrip test compares only name, type and bytes, concealing the loss behind the task's field-preservation claim. Serialize and validate the supplied Content-ID, and compare complete attachment metadata in the roundtrip test. Reproduced by `supplied_attachment_content_id_survives_build` in the probe log.
- crates/ely-mail-engine/src/mime.rs:186 — [P2] The missing Content-Type default is wrong for children of `multipart/digest`. A digest containing an enclosed message with no explicit part Content-Type is parsed as a message by mail-parser, but `leaf` labels its bytes `text/plain`. Consumers therefore receive the wrong attachment type, and rebuilding from that metadata emits a text attachment instead of an attached message. The default in this container is `message/rfc822` ([RFC 2046 §5.1.5](https://www.rfc-editor.org/rfc/rfc2046.html#section-5.1.5)). Derive the effective type from the parent context or the parsed message variant, retaining `text/plain` for ordinary untyped parts. Add a digest regression; `a_digest_child_defaults_to_message_rfc822` reproduces the mismatch.
+17
View File
@@ -0,0 +1,17 @@
FAIL
- crates/ely-mail-engine/src/mime/walk.rs:43 — [P1] Unbounded MIME traversal can abort the application. A valid 138,017-byte message containing 2,048 nested multipart/mixed containers parses into 2,049 parts successfully, but `read` then overflows the stack and terminates the Windows test process with `0xc00000fd`. This is a process abort, not a recoverable `Error::Mime`, and the mail engine runs inside the application. Replace recursive traversal with an explicit stack, or enforce a checked nesting limit that returns an error before exhausting the stack. Add a regression covering deeply nested input. Reproduced in `.work/C12.2-r2/depth.log`; the existing 46-test mail-engine suite passes.
- Cargo.toml:62 — [P2] The mail-parser dependency omits `full_encoding`, whose default is disabled. Its multibyte charset decoders consequently use lossy UTF-8 conversion. A valid `Subject: =?GB2312?B?1tDOxA==?=` and a text/plain body declaring `charset=gb2312` with base64 `1tDOxA==` both read as four replacement characters instead of `中文`, without a decoding warning. Enable the dependency's charset support and test multibyte encoded headers and bodies. The same isolated probe passes when only `mail-parser/full_encoding` is enabled; see `.work/C12.2-r2/encoding-before.log` and `encoding-feature.log`.
- crates/ely-mail-engine/src/mime/walk.rs:133 — [P2] Preserving text attachment octets now loses the charset needed to interpret those octets. `leaf` discards all Content-Type parameters. Reading a `text/plain; charset=iso-8859-1` attachment containing `Caf=E9`, copying its returned `Part` into `Draft.attachments`, and building it emits `Content-Type: text/plain` with the Latin-1 bytes unchanged. Parsing the forwarded attachment yields `Caf�`. Without the charset, the declared default is US-ASCII ([RFC 2046 §4.1.2](https://www.rfc-editor.org/rfc/rfc2046.html#section-4.1.2)). Preserve and serialize the parameters required to interpret attachment bytes, and test the recipient's decoded attachment text after forwarding. The existing byte-equality regression cannot detect this loss. Reproduced in `.work/C12.2-r2/probes.log`.
- crates/ely-mail-engine/src/mime.rs:159 — [P2] Multipart attachments returned by `read` cannot be forwarded as valid MIME. Their `bytes` include the entity headers, but `build` treats those bytes as a binary leaf: an attached multipart/alternative becomes `Content-Type: multipart/alternative` without a boundary, followed by a base64 encoding of its original headers and body. The recipient parser sees a binary leaf instead of the original multipart structure. Base64 transfer encoding is prohibited on multipart entities ([RFC 2045 §6.4](https://www.rfc-editor.org/rfc/rfc2045.html#section-6.4)). Serialize these attachments as complete MIME entities with their boundary and children and a permitted transfer encoding. Add a read/build/parse regression for the attached multipart covered by round 1. Reproduced in `.work/C12.2-r2/probes.log`.
- crates/ely-mail-engine/src/mime/walk.rs:104 — [P2] The round 1 related-disposition fix still excludes the related root. A multipart/related with `type="text/html"` whose first child is text/html with `Content-Disposition: attachment; filename=body.html` returns `html: None` and treats the root as an attachment, because `related` sends it through `body`'s attachment guard. Related processing must ignore the disposition type for the root as well as its sibling resources ([RFC 2387 §4](https://www.rfc-editor.org/rfc/rfc2387.html#section-4)). Carry that container context into root selection, while preserving the rules for the root's descendants, and add this root-disposition regression. Reproduced in `.work/C12.2-r2/related.log`.
- crates/ely-mail-engine/src/mime/walk.rs:114 — [P2] A multipart related resource loses its own entity, Content-ID and filename. With an HTML root and a sibling multipart/alternative carrying `Content-ID: <entity@example.com>` and `filename=entity.mime`, `resource` recursively flattens the sibling before inspecting its metadata. The result has no inline target for `entity@example.com`; only an unnamed text/plain descendant remains in attachments. The compound resource can neither be resolved by its CID nor saved with its original contents. Related processing must retain the correspondence between an entity, its headers and its Content-ID ([RFC 2387 §6.1](https://www.rfc-editor.org/rfc/rfc2387.html#section-6.1)). Preserve the resource entity and metadata when traversing composite resources, and add this regression. Reproduced in `.work/C12.2-r2/related.log`.
- crates/ely-mail-engine/src/mime/walk.rs:45 — [P2] The claimed non-related CID handling is bypassed for text/plain and text/html parts. In multipart/mixed, a plain body followed by a text/html resource with `Content-ID: <frame@example.com>` and inline disposition returns that resource as `Mail.html`, with an empty `inline` list. Its Content-ID and original part bytes are unavailable to callers, so a CID link to it cannot be resolved. This contradicts C12.2's explicit claim that non-attachment CID parts outside related containers become CID targets. Classify text resources using their container context before consuming them as body strings, and test text CID targets as well as binary images. Reproduced in `.work/C12.2-r2/related.log`.
- crates/ely-mail-engine/src/mime/walk.rs:170 — [P2] Unrecognized disposition types are silently treated as inline body text. A multipart/mixed containing plain `body` followed by text/plain with `Content-Disposition: x-file; filename=file.txt` and `private attachment` reads as `text: Some("body\nprivate attachment")` with zero attachments. Unrecognized disposition types should be handled as attachments, retaining their filename ([RFC 2183 §2.8](https://www.rfc-editor.org/rfc/rfc2183.html#section-2.8)). Apply that classification outside the related-container override and add an unknown-disposition regression. Reproduced in `.work/C12.2-r2/probes.log`.
+7
View File
@@ -0,0 +1,7 @@
FAIL
- crates/ely-mail-engine/src/mime/walk.rs:35 — [P1] The round 2 nesting fix still permits a process-aborting stack overflow through embedded messages. `sub_parts()` visits only multipart children, so 128 nested `message/rfc822` entities return `Ok` despite the claimed 64-level limit. With 8,192 such entities (262,188 bytes), `read` terminates the Windows process with `0xc00000fd`. The parser itself completes; dropping its recursively nested `PartType::Message` result overflows the stack. Count embedded-message nesting as well as multipart nesting, and ensure over-limit parser results can be rejected and disposed of without recursive stack exhaustion. Add an embedded-message regression; the current depth test covers only multipart containers. Reproduced in `.work/C12.2-r3/depth-128.log`, `depth-8192.log`, `depth-parse-forget.log`, and `depth-parse-drop.log`.
- crates/ely-mail-engine/src/mime/walk.rs:65 — [P2] Non-related multipart CID targets still disappear. In a `multipart/mixed` containing an HTML reference to `cid:compound@example.com` and an inline `multipart/alternative` sibling with that Content-ID and `filename=compound.mime`, `read` returns empty `inline` and `attachments` lists; only the sibling's descendant text survives. The multipart branch descends without retaining the containing entity, so its ID, filename and original bytes are unavailable. This leaves round 2's CID fix incomplete for composite parts and contradicts C12.2's non-attachment CID handling. Retain the composite entity as a CID target before traversing its contents, respecting the related-root exception, and test this structure. [RFC 2392 §2](https://www.rfc-editor.org/rfc/rfc2392.html#section-2) permits CID references to MIME entities. Reproduced in `.work/C12.2-r3/cid.log`.
- crates/ely-mail-engine/src/smtp.rs:104 — [P2] The new unencoded composite-forwarding path can send internationalized headers without SMTPUTF8. Build a draft with ASCII envelope and outer headers and a `message/global` attachment containing `Subject: 您好`. The attachment is emitted as 8bit, but `send` inspects only the outer header block for SMTPUTF8: against a TLS test server advertising 8BITMIME without SMTPUTF8, it sends the complete message with only `BODY=8BITMIME` and returns `Ok(())`. Such headers require SMTPUTF8 at any MIME depth ([RFC 6531 §3.2 and §3.4](https://www.rfc-editor.org/rfc/rfc6531.html#section-3.2)); 8BITMIME alone does not authorize them. Carry or determine the complete MIME message's SMTPUTF8 requirement before sending, reject unsupported servers, and include the parameter when supported. Add a built-message/SMTP regression covering embedded UTF-8 headers. Reproduced in `.work/C12.2-r3/embedded-header.log`.