C12.6: IMAP sync

tasks/TASKS.md

@@ -139,7 +139,7 @@
 - [x] C12.3 本地存储（§12.3、§18.1、§18.3）：SQLite 索引（账号、邮箱、消息身份为账号加邮箱加 UIDVALIDITY 加 UID、标记、头字段）、原文按内容存进内容目录、FTS5 全文索引（可重建）、结构版本与升级前备份、升级失败只读；各类大小可查。已做：`Store`（rusqlite，自带 SQLite 3.53.2，FTS5 与 dbstat）在给定文件夹开 `mail.db`（WAL，外键开）；表 accounts、mailboxes（账号加名字唯一）、messages（身份为邮箱加 UIDVALIDITY 加 UID，POP3 为邮箱加 UIDL，两者恰居其一，各有部分唯一索引；头字段：Message-ID、In-Reply-To、References、主题、发件人、收件人、抄送（JSON）、日期；附件数在整封到达前为空）、flags（每标记一行）；原文按 SHA-256 存在 `content/前两位/散列`（暂存后改名，同内容只存一份，读时核对散列，不符即报“已损坏”，最后一封引用它的邮件删去时删文件）；同一身份再来头部只改标记，整封到达即补原文、附件数与正文索引；FTS5 表 `search`（主题、人、正文、附件名，无内容副本，trigram 分词，删邮件的触发器同删索引行），HTML 独有的正文以 html5ever 分词器取文字（略去 script 与 style，标签处断词）；`rebuild_search` 由索引行与原文重建；`PRAGMA user_version` 记结构版本，升级前以 `VACUUM INTO` 备份到 `backups/`，各步在一个事务里，备份或升级失败即以只读重开、写入报 `ReadOnly` 并带原因，更新的版本写的库同样只读；`sizes` 给索引、全文索引（dbstat）与原文三类字节数。测试：结构版本 5 个（新库、升级先备份、失败回滚且只读、更新的版本只读、无备份不升级），存储 6 个（整封、先头后整封、UIDVALIDITY 与 POP 身份、共用原文、损坏原文、HTML 正文与重建），去掉各自的规则即失败；引擎共 72 个测试。第 1 轮审查后：重建先删去整张全文表再按库里记的定义重建（损坏的段删不掉）；存原文时已有的文件先核对散列，坏了即以本次收到的字节改写；写库失败时删去本次新建且无人引用的原文；`sizes` 的索引按 `mail.db`、`-wal`、`-shm` 的实际字节减去全文页；HTML 取文字照树构建器给分词器的状态（script 为脚本数据，style、iframe、noembed、noframes 为原始文本，title 与 textarea 为 RCDATA，xmp 与 plaintext 照显示，自闭合在 HTML 元素上无效；SVG 与 MathML 里自闭合有效、script 与 style 的文字略去）；给不存在的邮件设标记即报错。`tests/store_repair.rs` 6 个，去掉各自的修复即失败；引擎共 78 个测试。第 2 轮审查后：HTML 取文字改走 html5ever 的整个树构建器（`store/html.rs`，自建的节点表作 `TreeSink`，脚本关闭），命名空间、整合点与插入模式全由它定，不再自仿分词器状态；不画的元素照 HTML 渲染规则（script、style、title、head、iframe、template 等）略去；树构建器每个开始标签都扫一遍开放元素栈，嵌套越深越慢（实测 release：十万层 div 28 秒），故按 4 KB 分块喂入，元素深过 512 层（同 Blink 的上限）即停，已读的文字照常入索引，警告写明哪封邮件；暂存原文写入或改名失败时删去暂存文件，删不掉即连同原因报错。`tests/store_repair.rs` 共 8 个；去掉清理、不画元素、MathML 整合点、深度上限即各自失败。第 3 轮审查后（最后一轮，修完不再审查）：删除与替换邮件时，原文文件在同一事务提交之前删去，删不掉即整步回滚，邮件与原文都留着、可重试（原先先提交再删文件，失败后失去引用，文件永久残留）；提交在删文件之后失败时，下次清理遇文件已不在只记警告；两处超长注释缩短。`tests/store_release.rs`（Windows，以只读共享占住文件）2 个，删去与替换各一，改回先提交即失败；引擎共 82 个测试
 - [x] C12.4 凭据与 OAuth（MAIL-02，§18.2）：密码与刷新令牌只进系统凭据库（不可用即只留到本次退出并写明）；Gmail 与 Microsoft 的授权码加 PKCE、本机回环、换令牌与刷新；客户端 ID 编进构建，缺 ID 即写明不可用；刷新被拒转为“需重新登录”；替身授权端点测试。已做：`Vault`（keyring：钥匙串、Windows 凭据、Secret Service；拒绝时只在内存，`Kept::Session` 带原因）；`oauth::SignIn` 在 127.0.0.1 随机端口等回跳（Google 写 127.0.0.1，Microsoft 写 localhost），state 与 S256 PKCE 各 32 随机字节，回跳的 state 不符或带 error 即报错，其余请求答 404；换令牌与刷新经 ureq（后台线程，30 秒），刷新答 invalid_grant 或 interaction_required 即 `SignInAgain`，其余为 `SignIn`；邮箱取 ID 令牌的 email，Microsoft 无则 preferred_username；客户端 ID 由 build.rs 读环境变量或项目根 `oauth.env`（坏行、未知键即构建失败），缺即 `Unavailable`；`tests/oauth.rs` 以替身令牌端点与替身浏览器核对，`tests/vault.rs` 以替身凭据库核对。审查三轮：第 1 轮四处（Google 授权带 `access_type=offline` 与 `prompt=consent`、换码无刷新令牌即报错；Microsoft 另要 `profile`；`oauth.env` 读不出即构建失败、不存在为未配置；超长注释）、第 2 轮一处（ureq 在 trace 级写出令牌与客户端密钥，`ely_platform::logging` 不记 `ureq`、`ureq_proto` 低于 info 的日志；`tests/private_oauth_logs.rs` 以真日志器在 trace 下核对，改回即失败）已修，第 3 轮通过
 - [x] C12.5 账号（MAIL-01、02）：ElyMail 主窗口与添加账号对话框：Gmail、Microsoft、iCloud（应用专用密码向导）、其他 IMAP/SMTP、POP3/SMTP；服务器、端口、TLS 方式、认证方式；连通性检查逐项报结果；多账号；删除账号先写明删去的本地缓存、草稿与凭据，不动服务端。已做：引擎 `Account`（地址、用户名、`Auth` 为密码或 Google、Microsoft 的 OAuth，`Protocol` 为 IMAP 或 POP3，收发两个 `Server`），以 JSON 存在 accounts 表的 settings 列（结构 v1 未发布，直接改，不加升级步）；`valid` 照 SMTP 的 Mailbox 核对地址，用户名不空不带换行，主机不空不带空白，端口非零；`check` 同时登录收发两个服务器再退出，各自报结果；`Store::cached` 给该账号的邮件数与只属它的原文字节，`remove_account` 在一个事务里删账号（邮箱、邮件、标记随外键删去）与只属它的原文，原文在提交前删、删不掉即回滚，与别的账号共用的原文留着；凭据库名为 `account:地址`。应用：主窗口左栏列出账号（地址与协议、收信主机），中间为所选账号的地址、用户名、收发服务器、登录方式与“删除账号…”，无账号时为空状态与“添加账号…”；“账号”菜单两项。添加对话框分 Gmail、Microsoft、iCloud、IMAP、POP3：Gmail 与 Microsoft 填可选的地址作登录提示，“以 Google/Microsoft 登录”开浏览器，回来后以 ID 令牌的地址与访问令牌检查两台服务器，构建缺客户端 ID 即提示条写明并禁用按钮；iCloud 写明生成应用专用密码的步骤并可开 Apple 账户，服务器固定（imap.mail.me.com:993 TLS，smtp.mail.me.com:587 STARTTLS，两者用户名都用完整地址，Apple 写明可用）；IMAP 与 POP3 填地址、登录方式（密码或两种 OAuth）、用户名、密码、收发主机、端口与加密，改加密时端口随之换成惯用的，手填的端口不动。“检查”逐项报收信与发信服务器（连不上、TLS 失败、拒绝登录及服务器原话等，按界面语言），通过才可“添加账号”，之后改任一字段即回到编辑；后到的检查与登录按编号丢弃，关对话框即中止网络任务；同一地址（不分大小写）不重复添加。保存先写库再存凭据，凭据库拒绝即提示只保留到退出。删除先在后台量出缓存，对话框写明删去的邮件数与大小、草稿与保存的密码或登录、服务器上的邮件不动；先删凭据，删不掉即不删账号并报错。库与网络都在后台（库在锁后，网络在两线程的 tokio 运行时），提示文字不进日志（可能含地址）。测试：引擎 `tests/accounts.rs` 6 个（往返、地址唯一、删去时留下共用原文与量缓存、两台服务器都登录、各自报结果、不合的设置），应用窗口测试 8 个（缺项逐个写明、端口随加密、检查逐项报连不上且改字段回到编辑、iCloud 的服务器、缺客户端 ID、凭据库拒绝仍保存并留在本次、保存与重开读回再删去且凭据随之删、凭据删不掉即留账号），去掉各自的规则即失败；真窗口冒烟（`ELY_SMOKE=1`）画出一帧。未验证：真实的 Gmail、Microsoft、iCloud 与其他服务器（须用户的客户端 ID 与账号）。第 1 轮审查后：删去或替换的原文先改名留在原处，提交后才删、失败即改回，打开库时收拾中断留下的（引擎 `tests/aside.rs`：第二个原文删不掉时两个都还读得到，重开后放回或删去）；删除确认按钮带着所删的账号（原先关闭回调先清掉状态，按了什么也不删），测试改按真的按钮；端口以上次自动填入的值判断是否键入（原先把惯用端口当未键入，IMAP 收信填 995 会被冲掉）；请求在整个窗口编号，保存的结果只作用于仍在等它的对话框（原先会关掉或改写之后新开的对话框），测试 `a_late_save…`、`a_late_failed_save…`；去掉各修复即失败。第 2 轮审查后：本地删除失败即把读出的凭据放回（凭据库拒绝时只放回本次并写明），账号仍可登录（`a_failed_removal_puts_the_secret_back`：原文的停放处被占，删除失败后凭据、账号与原文都在）；地址唯一由库保证（`address` 不分 ASCII 大小写，`add_account` 遇冲突报 `Error::Taken`，对话框写明已添加），两个在途的保存不再都成功（`two_saves_in_flight_add_an_address_once`）；左栏账号列表改用组件库 `SelectableList`（一个 Tab 停靠点，上下键选账号，`the_keyboard_picks_another_account`）；去掉各修复即失败。第 3 轮审查后：凭据库删不掉时内存里的凭据照留（`Vault::remove` 先删凭据库，成功才清内存），凭据库不可用时添加的账号删除失败后仍能取回凭据（`a_Line truncated
-- [ ] C12.6 IMAP 同步（MAIL-14，§12.3，QA-18）：文件夹列表与特殊用途；邮件头全部、正文只取 90 天内；UIDVALIDITY 变化即重建该邮箱；服务端删除与标记变化随之；后台任务执行，界面显示最后同步时间、下载范围、未缓存的数量与失败原因
+- [x] C12.6 IMAP 同步（MAIL-14，§12.3，QA-18）：文件夹列表与特殊用途；邮件头全部、正文只取 90 天内；UIDVALIDITY 变化即重建该邮箱；服务端删除与标记变化随之；后台任务执行，界面显示最后同步时间、下载范围、未缓存的数量与失败原因。已做：引擎 `Imap::folders` 先问 CAPABILITY，有 LIST-EXTENDED 与 SPECIAL-USE 即以 `LIST "" * RETURN (SPECIAL-USE)` 取角色，否则照常 LIST（INBOX 按名，其余按 RFC 6154 的属性；`\Noselect`、`\NonExistent` 不可选）；`examine` 读 UIDVALIDITY 与 EXISTS（缺任一即报错，不当作空邮箱）；`flags`、`headers`、`bodies`、`structures`、`text` 按 UID 取，同一 UID 的多条 FETCH 回复合并（别的客户端改标记时插来的标记行照收），命令结束后才查所要的段（`\Recent` 属本次会话，不存）；`since` 以 UID SEARCH SINCE 找 90 天内的邮件。`sync::imap` 一轮：列出的可选文件夹存为邮箱（带角色），不再列出的连同邮件删去，库里 INBOX 不分大小写（`Store::mailbox`、`set_folders`）；每个邮箱 UIDVALIDITY 变了即清空重建；服务端已无的 UID 删去，标记不同即改；更早的邮件只取邮件头；90 天内的先读 BODYSTRUCTURE，无附件即取整封，有附件只取邮件头、各部分的 MIME 头与正文部分（文字，related 里的内嵌图片），按原结构重组，附件内容为空，部分号记在 `missing`（`Fetched::Text`、`Stored::missing`），附件按需（C12.17）；已有邮件头的在进入 90 天时补正文；每次 FETCH 至多 100 个 UID；任何一步失败即停下报错，上次同步时间不变，成功才记下；`sync::secret` 取密码，OAuth 以刷新令牌换访问令牌（新的刷新令牌随即存下）。应用：窗口打开时与添加账号后各同步一次 IMAP 账号（POP3 随 C12.7），在应用的 tokio 运行时上；删除账号先中止其同步；账号详情写明上次同步时间（或正在同步、尚未同步）、下载范围、只有邮件头的邮件数与附件未下载的邮件数，失败以提示条写出原因。测试：引擎 `tests/sync.rs` 9 个（首次同步取邮件头与近期正文并记角色与时间；删除、标记与不再列出的文件夹随服务端；新 UIDVALIDITY 重建；失败不记时间；缺 EXISTS 即失败且邮件留着；取信中插来的标记行；INBOX 换大小写不删邮件；扩展 LIST 取角色；有附件的只存正文、附件为空且记部分号），应用 `tests/syncs.rs` 3 个（打开即同步并写明连不上、POP3 不同步；凭据库里没有即写明）；去掉各规则即失败。未验证：真实服务器；令牌失效（`SignInAgain`）后重新登录的界面未做，现只写明原因。第 2 轮审查后：结构 v1 照已提交的原样，同步所需的列由 v2 升级加上（`schema::V2`：accounts.synced、mailboxes.role 与 uidvalidity、messages.missing），已存的 v1 库升级后邮件照留并能同步（`a_v1_store_gains_sync_and_keeps_its_mail`）；处置为 attachment 的多部分整个是一个按需的实体，部分号记在 `missing`，不再逐个子部分判（`an_attached_multipart_waits_whole`）；related 里不看处置：根照常读，带 Content-ID 的其余部分即正文资源，标为 attachment 的内嵌图片照样取下（`a_related_image_marked_attachment_comes_down`、`a_related_start_root_ignores_its_disposition`）；本机 TLS 初始化失败即为该账号的同步失败，提示条写出原因（`a_missing_tls_is_the_sync_failure`）；部分选择单测在 `imap/parts/tests.rs`（5 个，以真实的 BODYSTRUCTURE 解析后重组再经 MIME 读回）；去掉各修复即失败。第 3 轮审查后：邮箱的 UIDVALIDITY 不同于所记（含 v2 升级后尚未记下）时只删去别的 UIDVALIDITY 下的邮件，同一 UIDVALIDITY 的照留，升级后的首次同步不再删去已存的正文（`a_v1_store_gains_sync_and_keeps_its_mail` 接着打开邮箱）；related 的 `start` 在 BODYSTRUCTURE 里找不到（多部分不带 Content-ID）而有多部分子项时，各多部分子项照正文读，根的文字与 HTML 照样取下（`a_related_start_may_name_a_multipart`）；去掉各修复即失败；三轮已满，未再审查
 - [ ] C12.7 POP3 收取（MAIL-01，§12.3）：按 UIDL 去重下载到本地收件箱，“保留服务端副本”默认开，不做伪文件夹同步
 - [ ] C12.8 同步控制（MAIL-14）：手动刷新、运行期间定时同步、暂停、按文件夹完整离线下载
 - [ ] C12.9 邮箱视图（MAIL-03）：账号与文件夹、邮件列表、阅读区三栏；统一收件箱与账号视图；未读、标记、带附件筛选；列表只画可见行，10 万封滚动不掉帧

tasks/reviews/C12.6-r1.md

@@ -0,0 +1,11 @@
+FAIL
+
+1. crates/ely-mail-engine/src/imap.rs:139 — Missing `EXISTS` silently becomes an empty mailbox. If `EXAMINE` returns UIDVALIDITY and tagged OK but omits the required message count, `sync::one` deletes every cached message and records a successful sync. The independent probe returned `Ok(Report { removed: 1, .. })`, zero cached messages and a new last-sync time. Keep the count optional until validated and return a protocol error before changing local data when it is absent. Add a regression asserting both failure and cache preservation. Evidence: `.work/C12.6-r1-probes.log`, `an_examine_without_exists_does_not_delete_mail`; [RFC 3501 §6.3.1–6.3.2](https://www.rfc-editor.org/rfc/rfc3501.html#section-6.3.1).
+
+2. crates/ely-mail-engine/src/imap.rs:285 — A concurrent flag notification makes a valid header/body fetch fail. Another client can change an asked UID's flags while it is being downloaded; a valid body response followed by `* 1 FETCH (UID 1 FLAGS (\Seen))` is rejected as `UID 1 came without a body`, aborting synchronization and discarding the fetched batch. Unsolicited FETCH responses can include UID. Accumulate attributes by UID, process flag updates separately, and validate the requested section after the command completes rather than requiring it in every response. Evidence: `.work/C12.6-r1-probes.log`, `concurrent_flags_do_not_make_a_body_fetch_fail`; [RFC 3501 §7](https://www.rfc-editor.org/rfc/rfc3501.html#section-7).
+
+3. crates/ely-mail-engine/src/store/folders.rs:43 — Folder reconciliation treats INBOX's case variants as different identities. After caching `INBOX`, a subsequent LIST spelling it `inbox` deletes its messages and originals even with unchanged UIDVALIDITY and UIDs. Messages older than 90 days then return only as headers, losing their offline bodies. The probe's previously cached MessageId became absent after this case change. Normalize INBOX consistently for reconciliation and mailbox lookup while preserving other mailbox names' case distinctions, and test retention of the original bytes. Evidence: `.work/C12.6-r1-probes.log`, `inbox_casing_does_not_destroy_cached_mail`; [RFC 3501 §5.1](https://www.rfc-editor.org/rfc/rfc3501.html#section-5.1).
+
+4. crates/ely-mail-engine/src/imap.rs:123 — Special-use discovery relies entirely on optional attributes in ordinary LIST responses. A server may advertise `LIST-EXTENDED SPECIAL-USE` and return its Sent/Drafts/Trash roles only when requested with `RETURN (SPECIAL-USE)`. This implementation never discovers capabilities or requests those attributes, so such folders are stored with `role = None`; the probe returned no Sent role despite the advertised capability. Discover the supported capabilities and request special-use attributes when available, retaining ordinary LIST for servers without the extension. Add coverage for a server that omits roles from ordinary LIST. Evidence: `.work/C12.6-r1-probes.log`, `roles_are_discovered_when_only_extended_list_carries_them`; [RFC 6154 §2](https://www.rfc-editor.org/rfc/rfc6154.html#section-2).
+
+5. crates/ely-mail-engine/src/imap.rs:186 — Default body synchronization downloads attachments without a request. `BODY.PEEK[]` retrieves the entire MIME message, and `sync.rs:148` persists it as `Fetched::Whole`. With the existing multipart sample classified as recent, the independent sync probe cached `report.pdf` and all nine attachment bytes automatically. Larger attachments are likewise transferred and stored, contrary to PRD §12.3's attachment-on-demand default. Fetch and cache the message's body sections separately from attachment payloads; retrieve attachment bytes only on request or explicit complete offline retention, and test that default synchronization leaves those payloads uncached. Evidence: `.work/C12.6-r1-probes.log`, `a_default_recent_body_sync_does_not_fetch_attachments`.

tasks/reviews/C12.6-r2.md

@@ -0,0 +1,9 @@
+FAIL
+
+1. crates/ely-mail-engine/src/store/schema.rs:2 — The schema changes overwrite V1 without adding a migration. A database created by the completed C12.3/C12.5 implementation already has `user_version = 1`, so opening it skips upgrading and leaves `synced`, `role`, `uidvalidity` and `missing` absent. The independent probe opened the exact HEAD schema successfully, then `Store::synced` failed with `no such column: synced`; folder synchronization and message reads likewise reference absent columns. Preserve the existing V1 and append a versioned migration, with a regression that reopens a populated previous-version database and verifies its data and synchronization. Evidence: `.work/C12.6-r2-probes.log`, `probe_an_existing_v1_store_is_upgraded`.
+
+2. crates/ely-mail-engine/src/imap/parts.rs:110 — Multipart attachment dispositions are ignored before descending into their children, leaving round 1 defect 5 incompletely fixed. For a recent multipart/mixed message containing an attached multipart/alternative whose child is text/plain, every leaf is classified as body content, `Node::complete()` returns true, and sync downloads and stores the whole attachment through `BODY.PEEK[]`. The independent probe cached `nested.mime`, including its private attachment payload, without a request. Classify an attached multipart as one omitted entity before descending, retain its part number in `missing`, and cover multipart attachments in the synchronization regression. This matches the existing MIME reader and [RFC 2183 §2.9](https://www.rfc-editor.org/rfc/rfc2183.html#section-2.9). Evidence: `.work/C12.6-r2-probes.log`, `probe_multipart_attachments_stay_on_demand`.
+
+3. crates/ely-mail-engine/src/imap/parts.rs:142 — The attachment check overrides multipart/related semantics and omits inline body content. A recent HTML message with a CID PNG marked `Content-Disposition: attachment` is valid and is classified as an inline image by the existing MIME reader, but synchronization never requests its payload. The probe completed synchronization with empty image bytes instead of the PNG bytes; the message is then marked body-cached and subsequent default syncs skip it. Apply related-container semantics before disposition checks when selecting body parts, and test related roots and CID images carrying attachment dispositions. [RFC 2387 §4](https://www.rfc-editor.org/rfc/rfc2387.html#section-4) requires ignoring the disposition type within multipart/related. Evidence: `.work/C12.6-r2-probes.log`, `probe_related_images_marked_attachment_are_cached`.
+
+4. apps/ely-mail/src/syncing.rs:35 — A TLS initialization failure is discarded instead of appearing as a synchronization failure. If the local store opens but platform TLS initialization returns an error, startup synchronization returns after a generic log message, creates no `Syncing` entry, and the account view displays neither synchronization status nor the actual failure reason required by §12.3. The GPUI probe injected `platform trust store unavailable` and confirmed that the account had no failure state. Preserve the original TLS error in the account's synchronization state and render it through the existing failure banner; add startup coverage with failed TLS initialization. Evidence: `.work/C12.6-r2-app-probes.log`, `probe_tls_initialization_failure_is_shown_as_a_sync_failure`.

tasks/reviews/C12.6-r3.md

@@ -0,0 +1,5 @@
+FAIL
+
+1. crates/ely-mail-engine/src/store/folders.rs:106 — The first sync deletes an upgraded cache even when UIDVALIDITY has not changed. V2 leaves `mailboxes.uidvalidity` NULL, so `open_mailbox` calls `without_mail` and deletes every existing message and original before comparing their stored UIDVALIDITY with the server's. The independent probe upgraded a populated v1 database, confirmed its cached body survived opening, then called the same `open_mailbox(INBOX, 7)` used by sync: the message with UIDVALIDITY 7 disappeared and the message count became zero. For mail older than 90 days, synchronization restores only headers, losing its offline body. Reconcile existing message identities with the reported UIDVALIDITY when initializing the new column; preserve matching cached messages. Extend the migration regression through the first synchronization, including an older cached body. Evidence: `.work/C12.6-r3/probes.log`, `probe_a_migrated_body_survives_its_first_examine`.
+
+2. crates/ely-mail-engine/src/imap/parts.rs:159 — Related-root selection cannot resolve a multipart child named by `start`. It uses `id(child)`, which always returns None for multipart nodes, then falls back to the first child and marks the actual root as an omitted attachment. A valid multipart/related message with a CID image first and a multipart/alternative root second (`Content-ID: <root>`, `start="<root>"`) loses both its plain and HTML bodies. The independent reconstruction probe read `hello` from the full message, but the selected sections omitted part 2 and the cached message had neither text nor HTML. Sync nevertheless stores it as body-cached, so later default syncs skip repairing it. Resolve related roots using their MIME headers when BODYSTRUCTURE cannot supply Content-ID, and cover a non-first multipart root through reconstruction and synchronization. Evidence: `.work/C12.6-r3/probes.log`, `probe_related_multipart_start_root_is_cached`.