documents: tables keep their line ending; tabs count four

Table rewrites find their prefix and rows by CR, LF and CRLF alike and
write the table's own line ending; insert_table takes the text's. A
cell's width counts a tab as 4 and a control character as 0, as
Typora's wcswidth does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -13,7 +13,7 @@ Ely GPUI Component. A component library for GPUI, in light and dark.
 - Web views: `wry` 0.57 (Apache-2.0 OR MIT, default features off, macOS only) lays a child WKWebView over the window.
 - Terminal: `alacritty_terminal` 0.26 (Apache-2.0, default features off, native only) for the grid, its parser and the pseudo-terminal; `futures` carries its events. `vte` 0.15 (Apache-2.0 OR MIT), the parser it re-exports, reads `AnsiText`'s codes alone, so colored output builds for wasm32.
 - Diffs: `similar` 3.2 (Apache-2.0) for line and word diffs and three-way merges; its `unicode` feature splits words at punctuation.
-- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` turns to its emoji as GitHub reads it (the `emojis` crate's names, GitHub's today, so `:cricket:` is the insect; read after escapes and references, so `\:smile:` turns too; inside an autolink or `<code>` it stays text; within one DOM text node, which filtered tags do not split, the first `:name:`-shaped run decides for all: blank text or a space before it turns every shortcode, else none, so `x:smile: :smile:` stays), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` draw as what they mean GFM's filtered tags (`script`, `style`, `title`, `textarea`, `iframe`, `xmp`, `noembed`, `noframes`, `plaintext`) show as text, and other tags, hyphenated names too, vanish, leaving their text; a tag's name runs to white space or `/`, as HTML5 reads it; attribute values split on ASCII whitespace alone and decode as HTML decodes them; each Markdown container is the element GitHub writes (`em`, `strong`, `del`, `a`), on one stack with the tags and repaired as HTML5 repairs: an end tag closes the innermost element of its name and all above it, of which only formatting elements (`a`, `b`, `code`, `em`, `i`, `s`, `strike`, `strong`, `u`) reopen, a new `a` ends the open one and an `a` without `href` links nothing; other tags are not tracked, and `</br>` breaks as `<br>` does; and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. At the root, blocks read as GitHub shows them: a quote whose first line is plain text reading `[!NOTE]`, `[!TIP]`, `[!IMPORTANT]`, `[!WARNING]` or `[!CAUTION]` in any case, after escapes and references, spaces and tabs around it, with more after it, is an alert under its title (Important in the palette's violet, `syntax.keyword`, read at 4.5:1); a paragraph that is `[toc]` alone, as Typora reads it, lists every heading by depth, and a press on one reveals it as a link press does (`MarkdownDocument`'s `reveal`), the heading found where `CodeEditor` anchors carried it through each edit since the read, gone if an edit or an undo took it, and a secondary press acts on press, since a block writing back moves the list; front matter shows in a box without its fences; an HTML block shows its text with the inline tags above, white space collapsed and owed as a browser lays it, block elements breaking lines and closing what opened inside them, HTML5's start tags ending an open `p`, `fieldset` and `menu` too, and an `li` start an `li`, a `dd` or `dt` start either, as HTML5 ends them, comments dropped, the filtered tags shown as text, and an unclosed tag dropping the rest; a rule is a 2px `border_strong` line; bullets go •, ◦, ▪ by list depth, ordered lists counted. Exports read a tree as the drawing does through `MarkdownReader` (`block` gives its `MarkdownBlock`s, `inline` its text, `note` and `label` its footnotes, `headings` the `[toc]` list), one fresh stack per paragraph, heading or cell. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of its nodes, not its bytes: in release, 128 KiB of one-letter paragraphs takes 1.7 s, 16 KiB 42 ms, while 128 KiB of prose takes 20 ms; a block-heavy text under the cutoff can still hold the UI thread past 100 ms, a known limit). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the listLine truncated
+- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` turns to its emoji as GitHub reads it (the `emojis` crate's names, GitHub's today, so `:cricket:` is the insect; read after escapes and references, so `\:smile:` turns too; inside an autolink or `<code>` it stays text; within one DOM text node, which filtered tags do not split, the first `:name:`-shaped run decides for all: blank text or a space before it turns every shortcode, else none, so `x:smile: :smile:` stays), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` draw as what they mean GFM's filtered tags (`script`, `style`, `title`, `textarea`, `iframe`, `xmp`, `noembed`, `noframes`, `plaintext`) show as text, and other tags, hyphenated names too, vanish, leaving their text; a tag's name runs to white space or `/`, as HTML5 reads it; attribute values split on ASCII whitespace alone and decode as HTML decodes them; each Markdown container is the element GitHub writes (`em`, `strong`, `del`, `a`), on one stack with the tags and repaired as HTML5 repairs: an end tag closes the innermost element of its name and all above it, of which only formatting elements (`a`, `b`, `code`, `em`, `i`, `s`, `strike`, `strong`, `u`) reopen, a new `a` ends the open one and an `a` without `href` links nothing; other tags are not tracked, and `</br>` breaks as `<br>` does; and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. At the root, blocks read as GitHub shows them: a quote whose first line is plain text reading `[!NOTE]`, `[!TIP]`, `[!IMPORTANT]`, `[!WARNING]` or `[!CAUTION]` in any case, after escapes and references, spaces and tabs around it, with more after it, is an alert under its title (Important in the palette's violet, `syntax.keyword`, read at 4.5:1); a paragraph that is `[toc]` alone, as Typora reads it, lists every heading by depth, and a press on one reveals it as a link press does (`MarkdownDocument`'s `reveal`), the heading found where `CodeEditor` anchors carried it through each edit since the read, gone if an edit or an undo took it, and a secondary press acts on press, since a block writing back moves the list; front matter shows in a box without its fences; an HTML block shows its text with the inline tags above, white space collapsed and owed as a browser lays it, block elements breaking lines and closing what opened inside them, HTML5's start tags ending an open `p`, `fieldset` and `menu` too, and an `li` start an `li`, a `dd` or `dt` start either, as HTML5 ends them, comments dropped, the filtered tags shown as text, and an unclosed tag dropping the rest; a rule is a 2px `border_strong` line; bullets go •, ◦, ▪ by list depth, ordered lists counted. Exports read a tree as the drawing does through `MarkdownReader` (`block` gives its `MarkdownBlock`s, `inline` its text, `note` and `label` its footnotes, `headings` the `[toc]` list), one fresh stack per paragraph, heading or cell. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of its nodes, not its bytes: in release, 128 KiB of one-letter paragraphs takes 1.7 s, 16 KiB 42 ms, while 128 KiB of prose takes 20 ms; a block-heavy text under the cutoff can still hold the UI thread past 100 ms, a known limit). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the listLine truncated
 - Pictures: `image` 0.25 (MIT OR Apache-2.0), the crate gpui decodes with, default features off; it turns decoded frames and rims map tiles.
 - Math and diagrams: RaTeX 0.1.14 (`ratex-parser`, `ratex-layout`, `ratex-svg` with `embed-fonts`, `ratex-types`; MIT, KaTeX's fonts OFL) typesets `Latex` to SVG with outlined glyphs; `mermaid-rs-renderer` 0.3.1 (MIT, default features off) draws `Diagram`. Both go through gpui's `SvgRenderer` at the window's scale, loaded as gpui assets off the main thread; `usvg` 0.46, gpui's own, reads a diagram's size. A RaTeX or Mermaid panic on parsed input becomes the element's error. Each element holds its picture through `primitives::held`: a new source, or the element leaving the window, removes the old asset from gpui's cache and drops its image from the atlas after the frame.
 - Maps: Natural Earth's 110m countries (public domain). `assets/maps/countries.geojson`, trimmed by `scripts/countries.py` to an ISO code, a name and positions to 0.01°, backs `maps::WorldMap`; the gallery's tiles are cut from the same file, z0 to z3, by `scripts/tiles.py` with Pillow.

src/documents/document/tables.rs

@@ -6,6 +6,7 @@ use unicode_width::UnicodeWidthStr;
 
 use super::{
     MarkdownDocument,
+    block::{eol, line_end, line_start, next_line, prev_line},
     format::{Change, EditRefusal},
 };
 use crate::documents::{
@@ -59,14 +60,15 @@ pub(super) fn edit_table(text: &str, at: usize, edit: TableEdit) -> Result<Chang
             _ => None,
         })
         .ok_or(TableRefusal::Outside)?;
-    let prefix = &text[text[..range.start].rfind('\n').map_or(0, |found| found + 1)..range.start];
-    if text[range.clone()]
-        .split('\n')
-        .skip(1)
-        .any(|line| !line.starts_with(prefix))
-    {
-        return Err(TableRefusal::Mixed);
+    let prefix = &text[line_start(text, range.start)..range.start];
+    let mut line = range.start;
+    while let Some(next) = next_line(text, line).filter(|next| *next < range.end) {
+        if !text[next..].starts_with(prefix) {
+            return Err(TableRefusal::Mixed);
+        }
+        line = next;
     }
+    let eol = eol(&text[range.clone()]);
     let rows: Vec<&mdast::Node> = table.children.iter().collect();
     let row = rows
         .iter()
@@ -157,7 +159,7 @@ pub(super) fn edit_table(text: &str, at: usize, edit: TableEdit) -> Result<Chang
         TableEdit::Tidy => (row, column, offset),
         _ => return Err(TableRefusal::Kept),
     };
-    let (out, caret) = table_text(&grid, &aligns, caret, prefix);
+    let (out, caret) = table_text(&grid, &aligns, caret, prefix, eol);
     let caret = range.start + caret;
     Ok((vec![(range, out)], caret..caret))
 }
@@ -183,18 +185,29 @@ fn within(cell: &mdast::Node, at: usize) -> usize {
     }
 }
 
+/// Display columns, a tab four and controls none, as Typora's.
+fn width(cell: &str) -> usize {
+    let tabs = cell.matches('\t').count();
+    let shown: usize = cell
+        .split(char::is_control)
+        .map(UnicodeWidthStr::width)
+        .sum();
+    shown + 4 * tabs
+}
+
 /// The table's text and the caret at row, column, byte.
 fn table_text(
     grid: &[Vec<String>],
     aligns: &[AlignKind],
     at: (usize, usize, usize),
     prefix: &str,
+    eol: &str,
 ) -> (String, usize) {
     // Typora pads columns 4 to 60 display columns wide.
     let widths: Vec<usize> = (0..aligns.len())
         .map(|column| {
             (grid.iter())
-                .map(|cells| cells[column].width())
+                .map(|cells| width(&cells[column]))
                 .fold(4, usize::max)
                 .min(60)
         })
@@ -206,7 +219,7 @@ fn table_text(
         for (column, cell) in cells.iter().enumerate() {
             let room = widths
                 .get(column)
-                .map_or(0, |width| width.saturating_sub(cell.width()));
+                .map_or(0, |widest| widest.saturating_sub(width(cell)));
             let before = match aligns.get(column) {
                 Some(AlignKind::Right) => room,
                 Some(AlignKind::Center) => room / 2,
@@ -222,7 +235,7 @@ fn table_text(
             out.push_str(" |");
         }
         if row == 0 {
-            out.push('\n');
+            out.push_str(eol);
             out.push_str(prefix);
             out.push('|');
             for (align, width) in aligns.iter().zip(&widths) {
@@ -236,7 +249,7 @@ fn table_text(
             }
         }
         if row + 1 < grid.len() {
-            out.push('\n');
+            out.push_str(eol);
             out.push_str(prefix);
         }
     }
@@ -252,26 +265,22 @@ impl MarkdownDocument {
         self.commit(cx);
         let text = self.editor.read(cx).text().to_string();
         let at = self.editor.read(cx).primary().range().start;
-        let end = text[at..].find('\n').map_or(text.len(), |found| at + found);
-        let start = text[..at].rfind('\n').map_or(0, |found| found + 1);
+        let (start, end, eol) = (line_start(&text, at), line_end(&text, at), eol(&text));
         // A blank line gives way; a written one keeps.
         let blank = text[start..end].trim().is_empty();
-        let above = text[..start.saturating_sub(1)]
-            .rsplit('\n')
-            .next()
-            .unwrap_or_default();
+        let above = prev_line(&text, start).map_or("", |line| &text[line..line_end(&text, line)]);
         let (from, before) = match (blank, above.trim().is_empty()) {
-            (true, true) => (start, ""),
-            (true, false) => (start, "\n"),
-            (false, _) => (end, "\n\n"),
+            (true, true) => (start, String::new()),
+            (true, false) => (start, eol.to_string()),
+            (false, _) => (end, eol.repeat(2)),
         };
-        let after = if text[end..].starts_with("\n\n") || end == text.len() {
+        let after = if text[end..].starts_with(&eol.repeat(2)) || end == text.len() {
             ""
         } else {
-            "\n"
+            eol
         };
         let grid = vec![vec![String::new(); 3]; 3];
-        let (table, caret) = table_text(&grid, &[AlignKind::None; 3], (0, 0, 0), "");
+        let (table, caret) = table_text(&grid, &[AlignKind::None; 3], (0, 0, 0), "", eol);
         let caret = from + before.len() + caret;
         log::info!("markdown document: a table goes in");
         self.editor.update(cx, |editor, cx| {
@@ -378,6 +387,39 @@ mod tests {
         );
     }
 
+    #[test]
+    fn a_rewrite_keeps_the_tables_line_ending() {
+        let crlf = "Before\r\n\r\n| A | B |\r\n| - | - |\r\n| 1 | 2 |\r\n\r\nAfter\r\n";
+        assert_eq!(
+            edited_in(crlf, "1", TableEdit::Tidy),
+            Ok("Before\r\n\r\n| A    | B    |\r\n| ---- | ---- |\r\n| ‸1    | 2    |\r\n\r\nAfter\r\n".into())
+        );
+        let cr = "Before\r\r| A | B |\r| - | - |\r| 1 | 2 |\r\rAfter\r";
+        assert_eq!(
+            edited_in(cr, "1", TableEdit::Tidy),
+            Ok("Before\r\r| A    | B    |\r| ---- | ---- |\r| ‸1    | 2    |\r\rAfter\r".into())
+        );
+        let moved = "| A | B |\r\n| - | - |\r\n| 1 | 2 |\r\n";
+        assert_eq!(
+            edited_in(moved, "1", TableEdit::MoveRowUp),
+            Ok("| ‸1    | 2    |\r\n| ---- | ---- |\r\n| A    | B    |\r\n".into())
+        );
+        let quoted = "> | A |\r> | - |\r> | 1 |\r";
+        assert_eq!(
+            edited_in(quoted, "1", TableEdit::RowBelow),
+            Ok("> | A    |\r> | ---- |\r> | 1    |\r> | ‸     |\r".into())
+        );
+    }
+
+    #[test]
+    fn a_tab_counts_four_columns_as_in_typora() {
+        let text = "| a\tb | x |\n| - | - |\n| q | r |\n";
+        assert_eq!(
+            edited_in(text, "q", TableEdit::Tidy),
+            Ok("| a\tb | x    |\n| ------ | ---- |\n| ‸q      | r    |\n".into())
+        );
+    }
+
     #[test]
     fn the_head_stays_and_outside_is_no_table() {
         assert_eq!(edited("A", TableEdit::DeleteRow), Err(TableRefusal::Kept));

src/documents/document/tests/tools.rs

@@ -291,6 +291,37 @@ fn a_table_edit_at_an_edge_refuses_by_event(cx: &mut TestAppContext) {
     assert_eq!(text(&document, cx), SMALL);
 }
 
+#[gpui::test]
+fn tidy_in_cr_text_keeps_what_comes_before(cx: &mut TestAppContext) {
+    let source = "Before\r\r| A | B |\r| - | - |\r| 1 | 2 |\r\rAfter\r";
+    let (document, cx) = open(source, MarkdownMode::Source, cx);
+    let at = source.find('1').expect("a cell");
+    let editor = document.read_with(cx, |document, _| document.editor().clone());
+    editor.update(cx, |editor, cx| editor.select(std::iter::once(at..at), cx));
+    document.update(cx, |document, cx| {
+        document
+            .edit_table(crate::documents::TableEdit::Tidy, cx)
+            .expect("taken")
+    });
+    cx.run_until_parked();
+    assert_eq!(
+        text(&document, cx),
+        "Before\r\r| A    | B    |\r| ---- | ---- |\r| 1    | 2    |\r\rAfter\r"
+    );
+}
+
+#[gpui::test]
+fn a_new_table_takes_the_texts_line_ending(cx: &mut TestAppContext) {
+    let (document, cx) = open("Intro\rNext\r", MarkdownMode::Visual, cx);
+    document.update(cx, |document, cx| {
+        document.insert_table(cx).expect("inserted")
+    });
+    assert_eq!(
+        text(&document, cx),
+        "Intro\r\r|      |      |      |\r| ---- | ---- | ---- |\r|      |      |      |\r|      |      |      |\r\rNext\r"
+    );
+}
+
 #[gpui::test]
 fn tidy_in_an_open_table_keeps_it_open(cx: &mut TestAppContext) {
     let (document, cx) = open(SMALL, MarkdownMode::Visual, cx);