documents: emoji shortcodes, inline HTML tags and numbered footnote references draw as Typora draws them

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -13,7 +13,7 @@ Ely GPUI Component. A component library for GPUI, in light and dark.
 - Web views: `wry` 0.57 (Apache-2.0 OR MIT, default features off, macOS only) lays a child WKWebView over the window.
 - Terminal: `alacritty_terminal` 0.26 (Apache-2.0, default features off, native only) for the grid, its parser and the pseudo-terminal; `futures` carries its events. `vte` 0.15 (Apache-2.0 OR MIT), the parser it re-exports, reads `AnsiText`'s codes alone, so colored output builds for wasm32.
 - Diffs: `similar` 3.2 (Apache-2.0) for line and word diffs and three-way merges; its `unicode` feature splits words at punctuation.
-- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written.
+- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` turns to its emoji (the `emojis` crate's GitHub names), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` draw as what they mean and other tags vanish, leaving their text, a tag left open ends with the Markdown container it opened in, and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written.
 - Pictures: `image` 0.25 (MIT OR Apache-2.0), the crate gpui decodes with, default features off; it turns decoded frames and rims map tiles.
 - Math and diagrams: RaTeX 0.1.14 (`ratex-parser`, `ratex-layout`, `ratex-svg` with `embed-fonts`, `ratex-types`; MIT, KaTeX's fonts OFL) typesets `Latex` to SVG with outlined glyphs; `mermaid-rs-renderer` 0.3.1 (MIT, default features off) draws `Diagram`. Both go through gpui's `SvgRenderer` at the window's scale, loaded as gpui assets off the main thread; `usvg` 0.46, gpui's own, reads a diagram's size. A RaTeX or Mermaid panic on parsed input becomes the element's error. Each element holds its picture through `primitives::held`: a new source, or the element leaving the window, removes the old asset from gpui's cache and drops its image from the atlas after the frame.
 - Maps: Natural Earth's 110m countries (public domain). `assets/maps/countries.geojson`, trimmed by `scripts/countries.py` to an ISO code, a name and positions to 0.01°, backs `maps::WorldMap`; the gallery's tiles are cut from the same file, z0 to z3, by `scripts/tiles.py` with Pillow.

src/documents/render.rs

@@ -149,6 +149,18 @@ impl Draw<'_> {
                         color: Some(colors.fg_subtle),
                         ..HighlightStyle::default()
                     },
+                    Style::Underline => HighlightStyle {
+                        underline: Some(UnderlineStyle {
+                            thickness: rule,
+                            color: None,
+                            wavy: false,
+                        }),
+                        ..HighlightStyle::default()
+                    },
+                    Style::Mark => HighlightStyle {
+                        background_color: Some(colors.warning_subtle),
+                        ..HighlightStyle::default()
+                    },
                 };
                 (range.clone(), highlight)
             })

src/documents/tree/html.rs

@@ -0,0 +1,158 @@
+use super::Style;
+
+/// One HTML tag as written: its lowercase name and attributes.
+#[derive(Clone, Debug, PartialEq)]
+pub(crate) struct Tag {
+    pub name: String,
+    pub closing: bool,
+    pub attributes: Vec<(String, String)>,
+}
+
+impl Tag {
+    pub fn attribute(&self, name: &str) -> Option<&str> {
+        self.attributes
+            .iter()
+            .find(|(key, _)| key == name)
+            .map(|(_, value)| value.as_str())
+    }
+}
+
+/// How Typora shows an inline tag; others show their text.
+pub(crate) fn inline_style(name: &str) -> Option<Style> {
+    match name {
+        "u" => Some(Style::Underline),
+        "b" | "strong" => Some(Style::Strong),
+        "i" | "em" => Some(Style::Emphasis),
+        "s" | "del" | "strike" => Some(Style::Strike),
+        "code" | "kbd" => Some(Style::Code),
+        "mark" => Some(Style::Mark),
+        "a" => Some(Style::Link),
+        _ => None,
+    }
+}
+
+/// `html` read as one tag; comments and the rest are none.
+pub(crate) fn tag(html: &str) -> Option<Tag> {
+    let inner = html.trim().strip_prefix('<')?.strip_suffix('>')?;
+    let (closing, inner) = match inner.strip_prefix('/') {
+        Some(rest) => (true, rest),
+        None => (false, inner),
+    };
+    let named = inner
+        .find(|ch: char| !ch.is_ascii_alphanumeric())
+        .unwrap_or(inner.len());
+    if named == 0 || !inner.starts_with(|ch: char| ch.is_ascii_alphabetic()) {
+        return None;
+    }
+    Some(Tag {
+        name: inner[..named].to_ascii_lowercase(),
+        closing,
+        attributes: attributes(inner[named..].trim_end_matches('/')),
+    })
+}
+
+/// `name="value"`, `name='value'`, `name=value` and bare names.
+fn attributes(mut rest: &str) -> Vec<(String, String)> {
+    let mut out = Vec::new();
+    loop {
+        rest = rest.trim_start();
+        let named = rest
+            .find(|ch: char| ch.is_whitespace() || ch == '=')
+            .unwrap_or(rest.len());
+        if named == 0 {
+            return out;
+        }
+        let name = rest[..named].to_ascii_lowercase();
+        rest = rest[named..].trim_start();
+        let Some(after) = rest.strip_prefix('=') else {
+            out.push((name, String::new()));
+            continue;
+        };
+        let after = after.trim_start();
+        let (value, left) = match after.chars().next() {
+            Some(quote @ ('"' | '\'')) => match after[1..].find(quote) {
+                Some(end) => (&after[1..1 + end], &after[2 + end..]),
+                None => (&after[1..], ""),
+            },
+            _ => {
+                let end = after.find(char::is_whitespace).unwrap_or(after.len());
+                (&after[..end], &after[end..])
+            }
+        };
+        out.push((name, value.to_string()));
+        rest = left;
+    }
+}
+
+/// `:shortcode:` written as its emoji, as Typora shows it.
+pub(crate) fn emoji(text: &str) -> String {
+    let mut out = String::with_capacity(text.len());
+    let mut rest = text;
+    while let Some(open) = rest.find(':') {
+        out.push_str(&rest[..open]);
+        let after = &rest[open + 1..];
+        let code = after
+            .find(':')
+            .map(|close| &after[..close])
+            .filter(|code| !code.is_empty() && code.chars().all(shortcode_char));
+        match code.and_then(emojis::get_by_shortcode) {
+            Some(found) => {
+                out.push_str(found.as_str());
+                rest = &after[code.expect("a matched code").len() + 1..];
+            }
+            None => {
+                out.push(':');
+                rest = after;
+            }
+        }
+    }
+    out.push_str(rest);
+    out
+}
+
+fn shortcode_char(ch: char) -> bool {
+    ch.is_ascii_alphanumeric() || matches!(ch, '_' | '+' | '-')
+}
+
+/// `n` in superscript digits.
+pub(crate) fn superscript(n: usize) -> String {
+    const DIGITS: [char; 10] = ['⁰', '¹', '²', '³', '⁴', '⁵', '⁶', '⁷', '⁸', '⁹'];
+    n.to_string()
+        .bytes()
+        .map(|digit| DIGITS[usize::from(digit - b'0')])
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn a_tag_reads_its_name_and_attributes() {
+        let link = tag("<A HREF=\"https://ely.dev\" title='x' hidden>").expect("a tag");
+        assert_eq!((link.name.as_str(), link.closing), ("a", false));
+        assert_eq!(link.attribute("href"), Some("https://ely.dev"));
+        assert_eq!(link.attribute("title"), Some("x"));
+        assert_eq!(link.attribute("hidden"), Some(""));
+        assert_eq!(
+            tag("</u>").map(|t| (t.name, t.closing)),
+            Some(("u".into(), true))
+        );
+        assert_eq!(tag("<br/>").map(|t| t.name), Some("br".into()));
+        assert_eq!(tag("<!-- note -->"), None);
+        assert_eq!(tag("<1>"), None);
+    }
+
+    #[test]
+    fn shortcodes_become_emoji_and_others_stay() {
+        assert_eq!(emoji("hi :smile: :+1:"), "hi 😄 👍");
+        assert_eq!(emoji(":nope: at 10:30 and ::"), ":nope: at 10:30 and ::");
+        assert_eq!(emoji("a:smile:b"), "a😄b");
+    }
+
+    #[test]
+    fn numbers_go_up() {
+        assert_eq!(superscript(1), "¹");
+        assert_eq!(superscript(120), "¹²⁰");
+    }
+}

src/documents/tree/mod.rs

@@ -4,6 +4,8 @@ use markdown::mdast::{self, AlignKind};
 
 use super::markdown::{span, tree, walk};
 
+mod html;
+
 /// How a stretch of inline text prints.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub enum Style {
@@ -14,6 +16,9 @@ pub enum Style {
     Link,
     Math,
     Footnote,
+    Underline,
+    /// A highlighted run, as `<mark>`.
+    Mark,
 }
 
 /// Inline text, its styled stretches and link addresses.
@@ -63,13 +68,23 @@ pub enum Node {
 pub(crate) struct Reader<'a> {
     source: &'a str,
     urls: HashMap<String, String>,
+    /// Footnotes numbered by their first reference.
+    notes: HashMap<String, usize>,
+}
+
+/// A style open in a run; `tag` names an HTML one.
+struct Opened {
+    start: usize,
+    style: Style,
+    url: Option<String>,
+    tag: Option<String>,
 }
 
 /// Inline text being built, and the containers open around it.
 #[derive(Default)]
 struct Run {
     text: Inline,
-    open: Vec<(usize, Style, Option<String>)>,
+    open: Vec<Opened>,
 }
 
 impl Run {
@@ -77,21 +92,65 @@ impl Run {
     fn cut(&mut self) -> Inline {
         let mut done = std::mem::take(&mut self.text);
         let end = done.text.len();
-        for (start, style, url) in self.open.iter_mut().rev() {
-            if *start < end {
-                done.styles.push((*start..end, *style));
-                if let Some(url) = url {
-                    done.links.push((*start..end, url.clone()));
-                }
-            }
-            *start = 0;
+        for opened in self.open.iter_mut().rev() {
+            close(&mut done, opened, end);
+            opened.start = 0;
         }
         done
     }
 
     /// The innermost open link's address.
     fn link(&self) -> Option<String> {
-        self.open.iter().rev().find_map(|(_, _, url)| url.clone())
+        self.open.iter().rev().find_map(|opened| opened.url.clone())
+    }
+
+    /// Opens or closes a style for an inline HTML tag.
+    fn html(&mut self, written: &str) {
+        let Some(tag) = html::tag(written) else {
+            return;
+        };
+        if tag.name == "br" {
+            return self.text.text.push('\n');
+        }
+        let Some(style) = html::inline_style(&tag.name) else {
+            return;
+        };
+        // Only tags inside the innermost container match.
+        let floor = self
+            .open
+            .iter()
+            .rposition(|opened| opened.tag.is_none())
+            .map_or(0, |at| at + 1);
+        if tag.closing {
+            let found = self.open[floor..]
+                .iter()
+                .rposition(|opened| opened.tag.as_ref() == Some(&tag.name));
+            if let Some(at) = found {
+                let (opened, end) = (self.open.remove(floor + at), self.text.text.len());
+                close(&mut self.text, &opened, end);
+            }
+            return;
+        }
+        let url = tag.attribute("href").map(str::to_string);
+        if style == Style::Link && url.is_none() {
+            return;
+        }
+        self.open.push(Opened {
+            start: self.text.text.len(),
+            style,
+            url,
+            tag: Some(tag.name),
+        });
+    }
+}
+
+/// Records `opened` as styled up to `end`.
+fn close(text: &mut Inline, opened: &Opened, end: usize) {
+    if opened.start < end {
+        text.styles.push((opened.start..end, opened.style));
+        if let Some(url) = &opened.url {
+            text.links.push((opened.start..end, url.clone()));
+        }
     }
 }
 
@@ -111,14 +170,25 @@ impl<'a> Reader<'a> {
     pub(crate) fn new(source: &'a str, root: &mdast::Node) -> Self {
         let mut nodes = Vec::new();
         walk(root, &mut nodes);
-        let mut urls = HashMap::new();
+        let (mut urls, mut notes) = (HashMap::new(), HashMap::new());
         for node in nodes {
-            if let mdast::Node::Definition(definition) = node {
-                urls.entry(definition.identifier.clone())
-                    .or_insert_with(|| definition.url.clone());
+            match node {
+                mdast::Node::Definition(definition) => {
+                    urls.entry(definition.identifier.clone())
+                        .or_insert_with(|| definition.url.clone());
+                }
+                mdast::Node::FootnoteReference(note) => {
+                    let next = notes.len() + 1;
+                    notes.entry(note.identifier.clone()).or_insert(next);
+                }
+                _ => {}
             }
         }
-        Self { source, urls }
+        Self {
+            source,
+            urls,
+            notes,
+        }
     }
 
     fn blocks(&self, nodes: &[mdast::Node], out: &mut Vec<Node>) {
@@ -193,10 +263,13 @@ impl<'a> Reader<'a> {
             }
             mdast::Node::ThematicBreak(_) => out.push(Node::Rule),
             mdast::Node::FootnoteDefinition(note) => out.push(Node::Footnote {
-                label: note
-                    .label
-                    .clone()
-                    .unwrap_or_else(|| note.identifier.clone()),
+                label: match self.notes.get(&note.identifier) {
+                    Some(number) => number.to_string(),
+                    None => note
+                        .label
+                        .clone()
+                        .unwrap_or_else(|| note.identifier.clone()),
+                },
                 body: self.nested(&note.children),
             }),
             mdast::Node::Html(_) | mdast::Node::Definition(_) => {}
@@ -232,14 +305,14 @@ impl<'a> Reader<'a> {
         let (children, style, url) = match node {
             mdast::Node::Text(words) => {
                 let soft = words.value.replace("\r\n", " ").replace(['\r', '\n'], " ");
-                return text.text.push_str(&soft);
+                return text.text.push_str(&html::emoji(&soft));
             }
             mdast::Node::Break(_) => return text.text.push('\n'),
             mdast::Node::InlineCode(code) => return styled(text, &code.value, Style::Code),
             mdast::Node::InlineMath(math) => return styled(text, &math.value, Style::Math),
             mdast::Node::FootnoteReference(note) => {
-                let label = note.label.as_ref().unwrap_or(&note.identifier);
-                return styled(text, &format!("[{label}]"), Style::Footnote);
+                let number = self.notes[&note.identifier];
+                return styled(text, &html::superscript(number), Style::Footnote);
             }
             mdast::Node::Image(_) | mdast::Node::ImageReference(_) => {
                 let (alt, url) = match node {
@@ -259,7 +332,7 @@ impl<'a> Reader<'a> {
                     Images::After(out) => out.push(Node::Image { alt, url, link }),
                 };
             }
-            mdast::Node::Html(_) => return,
+            mdast::Node::Html(written) => return run.html(&written.value),
             mdast::Node::Strong(strong) => (&strong.children, Style::Strong, None),
             mdast::Node::Emphasis(emphasis) => (&emphasis.children, Style::Emphasis, None),
             mdast::Node::Delete(delete) => (&delete.children, Style::Strike, None),
@@ -271,17 +344,24 @@ impl<'a> Reader<'a> {
             ),
             other => unreachable!("{other:?} is no inline outside MDX"),
         };
-        run.open.push((run.text.text.len(), style, url));
+        run.open.push(Opened {
+            start: run.text.text.len(),
+            style,
+            url,
+            tag: None,
+        });
         for child in children {
             self.inline_into(child, run, images);
         }
-        let (start, style, url) = run.open.pop().expect("the container opened above");
+        // Tags left open end with their container.
+        let at = run
+            .open
+            .iter()
+            .rposition(|opened| opened.tag.is_none())
+            .expect("the container opened above");
         let end = run.text.text.len();
-        if start < end {
-            run.text.styles.push((start..end, style));
-            if let Some(url) = url {
-                run.text.links.push((start..end, url));
-            }
+        for opened in run.open.drain(at..) {
+            close(&mut run.text, &opened, end);
         }
     }
 }

src/documents/tree/tests.rs

@@ -189,7 +189,75 @@ fn footnotes_number_their_references_and_hold_their_body() {
     let Node::Paragraph(line) = &nodes[0] else {
         panic!("a paragraph, not {:?}", nodes[0])
     };
-    assert_eq!(line.text, "Lift[a].");
-    assert_eq!(line.styles, [(4..7, Style::Footnote)]);
-    assert!(matches!(&nodes[1], Node::Footnote { label, body } if label == "a" && body.len() == 1));
+    assert_eq!(line.text, "Lift¹.");
+    assert_eq!(line.styles, [(4..6, Style::Footnote)]);
+    assert!(matches!(&nodes[1], Node::Footnote { label, body } if label == "1" && body.len() == 1));
+    let alone = parse(
+        "[^z]: Unread.
+",
+    );
+    assert!(matches!(&alone[0], Node::Footnote { label, .. } if label == "z"));
+}
+
+fn line(source: &str) -> Inline {
+    match parse(source).into_iter().next() {
+        Some(Node::Paragraph(text)) => text,
+        other => panic!("a paragraph, not {other:?}"),
+    }
+}
+
+#[test]
+fn inline_html_shows_as_typora_shows_it() {
+    let text = line(
+        "a <u>un</u> <b>bo</b> <kbd>K</kbd> <mark>hi</mark> <a href=\"https://ely.dev\">x</a><br>end <span>sp</span> <a name=\"n\">no</a>",
+    );
+    assert_eq!(text.text, "a un bo K hi x\nend sp no");
+    assert_eq!(
+        text.styles,
+        [
+            (2..4, Style::Underline),
+            (5..7, Style::Strong),
+            (8..9, Style::Code),
+            (10..12, Style::Mark),
+            (13..14, Style::Link)
+        ]
+    );
+    assert_eq!(text.links, [(13..14, "https://ely.dev".to_string())]);
+}
+
+#[test]
+fn a_tag_left_open_ends_with_its_container() {
+    let text = line("**a <u>b** c</u> d");
+    assert_eq!(text.text, "a b c d");
+    assert!(text.styles.contains(&(2..3, Style::Underline)), "{text:?}");
+    assert!(text.styles.contains(&(0..3, Style::Strong)), "{text:?}");
+    assert_eq!(text.styles.len(), 2, "the stray close is ignored: {text:?}");
+}
+
+#[test]
+fn shortcodes_draw_as_emoji_outside_code() {
+    let text = line("go :rocket: `:rocket:`");
+    assert_eq!(text.text, "go 🚀 :rocket:");
+}
+
+#[test]
+fn footnotes_number_by_their_first_reference() {
+    let nodes = parse("x[^b] y[^a] z[^b]\n\n[^a]: A\n\n[^b]: B\n");
+    let Node::Paragraph(text) = &nodes[0] else {
+        panic!("a paragraph, not {:?}", nodes[0])
+    };
+    assert_eq!(text.text, "x¹ y² z¹");
+    let labels: Vec<&str> = nodes[1..]
+        .iter()
+        .map(|node| match node {
+            Node::Footnote { label, .. } => label.as_str(),
+            other => panic!("a footnote, not {other:?}"),
+        })
+        .collect();
+    assert_eq!(labels, ["2", "1"]);
+}
+
+#[test]
+fn one_tilde_still_strikes() {
+    assert_eq!(line("H~2~O").styles, [(1..2, Style::Strike)]);
 }