documents: shortcodes read from the source, so escapes, references and autolinks keep them; whole tag names, decoded attributes, only anchors link

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -13,7 +13,7 @@ Ely GPUI Component. A component library for GPUI, in light and dark.
 - Web views: `wry` 0.57 (Apache-2.0 OR MIT, default features off, macOS only) lays a child WKWebView over the window.
 - Terminal: `alacritty_terminal` 0.26 (Apache-2.0, default features off, native only) for the grid, its parser and the pseudo-terminal; `futures` carries its events. `vte` 0.15 (Apache-2.0 OR MIT), the parser it re-exports, reads `AnsiText`'s codes alone, so colored output builds for wasm32.
 - Diffs: `similar` 3.2 (Apache-2.0) for line and word diffs and three-way merges; its `unicode` feature splits words at punctuation.
-- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` turns to its emoji (the `emojis` crate's GitHub names), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` draw as what they mean and other tags vanish, leaving their text, a tag left open ends with the Markdown container it opened in, and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written.
+- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` written plainly turns to its emoji (the `emojis` crate's GitHub names; escaped, written through a character reference or inside an autolink it stays text), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` (its references decoded) draw as what they mean and other tags, hyphenated names too, vanish, leaving their text, a tag left open ends with the Markdown container it opened in, and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written.
 - Pictures: `image` 0.25 (MIT OR Apache-2.0), the crate gpui decodes with, default features off; it turns decoded frames and rims map tiles.
 - Math and diagrams: RaTeX 0.1.14 (`ratex-parser`, `ratex-layout`, `ratex-svg` with `embed-fonts`, `ratex-types`; MIT, KaTeX's fonts OFL) typesets `Latex` to SVG with outlined glyphs; `mermaid-rs-renderer` 0.3.1 (MIT, default features off) draws `Diagram`. Both go through gpui's `SvgRenderer` at the window's scale, loaded as gpui assets off the main thread; `usvg` 0.46, gpui's own, reads a diagram's size. A RaTeX or Mermaid panic on parsed input becomes the element's error. Each element holds its picture through `primitives::held`: a new source, or the element leaving the window, removes the old asset from gpui's cache and drops its image from the atlas after the frame.
 - Maps: Natural Earth's 110m countries (public domain). `assets/maps/countries.geojson`, trimmed by `scripts/countries.py` to an ISO code, a name and positions to 0.01°, backs `maps::WorldMap`; the gallery's tiles are cut from the same file, z0 to z3, by `scripts/tiles.py` with Pillow.

src/documents/tree/html.rs

@@ -1,3 +1,5 @@
+use markdown::{decode_named, decode_numeric};
+
 use super::Style;
 
 /// One HTML tag as written: its lowercase name and attributes.
@@ -17,7 +19,7 @@ impl Tag {
     }
 }
 
-/// How Typora shows an inline tag; others show their text.
+/// How Typora shows an inline tag; others show text.
 pub(crate) fn inline_style(name: &str) -> Option<Style> {
     match name {
         "u" => Some(Style::Underline),
@@ -31,23 +33,23 @@ pub(crate) fn inline_style(name: &str) -> Option<Style> {
     }
 }
 
-/// `html` read as one tag; comments and the rest are none.
+/// `html` as one tag; comments and the rest are none.
 pub(crate) fn tag(html: &str) -> Option<Tag> {
     let inner = html.trim().strip_prefix('<')?.strip_suffix('>')?;
     let (closing, inner) = match inner.strip_prefix('/') {
         Some(rest) => (true, rest),
         None => (false, inner),
     };
-    let named = inner
-        .find(|ch: char| !ch.is_ascii_alphanumeric())
-        .unwrap_or(inner.len());
-    if named == 0 || !inner.starts_with(|ch: char| ch.is_ascii_alphabetic()) {
+    if !inner.starts_with(|ch: char| ch.is_ascii_alphabetic()) {
         return None;
     }
+    let named = inner
+        .find(|ch: char| !(ch.is_ascii_alphanumeric() || ch == '-'))
+        .unwrap_or(inner.len());
     Some(Tag {
         name: inner[..named].to_ascii_lowercase(),
         closing,
-        attributes: attributes(inner[named..].trim_end_matches('/')),
+        attributes: attributes(&inner[named..]),
     })
 }
 
@@ -56,8 +58,13 @@ fn attributes(mut rest: &str) -> Vec<(String, String)> {
     let mut out = Vec::new();
     loop {
         rest = rest.trim_start();
+        // A slash between attributes closes the tag itself.
+        if let Some(after) = rest.strip_prefix('/') {
+            rest = after;
+            continue;
+        }
         let named = rest
-            .find(|ch: char| ch.is_whitespace() || ch == '=')
+            .find(|ch: char| ch.is_whitespace() || ch == '=' || ch == '/')
             .unwrap_or(rest.len());
         if named == 0 {
             return out;
@@ -79,30 +86,26 @@ fn attributes(mut rest: &str) -> Vec<(String, String)> {
                 (&after[..end], &after[end..])
             }
         };
-        out.push((name, value.to_string()));
+        out.push((name, decoded(value)));
         rest = left;
     }
 }
 
-/// `:shortcode:` written as its emoji, as Typora shows it.
-pub(crate) fn emoji(text: &str) -> String {
+/// `text` with its character references decoded.
+fn decoded(text: &str) -> String {
     let mut out = String::with_capacity(text.len());
     let mut rest = text;
-    while let Some(open) = rest.find(':') {
-        out.push_str(&rest[..open]);
-        let after = &rest[open + 1..];
-        let code = after
-            .find(':')
-            .map(|close| &after[..close])
-            .filter(|code| !code.is_empty() && code.chars().all(shortcode_char));
-        match code.and_then(emojis::get_by_shortcode) {
-            Some(found) => {
-                out.push_str(found.as_str());
-                rest = &after[code.expect("a matched code").len() + 1..];
+    while let Some(at) = rest.find('&') {
+        out.push_str(&rest[..at]);
+        rest = &rest[at..];
+        match reference(rest) {
+            Some((value, len)) => {
+                out.push_str(&value);
+                rest = &rest[len..];
             }
             None => {
-                out.push(':');
-                rest = after;
+                out.push('&');
+                rest = &rest[1..];
             }
         }
     }
@@ -110,6 +113,98 @@ pub(crate) fn emoji(text: &str) -> String {
     out
 }
 
+/// The reference `text` starts with, as CommonMark reads one.
+fn reference(text: &str) -> Option<(String, usize)> {
+    let body = text.strip_prefix('&')?;
+    let end = body.find(';')?;
+    let name = &body[..end];
+    let value = match name.strip_prefix('#') {
+        Some(number) => {
+            let (digits, radix, most) = match number.strip_prefix(['x', 'X']) {
+                Some(hex) => (hex, 16, 6),
+                None => (number, 10, 7),
+            };
+            let valid =
+                (1..=most).contains(&digits.len()) && digits.chars().all(|ch| ch.is_digit(radix));
+            valid.then(|| decode_numeric(digits, radix))?
+        }
+        None if !name.is_empty() && name.chars().all(|ch| ch.is_ascii_alphanumeric()) => {
+            decode_named(name, true)?
+        }
+        None => return None,
+    };
+    Some((value, end + 2))
+}
+
+/// `value` with each `:shortcode:` written plainly in `source` as emoji.
+pub(crate) fn emoji(source: &str, value: &str) -> String {
+    let mut codes = literal_codes(source).into_iter().peekable();
+    let mut out = String::with_capacity(value.len());
+    let (mut colons, mut rest) = (0, value);
+    while let Some(at) = rest.find(':') {
+        out.push_str(&rest[..at]);
+        rest = &rest[at..];
+        match codes.peek() {
+            Some((ordinal, code, glyph))
+                if *ordinal == colons
+                    && rest[1..].starts_with(code)
+                    && rest[1 + code.len()..].starts_with(':') =>
+            {
+                out.push_str(glyph);
+                rest = &rest[code.len() + 2..];
+                colons += 2;
+                codes.next();
+            }
+            _ => {
+                out.push(':');
+                rest = &rest[1..];
+                colons += 1;
+                while codes.next_if(|(ordinal, _, _)| *ordinal < colons).is_some() {}
+            }
+        }
+    }
+    out.push_str(rest);
+    out
+}
+
+/// Plain shortcodes in `source`, with colons counted before each.
+fn literal_codes(source: &str) -> Vec<(usize, &str, &'static str)> {
+    let (mut out, mut colons, mut at) = (Vec::new(), 0, 0);
+    while let Some(ch) = source[at..].chars().next() {
+        let (rest, after) = (&source[at..], &source[at + ch.len_utf8()..]);
+        if ch == '\\'
+            && let Some(next) = after.chars().next().filter(char::is_ascii_punctuation)
+        {
+            colons += usize::from(next == ':');
+            at += 1 + next.len_utf8();
+            continue;
+        }
+        if ch == '&'
+            && let Some((text, len)) = reference(rest)
+        {
+            colons += text.matches(':').count();
+            at += len;
+            continue;
+        }
+        if ch == ':' {
+            let found = after
+                .find(':')
+                .map(|end| &after[..end])
+                .filter(|code| !code.is_empty() && code.chars().all(shortcode_char))
+                .and_then(|code| emojis::get_by_shortcode(code).map(|found| (code, found)));
+            if let Some((code, found)) = found {
+                out.push((colons, code, found.as_str()));
+                colons += 2;
+                at += code.len() + 2;
+                continue;
+            }
+            colons += 1;
+        }
+        at += ch.len_utf8();
+    }
+    out
+}
+
 fn shortcode_char(ch: char) -> bool {
     ch.is_ascii_alphanumeric() || matches!(ch, '_' | '+' | '-')
 }
@@ -139,15 +234,39 @@ mod tests {
             Some(("u".into(), true))
         );
         assert_eq!(tag("<br/>").map(|t| t.name), Some("br".into()));
+        assert_eq!(tag("<u-box>").map(|t| t.name), Some("u-box".into()));
         assert_eq!(tag("<!-- note -->"), None);
         assert_eq!(tag("<1>"), None);
     }
 
     #[test]
-    fn shortcodes_become_emoji_and_others_stay() {
-        assert_eq!(emoji("hi :smile: :+1:"), "hi 😄 👍");
-        assert_eq!(emoji(":nope: at 10:30 and ::"), ":nope: at 10:30 and ::");
-        assert_eq!(emoji("a:smile:b"), "a😄b");
+    fn attribute_values_keep_slashes_and_decode_references() {
+        let bare = tag("<a href=/path/>").expect("a tag");
+        assert_eq!(bare.attribute("href"), Some("/path/"));
+        let closed = tag("<img src=\"a.png\" />").expect("a tag");
+        assert_eq!(closed.attribute("src"), Some("a.png"));
+        let query = tag("<a href=\"https://x.test/?q=1&amp;p=2&#38;r=&#x33;&nope;\">");
+        let query = query.expect("a tag");
+        assert_eq!(
+            query.attribute("href"),
+            Some("https://x.test/?q=1&p=2&r=3&nope;")
+        );
+    }
+
+    #[test]
+    fn shortcodes_written_plainly_become_emoji() {
+        let same = |text: &str| emoji(text, text);
+        assert_eq!(same("hi :smile: :+1:"), "hi 😄 👍");
+        assert_eq!(same(":nope: at 10:30 and ::"), ":nope: at 10:30 and ::");
+        assert_eq!(same("a:smile:b"), "a😄b");
+        assert_eq!(same("é :smile: 中\\é"), "é 😄 中\\é");
+    }
+
+    #[test]
+    fn escaped_or_referenced_colons_stay() {
+        assert_eq!(emoji("\\:smile: :smile:", ":smile: :smile:"), ":smile: 😄");
+        assert_eq!(emoji("&#58;smile: :tada:", ":smile: :tada:"), ":smile: 🎉");
+        assert_eq!(emoji(":sm&#105;le:", ":smile:"), ":smile:");
     }
 
     #[test]

src/documents/tree/mod.rs

@@ -72,7 +72,7 @@ pub(crate) struct Reader<'a> {
     notes: HashMap<String, usize>,
 }
 
-/// A style open in a run; `tag` names an HTML one.
+/// An open style; `tag` names an HTML one.
 struct Opened {
     start: usize,
     style: Style,
@@ -85,6 +85,8 @@ struct Opened {
 struct Run {
     text: Inline,
     open: Vec<Opened>,
+    /// Inside an autolink: its text stays as written.
+    verbatim: bool,
 }
 
 impl Run {
@@ -131,10 +133,13 @@ impl Run {
             }
             return;
         }
-        let url = tag.attribute("href").map(str::to_string);
-        if style == Style::Link && url.is_none() {
-            return;
-        }
+        let url = match style {
+            Style::Link => match tag.attribute("href") {
+                Some(href) => Some(href.to_string()),
+                None => return,
+            },
+            _ => None,
+        };
         self.open.push(Opened {
             start: self.text.text.len(),
             style,
@@ -304,8 +309,12 @@ impl<'a> Reader<'a> {
         let text = &mut run.text;
         let (children, style, url) = match node {
             mdast::Node::Text(words) => {
-                let soft = words.value.replace("\r\n", " ").replace(['\r', '\n'], " ");
-                return text.text.push_str(&html::emoji(&soft));
+                let shown = match run.verbatim {
+                    true => words.value.clone(),
+                    false => html::emoji(&self.source[span(node)], &words.value),
+                };
+                let soft = shown.replace("\r\n", " ").replace(['\r', '\n'], " ");
+                return run.text.text.push_str(&soft);
             }
             mdast::Node::Break(_) => return text.text.push('\n'),
             mdast::Node::InlineCode(code) => return styled(text, &code.value, Style::Code),
@@ -350,9 +359,14 @@ impl<'a> Reader<'a> {
             url,
             tag: None,
         });
+        let autolink =
+            matches!(node, mdast::Node::Link(_)) && !self.source[span(node)].starts_with('[');
+        let outer = run.verbatim;
+        run.verbatim |= autolink;
         for child in children {
             self.inline_into(child, run, images);
         }
+        run.verbatim = outer;
         // Tags left open end with their container.
         let at = run
             .open

src/documents/tree/tests.rs

@@ -261,3 +261,29 @@ fn footnotes_number_by_their_first_reference() {
 fn one_tilde_still_strikes() {
     assert_eq!(line("H~2~O").styles, [(1..2, Style::Strike)]);
 }
+
+#[test]
+fn escapes_references_and_autolinks_keep_their_shortcodes() {
+    assert_eq!(line(r"Show \:smile: now.").text, "Show :smile: now.");
+    assert_eq!(line("Show :smile: now.").text, "Show :smile: now.");
+    assert_eq!(
+        line("<https://x.test/:smile:>").text,
+        "https://x.test/:smile:"
+    );
+    assert_eq!(line("[go :smile:](https://x.test)").text, "go 😄");
+}
+
+#[test]
+fn unknown_tags_show_text_and_only_anchors_link() {
+    let custom = line("a <u-box>word</u-box> <br-box>in</br-box> end");
+    assert_eq!(
+        (custom.text.as_str(), custom.styles.len()),
+        ("a word in end", 0)
+    );
+    let bold = line("See <b href=\"https://x.test/\">bold</b>.");
+    assert_eq!(bold.styles, [(4..8, Style::Strong)]);
+    assert!(bold.links.is_empty(), "{bold:?}");
+    let query = line("<a href=\"https://x.test/?q=1&amp;p=2\">r</a> <a href=/path/>s</a>");
+    let urls: Vec<&str> = query.links.iter().map(|(_, url)| url.as_str()).collect();
+    assert_eq!(urls, ["https://x.test/?q=1&p=2", "/path/"]);
+}