diff --git a/AGENTS.md b/AGENTS.md index 6690797..684cbb2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ Ely GPUI Component. A component library for GPUI, in light and dark. - Web views: `wry` 0.57 (Apache-2.0 OR MIT, default features off, macOS only) lays a child WKWebView over the window. - Terminal: `alacritty_terminal` 0.26 (Apache-2.0, default features off, native only) for the grid, its parser and the pseudo-terminal; `futures` carries its events. `vte` 0.15 (Apache-2.0 OR MIT), the parser it re-exports, reads `AnsiText`'s codes alone, so colored output builds for wasm32. - Diffs: `similar` 3.2 (Apache-2.0) for line and word diffs and three-way merges; its `unicode` feature splits words at punctuation. -- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` written plainly turns to its emoji (the `emojis` crate's GitHub names; escaped, written through a character reference or inside an autolink it stays text), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` (its references decoded) draw as what they mean and other tags, hyphenated names too, vanish, leaving their text, a tag left open ends with the Markdown container it opened in, and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written. +- Markdown: `markdown` 1.0.0 (markdown-rs, MIT) reads GitHub's flavor and math into a syntax tree with byte positions for `MarkdownRenderer`, `MarkdownEditor`, the highlights, block ranges and outline. `MarkdownRenderer` draws display math through `typography::Latex`, a ```` ```mermaid ```` block through `documents::Diagram`, and pictures from the owner's folder (`base`) or `data:` addresses (`base64` 0.22, MIT OR Apache-2.0); web pictures stay unloaded. Inline math shows as TeX: gpui text carries no inline pictures. Inline it reads as Typora's defaults: `:shortcode:` written plainly turns to its emoji (the `emojis` crate's names, GitHub's today, so `:cricket:` is the insect; escaped, written through a character reference or inside an autolink it stays text), inline HTML's `u`, `b`/`strong`, `i`/`em`, `s`/`del`/`strike`, `code`/`kbd`, `mark`, `br` and `a href` draw as what they mean and other tags, hyphenated names too, vanish, leaving their text; attribute values split on ASCII whitespace alone and decode as HTML decodes them; tags and Markdown emphasis cross as a browser repairs CommonMark's HTML, as GitHub shows it, so a close meets its open anywhere in the line and a tag opened inside emphasis runs past it, and `
` breaks as `
` does; and a footnote reference is its number in superscript, numbered by first reference, as its definition is; one `~` still strikes. `MarkdownDocument` holds a whole file in one `CodeEditor` (text and undo, colored by `CodeEditor::highlighter` with `markdown_highlights`): visual draws the file's top-level blocks in a gpui `list`, a pressed block opening as its source in a `TextInput` whose typing is the document's text at once and whose write-back is one undo step; source is the editor; preview draws without presses. A written-back block, a mode switch or a new text is read at once up to 128 KiB, longer ones on the background executor; typing in source or split reads 300 ms after the last key, off the main thread (markdown-rs grows with the square of a paragraph's lines: 40,000 lines take 0.9 s). Texts past `LARGEST_READ` (2 MiB) are never read: no blocks, no outline. A new read keeps the list's blocks alike at both ends. One caret serves every mode: the editor's selection, which a written-back block moves; `set_caret` and `set_top_line` set caret and view apart; a top line asked (numbered, so a late frame callback finishes only its own ask; new text places it again, clamped to the text as it then is) waits for the read, lands on its block, then a frame later at its share of the block's height; `reveal` is such an ask with the caret at its line; a switch between source and the list keeps the top line, one between visual and preview keeps the list's scroll as it is. A block opens only when the read is of the text as it is now. A pressed block keeps its identity across the write-back of the one before it. Blocks are Tab stops that Enter or Space opens; preview takes no presses, undo or redo. `CodeEditor` keeps at most 64 MiB of undo and redo text, dropping the oldest undo and then the farthest redo after every record, composition, undo and redo. `BlockData::from_markdown` and its export still read with `pulldown-cmark` 0.13 (MIT, default features off); it adds GitHub's alerts, a callout's form, and reads each field's inline Markdown from the source by offset, as written. - Pictures: `image` 0.25 (MIT OR Apache-2.0), the crate gpui decodes with, default features off; it turns decoded frames and rims map tiles. - Math and diagrams: RaTeX 0.1.14 (`ratex-parser`, `ratex-layout`, `ratex-svg` with `embed-fonts`, `ratex-types`; MIT, KaTeX's fonts OFL) typesets `Latex` to SVG with outlined glyphs; `mermaid-rs-renderer` 0.3.1 (MIT, default features off) draws `Diagram`. Both go through gpui's `SvgRenderer` at the window's scale, loaded as gpui assets off the main thread; `usvg` 0.46, gpui's own, reads a diagram's size. A RaTeX or Mermaid panic on parsed input becomes the element's error. Each element holds its picture through `primitives::held`: a new source, or the element leaving the window, removes the old asset from gpui's cache and drops its image from the atlas after the frame. - Maps: Natural Earth's 110m countries (public domain). `assets/maps/countries.geojson`, trimmed by `scripts/countries.py` to an ISO code, a name and positions to 0.01°, backs `maps::WorldMap`; the gallery's tiles are cut from the same file, z0 to z3, by `scripts/tiles.py` with Pillow. diff --git a/src/documents/tree/html.rs b/src/documents/tree/html.rs index 59cba62..0468ee8 100644 --- a/src/documents/tree/html.rs +++ b/src/documents/tree/html.rs @@ -57,32 +57,32 @@ pub(crate) fn tag(html: &str) -> Option { fn attributes(mut rest: &str) -> Vec<(String, String)> { let mut out = Vec::new(); loop { - rest = rest.trim_start(); + rest = rest.trim_start_matches(spacing); // A slash between attributes closes the tag itself. if let Some(after) = rest.strip_prefix('/') { rest = after; continue; } let named = rest - .find(|ch: char| ch.is_whitespace() || ch == '=' || ch == '/') + .find(|ch: char| spacing(ch) || ch == '=' || ch == '/') .unwrap_or(rest.len()); if named == 0 { return out; } let name = rest[..named].to_ascii_lowercase(); - rest = rest[named..].trim_start(); + rest = rest[named..].trim_start_matches(spacing); let Some(after) = rest.strip_prefix('=') else { out.push((name, String::new())); continue; }; - let after = after.trim_start(); + let after = after.trim_start_matches(spacing); let (value, left) = match after.chars().next() { Some(quote @ ('"' | '\'')) => match after[1..].find(quote) { Some(end) => (&after[1..1 + end], &after[2 + end..]), None => (&after[1..], ""), }, _ => { - let end = after.find(char::is_whitespace).unwrap_or(after.len()); + let end = after.find(spacing).unwrap_or(after.len()); (&after[..end], &after[end..]) } }; @@ -91,14 +91,19 @@ fn attributes(mut rest: &str) -> Vec<(String, String)> { } } -/// `text` with its character references decoded. +/// HTML's whitespace, which is ASCII alone. +fn spacing(ch: char) -> bool { + matches!(ch, ' ' | '\t' | '\n' | '\r' | '\x0C') +} + +/// An attribute value with references decoded as HTML does. fn decoded(text: &str) -> String { let mut out = String::with_capacity(text.len()); let mut rest = text; while let Some(at) = rest.find('&') { out.push_str(&rest[..at]); rest = &rest[at..]; - match reference(rest) { + match attribute_reference(rest) { Some((value, len)) => { out.push_str(&value); rest = &rest[len..]; @@ -113,10 +118,79 @@ fn decoded(text: &str) -> String { out } +/// Longest reference name HTML knows, in bytes. +const LONGEST: usize = 32; + +/// A reference at `text`'s start, as HTML reads values. +fn attribute_reference(text: &str) -> Option<(String, usize)> { + let body = text.strip_prefix('&')?; + if let Some(number) = body.strip_prefix('#') { + let (digits, radix) = match number.strip_prefix(['x', 'X']) { + Some(hex) => (hex, 16), + None => (number, 10), + }; + let len = digits + .find(|ch: char| !ch.is_digit(radix)) + .unwrap_or(digits.len()); + if len == 0 { + return None; + } + let code = digits[..len].chars().try_fold(0u32, |code, ch| { + code.checked_mul(radix)?.checked_add(ch.to_digit(radix)?) + }); + let ended = usize::from(digits[len..].starts_with(';')); + let used = 2 + (number.len() - digits.len()) + len + ended; + return Some((numeric(code).to_string(), used)); + } + let len = body + .find(|ch: char| !ch.is_ascii_alphanumeric()) + .unwrap_or(body.len()); + let name = &body[..len]; + if (1..=LONGEST).contains(&len) + && body[len..].starts_with(';') + && let Some(value) = decode_named(name, true) + { + return Some((value, len + 2)); + } + let bare = !body[len..].starts_with(['=', ';']); + (bare && legacy(name)).then(|| (decode_named(name, true).expect("a legacy name"), len + 1)) +} + +/// A name HTML decodes without its semicolon. +fn legacy(name: &str) -> bool { + if matches!(name, "AMP" | "LT" | "GT" | "QUOT" | "COPY" | "REG") { + return true; + } + decode_named(name, false).is_some_and(|value| { + let mut chars = value.chars(); + let one = chars.next().filter(|_| chars.next().is_none()); + one.is_some_and(|ch| matches!(ch, '&' | '<' | '>' | '"' | '\u{A0}'..='\u{FF}')) + }) +} + +/// The character a numeric reference names, as HTML maps it. +fn numeric(code: Option) -> char { + // Windows-1252 for the C1 block, as browsers do. + const C1: [u32; 32] = [ + 0x20AC, 0x81, 0x201A, 0x0192, 0x201E, 0x2026, 0x2020, 0x2021, 0x02C6, 0x2030, 0x0160, + 0x2039, 0x0152, 0x8D, 0x017D, 0x8F, 0x90, 0x2018, 0x2019, 0x201C, 0x201D, 0x2022, 0x2013, + 0x2014, 0x02DC, 0x2122, 0x0161, 0x203A, 0x0153, 0x9D, 0x017E, 0x0178, + ]; + match code { + Some(code @ 0x80..=0x9F) => char::from_u32(C1[(code - 0x80) as usize]), + Some(0) | None => None, + Some(code) => char::from_u32(code), + } + .unwrap_or(char::REPLACEMENT_CHARACTER) +} + /// The reference `text` starts with, as CommonMark reads one. fn reference(text: &str) -> Option<(String, usize)> { let body = text.strip_prefix('&')?; - let end = body.find(';')?; + let end = body + .bytes() + .take(LONGEST + 1) + .position(|byte| byte == b';')?; let name = &body[..end]; let value = match name.strip_prefix('#') { Some(number) => { @@ -187,10 +261,11 @@ fn literal_codes(source: &str) -> Vec<(usize, &str, &'static str)> { continue; } if ch == ':' { - let found = after - .find(':') - .map(|end| &after[..end]) - .filter(|code| !code.is_empty() && code.chars().all(shortcode_char)) + let len = after + .find(|ch: char| !shortcode_char(ch)) + .unwrap_or(after.len()); + let found = Some(&after[..len]) + .filter(|code| !code.is_empty() && after[len..].starts_with(':')) .and_then(|code| emojis::get_by_shortcode(code).map(|found| (code, found))); if let Some((code, found)) = found { out.push((colons, code, found.as_str())); @@ -253,6 +328,43 @@ mod tests { ); } + #[test] + fn values_decode_as_a_browser_decodes_them() { + let href = |html: &str| { + tag(html) + .expect("a tag") + .attribute("href") + .map(str::to_owned) + }; + let cases = [ + ("", "?q=1&page=2"), + ("", "/€Ÿ\u{81}"), + ( + "", + "\u{FFFD}\u{FFFD}\u{FFFD}", + ), + ( + "", + "& page&x&=∉&", + ), + ("", "/a\u{A0}b"), + ]; + for (html, want) in cases { + assert_eq!(href(html).as_deref(), Some(want), "{html}"); + } + } + + #[test] + fn long_runs_scan_in_linear_time() { + let (amps, colons) = ("&".repeat(1 << 18), ":zq".repeat(1 << 16)); + let started = web_time::Instant::now(); + assert_eq!(decoded(&s).len(), amps.len()); + assert_eq!(emoji(&s, &s).len(), amps.len()); + assert_eq!(emoji(&colons, &colons).len(), colons.len()); + let spent = started.elapsed(); + assert!(spent.as_secs() < 2, "256 KiB took {spent:?}"); + } + #[test] fn shortcodes_written_plainly_become_emoji() { let same = |text: &str| emoji(text, text); diff --git a/src/documents/tree/mod.rs b/src/documents/tree/mod.rs index 683f5e2..1d19592 100644 --- a/src/documents/tree/mod.rs +++ b/src/documents/tree/mod.rs @@ -117,18 +117,14 @@ impl Run { let Some(style) = html::inline_style(&tag.name) else { return; }; - // Only tags inside the innermost container match. - let floor = self - .open - .iter() - .rposition(|opened| opened.tag.is_none()) - .map_or(0, |at| at + 1); + // A close meets its open anywhere, as a browser repairs. if tag.closing { - let found = self.open[floor..] + let found = self + .open .iter() .rposition(|opened| opened.tag.as_ref() == Some(&tag.name)); if let Some(at) = found { - let (opened, end) = (self.open.remove(floor + at), self.text.text.len()); + let (opened, end) = (self.open.remove(at), self.text.text.len()); close(&mut self.text, &opened, end); } return; @@ -367,16 +363,14 @@ impl<'a> Reader<'a> { self.inline_into(child, run, images); } run.verbatim = outer; - // Tags left open end with their container. + // Tags opened inside stay open past it. let at = run .open .iter() .rposition(|opened| opened.tag.is_none()) .expect("the container opened above"); - let end = run.text.text.len(); - for opened in run.open.drain(at..) { - close(&mut run.text, &opened, end); - } + let (opened, end) = (run.open.remove(at), run.text.text.len()); + close(&mut run.text, &opened, end); } } diff --git a/src/documents/tree/tests.rs b/src/documents/tree/tests.rs index b7698ee..3595c4a 100644 --- a/src/documents/tree/tests.rs +++ b/src/documents/tree/tests.rs @@ -226,12 +226,28 @@ fn inline_html_shows_as_typora_shows_it() { } #[test] -fn a_tag_left_open_ends_with_its_container() { - let text = line("**a b** c d"); - assert_eq!(text.text, "a b c d"); - assert!(text.styles.contains(&(2..3, Style::Underline)), "{text:?}"); - assert!(text.styles.contains(&(0..3, Style::Strong)), "{text:?}"); - assert_eq!(text.styles.len(), 2, "the stray close is ignored: {text:?}"); +fn tags_and_emphasis_cross_as_a_browser_repairs_them() { + let opened_inside = line("**a b** c d"); + assert_eq!(opened_inside.text, "a b c d"); + let mut styles = opened_inside.styles.clone(); + styles.sort_by_key(|(range, _)| (range.start, range.end)); + assert_eq!(styles, [(0..3, Style::Strong), (2..5, Style::Underline)]); + let closed_inside = line("a *text outside* end"); + assert_eq!(closed_inside.text, "a text outside end"); + let mut styles = closed_inside.styles.clone(); + styles.sort_by_key(|(range, _)| (range.start, range.end)); + assert_eq!(styles, [(2..6, Style::Underline), (2..14, Style::Emphasis)]); + assert!(line("a b").styles.is_empty(), "a stray close"); +} + +#[test] +fn a_closing_br_breaks_as_html_reads_it() { + assert_eq!(line("before

after").text, "before\n\nafter"); +} + +#[test] +fn shortcodes_mean_what_github_means() { + assert_eq!(line(":cricket: :cricket_game:").text, "🦗 🏏"); } #[test]