docs(comments): rewrite comments across all crates to the guidelines

Sweep every first-party crate source (1956 .rs files) to the project comment
guidelines: delete redundant restatements, decorative banners, change
narration, and end-of-line comments; keep and tighten the crucial ones
(invariants, bug rationale, SAFETY blocks, ported-source attribution).

No functional code changed. Every edit is proven comment-only against the
prior tree by a comment-stripping lexer (string/char/raw-string aware) plus a
separate doctest-fence check. Where removing a comment made rustfmt or clippy
want to re-lay-out adjacent code, the minimal triggering comment is restored so
code tokens stay byte-identical.

Gates green: cargo fmt --all --check (0 diffs), cargo check and cargo clippy
--workspace --all-targets (0 warnings).

Adds scripts/check_codegen_comment_guidelines.py — the enforcement gate for
these guidelines (flags banners, end-of-line comments, change narration, and
commented-out code).
This commit is contained in:
ZacharyZhang-NY committed 2026-07-23 16:55:39 -04:00
1 parent ff0fb56c67
commit a02b555e66
1458 files changed
+10729 -21750

No files matched your search

+5 -12
View File
@@ -1,24 +1,17 @@
//! Token-count seam.
//!
//! Budgeting math in the shared engine needs a *trusted* token count, but the
//! two harnesses disagree on how to produce one:
//!
//! - Kigi chat has a real tokenizer (`TextTokenizer` / `ImageTokenizer`) and
//! counts whole turns via `KigiTurn::get_num_tokens`.
//! - kigi estimates with `bytes / 4`.
//!
//! Rather than bake either policy into the shared crate, callers supply an
//! [`ItemTokenCounter`]. This keeps the engine deterministic and testable
//! while letting each harness plug in its own counting strategy.
//! two harnesses disagree on how to produce one: Kigi chat has a real tokenizer
//! (`TextTokenizer` / `ImageTokenizer`) and counts whole turns via
//! `KigiTurn::get_num_tokens`, while kigi estimates with `bytes / 4`. Rather
//! than bake either policy into the shared crate, callers supply an
//! [`ItemTokenCounter`].
//!
//! There is intentionally **no** blanket `Arc` forwarding here: each harness
//! implements the counter directly for the item type its algorithms run on
//! (Kigi chat: `ItemTokenCounter<Arc<KigiTurn>>`), so exactly one mechanism
//! is in play.
/// Counts tokens for a single conversation item on behalf of the shared
/// budgeting logic.
pub trait ItemTokenCounter<T: ?Sized>: Send + Sync {
/// Trusted token count of `item`.
fn count_item_tokens(&self, item: &T) -> u32;
}