ZacharyZhang-NY
a02b555e66
Sweep every first-party crate source (1956 .rs files) to the project comment guidelines: delete redundant restatements, decorative banners, change narration, and end-of-line comments; keep and tighten the crucial ones (invariants, bug rationale, SAFETY blocks, ported-source attribution). No functional code changed. Every edit is proven comment-only against the prior tree by a comment-stripping lexer (string/char/raw-string aware) plus a separate doctest-fence check. Where removing a comment made rustfmt or clippy want to re-lay-out adjacent code, the minimal triggering comment is restored so code tokens stay byte-identical. Gates green: cargo fmt --all --check (0 diffs), cargo check and cargo clippy --workspace --all-targets (0 warnings). Adds scripts/check_codegen_comment_guidelines.py — the enforcement gate for these guidelines (flags banners, end-of-line comments, change narration, and commented-out code).
18 lines
784 B
Rust
18 lines
784 B
Rust
//! Token-count seam.
|
|
//!
|
|
//! Budgeting math in the shared engine needs a *trusted* token count, but the
|
|
//! two harnesses disagree on how to produce one: Kigi chat has a real tokenizer
|
|
//! (`TextTokenizer` / `ImageTokenizer`) and counts whole turns via
|
|
//! `KigiTurn::get_num_tokens`, while kigi estimates with `bytes / 4`. Rather
|
|
//! than bake either policy into the shared crate, callers supply an
|
|
//! [`ItemTokenCounter`].
|
|
//!
|
|
//! There is intentionally **no** blanket `Arc` forwarding here: each harness
|
|
//! implements the counter directly for the item type its algorithms run on
|
|
//! (Kigi chat: `ItemTokenCounter<Arc<KigiTurn>>`), so exactly one mechanism
|
|
//! is in play.
|
|
|
|
pub trait ItemTokenCounter<T: ?Sized>: Send + Sync {
|
|
fn count_item_tokens(&self, item: &T) -> u32;
|
|
}
|