docs(comments): rewrite comments across all crates to the guidelines
Sweep every first-party crate source (1956 .rs files) to the project comment guidelines: delete redundant restatements, decorative banners, change narration, and end-of-line comments; keep and tighten the crucial ones (invariants, bug rationale, SAFETY blocks, ported-source attribution). No functional code changed. Every edit is proven comment-only against the prior tree by a comment-stripping lexer (string/char/raw-string aware) plus a separate doctest-fence check. Where removing a comment made rustfmt or clippy want to re-lay-out adjacent code, the minimal triggering comment is restored so code tokens stay byte-identical. Gates green: cargo fmt --all --check (0 diffs), cargo check and cargo clippy --workspace --all-targets (0 warnings). Adds scripts/check_codegen_comment_guidelines.py — the enforcement gate for these guidelines (flags banners, end-of-line comments, change narration, and commented-out code).
This commit is contained in:
1 parent
ff0fb56c67
commit
a02b555e66
1458 files changed
+10729
-21750
No files matched your search
@@ -1,24 +1,17 @@
|
||||
//! Token-count seam.
|
||||
//!
|
||||
//! Budgeting math in the shared engine needs a *trusted* token count, but the
|
||||
//! two harnesses disagree on how to produce one:
|
||||
//!
|
||||
//! - Kigi chat has a real tokenizer (`TextTokenizer` / `ImageTokenizer`) and
|
||||
//! counts whole turns via `KigiTurn::get_num_tokens`.
|
||||
//! - kigi estimates with `bytes / 4`.
|
||||
//!
|
||||
//! Rather than bake either policy into the shared crate, callers supply an
|
||||
//! [`ItemTokenCounter`]. This keeps the engine deterministic and testable
|
||||
//! while letting each harness plug in its own counting strategy.
|
||||
//! two harnesses disagree on how to produce one: Kigi chat has a real tokenizer
|
||||
//! (`TextTokenizer` / `ImageTokenizer`) and counts whole turns via
|
||||
//! `KigiTurn::get_num_tokens`, while kigi estimates with `bytes / 4`. Rather
|
||||
//! than bake either policy into the shared crate, callers supply an
|
||||
//! [`ItemTokenCounter`].
|
||||
//!
|
||||
//! There is intentionally **no** blanket `Arc` forwarding here: each harness
|
||||
//! implements the counter directly for the item type its algorithms run on
|
||||
//! (Kigi chat: `ItemTokenCounter<Arc<KigiTurn>>`), so exactly one mechanism
|
||||
//! is in play.
|
||||
|
||||
/// Counts tokens for a single conversation item on behalf of the shared
|
||||
/// budgeting logic.
|
||||
pub trait ItemTokenCounter<T: ?Sized>: Send + Sync {
|
||||
/// Trusted token count of `item`.
|
||||
fn count_item_tokens(&self, item: &T) -> u32;
|
||||
}
|
||||
Reference in new issue
Block a user