收紧 token 估算到 2 chars/token (C3)

原先 4 chars/token 对中文 BPE 严重低估(汉字实际约 1.5-2 tokens/char),
120k 阈值只有在真实上下文已经爆窗之后才触发。2 chars/token 对拉丁文保守
一些,换来中文 agent 不再漏触发压缩。pi-ai 0.67.68 没暴露真 tokenizer,
换 tiktoken/anthropic-tokenizer 的跟进方案见 docs/proposals/C3-real-tokenizer-followup.md。
This commit is contained in:
ZacharyZhang-NY committed 2026-04-20 10:59:12 -04:00
1 parent 4ff751b3af
commit c59f62ba84
1 file changed
+11 -2
+11 -2
View File
@@ -28,8 +28,17 @@ const COMPACTION_TRIGGER_TOKENS = 120_000;
* losing mid-stream state (tool-call pairs, user intent).
*/
const RECENT_MESSAGE_TAIL = 20;
/** Rough tokens-per-char estimator; good enough for gating. */
const CHARS_PER_TOKEN = 4;
/**
* Conservative tokens-per-char estimator. The former 4 chars/token value
* under-counts CJK traffic by roughly 2x — Chinese BPE tokenizers land
* closer to 1.5-2 tokens per character, so a 120k-token gate would
* silently trip only after the real context already blew past the
* window. 2 chars/token is a deliberate over-estimate for latin text in
* exchange for not under-triggering on Chinese agent runs. The proper
* fix is a real tokenizer — pi-ai 0.67.68 does not export one yet, so
* see docs/proposals/C3-real-tokenizer-followup.md for the upgrade path.
*/
const CHARS_PER_TOKEN = 2;
/** Per-assistant-message summary cap to keep the compacted turn readable. */
const MAX_SUMMARY_CHARS_PER_MESSAGE = 160;