收紧 token 估算到 2 chars/token (C3)
原先 4 chars/token 对中文 BPE 严重低估(汉字实际约 1.5-2 tokens/char), 120k 阈值只有在真实上下文已经爆窗之后才触发。2 chars/token 对拉丁文保守 一些,换来中文 agent 不再漏触发压缩。pi-ai 0.67.68 没暴露真 tokenizer, 换 tiktoken/anthropic-tokenizer 的跟进方案见 docs/proposals/C3-real-tokenizer-followup.md。
This commit is contained in:
1 parent
4ff751b3af
commit
c59f62ba84
1 file changed
+11
-2
@@ -28,8 +28,17 @@ const COMPACTION_TRIGGER_TOKENS = 120_000;
|
||||
* losing mid-stream state (tool-call pairs, user intent).
|
||||
*/
|
||||
const RECENT_MESSAGE_TAIL = 20;
|
||||
/** Rough tokens-per-char estimator; good enough for gating. */
|
||||
const CHARS_PER_TOKEN = 4;
|
||||
/**
|
||||
* Conservative tokens-per-char estimator. The former 4 chars/token value
|
||||
* under-counts CJK traffic by roughly 2x — Chinese BPE tokenizers land
|
||||
* closer to 1.5-2 tokens per character, so a 120k-token gate would
|
||||
* silently trip only after the real context already blew past the
|
||||
* window. 2 chars/token is a deliberate over-estimate for latin text in
|
||||
* exchange for not under-triggering on Chinese agent runs. The proper
|
||||
* fix is a real tokenizer — pi-ai 0.67.68 does not export one yet, so
|
||||
* see docs/proposals/C3-real-tokenizer-followup.md for the upgrade path.
|
||||
*/
|
||||
const CHARS_PER_TOKEN = 2;
|
||||
/** Per-assistant-message summary cap to keep the compacted turn readable. */
|
||||
const MAX_SUMMARY_CHARS_PER_MESSAGE = 160;
|
||||
|
||||
|
||||
Reference in new issue
Block a user