Give Chinese memories a way to be corroborated

An independent reviewer repeated the reviewer's own counterexample in
Chinese and found the corroboration channel was dead for it: fts_terms
counts one CJK run as one term, so a Chinese claim could never reach the
three words the rule demands, however many independent sources repeated
it. A run is now broken into its adjacent two-character pairs and three
are taken from across the sentence, so a second source has to agree in
three separate places rather than copy one stretch of characters. The
pairs belonging to an injection phrase are stripped too, so an
instruction cannot vouch for its own characters.

Corroboration stays term-level in both languages: a paraphrase that
keeps the meaning and changes the wording is not a second witness, and a
claim with fewer usable words than the rule demands is never believed.

The record was wrong in four places the same review measured: the token
rename breakdown summed to 11 while claiming 13 across 6 files (it is 13
across 7), the acceptance cell counted tests that no longer exist, the
gate cell quoted the previous round's totals, and a cited line had
drifted onto a blank. The section also stops calling the injection work
"sanitization": 19 literal phrases plus quarantine plus one prompt
sentence is detection, and a reworded directive walks straight past it.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-09 01:26:17 -04:00
1 parent 203b6d04af
commit ff2471d362
4 files changed
+89 -19

No files matched your search

+9 -3
View File
@@ -70,7 +70,7 @@ def fts_all_expression(terms: list[str]) -> str:
def _term_phrase(term: str) -> str:
return _cjk_phrase(term) if _is_cjk(term) else f"{_quote(term)}*"
return _cjk_phrase(term) if is_cjk(term) else f"{_quote(term)}*"
def fts_query(query: str) -> str:
@@ -146,10 +146,16 @@ def _bare(text: str) -> str:
return _SPACES.sub(" ", _SEPARATORS.sub(" ", spaced)).strip().lower()
def _is_cjk(term: str) -> bool:
def is_cjk(term: str) -> bool:
"""True when every character of this term is one the index stores on its own."""
return all(char in _CJK_CHARS for char in term)
def cjk_pairs(run: str) -> list[str]:
"""Every adjacent two-character run inside a CJK string, in order."""
return [run[start : start + 2] for start in range(len(run) - 1)]
def _phrase(run: str) -> str:
return '"' + " ".join(run) + '"'
@@ -162,7 +168,7 @@ def _cjk_phrase(run: str) -> str:
appear inside 我们的钥匙放在玄关抽屉里. Offering every adjacent pair as an alternative keeps
adjacency — 钥 and 匙 still have to touch — without demanding equal length.
"""
phrases = list(dict.fromkeys([_phrase(run)] + [_phrase(run[start : start + 2]) for start in range(len(run) - 1)]))
phrases = list(dict.fromkeys([_phrase(run)] + [_phrase(pair) for pair in cjk_pairs(run)]))
return phrases[0] if len(phrases) == 1 else "(" + " OR ".join(phrases) + ")"