Give Chinese memories a way to be corroborated
An independent reviewer repeated the reviewer's own counterexample in Chinese and found the corroboration channel was dead for it: fts_terms counts one CJK run as one term, so a Chinese claim could never reach the three words the rule demands, however many independent sources repeated it. A run is now broken into its adjacent two-character pairs and three are taken from across the sentence, so a second source has to agree in three separate places rather than copy one stretch of characters. The pairs belonging to an injection phrase are stripped too, so an instruction cannot vouch for its own characters. Corroboration stays term-level in both languages: a paraphrase that keeps the meaning and changes the wording is not a second witness, and a claim with fewer usable words than the rule demands is never believed. The record was wrong in four places the same review measured: the token rename breakdown summed to 11 while claiming 13 across 6 files (it is 13 across 7), the acceptance cell counted tests that no longer exist, the gate cell quoted the previous round's totals, and a cited line had drifted onto a blank. The section also stops calling the injection work "sanitization": 19 literal phrases plus quarantine plus one prompt sentence is detection, and a reworded directive walks straight past it.
This commit is contained in:
1 parent
203b6d04af
commit
ff2471d362
4 files changed
+89
-19
No files matched your search
@@ -70,7 +70,7 @@ def fts_all_expression(terms: list[str]) -> str:
|
||||
|
||||
|
||||
def _term_phrase(term: str) -> str:
|
||||
return _cjk_phrase(term) if _is_cjk(term) else f"{_quote(term)}*"
|
||||
return _cjk_phrase(term) if is_cjk(term) else f"{_quote(term)}*"
|
||||
|
||||
|
||||
def fts_query(query: str) -> str:
|
||||
@@ -146,10 +146,16 @@ def _bare(text: str) -> str:
|
||||
return _SPACES.sub(" ", _SEPARATORS.sub(" ", spaced)).strip().lower()
|
||||
|
||||
|
||||
def _is_cjk(term: str) -> bool:
|
||||
def is_cjk(term: str) -> bool:
|
||||
"""True when every character of this term is one the index stores on its own."""
|
||||
return all(char in _CJK_CHARS for char in term)
|
||||
|
||||
|
||||
def cjk_pairs(run: str) -> list[str]:
|
||||
"""Every adjacent two-character run inside a CJK string, in order."""
|
||||
return [run[start : start + 2] for start in range(len(run) - 1)]
|
||||
|
||||
|
||||
def _phrase(run: str) -> str:
|
||||
return '"' + " ".join(run) + '"'
|
||||
|
||||
@@ -162,7 +168,7 @@ def _cjk_phrase(run: str) -> str:
|
||||
appear inside 我们的钥匙放在玄关抽屉里. Offering every adjacent pair as an alternative keeps
|
||||
adjacency — 钥 and 匙 still have to touch — without demanding equal length.
|
||||
"""
|
||||
phrases = list(dict.fromkeys([_phrase(run)] + [_phrase(run[start : start + 2]) for start in range(len(run) - 1)]))
|
||||
phrases = list(dict.fromkeys([_phrase(run)] + [_phrase(pair) for pair in cjk_pairs(run)]))
|
||||
return phrases[0] if len(phrases) == 1 else "(" + " OR ".join(phrases) + ")"
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user