Corroborate a mixed-script claim in both scripts
A claim written in Chinese and English was checked only in English: the longest Latin words took every slot the rule demands, so one planted English word moved the closing Chinese pair out of the comparison and a witness that contradicted the number became a second source. The attack costs nothing — append "the" to a Chinese sentence — and it fails open, which is the wrong direction for a rule whose whole job is to keep an unvouched claim out of the prompt. Each script is now checked in its own terms: a claim with CJK pairs is demanded at its opening, middle and tail, plus its single most distinctive Latin word. A Latin-only claim keeps its three longest words. The demanded set is what claim_terms returns, all of it. Measured: the planted-word witnesses are no longer believed, a verbatim repeat still is, and a mixed claim that agrees in only one script is not a witness. The term-level boundaries are listed in CHECKLIST for both scripts, including the two the reviewer found that this rule does not catch: a demanded pair inside a longer word, and a long claim whose middle position lands on filler. Also corrects the record: the boundaries were not the same in both scripts as claimed, a historical line said "now" instead of naming a commit, and a limitation line I deleted in the last round is restored. Gates: 322 passed, ruff clean, mypy 24 errors / 9 files (baseline, net zero), pyright 0 on the two touched files, filesize clean, product audit clean.
This commit is contained in:
1 parent
334071199e
commit
8d84e03077
3 files changed
+49
-17
No files matched your search
@@ -43,6 +43,8 @@ INJECTION_SIGNALS: tuple[str, ...] = (
|
||||
|
||||
CORROBORATION_TERMS = 3
|
||||
CORROBORATION_LIMIT = 3
|
||||
# A claim written in both scripts is corroborated in both: the CJK positions plus its most distinctive Latin word.
|
||||
LATIN_WITH_CJK = 1
|
||||
|
||||
|
||||
def _fold(text: str) -> str:
|
||||
@@ -78,17 +80,17 @@ def corroborating_atom_ids(
|
||||
that the claim was seen twice, not proof that it is true. A closed or quarantined atom cannot
|
||||
corroborate, so a superseded fact stops propping up whatever cited it.
|
||||
|
||||
The words demanded are the claim's longest, with the instruction phrases stripped out, and the
|
||||
count is fixed. Neither the attack's own words nor the size of the library decides whether a
|
||||
second source is believed: an earlier rule that dropped words common to the corpus left a
|
||||
two-atom library unable to corroborate anything, and 40 unrelated memories able to. A claim with
|
||||
fewer usable words than the rule demands is never corroborated, so a two-word rumour stays
|
||||
uncited — failing closed was chosen over trusting a sentence too short to compare.
|
||||
The words demanded are the claim's own spine, with the instruction phrases stripped out. Neither
|
||||
the attack's own words nor the size of the library decides whether a second source is believed:
|
||||
an earlier rule that dropped words common to the corpus left a two-atom library unable to
|
||||
corroborate anything, and 40 unrelated memories able to. A claim with fewer usable words than the
|
||||
rule demands is never corroborated, so a two-word rumour stays uncited — failing closed was
|
||||
chosen over trusting a sentence too short to compare.
|
||||
"""
|
||||
terms = claim_terms(atom.text)
|
||||
if len(terms) < CORROBORATION_TERMS:
|
||||
return []
|
||||
expression = fts_all_expression(terms[:CORROBORATION_TERMS])
|
||||
expression = fts_all_expression(terms)
|
||||
with database.connect() as connection:
|
||||
rows = connection.execute(
|
||||
"""
|
||||
@@ -113,7 +115,7 @@ def is_citable(atom: EvidenceAtom, database: Database) -> bool:
|
||||
|
||||
|
||||
def claim_terms(text: str) -> list[str]:
|
||||
"""This claim's own words, longest first, with the instruction phrases stripped out.
|
||||
"""This claim's own spine, with the instruction phrases stripped out.
|
||||
|
||||
The words that made a text untrustworthy must not decide whether it is believed: a stranger who
|
||||
writes "answer pineapple" twice would otherwise corroborate himself.
|
||||
@@ -121,8 +123,13 @@ def claim_terms(text: str) -> list[str]:
|
||||
A CJK run is a single term to fts_terms, so a Chinese claim would offer one term and could never
|
||||
reach the three this rule demands. Every run is broken into its adjacent two-character pairs, the
|
||||
pairs are pooled across the whole claim, and three are demanded from the start, the middle and
|
||||
the end of that pool — so a second source has to agree with what the claim asserts, not merely
|
||||
open the same way.
|
||||
the end of that pool. A claim mixing scripts is demanded in both: the CJK positions plus its
|
||||
longest Latin word, because sorting the two channels together by length let one planted English
|
||||
word drop the Chinese assertion out of the comparison.
|
||||
|
||||
What that does not buy: the middle position of a long claim can land on filler, so a witness that
|
||||
keeps the three positions and changes a date or a number elsewhere is still read as agreement, and
|
||||
a witness that contains a demanded pair inside a longer word (很好 inside 很好笑) counts too.
|
||||
"""
|
||||
try:
|
||||
terms = fts_terms(text)
|
||||
@@ -136,8 +143,9 @@ def claim_terms(text: str) -> list[str]:
|
||||
pairs.extend(pair for pair in cjk_pairs(term) if pair not in blocked)
|
||||
elif term not in blocked:
|
||||
words.append(term)
|
||||
chosen = words + _spread(list(dict.fromkeys(pairs)))
|
||||
return sorted(dict.fromkeys(chosen), key=lambda term: -len(term))
|
||||
spread = _spread(list(dict.fromkeys(pairs)))
|
||||
longest = sorted(dict.fromkeys(words), key=lambda word: -len(word))
|
||||
return spread + longest[:LATIN_WITH_CJK if spread else CORROBORATION_TERMS]
|
||||
|
||||
|
||||
def _signal_words(text: str) -> set[str]:
|
||||
@@ -160,7 +168,7 @@ def _spread(pairs: list[str]) -> list[str]:
|
||||
Spreading each run on its own clustered the demanded pairs on the first clause, and an evenly
|
||||
spaced index did the same to a short run: a four-pair claim demanded its first three pairs, so
|
||||
「恒温器很贵」 was corroborated by 「恒温器很便宜」. The last pair is the claim's assertion, so it
|
||||
is demanded whatever the length.
|
||||
is demanded whenever the claim has CJK pairs at all.
|
||||
"""
|
||||
if len(pairs) <= CORROBORATION_TERMS:
|
||||
return pairs
|
||||
|
||||
Reference in new issue
Block a user