Stop a borrowed word from making a short claim comparable
A claim too short to compare in its own script was rescued by a Latin filler: 恒温器 the offered two Chinese pairs, the rule asked for three terms, and "the" completed the set — so the claim was corroborated by anything that happened to carry the same filler. English was isomorphic: Pineapple answer stayed uncited, Pineapple answer the did not. The shortness test is now per script, and function words are out of the demanded set. A mixed claim still asks for its three Chinese positions plus its longest real Latin word, so dropping "the" from the demand no longer blocks a legitimate witness: 档案室的恒温器保持十八度 the is now corroborated by 恒温器一直放在档案室,读数十八度, which the last review round measured as False. A witness that translates the demanded Latin word is still read as disagreement, and a witness that inflects a demanded verb (holds / hold) is too: the rule is term-level, and the checklist now says so with both measurements instead of the earlier blanket claim. Gates: 324 passed, 59 in the four acceptance files, ruff clean, mypy cold cache 24 errors in 9 files (the recorded baseline, net zero), pyright 0 on both touched files, filesize clean, product audit clean, MCP write smoke passes.
This commit is contained in:
1 parent
8d84e03077
commit
2ea48d8530
3 files changed
+49
-15
No files matched your search
@@ -45,6 +45,11 @@ CORROBORATION_TERMS = 3
|
||||
CORROBORATION_LIMIT = 3
|
||||
# A claim written in both scripts is corroborated in both: the CJK positions plus its most distinctive Latin word.
|
||||
LATIN_WITH_CJK = 1
|
||||
# A function word cannot be one of the places a witness has to match. The list is fixed and knows nothing
|
||||
# about the corpus, so it cannot decide belief by size the way a rare-term cutoff did.
|
||||
STOPWORDS: frozenset[str] = frozenset(
|
||||
["a", "an", "the", "and", "or", "but", "if", "then", "than", "of", "to", "in", "on", "for", "with", "without", "within", "at", "by", "from", "into", "over", "under", "above", "as", "that", "this", "these", "those", "it", "its", "he", "she", "they", "them", "we", "you", "i", "not", "no", "yes", "very", "just", "also", "too", "am", "is", "are", "was", "were", "be", "been", "being", "do", "does", "did", "have", "has", "had", "having", "will", "would", "shall", "should", "can", "could", "may", "might", "must", "there", "here", "where", "when", "who", "whom", "whose", "which", "what", "how", "why"]
|
||||
)
|
||||
|
||||
|
||||
def _fold(text: str) -> str:
|
||||
@@ -80,12 +85,13 @@ def corroborating_atom_ids(
|
||||
that the claim was seen twice, not proof that it is true. A closed or quarantined atom cannot
|
||||
corroborate, so a superseded fact stops propping up whatever cited it.
|
||||
|
||||
The words demanded are the claim's own spine, with the instruction phrases stripped out. Neither
|
||||
the attack's own words nor the size of the library decides whether a second source is believed:
|
||||
The words demanded are the claim's own spine, with the instruction phrases and function words
|
||||
stripped out. Neither the attack's own words nor the size of the library decides whether a second
|
||||
source is believed:
|
||||
an earlier rule that dropped words common to the corpus left a two-atom library unable to
|
||||
corroborate anything, and 40 unrelated memories able to. A claim with fewer usable words than the
|
||||
rule demands is never corroborated, so a two-word rumour stays uncited — failing closed was
|
||||
chosen over trusting a sentence too short to compare.
|
||||
corroborate anything, and 40 unrelated memories able to. A claim too short to compare in its own
|
||||
script is never corroborated, and a word borrowed from the other script cannot rescue it — failing
|
||||
closed was chosen over trusting a sentence too short to compare.
|
||||
"""
|
||||
terms = claim_terms(atom.text)
|
||||
if len(terms) < CORROBORATION_TERMS:
|
||||
@@ -127,9 +133,15 @@ def claim_terms(text: str) -> list[str]:
|
||||
longest Latin word, because sorting the two channels together by length let one planted English
|
||||
word drop the Chinese assertion out of the comparison.
|
||||
|
||||
Each script is measured on its own terms. A claim with fewer than three usable pairs is not made
|
||||
comparable by a borrowed Latin word, and a demanded word is only counted if it carries content —
|
||||
"恒温器 the" repeated verbatim, or "Pineapple answer the", is too short to corroborate anything.
|
||||
|
||||
What that does not buy: the middle position of a long claim can land on filler, so a witness that
|
||||
keeps the three positions and changes a date or a number elsewhere is still read as agreement, and
|
||||
a witness that contains a demanded pair inside a longer word (很好 inside 很好笑) counts too.
|
||||
a witness that contains a demanded pair inside a longer word (很好 inside 很好笑) counts too. A mixed
|
||||
claim also demands its longest Latin word, so a witness that translates that word instead of
|
||||
keeping it is read as disagreement.
|
||||
"""
|
||||
try:
|
||||
terms = fts_terms(text)
|
||||
@@ -141,11 +153,15 @@ def claim_terms(text: str) -> list[str]:
|
||||
for term in terms:
|
||||
if is_cjk(term):
|
||||
pairs.extend(pair for pair in cjk_pairs(term) if pair not in blocked)
|
||||
elif term not in blocked:
|
||||
elif term not in blocked and term not in STOPWORDS:
|
||||
words.append(term)
|
||||
spread = _spread(list(dict.fromkeys(pairs)))
|
||||
unique = list(dict.fromkeys(pairs))
|
||||
longest = sorted(dict.fromkeys(words), key=lambda word: -len(word))
|
||||
return spread + longest[:LATIN_WITH_CJK if spread else CORROBORATION_TERMS]
|
||||
if unique:
|
||||
if len(unique) < CORROBORATION_TERMS:
|
||||
return []
|
||||
return _spread(unique) + longest[:LATIN_WITH_CJK]
|
||||
return longest[:CORROBORATION_TERMS] if len(longest) >= CORROBORATION_TERMS else []
|
||||
|
||||
|
||||
def _signal_words(text: str) -> set[str]:
|
||||
|
||||
Reference in new issue
Block a user