Give wordless pictures a way to be found

A screenshot with no OCR-able words used to be a memory nothing could
reach. It now has a vector, and a question can be asked with it.

- picture space: Xenova/chinese-clip-vit-base-patch16 fp16 ONNX, 361MB,
  sha256 0fad978faa89b85f…, CPU-runnable on the onnxruntime the repo
  already ships. Text 76ms, image 29ms/image at batch 8. The ONNX contract
  was measured, not assumed: it is the fused contrastive graph, so the
  unused tower is fed a placeholder row, and the tokenizer must be
  BertTokenizer — a CLIP tokenizer emits ids past the embedding table.
- visual_index.py keeps picture vectors in their own family beside the
  text ones, keyed by atom id, and refuses to answer when the map and the
  array disagree. retrieval.py stays under 500 lines.
- a search can be asked with a picture, and when the words come up thin
  the picture ranking joins the fusion — but only then, so a text query
  never pays for the encoder.
- visual tokens are counted at 32x32 patches (migration v7): 21,325 image
  atoms went from 5,331 to 34,618 tokens, and the library from 29.9M to
  84.2M. The 100M capsule fits this 50,278-file corpus and nothing much
  larger — that is what the claim actually means.
- the vision-language describer is written and gated: with no snapshot an
  atom records "deferred: no vision-language model snapshot"; a broken
  model becomes a note on the atom instead of a dead ingest.
- forget now voids the picture index and the copy this process holds, so
  a forgotten picture stops answering from RAM.

Measured, six arms, ScreenSpot-v2 (554 icon crops, 408 whole screenshots),
bench/reports/VISUAL.md:
  icon pool      picture 0.208  words 0.175  chance 0.018
  screenshot pool words 0.384   picture 0.215
  screenshot to icon 0.108  — CLIP is a whole-image model, not a grounding
  model: it cannot point at one small icon inside a screen.
  within one screen MRR 0.908 (recall@10 is 1.000 there only because
  chance is 1.000 too, so MRR is the only honest number in that row).
ScreenSpot-v2 instructions are all English, so this run does not test the
bilingual claim and the report says so.

Gates: 199 tests, ruff clean on 17 files, mypy clean on 16 (fresh cache),
the repo's own pyright 0 errors on the 12 files touched.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 20:41:19 -04:00
1 parent 7e1fe649f6
commit 03ca35763f
20 files changed
+1702 -22

No files matched your search

+28 -1
View File
@@ -10,11 +10,13 @@ upgraded in place rather than rebuilt, because the only copy of the memory is th
from __future__ import annotations
import json
import sqlite3
from .storage import token_equivalent, visual_token_equivalent
from .text_index import index_text
SCHEMA_VERSION = 6
SCHEMA_VERSION = 7
ATOM_COLUMNS_V2 = """
atom_id TEXT PRIMARY KEY,
@@ -204,6 +206,29 @@ def _upgrade_v1_to_v2(connection: sqlite3.Connection) -> None:
connection.execute(f"ALTER TABLE atoms ADD COLUMN {name} {definition}")
def _recount_visual_tokens(connection: sqlite3.Connection) -> None:
"""Re-price raster atoms at 32x32 patches; a 1280x720 frame cost 18 tokens under the old rule.
An atom whose layout carries no size keeps its stored number: there is no region to price,
and inventing one would move a figure we cannot verify.
"""
rows = connection.execute(
"SELECT atom_id, text, layout_json FROM atoms WHERE modality IN ('image', 'video_frame', 'ui_screenshot')"
).fetchall()
updates: list[tuple[int, str]] = []
for row in rows:
try:
layout = json.loads(row["layout_json"] or "{}")
except json.JSONDecodeError as error:
raise SystemExit(f"atom {row['atom_id']}: its stored layout is unreadable ({error})") from error
width = float(layout.get("w") or 0)
height = float(layout.get("h") or 0)
if width <= 0 or height <= 0:
continue
updates.append((token_equivalent(row["text"]) + visual_token_equivalent(width, height), str(row["atom_id"])))
connection.executemany("UPDATE atoms SET token_equivalent = ? WHERE atom_id = ?", updates)
def apply(connection: sqlite3.Connection) -> None:
"""Bring the schema to SCHEMA_VERSION, in place, without losing rows."""
version = int(connection.execute("PRAGMA user_version").fetchone()[0])
@@ -220,6 +245,8 @@ def apply(connection: sqlite3.Connection) -> None:
_create_v2(connection)
if version < 3:
_rebuild_fts(connection)
if version < 7:
_recount_visual_tokens(connection)
connection.executescript(VIEWS)
connection.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")