Rank memory search in SQL instead of rebuilding BM25 in Python
The product path pulled every atom into Python and rebuilt a BM25Okapi
index per query: p50 20.16s over 165,973 atoms. It now runs one SQLite
statement with LIMIT/OFFSET.
- text_index.py: one normalizer for both sides of the index. CJK runs are
split per character (unicode61 cannot segment them), camelCase and
separators are split so "search atoms" finds SearchAtoms, latin tokens
get a prefix match. Rows written before this existed hold raw text in
atom_fts, so schema v3 rebuilds the index on upgrade.
- retrieval.py: filters (holder / scenario / source / modality / since /
until / min_trust), a base64 cursor, and reciprocal rank fusion in place
of the 0.55/0.35/0.10 blend whose graph term was always zero.
- An empty result is {status: "no_match"}, not a bare array.
- Go: search_atoms and compile_context rank through POST /api/search. The
second Go tokenizer is deleted; two implementations could drift, which
is what the MCP/HTTP parity requirement forbids.
- db.py passed 500 lines, so the row codec moved to row_codec.py.
- Tooling: installed the repo's own pyright. pyrightconfig.json had no
exclude and swept the downloaded corpus (1171 errors); with excludes the
count is 62, all pre-existing (evals 48, profiles 4, training 4,
ingestion 3, runtime 2, cartridge_assets 1). backend/pyproject.toml
carried a second [tool.pyright] that shadowed the root config; deleted.
- bench/scale_check.py measures the product search at corpus size. The
100k-atom p50 number is not written until it is measured on an idle
machine.
Verification: 112 pytest passed; go build/vet/test clean, gofmt clean;
ruff clean on every touched file; mypy clean on the 14 touched Python
files; repo pyright 62 errors, unchanged per-file ledger.
This commit is contained in:
1 parent
3d792417cf
commit
674c88e84e
29 files changed
+1097
-356
No files matched your search
@@ -24,8 +24,9 @@ from .ingestion import IngestionService
|
||||
from .memory import MemoryService
|
||||
from .profiles import runtime_profile_report
|
||||
from .research_modules import ResearchModuleService
|
||||
from .retrieval import RetrievalService
|
||||
from .runtime import QwenRuntime
|
||||
from .schemas import MemoryOrigin, TimePrecision, TrainingAdapterKind
|
||||
from .schemas import MemoryOrigin, SearchFilters, TimePrecision, TrainingAdapterKind
|
||||
from .training import TrainingService
|
||||
|
||||
app = typer.Typer(no_args_is_help=True)
|
||||
@@ -79,6 +80,36 @@ def _moment(value: str, option: str) -> datetime | None:
|
||||
raise ValueError(f"{option} {value!r} is not an ISO timestamp: {error}") from error
|
||||
|
||||
|
||||
@app.command()
|
||||
def search(
|
||||
query: str,
|
||||
top_k: int = typer.Option(0, "--top-k", help="hits to return; 0 uses the configured default"),
|
||||
holder: str = typer.Option("", "--holder", help="only this holder's memories may answer"),
|
||||
scenario: str = typer.Option("", "--scenario"),
|
||||
source_id: str = typer.Option("", "--source-id"),
|
||||
since: str = typer.Option("", "--since", help="event time at or after, ISO with offset"),
|
||||
until: str = typer.Option("", "--until"),
|
||||
min_trust: float = typer.Option(-1.0, "--min-trust", help="drop memories whose trust falls below this"),
|
||||
cursor: str = typer.Option("", "--cursor", help="next_cursor from the previous page"),
|
||||
) -> None:
|
||||
"""Rank the memories that may answer, and say so when none may."""
|
||||
settings = get_settings()
|
||||
result = RetrievalService(settings).search(
|
||||
query,
|
||||
top_k or None,
|
||||
filters=SearchFilters(
|
||||
holder=holder or None,
|
||||
scenario=scenario or None,
|
||||
source_id=source_id or None,
|
||||
since=_moment(since, "--since"),
|
||||
until=_moment(until, "--until"),
|
||||
min_trust=None if min_trust < 0 else min_trust,
|
||||
),
|
||||
cursor=cursor or None,
|
||||
)
|
||||
typer.echo(result.model_dump_json(indent=2))
|
||||
|
||||
|
||||
@app.command()
|
||||
def remember(
|
||||
text: str,
|
||||
@@ -289,7 +320,7 @@ def build_visual_training_data(
|
||||
prd_path: Path = Path("PRD.md"),
|
||||
output_path: Path | None = None,
|
||||
cartridge_id: str | None = None,
|
||||
image_path: list[Path] | None = typer.Option(None, "--image-path"),
|
||||
image_path: list[Path] | None = typer.Option(None, "--image-path"), # noqa: B008 # typer reads option defaults itself
|
||||
) -> None:
|
||||
path = TrainingService(get_settings()).build_visual_dataset(
|
||||
prd_path=prd_path,
|
||||
|
||||
Reference in new issue
Block a user