diff --git a/CHECKLIST.md b/CHECKLIST.md index f95d92c..23d0734 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -223,14 +223,24 @@ - [x] p1-3.1 按卡带/场景解析 6 个 adapter，不再只有 `runtime.py:317` - [x] p1-3.2 加载结果可观测：日志 + `GET /api/runtime/adapters` - 验收：状态端点列每个 adapter 的 loaded/deferred + 原因；桩权重测试证明加载分支执行 - - `backend/src/ely_eye/adapters.py` 是唯一的 adapter 解析处。`runtime.py` 原先在 7 处调 `resolve_runtime_adapter`，现在统一走 `_generation_adapter_fields()`；`peft` 只在 `adapters.load_into_model` 里懒加载，`RuntimeAdapter` 与 `resolve_runtime_adapter` 已删除。 + - `backend/src/ely_eye/adapters.py` 是唯一的 adapter 解析处。`runtime.py` 原先在 7 处调 `resolve_runtime_adapter`，现在统一走 `_generation_adapter_fields()`；生成路径上的 `peft` 只在 `adapters.load_into_model` 里懒加载（训练路径 `training.py:443`、`training.py:767`、`proofs/training_trace.py:33` 仍直接 import peft，那是训练不是生成），`RuntimeAdapter` 与 `resolve_runtime_adapter` 已删除。 - 「场景」= 挂载的卡带：`AdapterManifest` 和 `CartridgeManifest` 都没有 scenario 字段（读出来的事实），所以解析按 `cartridge_store.mounted_ids` 选，报告写明来自哪个卡带，没有新造字段。 - 修掉的真 bug：`CartridgeService.attach_adapter`（`cartridge.py:230`）把 adapter 拷进 `<卡带>/adapters/{kind}/`，但解析从来不看卡带目录，挂上去的东西运行时根本不加载。现在挂载卡带里的副本优先于库里的同名 adapter。 - 测试写出来才发现的第二个 bug：`ready.sort(key=_preference)` 丢了原来的 `reverse=True`，库里的 adapter 反而赢了挂载副本。 - - 状态只由事实支撑：loaded（有 LoadRecord）/ available（manifest 与 `adapter_model.safetensors` 都在）/ deferred（base_model 不匹配或缺权重文件，原因里写两个模型名）/ missing（磁盘上没有该 kind）。落选者单独一行 `shadowed=True`。 - - 验收命令：`pytest backend/tests/test_adapters.py -q` → 10 passed。`test_stub_weights_prove_the_load_branch_runs` 用桩权重断言 `attached == [(ANY, 路径)]`、`model.evalled is True`、LoadRecord 存在、端点该行 `state == "loaded"`；`test_unload_clears_the_loaded_state` 断言 unload 后回到 available。 + - 状态只由事实支撑：loaded（模型还活着——`record_load` 存 `weakref`，且记录挂在具体 `QwenRuntime._attached` 上，不是进程全局）/ available（manifest 与 `adapter_model.safetensors` 都在；模型没了就说「attached earlier in this process … that model is gone」）/ deferred（base_model 不匹配或缺权重文件，原因里写两个模型名）/ missing（磁盘上没有该 kind）。落选者单独一行 `shadowed=True`，且**只在真有赢家时**才有这行，原因点名赢家。 + - 排序键是三个：挂载卡带 > manifest 带 `cartridge_id` > 权重 mtime。第一个版本漏了中间那个，review 用 `/tmp/p13_pref.py` 量出 parent `9f0f799` 与 HEAD 选中的文件不同——`training.py:106` 写库内 adapter 时会带卡带标记，所以这不是风格问题。 + - 验收命令：`pytest backend/tests/test_adapters.py -q` → 13 passed。`test_stub_weights_prove_the_load_branch_runs` 用桩权重断言 `attached == [(ANY, 路径)]`、`model.evalled is True`、LoadRecord 存在、端点该行 `state == "loaded"`；`test_generation_reports_the_adapter_it_attached` 走完 transformers 生成路径，断言答案里的 adapter 路径 / kind / id / sha256 与 `output_tokens`；`test_unload_stops_claiming_a_model_that_is_gone` 断言 unload 后回到 available 并说明模型已消失，同时断言另一个实例不会继承这个声明。 + - 证明与生成解析一致：`proofs/checks_runtime.py` 的 `_generation_adapter()` 用 `discover` + `resolve`（含挂载卡带），不再用只看库的解析，否则同一份证明里 `resolved_adapter` 与 `adapter` 两块会指向不同副本还静默。 - 新文件 `backend/src/ely_eye/runtime_schemas.py`（30 行）：`schemas.py` 已 498 行，离 500 行红线 2 行，报告类型不放进去。 - - 门禁：`pytest -q` 222 passed；ruff 干净；mypy 冷缓存查 7 个改动文件 → 2 errors，与 HEAD 的 `runtime.py` 逐条相同（`qwen_vl_utils` 无 stubs、`model_class` 重赋值），净新增 0；pyright 同 7 文件 → 2 errors，HEAD 也是 2 条，都是 `process_vision_info` 的双返回（`vision_process.py:532-534` 由 `return_video_kwargs` 决定返回 2 或 3 元组，产品用默认值取 2 个，运行时正确）。 + - 门禁：`pytest -q` 225 passed；ruff 干净；mypy 冷缓存查 7 个改动文件 → 2 errors，与 HEAD 的 `runtime.py` 逐条相同（`qwen_vl_utils` 无 stubs、`model_class` 重赋值），净新增 0；pyright 同 7 文件 → 2 errors，HEAD 也是 2 条，都是 `process_vision_info` 的双返回（`vision_process.py:532-534` 由 `return_video_kwargs` 决定返回 2 或 3 元组，产品用默认值取 2 个，运行时正确）。 + - 独立 review（同一个 `p1review` 里的 pi，模型 `strata/qwen3.8-flash-next-abliterated-iq3_s:high`）第一轮 **FAIL 6 条**，全部核实并改掉： + 1. `plan()` 用 `chosen_paths` 判落选，把「被拒绝的候选」也当成落选者，于是没有任何 adapter 胜出时却说「another adapter of this kind wins」，还把真正的 deferred 原因盖掉了（reviewer 挂载 cart-home 复现，`hme_router`、`hme_vision_lora` 两行都在撒谎）。 + 2. `_LOADED` 是进程全局，而原因写「attached to the live model」——`api.py` 每个请求新建一个 `QwenRuntime`，`_model is None` 也报 loaded。改成 weakref + 实例字段。 + 3. CHECKLIST 那句「peft 只在 `adapters.load_into_model` 里」在全仓范围不真（training.py 两处、training_trace.py 一处），已限定为生成路径。 + 4. `LoadRecord.loaded_at` 写了没人读 → 删。 + 5. `_preference` 丢了 `manifest.cartridge_id is not None` 这个次级键，换掉了原来会加载哪个 adapter（见上）。 + 6. 证明检查与生成解析不一致（见上）。 + reviewer 还指出 4 条生成路径与 `status()` 零覆盖，所以「没丢函数」当时只是读出来的——现在有了走完生成路径的测试。 - CLI 面：`ely-eye adapters` 打印同一份报告。 - [ ] **p1-4** 工程地基 - [ ] p1-4.1 pytest 从零到有（后端） diff --git a/backend/src/ely_eye/adapters.py b/backend/src/ely_eye/adapters.py index e369b2a..047a438 100644 --- a/backend/src/ely_eye/adapters.py +++ b/backend/src/ely_eye/adapters.py @@ -3,8 +3,8 @@ from __future__ import annotations import logging +import weakref from dataclasses import dataclass -from datetime import datetime from pathlib import Path from typing import Any @@ -34,16 +34,16 @@ class AdapterSource: @dataclass(frozen=True) class LoadRecord: - """An adapter this process actually attached to a live model.""" + """An adapter this process attached to a model.""" kind: TrainingAdapterKind adapter_id: str path: str sha256: str - loaded_at: str _LOADED: dict[str, LoadRecord] = {} +_LIVE: dict[str, weakref.ReferenceType[object]] = {} def read_manifest(path: Path) -> AdapterManifest: @@ -143,36 +143,41 @@ def resolve( return chosen, "available", "selected from the adapter library" -def resolve_library_adapter( - settings: Settings, - kind: TrainingAdapterKind = TrainingAdapterKind.hme_core_lora, -) -> tuple[AdapterSource | None, AdapterState, str]: - """Resolution without a database: the library only, no mounted cartridges.""" - return resolve(library_sources(settings), settings, kind) - - -def _preference(source: AdapterSource) -> tuple[bool, float]: +def _preference(source: AdapterSource) -> tuple[bool, bool, float]: weights = source.path / WEIGHTS_FILE - return (source.origin == "cartridge", weights.stat().st_mtime if weights.exists() else 0.0) + return ( + source.origin == "cartridge", + source.manifest.cartridge_id is not None, + weights.stat().st_mtime if weights.exists() else 0.0, + ) def load_records() -> list[LoadRecord]: return list(_LOADED.values()) +def live_kinds() -> set[str]: + """Kinds whose attached model is still alive, not merely attached once.""" + alive = {kind for kind, reference in _LIVE.items() if reference() is not None} + for kind in [kind for kind, reference in _LIVE.items() if reference() is None]: + del _LIVE[kind] + return alive + + def clear_loads() -> None: _LOADED.clear() + _LIVE.clear() -def record_load(source: AdapterSource) -> LoadRecord: +def record_load(source: AdapterSource, model: Any) -> LoadRecord: record = LoadRecord( kind=source.manifest.kind, adapter_id=source.manifest.adapter_id, path=str(source.path), sha256=source.manifest.sha256, - loaded_at=datetime.now().astimezone().isoformat(), ) _LOADED[record.kind.value] = record + _LIVE[record.kind.value] = weakref.ref(model) logger.info("loaded adapter %s (%s) from %s", record.adapter_id, record.kind.value, record.path) return record @@ -191,12 +196,18 @@ def plan(settings: Settings, database: Database) -> AdapterLoadReport: """One row per adapter kind, plus a row for every adapter that lost the choice.""" sources, mounted = discover(settings, database) rows: list[AdapterChoice] = [] + winners: dict[str, AdapterSource | None] = {} + alive = live_kinds() for kind in TrainingAdapterKind: source, state, reason = resolve(sources, settings, kind) + winners[kind.value] = source record = next((item for item in load_records() if item.kind == kind), None) if record is not None: - reason = f"attached to the live model from {record.path}" - state = "loaded" + if kind.value in alive: + state, reason = "loaded", f"attached to the live model from {record.path}" + else: + state = "available" + reason = f"attached earlier in this process from {record.path}; that model is gone" rows.append( AdapterChoice( kind=kind, @@ -209,15 +220,15 @@ def plan(settings: Settings, database: Database) -> AdapterLoadReport: cartridge_id=source.cartridge_id if source else None, ) ) - chosen_paths = {row.path for row in rows if row.path} for source in sources: - if str(source.path) in chosen_paths: + winner = winners.get(source.manifest.kind.value) + if winner is None or source.path == winner.path: continue rows.append( AdapterChoice( kind=source.manifest.kind, state="deferred", - reason="another adapter of this kind wins for the current mount", + reason=f"shadowed by {winner.manifest.adapter_id} for the current mount", path=str(source.path), adapter_id=source.manifest.adapter_id, base_model=source.manifest.base_model, diff --git a/backend/src/ely_eye/proofs/checks_runtime.py b/backend/src/ely_eye/proofs/checks_runtime.py index e5032d1..1cceacf 100644 --- a/backend/src/ely_eye/proofs/checks_runtime.py +++ b/backend/src/ely_eye/proofs/checks_runtime.py @@ -19,6 +19,7 @@ from ..evaluation_schemas import ( from ..evidence_graph import EvidenceGraphService from ..profiles import runtime_profile_report from ..runtime import QwenRuntime +from ..runtime_schemas import AdapterState from ..schemas import ( RuntimeProfile, TrainingAdapterKind, @@ -40,6 +41,16 @@ from .training_trace import ( ) +def _generation_adapter( + settings: Settings, + database: Database, + kind: TrainingAdapterKind = TrainingAdapterKind.hme_core_lora, +) -> tuple[adapters.AdapterSource | None, AdapterState, str]: + """Resolve the way generation does, mounted cartridges included, so proof and generation name one copy.""" + sources, _mounted = adapters.discover(settings, database) + return adapters.resolve(sources, settings, kind) + + class RuntimeChecks: """Runtime adapter binding, gold-adapter generation, launch profiles, and the removed module check.""" @@ -59,7 +70,7 @@ class RuntimeChecks: return deferred runtime = QwenRuntime(self.settings) status = runtime.status() - adapter, _adapter_state, _adapter_reason = adapters.resolve_library_adapter(self.settings) + adapter, _adapter_state, _adapter_reason = _generation_adapter(self.settings, self.db) weight_path = adapter.path / "adapter_model.safetensors" if adapter else None payload = { "runtime_status": status.model_dump(mode="json"), @@ -134,7 +145,7 @@ class RuntimeChecks: ) payload["generation_question"] = generation_question status = runtime.status() - adapter, _adapter_state, _adapter_reason = adapters.resolve_library_adapter(proof_settings) + adapter, _adapter_state, _adapter_reason = _generation_adapter(proof_settings, self.db) generation = runtime.generate(generation_question, context) answer, citations, citation_report = Verifier().verify_answer(generation.text, context) claim_id = EvidenceGraphService(proof_settings, self.db).record_answer( diff --git a/backend/src/ely_eye/runtime.py b/backend/src/ely_eye/runtime.py index e39e8ac..7ed7b00 100644 --- a/backend/src/ely_eye/runtime.py +++ b/backend/src/ely_eye/runtime.py @@ -42,6 +42,7 @@ class QwenRuntime: self.database = Database(self.settings) self._processor: Any | None = None self._model: Any | None = None + self._attached: adapters.LoadRecord | None = None def _adapter_choice(self) -> tuple[adapters.AdapterSource | None, str]: """The adapter this runtime would attach, plus the reason if it would not attach one.""" @@ -50,7 +51,8 @@ class QwenRuntime: return source, reason def _loaded_adapter(self) -> adapters.LoadRecord | None: - return next((item for item in adapters.load_records() if item.kind == TrainingAdapterKind.hme_core_lora), None) + """What this instance attached; another instance's model is not this one's claim.""" + return self._attached def _generation_adapter_fields(self) -> dict[str, str | None]: record = self._loaded_adapter() @@ -314,7 +316,7 @@ class QwenRuntime: logger.info("generating without an adapter: %s", reason) else: model = adapters.load_into_model(model, source.path) - adapters.record_load(source) + self._attached = adapters.record_load(source, model) model.eval() self._model, self._processor = model, processor return model, processor @@ -345,7 +347,7 @@ class QwenRuntime: def unload(self) -> None: self._model = None self._processor = None - adapters.clear_loads() + self._attached = None try: import torch diff --git a/backend/tests/test_adapters.py b/backend/tests/test_adapters.py index da97845..40b9e85 100644 --- a/backend/tests/test_adapters.py +++ b/backend/tests/test_adapters.py @@ -3,12 +3,15 @@ from __future__ import annotations import json +import sys from datetime import UTC, datetime from pathlib import Path +from types import SimpleNamespace from typing import Any from unittest import mock import pytest +import torch from fastapi.testclient import TestClient from ely_eye import adapters, api @@ -92,10 +95,25 @@ def _cartridge(settings: Settings, database: Database, cartridge_id: str) -> Pat class _FakeModel: def __init__(self) -> None: self.evalled = False + self.device = "cpu" def eval(self) -> None: self.evalled = True + def generate(self, **_kwargs: Any) -> torch.Tensor: + return torch.zeros((1, 8), dtype=torch.long) + + +class _FakeProcessor: + def apply_chat_template(self, _messages: Any, **_kwargs: Any) -> str: + return "prompt" + + def __call__(self, **_kwargs: Any) -> dict[str, torch.Tensor]: + return {"input_ids": torch.zeros((1, 5), dtype=torch.long)} + + def batch_decode(self, _generated: Any, **_kwargs: Any) -> list[str]: + return ["three tokens"] + @pytest.fixture(autouse=True) def _no_stale_loads(): @@ -107,7 +125,7 @@ def _no_stale_loads(): def test_a_library_adapter_is_selected_for_its_kind(tmp_path: Path) -> None: settings = _settings(tmp_path) chosen = _write_adapter(settings.adapters_dir / "core") - source, state, reason = adapters.resolve_library_adapter(settings) + source, state, reason = adapters.resolve(adapters.library_sources(settings), settings, CORE) assert source is not None and source.path == chosen assert state == "available" assert reason == "selected from the adapter library" @@ -116,7 +134,7 @@ def test_a_library_adapter_is_selected_for_its_kind(tmp_path: Path) -> None: def test_an_adapter_on_another_base_model_is_deferred_naming_both_models(tmp_path: Path) -> None: settings = _settings(tmp_path) _write_adapter(settings.adapters_dir / "router", kind=TrainingAdapterKind.hme_router, base_model="Qwen/Qwen3.5-0.8B") - source, state, reason = adapters.resolve_library_adapter(settings, TrainingAdapterKind.hme_router) + source, state, reason = adapters.resolve(adapters.library_sources(settings), settings, TrainingAdapterKind.hme_router) assert source is None assert state == "deferred" assert "trained on Qwen/Qwen3.5-0.8B" in reason and f"runtime holds {MODEL}" in reason @@ -125,7 +143,7 @@ def test_an_adapter_on_another_base_model_is_deferred_naming_both_models(tmp_pat def test_a_manifest_without_weights_is_deferred_naming_the_missing_file(tmp_path: Path) -> None: settings = _settings(tmp_path) _write_adapter(settings.adapters_dir / "core", weights=False) - source, state, reason = adapters.resolve_library_adapter(settings) + source, state, reason = adapters.resolve(adapters.library_sources(settings), settings, CORE) assert source is None and state == "deferred" assert "adapter_model.safetensors" in reason @@ -183,32 +201,70 @@ def test_stub_weights_prove_the_load_branch_runs(tmp_path: Path, monkeypatch) -> assert row.state == "loaded" and "attached to the live model" in row.reason -def test_unload_clears_the_loaded_state(tmp_path: Path, monkeypatch) -> None: +def test_unload_stops_claiming_a_model_that_is_gone(tmp_path: Path, monkeypatch) -> None: settings = _settings(tmp_path) _write_adapter(settings.adapters_dir / "core") monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel()) monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), object())) runtime = QwenRuntime(settings) runtime._load_transformers() - assert adapters.load_records(), "nothing was recorded as loaded" + database = Database(settings) + row = next(item for item in adapters.plan(settings, database).adapters if item.kind == CORE) + assert row.state == "loaded" and "live model" in row.reason + other = QwenRuntime(settings) + assert other._loaded_adapter() is None, "a second instance inherited another instance's claim" runtime.unload() - assert adapters.load_records() == [] - row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE) + row = next(item for item in adapters.plan(settings, database).adapters if item.kind == CORE) assert row.state == "available" + assert "that model is gone" in row.reason, row.reason -def test_the_loser_of_the_choice_is_reported_as_shadowed(tmp_path: Path) -> None: +def test_a_rejected_candidate_does_not_claim_another_adapter_won(tmp_path: Path) -> None: + settings = _settings(tmp_path) + _write_adapter(settings.adapters_dir / "router", kind=TrainingAdapterKind.hme_router, base_model="Qwen/Qwen3.5-0.8B") + report = adapters.plan(settings, Database(settings)) + router_rows = [row for row in report.adapters if row.kind == TrainingAdapterKind.hme_router] + assert len(router_rows) == 1, "a rejected candidate was published as a shadow row" + assert router_rows[0].state == "deferred" and "trained on Qwen/Qwen3.5-0.8B" in router_rows[0].reason + assert not any(row.shadowed for row in report.adapters), "nothing won, so nothing may claim a winner" + + +def test_the_shadow_row_names_the_adapter_that_won(tmp_path: Path) -> None: settings = _settings(tmp_path) _write_adapter(settings.adapters_dir / "first", adapter_id="written_first") _write_adapter(settings.adapters_dir / "second", adapter_id="written_second") report = adapters.plan(settings, Database(settings)) losers = [row for row in report.adapters if row.shadowed] assert [row.adapter_id for row in losers] == ["written_first"] - assert "another adapter of this kind wins" in losers[0].reason + assert losers[0].reason == "shadowed by written_second for the current mount" winner = next(row for row in report.adapters if row.adapter_id == "written_second") assert winner.shadowed is False and winner.state == "available" +def test_a_cartridge_tagged_adapter_outranks_an_untagged_one(tmp_path: Path) -> None: + settings = _settings(tmp_path) + tagged = _write_adapter(settings.adapters_dir / "tagged", adapter_id="tagged") + manifest = json.loads((tagged / "adapter_manifest.json").read_text(encoding="utf-8")) + manifest["cartridge_id"] = "cart-not-mounted" + (tagged / "adapter_manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + _write_adapter(settings.adapters_dir / "plain", adapter_id="plain") + source, _state, _reason = adapters.resolve(adapters.library_sources(settings), settings, CORE) + assert source is not None and source.manifest.adapter_id == "tagged" + + +def test_generation_reports_the_adapter_it_attached(tmp_path: Path, monkeypatch) -> None: + settings = _settings(tmp_path) + chosen = _write_adapter(settings.adapters_dir / "core") + monkeypatch.setitem(sys.modules, "qwen_vl_utils", SimpleNamespace(process_vision_info=lambda _messages: ([], None))) + monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel()) + monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), _FakeProcessor())) + generation = QwenRuntime(settings).generate_prompt("what does the cartridge hold?") + assert generation.adapter_path == str(chosen) + assert generation.adapter_kind == CORE.value and generation.adapter_id == "adapter-1" + assert generation.adapter_sha256 == "a" * 64 + assert generation.output_tokens == 3 and generation.text == "three tokens" + + def test_an_unreadable_manifest_names_the_file(tmp_path: Path) -> None: settings = _settings(tmp_path) broken = settings.adapters_dir / "broken"