diff --git a/CHECKLIST.md b/CHECKLIST.md index 29e4ba4..c12262e 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -229,10 +229,10 @@ - 测试写出来才发现的第二个 bug：`ready.sort(key=_preference)` 丢了原来的 `reverse=True`，库里的 adapter 反而赢了挂载副本。 - 状态只由事实支撑：loaded（模型还活着——`record_load` 存 `weakref`，且记录挂在具体 `QwenRuntime._attached` 上，不是进程全局）/ available（manifest 与 `adapter_model.safetensors` 都在；模型没了就说「attached earlier in this process … that model is gone」）/ deferred（base_model 不匹配或缺权重文件，原因里写两个模型名）/ missing（磁盘上没有该 kind）。落选者单独一行 `shadowed=True`，且**只在真有赢家时**才有这行，原因点名赢家。 - 排序键是三个：挂载卡带 > manifest 带 `cartridge_id` > 权重 mtime。第一个版本漏了中间那个，reviewer 在 /tmp 里对比 parent `9f0f799` 与本提交选中的文件，量出两者不同（可复现做法：先写一个带 `cartridge_id` 的 adapter，再写一个不带但 mtime 更新的，看 `resolve` 选哪个）——`training.py:106` 写库内 adapter 时会带卡带标记，所以这不是风格问题。 - - 验收命令：`pytest backend/tests/test_adapters.py -q` → 15 passed。`test_stub_weights_prove_the_load_branch_runs` 用桩权重断言 `attached == [(ANY, 路径)]`、`model.evalled is True`、LoadRecord 存在、端点该行 `state == "loaded"`；`test_generation_reports_the_adapter_it_attached` 走完 transformers 生成路径，断言答案里的 adapter 路径 / kind / id / sha256 与 `output_tokens`；`test_unload_stops_claiming_a_model_that_is_gone` 断言 unload 后回到 available 并说明模型已消失，同时断言另一个实例不会继承这个声明；`test_a_loaded_row_names_the_adapter_that_is_actually_attached` 与 `test_one_unload_does_not_declare_another_live_model_gone` 钉住第二轮那两条（行里的身份列必须跟加载记录一致；一个模型 unload 不能说成另一个活模型消失了）。两条都做了变异检验：把行身份改回按解析结果取 → 前者 FAILED；让 `live_attachments()` 忽略存活 → 后两个 FAILED。 + - 验收命令：`pytest backend/tests/test_adapters.py -q` → 16 passed。`test_stub_weights_prove_the_load_branch_runs` 用桩权重断言 `attached == [(ANY, 路径)]`、`model.evalled is True`、LoadRecord 存在、端点该行 `state == "loaded"`；`test_generation_reports_the_adapter_it_attached` 走完 transformers 生成路径，断言答案里的 adapter 路径 / kind / id / sha256 与 `output_tokens`；`test_unload_stops_claiming_a_model_that_is_gone` 断言 unload 后回到 available 并说明模型已消失，同时断言另一个实例不会继承这个声明；`test_a_loaded_row_names_the_adapter_that_is_actually_attached` 与 `test_one_unload_does_not_declare_another_live_model_gone` 钉住第二轮那两条（行里的身份列必须跟加载记录一致；一个模型 unload 不能说成另一个活模型消失了）。两条都做了变异检验：把行身份改回按解析结果取 → 前者 FAILED；让 `live_attachments()` 忽略存活 → 后两个 FAILED。`test_three_live_models_are_named_as_models_not_one_model` 钉住第三轮那条单复数 bug。 - 证明与生成解析一致：`proofs/checks_runtime.py` 的 `_generation_adapter()` 用 `discover` + `resolve`（含挂载卡带），不再用只看库的解析，否则同一份证明里 `resolved_adapter` 与 `adapter` 两块会指向不同副本还静默。 - 新文件 `backend/src/ely_eye/runtime_schemas.py`（30 行）：`schemas.py` 已 498 行，离 500 行红线 2 行，报告类型不放进去。 - - 门禁：`pytest -q` 227 passed；ruff 干净；mypy 冷缓存查 3 个改动源文件 → 2 errors，与 `git stash` 后同一命令在 HEAD 树上的实测完全同两条（`qwen_vl_utils` 无 stubs、`model_class` 重赋值），净新增 0；pyright 同文件 → 2 errors，HEAD 也是 2 条，都是 `process_vision_info` 的双返回（`vision_process.py:532-534` 由 `return_video_kwargs` 决定返回 2 或 3 元组，产品用默认值取 2 个，运行时正确）。 + - 门禁：`pytest -q` 228 passed；ruff 干净；mypy 冷缓存查 3 个改动源文件 → 2 errors，与 `git stash` 后同一命令在 HEAD 树上的实测完全同两条（`qwen_vl_utils` 无 stubs、`model_class` 重赋值），净新增 0；pyright 同文件 → 2 errors，HEAD 也是 2 条，都是 `process_vision_info` 的双返回（`vision_process.py:532-534` 由 `return_video_kwargs` 决定返回 2 或 3 元组，产品用默认值取 2 个，运行时正确）。 - 独立 review（同一个 `p1review` 里的 pi，模型 `strata/qwen3.8-flash-next-abliterated-iq3_s:high`）第一轮 **FAIL 6 条**，全部核实并改掉： 1. `plan()` 用 `chosen_paths` 判落选，把「被拒绝的候选」也当成落选者，于是没有任何 adapter 胜出时却说「another adapter of this kind wins」，还把真正的 deferred 原因盖掉了（reviewer 挂载 cart-home 复现，`hme_router`、`hme_vision_lora` 两行都在撒谎）。 2. `_LOADED` 是进程全局，而原因写「attached to the live model」——`api.py` 每个请求新建一个 `QwenRuntime`，`_model is None` 也报 loaded。改成 weakref + 实例字段。 @@ -242,6 +242,7 @@ 6. 证明检查与生成解析不一致（见上）。 reviewer 还指出 4 条生成路径与 `status()` 零覆盖，所以「没丢函数」当时只是读出来的——现在有了走完生成路径的测试。 - 第二轮 **FAIL 1 条**（同一 reviewer，它自己建 worktree + `PYTHONPATH` 跑 harness，并记下 editable install 指向主树、不加 `PYTHONPATH` 的 worktree 测试会假绿）：`plan()` 里 loaded 行的 `reason` 按加载记录写，而 `path`/`adapter_id`/`base_model`/`origin`/`cartridge_id` 按当前解析结果写——实测：活模型挂 X，磁盘上又写了个更新的 Y，端点就返回 `adapter_id=adapter-Y` 却解释「attached from …/x-attached」。同一个根因的第二个症状：登记按 kind 存、后写覆盖，两个 runtime 时 unload 其中一个会把另一个还活着的模型说成「that model is gone」。修法：`LoadRecord` 直接携带 `AdapterSource`，行身份从记录取（`_choice()` 只有一个身份来源）；登记改成按 attachment 列表 + 各自 weakref，`loaded` 只看是否还有活模型持有，多于一个时原因里写明另一个。 + - 第三轮 **FAIL 2 条**（都是措辞与重复字段，reviewer 用三个活 runtime 实测）：`adapters.py` 的 `plan()` 在三个以上模型同时持有时写「another live model still holds adapter-a and adapter-b」，单数主语把多个 adapter 塞给了一个模型；`LoadRecord.sha256` 仍是 `source.manifest.sha256` 的存储副本，正是这次刚清掉的「第二个真相来源」。修法：主语按数量分单复数，`sha256` 改成 property。两条都有变异检验：去掉复数分支 → 两个测试 FAILED；换回存储副本 → 生成路径测试 FAILED。reviewer 另记三条它判定不算缺陷的，这里补记：`base_model` 在「磁盘上没有该 kind」的行上从「运行时的模型 id」改成 `null`（更诚实：那一行根本没有 adapter）；同一 adapter id 可能同时出现在 loaded 行与 shadowed 行（它自已被挂着，但挂载里另有副本胜出，两行都真）；被加载的 adapter 从磁盘删掉且无解析结果时不再说「that model is gone」（`missing` 行身份为 null，没有假断言）。 - CLI 面：`ely-eye adapters` 打印同一份报告。 - [ ] **p1-4** 工程地基 - [ ] p1-4.1 pytest 从零到有（后端） diff --git a/backend/src/ely_eye/adapters.py b/backend/src/ely_eye/adapters.py index ed56c6a..df49b96 100644 --- a/backend/src/ely_eye/adapters.py +++ b/backend/src/ely_eye/adapters.py @@ -37,7 +37,6 @@ class LoadRecord: """One adapter this process attached to one model.""" source: AdapterSource - sha256: str @property def kind(self) -> TrainingAdapterKind: @@ -51,6 +50,10 @@ class LoadRecord: def path(self) -> str: return str(self.source.path) + @property + def sha256(self) -> str: + return self.source.manifest.sha256 + _ATTACHMENTS: list[tuple[LoadRecord, weakref.ReferenceType[object]]] = [] @@ -63,7 +66,9 @@ def read_manifest(path: Path) -> AdapterManifest: try: return AdapterManifest.model_validate_json(text) except ValidationError as error: - raise ValueError(f"{path}: the adapter manifest is not a valid AdapterManifest ({error})") from error + raise ValueError( + f"{path}: the adapter manifest is not a valid AdapterManifest ({error})" + ) from error def cartridge_roots(database: Database) -> dict[str, Path]: @@ -111,7 +116,9 @@ def discover(settings: Settings, database: Database) -> tuple[list[AdapterSource for cartridge_id in mounted: root = roots.get(cartridge_id) if root is None: - logger.warning("cartridge %s is mounted but its directory is not registered", cartridge_id) + logger.warning( + "cartridge %s is mounted but its directory is not registered", cartridge_id + ) continue for manifest_path in sorted((root / "adapters").glob(f"*/{MANIFEST_FILE}")): sources.append( @@ -144,7 +151,11 @@ def resolve( ) ready = [source for source in matching if (source.path / WEIGHTS_FILE).exists()] if not ready: - return None, "deferred", f"the {kind.value} manifests are present but none carries {WEIGHTS_FILE}" + return ( + None, + "deferred", + f"the {kind.value} manifests are present but none carries {WEIGHTS_FILE}", + ) ready.sort(key=_preference, reverse=True) chosen = ready[0] if chosen.origin == "cartridge": @@ -172,7 +183,7 @@ def live_attachments() -> list[LoadRecord]: def record_load(source: AdapterSource, model: Any) -> LoadRecord: - record = LoadRecord(source=source, sha256=source.manifest.sha256) + record = LoadRecord(source=source) _ATTACHMENTS[:] = [ (item, reference) for item, reference in _ATTACHMENTS if reference() is not None ] @@ -208,7 +219,8 @@ def plan(settings: Settings, database: Database) -> AdapterLoadReport: reason = f"attached to the live model from {newest.path}" held = sorted({record.adapter_id for record in live[:-1] if record.adapter_id != newest.adapter_id}) if held: - reason = f"{reason}; another live model still holds {' and '.join(held)}" + subject = "another live model still holds" if len(held) == 1 else "other live models still hold" + reason = f"{reason}; {subject} {' and '.join(held)}" rows.append(_choice(kind, state, reason, newest.source)) continue if earlier is not None and source is not None: diff --git a/backend/tests/test_adapters.py b/backend/tests/test_adapters.py index a11b812..35afd21 100644 --- a/backend/tests/test_adapters.py +++ b/backend/tests/test_adapters.py @@ -133,8 +133,14 @@ def test_a_library_adapter_is_selected_for_its_kind(tmp_path: Path) -> None: def test_an_adapter_on_another_base_model_is_deferred_naming_both_models(tmp_path: Path) -> None: settings = _settings(tmp_path) - _write_adapter(settings.adapters_dir / "router", kind=TrainingAdapterKind.hme_router, base_model="Qwen/Qwen3.5-0.8B") - source, state, reason = adapters.resolve(adapters.library_sources(settings), settings, TrainingAdapterKind.hme_router) + _write_adapter( + settings.adapters_dir / "router", + kind=TrainingAdapterKind.hme_router, + base_model="Qwen/Qwen3.5-0.8B", + ) + source, state, reason = adapters.resolve( + adapters.library_sources(settings), settings, TrainingAdapterKind.hme_router + ) assert source is None assert state == "deferred" assert "trained on Qwen/Qwen3.5-0.8B" in reason and f"runtime holds {MODEL}" in reason @@ -153,7 +159,9 @@ def test_the_mounted_cartridge_copy_wins_over_the_library(tmp_path: Path) -> Non database = Database(settings) _write_adapter(settings.adapters_dir / "core", adapter_id="library-copy") cartridge_root = _cartridge(settings, database, "cart-home") - cartridge_copy = _write_adapter(cartridge_root / "adapters" / "hme_core_lora", adapter_id="cartridge-copy") + cartridge_copy = _write_adapter( + cartridge_root / "adapters" / "hme_core_lora", adapter_id="cartridge-copy" + ) sources, mounted = adapters.discover(settings, database) assert mounted == ["cart-home"] source, state, reason = adapters.resolve(sources, settings, CORE) @@ -197,7 +205,9 @@ def test_stub_weights_prove_the_load_branch_runs(tmp_path: Path, monkeypatch) -> assert attached == [(mock.ANY, chosen)], "peft was never asked to attach the stub adapter" assert model.evalled is True assert [record.adapter_id for record in adapters.load_records()] == ["adapter-1"] - row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE) + row = next( + item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE + ) assert row.state == "loaded" and "attached to the live model" in row.reason @@ -219,7 +229,9 @@ def test_unload_stops_claiming_a_model_that_is_gone(tmp_path: Path, monkeypatch) assert "that model is gone" in row.reason, row.reason -def test_a_loaded_row_names_the_adapter_that_is_actually_attached(tmp_path: Path, monkeypatch) -> None: +def test_a_loaded_row_names_the_adapter_that_is_actually_attached( + tmp_path: Path, monkeypatch +) -> None: settings = _settings(tmp_path) attached = _write_adapter(settings.adapters_dir / "x", adapter_id="adapter-x") monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel()) @@ -227,9 +239,13 @@ def test_a_loaded_row_names_the_adapter_that_is_actually_attached(tmp_path: Path runtime = QwenRuntime(settings) runtime._load_transformers() _write_adapter(settings.adapters_dir / "y", adapter_id="adapter-y") - row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE) + row = next( + item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE + ) assert row.state == "loaded" - assert row.adapter_id == "adapter-x", f"the row promises {row.adapter_id} but {attached} is attached" + assert row.adapter_id == "adapter-x", ( + f"the row promises {row.adapter_id} but {attached} is attached" + ) assert row.path == str(attached) and row.reason.endswith(str(attached)), row.reason @@ -256,14 +272,39 @@ def test_one_unload_does_not_declare_another_live_model_gone(tmp_path: Path, mon assert row.state == "available" and "that model is gone" in row.reason, row.reason +def test_three_live_models_are_named_as_models_not_one_model(tmp_path: Path, monkeypatch) -> None: + settings = _settings(tmp_path) + monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel()) + monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), object())) + runtimes = [] + for name in ("a", "b", "c"): + _write_adapter(settings.adapters_dir / name, adapter_id=f"adapter-{name}") + runtime = QwenRuntime(settings) + runtime._load_transformers() + runtimes.append(runtime) + row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE) + assert row.state == "loaded" and row.adapter_id == "adapter-c" + assert "other live models still hold adapter-a and adapter-b" in row.reason, row.reason + assert "another live model" not in row.reason, row.reason + + def test_a_rejected_candidate_does_not_claim_another_adapter_won(tmp_path: Path) -> None: settings = _settings(tmp_path) - _write_adapter(settings.adapters_dir / "router", kind=TrainingAdapterKind.hme_router, base_model="Qwen/Qwen3.5-0.8B") + _write_adapter( + settings.adapters_dir / "router", + kind=TrainingAdapterKind.hme_router, + base_model="Qwen/Qwen3.5-0.8B", + ) report = adapters.plan(settings, Database(settings)) router_rows = [row for row in report.adapters if row.kind == TrainingAdapterKind.hme_router] assert len(router_rows) == 1, "a rejected candidate was published as a shadow row" - assert router_rows[0].state == "deferred" and "trained on Qwen/Qwen3.5-0.8B" in router_rows[0].reason - assert not any(row.shadowed for row in report.adapters), "nothing won, so nothing may claim a winner" + assert ( + router_rows[0].state == "deferred" + and "trained on Qwen/Qwen3.5-0.8B" in router_rows[0].reason + ) + assert not any(row.shadowed for row in report.adapters), ( + "nothing won, so nothing may claim a winner" + ) def test_the_shadow_row_names_the_adapter_that_won(tmp_path: Path) -> None: @@ -292,9 +333,15 @@ def test_a_cartridge_tagged_adapter_outranks_an_untagged_one(tmp_path: Path) -> def test_generation_reports_the_adapter_it_attached(tmp_path: Path, monkeypatch) -> None: settings = _settings(tmp_path) chosen = _write_adapter(settings.adapters_dir / "core") - monkeypatch.setitem(sys.modules, "qwen_vl_utils", SimpleNamespace(process_vision_info=lambda _messages: ([], None))) + monkeypatch.setitem( + sys.modules, + "qwen_vl_utils", + SimpleNamespace(process_vision_info=lambda _messages: ([], None)), + ) monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel()) - monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), _FakeProcessor())) + monkeypatch.setattr( + QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), _FakeProcessor()) + ) generation = QwenRuntime(settings).generate_prompt("what does the cartridge hold?") assert generation.adapter_path == str(chosen) assert generation.adapter_kind == CORE.value and generation.adapter_id == "adapter-1"