Resolve adapters by what is mounted, and say why each kind is not loaded

runtime.py resolved adapters at 7 call sites and only ever looked in the
adapter library, so an adapter attached to a cartridge (cartridge.py:230
copies it into <cartridge>/adapters/{kind}/) was never loaded by anything.
Resolution now lives in adapters.py: it reads the mounted cartridge set,
prefers a mounted cartridge's copy over the library, and answers every one
of the six kinds with a state the facts support — loaded, available,
deferred (base_model mismatch or missing weights, both models named in the
reason), missing — plus a shadowed row for each adapter that lost the
choice. A mounted cartridge is the scenario selection; neither manifest
carries a scenario field, so none was invented.

Two bugs the new tests caught: the preference sort lost reverse=True, so a
library adapter beat a mounted cartridge's copy, and the load branch had
never been executed by a test. Stub weights now prove peft is asked to
attach, that the returned model is put in eval mode, and that the endpoint
reports loaded until unload clears it.

Faces: GET /api/runtime/adapters, `ely-eye adapters`, and a log line naming
the reason whenever a generation runs without an adapter.

Gates: 222 passed; ruff clean; mypy cold cache on the 7 touched files gives
the same 2 errors as HEAD's runtime.py, and pyright gives the same 2 as
HEAD — both are qwen_vl_utils typing artifacts (process_vision_info returns
2 or 3 values depending on return_video_kwargs; the product takes 2).
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 23:09:28 -04:00
1 parent 9f0f7993e5
commit 2fbb4af96b
8 files changed
+585 -86

No files matched your search

+12 -3
View File
@@ -219,10 +219,19 @@
重跑 `bench.asr_check`(新进程)、重跑门禁、逐项核对 stamp 链与 18 条 CHECKLIST 断言, 重跑 `bench.asr_check`(新进程)、重跑门禁、逐项核对 stamp 链与 18 条 CHECKLIST 断言,
并确认旧数字(1.7 秒 / 3.55 / 61.6 / 31 段 / 210.5 / 51×)零残留。三轮都是同一个 reviewer, 并确认旧数字(1.7 秒 / 3.55 / 61.6 / 31 段 / 210.5 / 51×)零残留。三轮都是同一个 reviewer,
没有换人放水。 没有换人放水。
- [ ] **p1-3** adapter 加载 - [x] **p1-3** adapter 加载
- [ ] p1-3.1 按卡带/场景解析 6 个 adapter,不再只有 `runtime.py:317` - [x] p1-3.1 按卡带/场景解析 6 个 adapter,不再只有 `runtime.py:317`
- [ ] p1-3.2 加载结果可观测:日志 + `GET /api/runtime/adapters` - [x] p1-3.2 加载结果可观测:日志 + `GET /api/runtime/adapters`
- 验收:状态端点列每个 adapter 的 loaded/deferred + 原因;桩权重测试证明加载分支执行 - 验收:状态端点列每个 adapter 的 loaded/deferred + 原因;桩权重测试证明加载分支执行
- `backend/src/ely_eye/adapters.py` 是唯一的 adapter 解析处。`runtime.py` 原先在 7 处调 `resolve_runtime_adapter`,现在统一走 `_generation_adapter_fields()`;`peft` 只在 `adapters.load_into_model` 里懒加载,`RuntimeAdapter` 与 `resolve_runtime_adapter` 已删除。
- 「场景」= 挂载的卡带:`AdapterManifest` 和 `CartridgeManifest` 都没有 scenario 字段(读出来的事实),所以解析按 `cartridge_store.mounted_ids` 选,报告写明来自哪个卡带,没有新造字段。
- 修掉的真 bug:`CartridgeService.attach_adapter`(`cartridge.py:230`)把 adapter 拷进 `<卡带>/adapters/{kind}/`,但解析从来不看卡带目录,挂上去的东西运行时根本不加载。现在挂载卡带里的副本优先于库里的同名 adapter。
- 测试写出来才发现的第二个 bug:`ready.sort(key=_preference)` 丢了原来的 `reverse=True`,库里的 adapter 反而赢了挂载副本。
- 状态只由事实支撑:loaded(有 LoadRecord)/ available(manifest 与 `adapter_model.safetensors` 都在)/ deferred(base_model 不匹配或缺权重文件,原因里写两个模型名)/ missing(磁盘上没有该 kind)。落选者单独一行 `shadowed=True`。
- 验收命令:`pytest backend/tests/test_adapters.py -q` → 10 passed。`test_stub_weights_prove_the_load_branch_runs` 用桩权重断言 `attached == [(ANY, 路径)]`、`model.evalled is True`、LoadRecord 存在、端点该行 `state == "loaded"`;`test_unload_clears_the_loaded_state` 断言 unload 后回到 available。
- 新文件 `backend/src/ely_eye/runtime_schemas.py`(30 行):`schemas.py` 已 498 行,离 500 行红线 2 行,报告类型不放进去。
- 门禁:`pytest -q` 222 passed;ruff 干净;mypy 冷缓存查 7 个改动文件 → 2 errors,与 HEAD 的 `runtime.py` 逐条相同(`qwen_vl_utils` 无 stubs、`model_class` 重赋值),净新增 0;pyright 同 7 文件 → 2 errors,HEAD 也是 2 条,都是 `process_vision_info` 的双返回(`vision_process.py:532-534` 由 `return_video_kwargs` 决定返回 2 或 3 元组,产品用默认值取 2 个,运行时正确)。
- CLI 面:`ely-eye adapters` 打印同一份报告。
- [ ] **p1-4** 工程地基 - [ ] **p1-4** 工程地基
- [ ] p1-4.1 pytest 从零到有(后端) - [ ] p1-4.1 pytest 从零到有(后端)
- [ ] p1-4.2 go test 从零到有(MCP) - [ ] p1-4.2 go test 从零到有(MCP)
+229
View File
@@ -0,0 +1,229 @@
"""One place decides which adapter loads, and says out loud why the others do not."""
from __future__ import annotations
import logging
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from pydantic import ValidationError
from . import cartridge_store
from .config import Settings
from .db import Database
from .runtime_schemas import AdapterChoice, AdapterLoadReport, AdapterOrigin, AdapterState
from .schemas import AdapterManifest, TrainingAdapterKind
logger = logging.getLogger("ely_eye.adapters")
MANIFEST_FILE = "adapter_manifest.json"
WEIGHTS_FILE = "adapter_model.safetensors"
@dataclass(frozen=True)
class AdapterSource:
"""One adapter found on disk, plus where it was found."""
path: Path
manifest: AdapterManifest
origin: AdapterOrigin
cartridge_id: str | None
@dataclass(frozen=True)
class LoadRecord:
"""An adapter this process actually attached to a live model."""
kind: TrainingAdapterKind
adapter_id: str
path: str
sha256: str
loaded_at: str
_LOADED: dict[str, LoadRecord] = {}
def read_manifest(path: Path) -> AdapterManifest:
try:
text = path.read_text(encoding="utf-8")
except OSError as error:
raise ValueError(f"{path}: the adapter manifest cannot be read ({error})") from error
try:
return AdapterManifest.model_validate_json(text)
except ValidationError as error:
raise ValueError(f"{path}: the adapter manifest is not a valid AdapterManifest ({error})") from error
def cartridge_roots(database: Database) -> dict[str, Path]:
roots: dict[str, Path] = {}
for row in database.list_cartridges():
roots[str(row["cartridge_id"])] = Path(str(row["root_path"]))
return roots
def library_sources(settings: Settings) -> list[AdapterSource]:
"""Adapters in the library directory, for callers that have no database open."""
if settings.adapter_dir is not None:
pinned = settings.adapter_dir.expanduser()
if not (pinned / MANIFEST_FILE).exists():
logger.warning("adapter_dir %s holds no %s", pinned, MANIFEST_FILE)
return []
manifest = read_manifest(pinned / MANIFEST_FILE)
return [
AdapterSource(
path=pinned,
manifest=manifest,
origin="library",
cartridge_id=manifest.cartridge_id,
)
]
sources: list[AdapterSource] = []
for manifest_path in sorted(settings.adapters_dir.glob(f"*/{MANIFEST_FILE}")):
manifest = read_manifest(manifest_path)
sources.append(
AdapterSource(
path=manifest_path.parent,
manifest=manifest,
origin="library",
cartridge_id=manifest.cartridge_id,
)
)
return sources
def discover(settings: Settings, database: Database) -> tuple[list[AdapterSource], list[str]]:
"""Every adapter the runtime could use: the library plus the copies inside mounted cartridges."""
mounted = cartridge_store.mounted_ids(database)
roots = cartridge_roots(database)
sources = library_sources(settings)
for cartridge_id in mounted:
root = roots.get(cartridge_id)
if root is None:
logger.warning("cartridge %s is mounted but its directory is not registered", cartridge_id)
continue
for manifest_path in sorted((root / "adapters").glob(f"*/{MANIFEST_FILE}")):
sources.append(
AdapterSource(
path=manifest_path.parent,
manifest=read_manifest(manifest_path),
origin="cartridge",
cartridge_id=cartridge_id,
)
)
return sources, mounted
def resolve(
sources: list[AdapterSource],
settings: Settings,
kind: TrainingAdapterKind,
) -> tuple[AdapterSource | None, AdapterState, str]:
"""Pick the adapter to load for one kind, or say which state it is stuck in and why."""
candidates = [source for source in sources if source.manifest.kind == kind]
if not candidates:
return None, "missing", f"no adapter of kind {kind.value} is on disk"
matching = [source for source in candidates if source.manifest.base_model == settings.model_id]
if not matching:
trained_on = ", ".join(sorted({source.manifest.base_model for source in candidates}))
return (
None,
"deferred",
f"every {kind.value} adapter is trained on {trained_on}; the runtime holds {settings.model_id}",
)
ready = [source for source in matching if (source.path / WEIGHTS_FILE).exists()]
if not ready:
return None, "deferred", f"the {kind.value} manifests are present but none carries {WEIGHTS_FILE}"
ready.sort(key=_preference, reverse=True)
chosen = ready[0]
if chosen.origin == "cartridge":
return chosen, "available", f"selected from mounted cartridge {chosen.cartridge_id}"
return chosen, "available", "selected from the adapter library"
def resolve_library_adapter(
settings: Settings,
kind: TrainingAdapterKind = TrainingAdapterKind.hme_core_lora,
) -> tuple[AdapterSource | None, AdapterState, str]:
"""Resolution without a database: the library only, no mounted cartridges."""
return resolve(library_sources(settings), settings, kind)
def _preference(source: AdapterSource) -> tuple[bool, float]:
weights = source.path / WEIGHTS_FILE
return (source.origin == "cartridge", weights.stat().st_mtime if weights.exists() else 0.0)
def load_records() -> list[LoadRecord]:
return list(_LOADED.values())
def clear_loads() -> None:
_LOADED.clear()
def record_load(source: AdapterSource) -> LoadRecord:
record = LoadRecord(
kind=source.manifest.kind,
adapter_id=source.manifest.adapter_id,
path=str(source.path),
sha256=source.manifest.sha256,
loaded_at=datetime.now().astimezone().isoformat(),
)
_LOADED[record.kind.value] = record
logger.info("loaded adapter %s (%s) from %s", record.adapter_id, record.kind.value, record.path)
return record
def load_into_model(model: Any, path: Path) -> Any:
"""Attach one LoRA to a live model; the only place generation imports peft."""
from .training import bitsandbytes_gaudi_probe_guard
with bitsandbytes_gaudi_probe_guard():
from peft import PeftModel
return PeftModel.from_pretrained(model, str(path))
def plan(settings: Settings, database: Database) -> AdapterLoadReport:
"""One row per adapter kind, plus a row for every adapter that lost the choice."""
sources, mounted = discover(settings, database)
rows: list[AdapterChoice] = []
for kind in TrainingAdapterKind:
source, state, reason = resolve(sources, settings, kind)
record = next((item for item in load_records() if item.kind == kind), None)
if record is not None:
reason = f"attached to the live model from {record.path}"
state = "loaded"
rows.append(
AdapterChoice(
kind=kind,
state=state,
reason=reason,
path=str(source.path) if source else None,
adapter_id=source.manifest.adapter_id if source else None,
base_model=source.manifest.base_model if source else settings.model_id,
origin=source.origin if source else None,
cartridge_id=source.cartridge_id if source else None,
)
)
chosen_paths = {row.path for row in rows if row.path}
for source in sources:
if str(source.path) in chosen_paths:
continue
rows.append(
AdapterChoice(
kind=source.manifest.kind,
state="deferred",
reason="another adapter of this kind wins for the current mount",
path=str(source.path),
adapter_id=source.manifest.adapter_id,
base_model=source.manifest.base_model,
origin=source.origin,
cartridge_id=source.cartridge_id,
shadowed=True,
)
)
return AdapterLoadReport(model_id=settings.model_id, mounted=mounted, adapters=rows)
+6
View File
@@ -10,6 +10,7 @@ from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel from pydantic import BaseModel
from . import __version__ from . import __version__
from .adapters import plan as adapter_plan
from .api_cartridges import register as register_cartridge_routes from .api_cartridges import register as register_cartridge_routes
from .api_holders import register as register_holder_routes from .api_holders import register as register_holder_routes
from .benchmarks import ( from .benchmarks import (
@@ -41,6 +42,7 @@ from .proofs import HashHopEvaluator, PRDProofSuite
from .research_modules import ResearchModuleService from .research_modules import ResearchModuleService
from .retrieval import RetrievalService from .retrieval import RetrievalService
from .runtime import QwenRuntime, RuntimeUnavailableError from .runtime import QwenRuntime, RuntimeUnavailableError
from .runtime_schemas import AdapterLoadReport
from .schemas import ( from .schemas import (
AdapterManifest, AdapterManifest,
AdapterStatus, AdapterStatus,
@@ -277,6 +279,10 @@ def create_app() -> FastAPI:
def runtime_profiles() -> RuntimeProfileReport: def runtime_profiles() -> RuntimeProfileReport:
return runtime_profile_report() return runtime_profile_report()
@app.get("/api/runtime/adapters", response_model=AdapterLoadReport)
def runtime_adapters(settings: Annotated[Settings, Depends(settings_dep)]) -> AdapterLoadReport:
return adapter_plan(settings, Database(settings))
@app.get("/api/research/modules") @app.get("/api/research/modules")
def research_modules(settings: Annotated[Settings, Depends(settings_dep)]): def research_modules(settings: Annotated[Settings, Depends(settings_dep)]):
db = Database(settings) db = Database(settings)
+9
View File
@@ -6,6 +6,7 @@ from pathlib import Path
import typer import typer
import uvicorn import uvicorn
from .adapters import plan as adapter_plan
from .api import create_app from .api import create_app
from .benchmarks import ( from .benchmarks import (
benchmark_prediction_report, benchmark_prediction_report,
@@ -475,5 +476,13 @@ def attach_adapter(cartridge_id: str, adapter_dir: Path) -> None:
typer.echo(manifest.model_dump_json(indent=2)) typer.echo(manifest.model_dump_json(indent=2))
@app.command("adapters")
def adapter_status() -> None:
"""Which adapter the runtime would load, and why each other kind would not."""
settings = get_settings()
report = adapter_plan(settings, Database(settings))
typer.echo(json_dump(report.model_dump(mode="json")))
if __name__ == "__main__": if __name__ == "__main__":
app() app()
+4 -3
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from .. import adapters
from ..compiler import ContextCompiler from ..compiler import ContextCompiler
from ..config import Settings from ..config import Settings
from ..db import Database from ..db import Database
@@ -17,7 +18,7 @@ from ..evaluation_schemas import (
) )
from ..evidence_graph import EvidenceGraphService from ..evidence_graph import EvidenceGraphService
from ..profiles import runtime_profile_report from ..profiles import runtime_profile_report
from ..runtime import QwenRuntime, resolve_runtime_adapter from ..runtime import QwenRuntime
from ..schemas import ( from ..schemas import (
RuntimeProfile, RuntimeProfile,
TrainingAdapterKind, TrainingAdapterKind,
@@ -58,7 +59,7 @@ class RuntimeChecks:
return deferred return deferred
runtime = QwenRuntime(self.settings) runtime = QwenRuntime(self.settings)
status = runtime.status() status = runtime.status()
adapter = resolve_runtime_adapter(self.settings) adapter, _adapter_state, _adapter_reason = adapters.resolve_library_adapter(self.settings)
weight_path = adapter.path / "adapter_model.safetensors" if adapter else None weight_path = adapter.path / "adapter_model.safetensors" if adapter else None
payload = { payload = {
"runtime_status": status.model_dump(mode="json"), "runtime_status": status.model_dump(mode="json"),
@@ -133,7 +134,7 @@ class RuntimeChecks:
) )
payload["generation_question"] = generation_question payload["generation_question"] = generation_question
status = runtime.status() status = runtime.status()
adapter = resolve_runtime_adapter(proof_settings) adapter, _adapter_state, _adapter_reason = adapters.resolve_library_adapter(proof_settings)
generation = runtime.generate(generation_question, context) generation = runtime.generate(generation_question, context)
answer, citations, citation_report = Verifier().verify_answer(generation.text, context) answer, citations, citation_report = Verifier().verify_answer(generation.text, context)
claim_id = EvidenceGraphService(proof_settings, self.db).record_answer( claim_id = EvidenceGraphService(proof_settings, self.db).record_answer(
+68 -80
View File
@@ -2,6 +2,7 @@ from __future__ import annotations
import base64 import base64
import json import json
import logging
import subprocess import subprocess
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@@ -9,10 +10,14 @@ from typing import Any
import httpx import httpx
from . import adapters
from .config import Settings, get_settings from .config import Settings, get_settings
from .schemas import AdapterManifest, CompiledContext, RuntimeStatus, TrainingAdapterKind from .db import Database
from .schemas import CompiledContext, RuntimeStatus, TrainingAdapterKind
from .storage import ObjectStore from .storage import ObjectStore
logger = logging.getLogger("ely_eye.runtime")
class RuntimeUnavailableError(RuntimeError): class RuntimeUnavailableError(RuntimeError):
pass pass
@@ -34,12 +39,41 @@ class QwenRuntime:
def __init__(self, settings: Settings | None = None) -> None: def __init__(self, settings: Settings | None = None) -> None:
self.settings = settings or get_settings() self.settings = settings or get_settings()
self.objects = ObjectStore(self.settings) self.objects = ObjectStore(self.settings)
self.database = Database(self.settings)
self._processor: Any | None = None self._processor: Any | None = None
self._model: Any | None = None self._model: Any | None = None
def _adapter_choice(self) -> tuple[adapters.AdapterSource | None, str]:
"""The adapter this runtime would attach, plus the reason if it would not attach one."""
sources, _ = adapters.discover(self.settings, self.database)
source, _state, reason = adapters.resolve(sources, self.settings, TrainingAdapterKind.hme_core_lora)
return source, reason
def _loaded_adapter(self) -> adapters.LoadRecord | None:
return next((item for item in adapters.load_records() if item.kind == TrainingAdapterKind.hme_core_lora), None)
def _generation_adapter_fields(self) -> dict[str, str | None]:
record = self._loaded_adapter()
if record is not None:
return {
"adapter_path": record.path,
"adapter_kind": record.kind.value,
"adapter_id": record.adapter_id,
"adapter_sha256": record.sha256,
}
source, _ = self._adapter_choice()
if source is None:
return {"adapter_path": None, "adapter_kind": None, "adapter_id": None, "adapter_sha256": None}
return {
"adapter_path": str(source.path),
"adapter_kind": source.manifest.kind.value,
"adapter_id": source.manifest.adapter_id,
"adapter_sha256": source.manifest.sha256,
}
def status(self) -> RuntimeStatus: def status(self) -> RuntimeStatus:
gpu = gpu_status() gpu = gpu_status()
adapter = resolve_runtime_adapter(self.settings) fields = self._generation_adapter_fields()
if self.settings.runtime_backend == "sglang": if self.settings.runtime_backend == "sglang":
available, detail = self._sglang_status() available, detail = self._sglang_status()
else: else:
@@ -49,10 +83,10 @@ class QwenRuntime:
model_id=self.settings.model_id, model_id=self.settings.model_id,
available=available, available=available,
detail=detail, detail=detail,
adapter_path=str(adapter.path) if adapter else None, adapter_path=fields["adapter_path"],
adapter_kind=adapter.manifest.kind if adapter else None, adapter_kind=TrainingAdapterKind(fields["adapter_kind"]) if fields["adapter_kind"] else None,
adapter_id=adapter.manifest.adapter_id if adapter else None, adapter_id=fields["adapter_id"],
adapter_sha256=adapter.manifest.sha256 if adapter else None, adapter_sha256=fields["adapter_sha256"],
gpu_name=gpu.get("name"), gpu_name=gpu.get("name"),
cuda=gpu.get("cuda"), cuda=gpu.get("cuda"),
vram_total_mb=gpu.get("total_mb"), vram_total_mb=gpu.get("total_mb"),
@@ -84,15 +118,17 @@ class QwenRuntime:
import torch import torch
adapter_detail = "" adapter_detail = ""
adapter = resolve_runtime_adapter(self.settings) source, reason = self._adapter_choice()
if adapter is not None: if source is not None:
adapter_detail = f" Adapter selected: {adapter.path}." adapter_detail = f" Adapter selected: {source.path}."
elif self.settings.adapter_dir is not None: elif self.settings.adapter_dir is not None:
return False, f"Configured adapter path is missing: {self.settings.adapter_dir}" return False, f"Configured adapter path is missing: {self.settings.adapter_dir}"
if source is None:
adapter_detail = f" No adapter: {reason}."
if torch.cuda.is_available(): if torch.cuda.is_available():
return True, f"Transformers backend can use CUDA.{adapter_detail}" return True, f"Transformers backend can use CUDA.{adapter_detail}"
return True, f"Transformers backend is available on CPU.{adapter_detail}" return True, f"Transformers backend is available on CPU.{adapter_detail}"
except Exception as exc: except Exception as exc: # noqa: BLE001 # a status probe reports why the backend is unusable
return False, f"Transformers backend import failed: {exc}" return False, f"Transformers backend import failed: {exc}"
def _generate_sglang(self, question: str, context: CompiledContext) -> RuntimeGeneration: def _generate_sglang(self, question: str, context: CompiledContext) -> RuntimeGeneration:
@@ -122,15 +158,11 @@ class QwenRuntime:
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
raise RuntimeUnavailableError(f"SGLang generation failed: {exc}") from exc raise RuntimeUnavailableError(f"SGLang generation failed: {exc}") from exc
data = response.json() data = response.json()
adapter = resolve_runtime_adapter(self.settings)
usage = data.get("usage") if isinstance(data, dict) else None usage = data.get("usage") if isinstance(data, dict) else None
return RuntimeGeneration( return RuntimeGeneration(
text=data["choices"][0]["message"]["content"], text=data["choices"][0]["message"]["content"],
backend="sglang", backend="sglang",
adapter_path=str(adapter.path) if adapter else None, **self._generation_adapter_fields(),
adapter_kind=adapter.manifest.kind.value if adapter else None,
adapter_id=adapter.manifest.adapter_id if adapter else None,
adapter_sha256=adapter.manifest.sha256 if adapter else None,
input_tokens=usage.get("prompt_tokens") if isinstance(usage, dict) else None, input_tokens=usage.get("prompt_tokens") if isinstance(usage, dict) else None,
output_tokens=usage.get("completion_tokens") if isinstance(usage, dict) else None, output_tokens=usage.get("completion_tokens") if isinstance(usage, dict) else None,
) )
@@ -161,15 +193,11 @@ class QwenRuntime:
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
raise RuntimeUnavailableError(f"SGLang generation failed: {exc}") from exc raise RuntimeUnavailableError(f"SGLang generation failed: {exc}") from exc
data = response.json() data = response.json()
adapter = resolve_runtime_adapter(self.settings)
usage = data.get("usage") if isinstance(data, dict) else None usage = data.get("usage") if isinstance(data, dict) else None
return RuntimeGeneration( return RuntimeGeneration(
text=data["choices"][0]["message"]["content"], text=data["choices"][0]["message"]["content"],
backend="sglang", backend="sglang",
adapter_path=str(adapter.path) if adapter else None, **self._generation_adapter_fields(),
adapter_kind=adapter.manifest.kind.value if adapter else None,
adapter_id=adapter.manifest.adapter_id if adapter else None,
adapter_sha256=adapter.manifest.sha256 if adapter else None,
input_tokens=usage.get("prompt_tokens") if isinstance(usage, dict) else None, input_tokens=usage.get("prompt_tokens") if isinstance(usage, dict) else None,
output_tokens=usage.get("completion_tokens") if isinstance(usage, dict) else None, output_tokens=usage.get("completion_tokens") if isinstance(usage, dict) else None,
) )
@@ -218,14 +246,10 @@ class QwenRuntime:
input_length = int(inputs["input_ids"].shape[-1]) input_length = int(inputs["input_ids"].shape[-1])
generated = generated[:, input_length:] generated = generated[:, input_length:]
decoded = processor.batch_decode(generated, skip_special_tokens=True)[0] decoded = processor.batch_decode(generated, skip_special_tokens=True)[0]
adapter = resolve_runtime_adapter(self.settings)
return RuntimeGeneration( return RuntimeGeneration(
text=decoded, text=decoded,
backend="transformers", backend="transformers",
adapter_path=str(adapter.path) if adapter else None, **self._generation_adapter_fields(),
adapter_kind=adapter.manifest.kind.value if adapter else None,
adapter_id=adapter.manifest.adapter_id if adapter else None,
adapter_sha256=adapter.manifest.sha256 if adapter else None,
input_tokens=input_length, input_tokens=input_length,
output_tokens=int(generated.shape[-1]), output_tokens=int(generated.shape[-1]),
) )
@@ -273,14 +297,10 @@ class QwenRuntime:
input_length = int(inputs["input_ids"].shape[-1]) input_length = int(inputs["input_ids"].shape[-1])
generated = generated[:, input_length:] generated = generated[:, input_length:]
decoded = processor.batch_decode(generated, skip_special_tokens=True)[0] decoded = processor.batch_decode(generated, skip_special_tokens=True)[0]
adapter = resolve_runtime_adapter(self.settings)
return RuntimeGeneration( return RuntimeGeneration(
text=decoded, text=decoded,
backend="transformers", backend="transformers",
adapter_path=str(adapter.path) if adapter else None, **self._generation_adapter_fields(),
adapter_kind=adapter.manifest.kind.value if adapter else None,
adapter_id=adapter.manifest.adapter_id if adapter else None,
adapter_sha256=adapter.manifest.sha256 if adapter else None,
input_tokens=input_length, input_tokens=input_length,
output_tokens=int(generated.shape[-1]), output_tokens=int(generated.shape[-1]),
) )
@@ -288,6 +308,19 @@ class QwenRuntime:
def _load_transformers(self) -> tuple[Any, Any]: def _load_transformers(self) -> tuple[Any, Any]:
if self._model is not None and self._processor is not None: if self._model is not None and self._processor is not None:
return self._model, self._processor return self._model, self._processor
model, processor = self._load_base_model()
source, reason = self._adapter_choice()
if source is None:
logger.info("generating without an adapter: %s", reason)
else:
model = adapters.load_into_model(model, source.path)
adapters.record_load(source)
model.eval()
self._model, self._processor = model, processor
return model, processor
def _load_base_model(self) -> tuple[Any, Any]:
"""Load the chat model; tests replace this so the adapter branch runs without a 9B download."""
import torch import torch
from transformers import AutoProcessor from transformers import AutoProcessor
@@ -300,33 +333,25 @@ class QwenRuntime:
model_cls = AutoModelForCausalLM model_cls = AutoModelForCausalLM
self._processor = AutoProcessor.from_pretrained(self.settings.model_id, trust_remote_code=True) processor = AutoProcessor.from_pretrained(self.settings.model_id, trust_remote_code=True)
self._model = model_cls.from_pretrained( model = model_cls.from_pretrained(
self.settings.model_id, self.settings.model_id,
dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
device_map="auto", device_map="auto",
trust_remote_code=True, trust_remote_code=True,
) )
adapter = resolve_runtime_adapter(self.settings) return model, processor
if adapter is not None:
from .training import bitsandbytes_gaudi_probe_guard
with bitsandbytes_gaudi_probe_guard():
from peft import PeftModel
self._model = PeftModel.from_pretrained(self._model, adapter.path)
self._model.eval()
return self._model, self._processor
def unload(self) -> None: def unload(self) -> None:
self._model = None self._model = None
self._processor = None self._processor = None
adapters.clear_loads()
try: try:
import torch import torch
if torch.cuda.is_available(): if torch.cuda.is_available():
torch.cuda.empty_cache() torch.cuda.empty_cache()
except Exception: except Exception: # noqa: BLE001 # freeing the cache is best-effort, unloading must not raise
return return
def _context_images(self, context: CompiledContext) -> list[Path]: def _context_images(self, context: CompiledContext) -> list[Path]:
@@ -338,43 +363,6 @@ class QwenRuntime:
return paths return paths
@dataclass(frozen=True)
class RuntimeAdapter:
path: Path
manifest: AdapterManifest
def resolve_runtime_adapter(
settings: Settings,
kind: TrainingAdapterKind = TrainingAdapterKind.hme_core_lora,
) -> RuntimeAdapter | None:
if settings.adapter_dir is not None:
path = settings.adapter_dir.expanduser().resolve()
manifest_path = path / "adapter_manifest.json"
if path.exists() and manifest_path.exists():
return RuntimeAdapter(
path=path,
manifest=AdapterManifest.model_validate_json(manifest_path.read_text(encoding="utf-8")),
)
return None
candidates: list[RuntimeAdapter] = []
for manifest_path in settings.adapters_dir.glob("*/adapter_manifest.json"):
manifest = AdapterManifest.model_validate_json(manifest_path.read_text(encoding="utf-8"))
if manifest.kind == kind and manifest.base_model == settings.model_id:
candidates.append(RuntimeAdapter(path=manifest_path.parent.resolve(), manifest=manifest))
candidates.sort(
key=lambda item: (
item.manifest.cartridge_id is not None,
(item.path / "adapter_model.safetensors").stat().st_mtime
if (item.path / "adapter_model.safetensors").exists()
else 0.0,
),
reverse=True,
)
return candidates[0] if candidates else None
def build_prompt(question: str, context: CompiledContext) -> str: def build_prompt(question: str, context: CompiledContext) -> str:
contract = "\n".join(f"- {item}" for item in context.plan.verifier_contract) contract = "\n".join(f"- {item}" for item in context.plan.verifier_contract)
return f"""You are Ely-Eye, a local multimodal context operating system. return f"""You are Ely-Eye, a local multimodal context operating system.
+30
View File
@@ -0,0 +1,30 @@
"""Runtime observability shapes: what the model holds in memory right now."""
from typing import Literal
from pydantic import BaseModel
from .schemas import TrainingAdapterKind
AdapterState = Literal["loaded", "available", "deferred", "missing"]
AdapterOrigin = Literal["library", "cartridge"]
class AdapterChoice(BaseModel):
"""One adapter kind, and the honest reason it is or is not attached to the model."""
kind: TrainingAdapterKind
state: AdapterState
reason: str
path: str | None = None
adapter_id: str | None = None
base_model: str | None = None
origin: AdapterOrigin | None = None
cartridge_id: str | None = None
shadowed: bool = False
class AdapterLoadReport(BaseModel):
model_id: str
mounted: list[str]
adapters: list[AdapterChoice]
+227
View File
@@ -0,0 +1,227 @@
"""Which adapter the runtime loads, and the reason every other kind does not."""
from __future__ import annotations
import json
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from unittest import mock
import pytest
from fastapi.testclient import TestClient
from ely_eye import adapters, api
from ely_eye.cartridge_store import set_mounts
from ely_eye.config import Settings
from ely_eye.db import Database
from ely_eye.runtime import QwenRuntime
from ely_eye.schemas import CartridgeManifest, TrainingAdapterKind
CORE = TrainingAdapterKind.hme_core_lora
MODEL = "Qwen/Qwen3.5-9B"
def _manifest(kind: TrainingAdapterKind, base_model: str, adapter_id: str) -> dict[str, Any]:
return {
"adapter_id": adapter_id,
"kind": kind.value,
"created_at": "2026-05-01T00:00:00+00:00",
"base_model": base_model,
"dataset_path": "PRD.md",
"dataset_sha256": "d" * 64,
"sample_count": 8,
"max_steps": 16,
"max_length": 512,
"rank": 8,
"alpha": 16,
"target_modules": ["q_proj"],
"trainable_params": 1_000,
"total_params": 10_000,
"optimizer": "adamw_8bit",
"learning_rate": 0.0002,
"gradient_accumulation_steps": 1,
"bf16": True,
"final_loss": 0.01,
"artifacts": {},
"sha256": "a" * 64,
}
def _write_adapter(
root: Path,
kind: TrainingAdapterKind = CORE,
base_model: str = MODEL,
adapter_id: str = "adapter-1",
weights: bool = True,
) -> Path:
root.mkdir(parents=True, exist_ok=True)
manifest = _manifest(kind, base_model, adapter_id)
(root / "adapter_manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
if weights:
(root / "adapter_model.safetensors").write_bytes(b"stub weights")
return root
def _settings(tmp_path: Path) -> Settings:
settings = Settings(home=tmp_path / "home", model_id=MODEL)
settings.ensure_dirs()
return settings
def _cartridge(settings: Settings, database: Database, cartridge_id: str) -> Path:
root = settings.home / "cartridges" / cartridge_id
root.mkdir(parents=True, exist_ok=True)
now = datetime.now(UTC)
manifest = CartridgeManifest(
cartridge_id=cartridge_id,
name="home",
created_at=now,
model_base=settings.model_id,
source_count=0,
atom_count=0,
token_equivalent=0,
artifacts={},
dna=None,
)
database.write_cartridge(cartridge_id, "home", root, manifest.model_dump_json(), None, 0, now)
set_mounts(database, [cartridge_id], now)
return root
class _FakeModel:
def __init__(self) -> None:
self.evalled = False
def eval(self) -> None:
self.evalled = True
@pytest.fixture(autouse=True)
def _no_stale_loads():
adapters.clear_loads()
yield
adapters.clear_loads()
def test_a_library_adapter_is_selected_for_its_kind(tmp_path: Path) -> None:
settings = _settings(tmp_path)
chosen = _write_adapter(settings.adapters_dir / "core")
source, state, reason = adapters.resolve_library_adapter(settings)
assert source is not None and source.path == chosen
assert state == "available"
assert reason == "selected from the adapter library"
def test_an_adapter_on_another_base_model_is_deferred_naming_both_models(tmp_path: Path) -> None:
settings = _settings(tmp_path)
_write_adapter(settings.adapters_dir / "router", kind=TrainingAdapterKind.hme_router, base_model="Qwen/Qwen3.5-0.8B")
source, state, reason = adapters.resolve_library_adapter(settings, TrainingAdapterKind.hme_router)
assert source is None
assert state == "deferred"
assert "trained on Qwen/Qwen3.5-0.8B" in reason and f"runtime holds {MODEL}" in reason
def test_a_manifest_without_weights_is_deferred_naming_the_missing_file(tmp_path: Path) -> None:
settings = _settings(tmp_path)
_write_adapter(settings.adapters_dir / "core", weights=False)
source, state, reason = adapters.resolve_library_adapter(settings)
assert source is None and state == "deferred"
assert "adapter_model.safetensors" in reason
def test_the_mounted_cartridge_copy_wins_over_the_library(tmp_path: Path) -> None:
settings = _settings(tmp_path)
database = Database(settings)
_write_adapter(settings.adapters_dir / "core", adapter_id="library-copy")
cartridge_root = _cartridge(settings, database, "cart-home")
cartridge_copy = _write_adapter(cartridge_root / "adapters" / "hme_core_lora", adapter_id="cartridge-copy")
sources, mounted = adapters.discover(settings, database)
assert mounted == ["cart-home"]
source, state, reason = adapters.resolve(sources, settings, CORE)
assert source is not None and source.path == cartridge_copy
assert state == "available" and "mounted cartridge cart-home" in reason
report = adapters.plan(settings, database)
row = next(item for item in report.adapters if item.kind == CORE)
assert row.adapter_id == "cartridge-copy" and row.cartridge_id == "cart-home"
def test_the_endpoint_lists_every_kind_with_a_reason(tmp_path: Path, monkeypatch) -> None:
settings = _settings(tmp_path)
_write_adapter(settings.adapters_dir / "core")
monkeypatch.setattr(api, "get_settings", lambda: settings)
client = TestClient(api.create_app())
body = client.get("/api/runtime/adapters").json()
assert body["model_id"] == MODEL and body["mounted"] == []
by_kind = {row["kind"]: row for row in body["adapters"]}
assert set(by_kind) == {kind.value for kind in TrainingAdapterKind}
assert by_kind[CORE.value]["state"] == "available"
for kind in TrainingAdapterKind:
if kind == CORE:
continue
assert by_kind[kind.value]["state"] == "missing"
assert by_kind[kind.value]["reason"], f"{kind.value} carries no reason"
def test_stub_weights_prove_the_load_branch_runs(tmp_path: Path, monkeypatch) -> None:
settings = _settings(tmp_path)
chosen = _write_adapter(settings.adapters_dir / "core")
attached: list[tuple[Any, Path]] = []
def fake_attach(model: Any, path: Path) -> _FakeModel:
attached.append((model, path))
return _FakeModel()
monkeypatch.setattr(adapters, "load_into_model", fake_attach)
monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), object()))
runtime = QwenRuntime(settings)
model, _processor = runtime._load_transformers()
assert attached == [(mock.ANY, chosen)], "peft was never asked to attach the stub adapter"
assert model.evalled is True
assert [record.adapter_id for record in adapters.load_records()] == ["adapter-1"]
row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE)
assert row.state == "loaded" and "attached to the live model" in row.reason
def test_unload_clears_the_loaded_state(tmp_path: Path, monkeypatch) -> None:
settings = _settings(tmp_path)
_write_adapter(settings.adapters_dir / "core")
monkeypatch.setattr(adapters, "load_into_model", lambda model, path: _FakeModel())
monkeypatch.setattr(QwenRuntime, "_load_base_model", lambda self: (_FakeModel(), object()))
runtime = QwenRuntime(settings)
runtime._load_transformers()
assert adapters.load_records(), "nothing was recorded as loaded"
runtime.unload()
assert adapters.load_records() == []
row = next(item for item in adapters.plan(settings, Database(settings)).adapters if item.kind == CORE)
assert row.state == "available"
def test_the_loser_of_the_choice_is_reported_as_shadowed(tmp_path: Path) -> None:
settings = _settings(tmp_path)
_write_adapter(settings.adapters_dir / "first", adapter_id="written_first")
_write_adapter(settings.adapters_dir / "second", adapter_id="written_second")
report = adapters.plan(settings, Database(settings))
losers = [row for row in report.adapters if row.shadowed]
assert [row.adapter_id for row in losers] == ["written_first"]
assert "another adapter of this kind wins" in losers[0].reason
winner = next(row for row in report.adapters if row.adapter_id == "written_second")
assert winner.shadowed is False and winner.state == "available"
def test_an_unreadable_manifest_names_the_file(tmp_path: Path) -> None:
settings = _settings(tmp_path)
broken = settings.adapters_dir / "broken"
broken.mkdir(parents=True)
(broken / "adapter_manifest.json").write_text("{not json", encoding="utf-8")
with pytest.raises(ValueError, match="the adapter manifest is not a valid AdapterManifest"):
adapters.library_sources(settings)
def test_a_pinned_adapter_dir_without_a_manifest_says_the_path_is_missing(tmp_path: Path) -> None:
settings = Settings(home=tmp_path / "home", model_id=MODEL, adapter_dir=tmp_path / "empty")
settings.ensure_dirs()
assert adapters.library_sources(settings) == []
available, detail = QwenRuntime(settings)._transformers_status()
assert available is False
assert detail == f"Configured adapter path is missing: {tmp_path / 'empty'}"