perf(agents): cut codex usage collector memory with SQL filter and cache (#6780)
* perf(agents): cut codex usage collector memory with SQL filter and cache The codex collector scanned every row of opencode.db (1.7 GB, 55k+ rows) with Python-side json.loads, peaking around 716 MB of RSS on every run -- including the panel's refreshLimits() call, which passed --limits-only that the collector silently ignored. Filter rows in SQL (LIKE gates + json_valid + json_extract authority, mirroring the old Python filter semantics) so giant blobs are never parsed, and cache the local stats scan in XDG_CACHE_HOME following the claude collector's pattern (atomic writes, flock, schemaVersion). --force rescans, --limits-only and normal mode reuse a fresh cache and fall back to a full scan when it is missing, stale, or corrupt. Measured: cold scan 716 MB -> 158 MB peak; warm --limits-only ~85 MB and ~1.4 s. Output record schema and values are unchanged for the same data (parity verified against the old filter, including malformed rows). * Scope the codex scan cache's 15-minute reuse to --limits-only A no-flag run is the widget's periodic refresh, and refreshIntervalSec is configurable down to 30 seconds; holding every mode to a 15-minute cache meant stats could lag far behind the interval the user asked for. Mirror the claude collector: normal runs reuse a scan for ~20 seconds purely to dedup concurrent collectors, and only --limits-only, which promises just fresh limits, may reuse a scan for up to 15 minutes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Invalidate the codex scan cache across day boundaries The cached stats embed date-dependent fields (todayPrompts, todayTotalTokens, recentDays), but only the file's age was checked, so a cache written at 23:58 served yesterday's numbers as "today" for up to 15 minutes past midnight. Stamp the envelope with the scan's local date and treat any other date as a miss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reject codex scan caches with a future mtime A cache whose mtime is ahead of the clock has a negative age, which the freshness check accepted forever: setting the clock backwards froze the stats until real time caught up with the file. Require a non-negative age before trusting the cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Never cache an interrupted opencode scan A transient lock, schema migration, or corrupted database aborts the opencode scan mid-flight; the partial numbers still serve the current run, but persisting them let a single bad read suppress opencode usage for every cache reader until expiry. The claude collector already skips its opencode cache write on a database error; do the same here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make the json_valid guard order explicit in the opencode query The query relied on json_valid(data) evaluating before json_extract(), but SQLite does not promise that AND terms run left to right; a reordered plan would let json_extract raise on a malformed row and silently truncate the scan. Wrap each json_extract in a CASE so the guard is structural rather than positional. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drop a claude-collector comment that is false for codex "These caches were world-readable before" was copied from the claude collector; codex had no caches before this one existed. Explain the chmod on its own terms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: markbusking <marcosbustos.dev@gmail.com> Co-authored-by: David Heinemeier Hansson <david@hey.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
markbus-ai
co-authored by
Claude Fable 5
markbusking
David Heinemeier Hansson
Claude Fable 5
markbusking
David Heinemeier Hansson
parent
929849c7b9
commit
25c2b3aed2
@@ -128,3 +128,465 @@ pass "Codex collector counts OpenAI usage, reasoning included, from opencode ses
|
||||
[[ $(jq -c '.modelUsage' <<<"$result") == '{"gpt-5.2-codex":{"inputTokens":80,"outputTokens":45,"cacheReadInputTokens":30,"cacheCreationInputTokens":0}}' ]] ||
|
||||
fail "Codex collector ignores prefix-colliding providers, user messages, and malformed rows" "$result"
|
||||
pass "Codex collector ignores prefix-colliding providers, user messages, and malformed rows"
|
||||
|
||||
# A warm cache makes --limits-only cheap: local stats come from the last scan
|
||||
# instead of another walk over the opencode database, and --force bypasses it.
|
||||
CACHE_HOME=$(mktemp -d)
|
||||
trap 'rm -rf "$TEST_HOME" "$PI_HOME" "$OPENCODE_HOME" "$CACHE_HOME" "$FRESH_HOME"' EXIT
|
||||
mkdir -p "$CACHE_HOME/bin"
|
||||
cp "$TEST_HOME/bin/codex" "$CACHE_HOME/bin/codex"
|
||||
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE message (id text PRIMARY KEY, session_id text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, data text NOT NULL)")
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
def message(id, provider, model, role="assistant", input=0, output=0, reasoning=0, read=0, write=0):
|
||||
return (id, "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": role,
|
||||
"providerID": provider,
|
||||
"modelID": model,
|
||||
"tokens": {"input": input, "output": output, "reasoning": reasoning, "cache": {"read": read, "write": write}},
|
||||
"time": {"created": now_ms},
|
||||
}))
|
||||
|
||||
conn.executemany("INSERT INTO message VALUES (?, ?, ?, ?, ?)", [
|
||||
message("c_1", "openai", "gpt-5.2-codex", input=5),
|
||||
])
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "5" ]] ||
|
||||
fail "Codex collector writes a fresh local-stats cache on first scan" "$result"
|
||||
cache_file=$(ls "$CACHE_HOME/.cache/omarchy/agent-usage/"/codex-scan-*.json 2>/dev/null | head -n 1)
|
||||
[[ -n $cache_file && -s $cache_file ]] ||
|
||||
fail "Codex collector leaves a cache file behind" "$result"
|
||||
[[ $(stat -c %a "$cache_file") == "644" ]] ||
|
||||
fail "Codex collector keeps cache files readable" "$result"
|
||||
[[ $(jq -r '.schemaVersion' "$cache_file") == "1" && $(jq -r '.stats.todayTotalTokens' "$cache_file") == "5" ]] ||
|
||||
fail "Codex collector writes a versioned cache envelope" "$result"
|
||||
pass "Codex collector writes a local-stats cache on first scan"
|
||||
|
||||
# A corrupt-but-parseable cache (wrong shape) is a cache miss: rescan and
|
||||
# rewrite instead of emitting a garbage record.
|
||||
printf '[]' >"$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "5" ]] ||
|
||||
fail "Codex collector recovers from a corrupt cache file" "$result"
|
||||
[[ $(jq -r '.schemaVersion' "$cache_file") == "1" ]] ||
|
||||
fail "Codex collector rewrites the cache after a corrupt read" "$result"
|
||||
pass "Codex collector recovers from a corrupt cache file"
|
||||
|
||||
# A new opencode message changes what a scan would find; a --limits-only run
|
||||
# must reuse the cached stats instead of rescanning.
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"c_2", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 10, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "5" ]] ||
|
||||
fail "Codex collector --limits-only reuses cached local stats" "$result"
|
||||
[[ $(jq -c '.modelUsage' <<<"$result") == '{"gpt-5.2-codex":{"inputTokens":5,"outputTokens":0,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}}' ]] ||
|
||||
fail "Codex collector --limits-only emits a complete record from cache" "$result"
|
||||
pass "Codex collector --limits-only reuses cached local stats"
|
||||
|
||||
# --force must ignore the cache and pick up the new message.
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --force)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "15" ]] ||
|
||||
fail "Codex collector --force rescans past the cache" "$result"
|
||||
pass "Codex collector --force rescans past the cache"
|
||||
|
||||
# The forced scan refreshed the cache, so a following --limits-only sees it.
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "15" ]] ||
|
||||
fail "Codex collector --limits-only sees a refreshed cache after --force" "$result"
|
||||
pass "Codex collector --limits-only sees a refreshed cache after --force"
|
||||
|
||||
# An expired cache makes --limits-only rescan too: stale today* stats must
|
||||
# never be served under a fresh updatedAt.
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"c_3", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 10, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
touch -d "2 hours ago" "$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "25" ]] ||
|
||||
fail "Codex collector --limits-only rescans when the cache is stale" "$result"
|
||||
pass "Codex collector --limits-only rescans when the cache is stale"
|
||||
|
||||
# The 15-minute reuse window belongs to --limits-only alone. A no-flag run
|
||||
# (the widget's periodic refresh) reuses a scan only while it is young enough
|
||||
# to be a concurrent collector run; past that it rescans, so stats stay as
|
||||
# fresh as refreshIntervalSec, however low the user sets it.
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"c_4", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 10, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "25" ]] ||
|
||||
fail "Codex collector no-flag reuses a seconds-old cache" "$result"
|
||||
|
||||
# 30 seconds is the lowest refreshIntervalSec the widget supports, so a
|
||||
# cache that old must already be past the no-flag reuse window.
|
||||
touch -d "30 seconds ago" "$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "35" ]] ||
|
||||
fail "Codex collector no-flag rescans past the concurrent-run window" "$result"
|
||||
pass "Codex collector no-flag mode rescans instead of serving a stale cache"
|
||||
|
||||
# The same age from the other side: a cache far past the no-flag window but
|
||||
# well inside 15 minutes is still good enough for --limits-only.
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"c_5", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 10, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
touch -d "10 minutes ago" "$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "35" ]] ||
|
||||
fail "Codex collector --limits-only reuses a scan the no-flag mode would refresh" "$result"
|
||||
pass "Codex collector --limits-only reuses a scan the no-flag mode would refresh"
|
||||
|
||||
# A cache written on another local date holds another day's today* stats even
|
||||
# under a fresh mtime (midnight passed, or the clock moved backwards): the
|
||||
# envelope's scanDate must turn it into a miss.
|
||||
jq -c '.scanDate = "1999-01-01"' "$cache_file" >"$cache_file.tmp" && mv "$cache_file.tmp" "$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "45" ]] ||
|
||||
fail "Codex collector treats a cache from another day as a miss" "$result"
|
||||
[[ $(jq -r '.scanDate' "$cache_file") == "$(date +%Y-%m-%d)" ]] ||
|
||||
fail "Codex collector stamps the rewritten cache with the scan date" "$result"
|
||||
pass "Codex collector treats a cache from another day as a miss"
|
||||
|
||||
# A cache stamped in the future (the clock was set backwards after the write)
|
||||
# has no trustworthy age: it must be a miss, not fresh until the clock
|
||||
# catches up.
|
||||
python3 - "$CACHE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"c_6", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 10, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
touch -d "@$(( $(date +%s) + 3600 ))" "$cache_file"
|
||||
result=$(HOME="$CACHE_HOME" CODEX_HOME="$CACHE_HOME/.codex" XDG_CACHE_HOME="$CACHE_HOME/.cache" XDG_DATA_HOME="$CACHE_HOME/.local/share" \
|
||||
PATH="$CACHE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "55" ]] ||
|
||||
fail "Codex collector treats a future-dated cache as a miss" "$result"
|
||||
pass "Codex collector treats a future-dated cache as a miss"
|
||||
|
||||
# First --limits-only on a machine with no cache falls back to a full scan.
|
||||
FRESH_HOME=$(mktemp -d)
|
||||
trap 'rm -rf "$TEST_HOME" "$PI_HOME" "$OPENCODE_HOME" "$CACHE_HOME" "$FRESH_HOME"' EXIT
|
||||
mkdir -p "$FRESH_HOME/bin"
|
||||
cp "$TEST_HOME/bin/codex" "$FRESH_HOME/bin/codex"
|
||||
|
||||
python3 - "$FRESH_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE message (id text PRIMARY KEY, session_id text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, data text NOT NULL)")
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"f_1", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 7, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$FRESH_HOME" CODEX_HOME="$FRESH_HOME/.codex" XDG_CACHE_HOME="$FRESH_HOME/.cache" XDG_DATA_HOME="$FRESH_HOME/.local/share" \
|
||||
PATH="$FRESH_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "7" ]] ||
|
||||
fail "Codex collector --limits-only falls back to a full scan without a cache" "$result"
|
||||
pass "Codex collector --limits-only falls back to a full scan without a cache"
|
||||
|
||||
# A malformed opencode row must not abort the scan: json_valid() guards the
|
||||
# parse, so the good rows are still counted. Real opencode data also stores
|
||||
# compact JSON, so one row is serialized compactly here on purpose.
|
||||
MALFORMED_HOME=$(mktemp -d)
|
||||
trap 'rm -rf "$TEST_HOME" "$PI_HOME" "$OPENCODE_HOME" "$CACHE_HOME" "$FRESH_HOME" "$MALFORMED_HOME"' EXIT
|
||||
mkdir -p "$MALFORMED_HOME/bin"
|
||||
cp "$TEST_HOME/bin/codex" "$MALFORMED_HOME/bin/codex"
|
||||
|
||||
python3 - "$MALFORMED_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE message (id text PRIMARY KEY, session_id text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, data text NOT NULL)")
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
def message(id, input=0, compact=False):
|
||||
payload = {
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": input, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}
|
||||
if compact:
|
||||
return (id, "ses_1", now_ms, now_ms, json.dumps(payload, separators=(",", ":")))
|
||||
return (id, "ses_1", now_ms, now_ms, json.dumps(payload))
|
||||
|
||||
conn.executemany("INSERT INTO message VALUES (?, ?, ?, ?, ?)", [
|
||||
message("mm_1", input=5, compact=True),
|
||||
message("mm_2", input=7),
|
||||
])
|
||||
# Valid JSON followed by trailing garbage: without json_valid() this row
|
||||
# makes json_extract() raise and aborts the whole scan.
|
||||
good = json.dumps({"role": "assistant", "providerID": "openai", "modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 999, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms}})
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", ("mm_3", "ses_1", now_ms, now_ms, good + " trailing-garbage"))
|
||||
# Completely broken row: not JSON at all.
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", ("mm_4", "ses_1", now_ms, now_ms, "this is not json"))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$MALFORMED_HOME" CODEX_HOME="$MALFORMED_HOME/.codex" XDG_CACHE_HOME="$MALFORMED_HOME/.cache" XDG_DATA_HOME="$MALFORMED_HOME/.local/share" \
|
||||
PATH="$MALFORMED_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "12" ]] ||
|
||||
fail "Codex collector counts good opencode rows past malformed ones" "$result"
|
||||
pass "Codex collector counts good opencode rows past malformed ones"
|
||||
|
||||
# An unwritable cache must not kill the collector: the record is the contract.
|
||||
UNWRITABLE_HOME=$(mktemp -d)
|
||||
trap 'rm -rf "$TEST_HOME" "$PI_HOME" "$OPENCODE_HOME" "$CACHE_HOME" "$FRESH_HOME" "$MALFORMED_HOME" "$UNWRITABLE_HOME"' EXIT
|
||||
mkdir -p "$UNWRITABLE_HOME/bin"
|
||||
cp "$TEST_HOME/bin/codex" "$UNWRITABLE_HOME/bin/codex"
|
||||
|
||||
python3 - "$UNWRITABLE_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE message (id text PRIMARY KEY, session_id text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, data text NOT NULL)")
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"u_1", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 3, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
# XDG_CACHE_HOME points at a regular file, so mkdir inside cache_root fails.
|
||||
touch "$UNWRITABLE_HOME/not-a-dir"
|
||||
result=$(HOME="$UNWRITABLE_HOME" CODEX_HOME="$UNWRITABLE_HOME/.codex" XDG_CACHE_HOME="$UNWRITABLE_HOME/not-a-dir" XDG_DATA_HOME="$UNWRITABLE_HOME/.local/share" \
|
||||
PATH="$UNWRITABLE_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "3" ]] ||
|
||||
fail "Codex collector still prints a complete record when the cache is unwritable" "$result"
|
||||
pass "Codex collector still prints a complete record when the cache is unwritable"
|
||||
|
||||
# A scan cut short by a database error (schema migration, transient lock,
|
||||
# corruption) must not be cached as the whole story, or the missing usage
|
||||
# would be suppressed for every reader until the cache expires.
|
||||
INTERRUPTED_HOME=$(mktemp -d)
|
||||
trap 'rm -rf "$TEST_HOME" "$PI_HOME" "$OPENCODE_HOME" "$CACHE_HOME" "$FRESH_HOME" "$MALFORMED_HOME" "$UNWRITABLE_HOME" "$INTERRUPTED_HOME"' EXIT
|
||||
mkdir -p "$INTERRUPTED_HOME/bin"
|
||||
cp "$TEST_HOME/bin/codex" "$INTERRUPTED_HOME/bin/codex"
|
||||
|
||||
# A database without the message table makes the scan fail mid-flight.
|
||||
python3 - "$INTERRUPTED_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE unrelated (id text PRIMARY KEY)")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$INTERRUPTED_HOME" CODEX_HOME="$INTERRUPTED_HOME/.codex" XDG_CACHE_HOME="$INTERRUPTED_HOME/.cache" XDG_DATA_HOME="$INTERRUPTED_HOME/.local/share" \
|
||||
PATH="$INTERRUPTED_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex")
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "0" ]] ||
|
||||
fail "Codex collector reports what it could read from a broken database" "$result"
|
||||
[[ -z $(ls "$INTERRUPTED_HOME/.cache/omarchy/agent-usage/"codex-scan-*.json 2>/dev/null) ]] ||
|
||||
fail "Codex collector must not cache an interrupted scan" "$result"
|
||||
|
||||
# Once the database is whole again, the very next --limits-only run scans it
|
||||
# instead of reusing a zero snapshot.
|
||||
python3 - "$INTERRUPTED_HOME/.local/share/opencode/opencode.db" <<'PY'
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
db = Path(sys.argv[1])
|
||||
conn = sqlite3.connect(db)
|
||||
conn.execute("CREATE TABLE message (id text PRIMARY KEY, session_id text NOT NULL, time_created integer NOT NULL, time_updated integer NOT NULL, data text NOT NULL)")
|
||||
now_ms = int(time.time() * 1000)
|
||||
conn.execute("INSERT INTO message VALUES (?, ?, ?, ?, ?)", (
|
||||
"i_1", "ses_1", now_ms, now_ms, json.dumps({
|
||||
"role": "assistant",
|
||||
"providerID": "openai",
|
||||
"modelID": "gpt-5.2-codex",
|
||||
"tokens": {"input": 9, "output": 0, "reasoning": 0, "cache": {"read": 0, "write": 0}},
|
||||
"time": {"created": now_ms},
|
||||
}),
|
||||
))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
result=$(HOME="$INTERRUPTED_HOME" CODEX_HOME="$INTERRUPTED_HOME/.codex" XDG_CACHE_HOME="$INTERRUPTED_HOME/.cache" XDG_DATA_HOME="$INTERRUPTED_HOME/.local/share" \
|
||||
PATH="$INTERRUPTED_HOME/bin:$PATH" "$ROOT/bin/omarchy-agent-usage-codex" --limits-only)
|
||||
|
||||
[[ $(jq -r '.todayTotalTokens' <<<"$result") == "9" ]] ||
|
||||
fail "Codex collector does not reuse a snapshot from an interrupted scan" "$result"
|
||||
pass "Codex collector does not cache an interrupted opencode scan"
|
||||
|
||||
Reference in New Issue
Block a user