omarchy/docs/testing.md

# Testing

How the non-graphical test suites are organized: what each runner owns, the
protocol test files speak, and the conventions that keep them runnable on any
machine — including headless CI sandboxes with no compositor. The graphical
acceptance suite is a separate thing that drives a live session in a disposable
VM; see [`agents/skills/acceptance-tests.md`](../agents/skills/acceptance-tests.md).

## Suite map

`./test/all` runs both suites below and keeps going when one fails, so a single
failure cannot hide the other suite behind it. It reports the failed suites at
the end and exits non-zero.

- **`./test/cli`** — one big script, one suite. It owns the CLI router: help
  and group rendering, route resolution, aliases, hidden commands, and the
  guarantee that a trailing `--help` never executes the target. It also owns
  the metadata lint — every `omarchy-*` executable under `bin/` is checked for a
  `# omarchy:summary=` header and against removed or redundant fields — plus
  the theme pipeline: template rendering (`omarchy-theme-set-templates`,
  `omarchy-theme-color`, `omarchy-theme-osc`), the theme sync commands
  (tmux, GNOME, VS Code, Pi, Claude) run against stub binaries and a fake
  `$HOME`.
- **`./test/shell`** — runs every `test/shell.d/*-test.sh` (except
  `base-test.sh` itself). Each file is an independent suite covering one area:
  a shell plugin, a `bin/` command, a config invariant, or a still-live migration. This is where new tests go.
- **Acceptance** — everything that needs a real desktop doing real things.
  Deliberately excluded from `./test/all`; it runs in a VM, not the
  development session.

A new shell test only needs the right name: drop `<area>-test.sh` into
`test/shell.d/` and `./test/shell` picks it up automatically. Shared fixtures
live under `test/shell.d/fixtures/`.

## The base-test.sh contract

Every shell test starts the same way:

```bash
#!/bin/bash

set -euo pipefail

source "$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/base-test.sh"
```

`base-test.sh` refuses to be executed directly — it is a library. It discovers
the repo root from its own location and exports it as `ROOT`, so tests
reference files as `$ROOT/bin/...` and never depend on the caller's working
directory or an installed Omarchy.

Assertions are TAP-flavored and blunt:

- `pass "description"` prints `ok - description`.
- `skip "description"` prints `ok - description # SKIP`. Include why the check could not run. Like `pass`, it returns normally; use `exit 0` afterwards only when the rest of the file cannot run either.
- `fail "description" [detail]` prints the optional detail and
  `not ok - description` to stderr, then **exits the file**. There is no
  counting or continuing within a file: the first failed assertion ends it,
  which keeps later assertions from reporting against state the failure
  already invalidated.
- `require_command <cmd>` fails the file when a needed tool is absent.

The runner compensates for that early exit: `./test/shell` continues past a failing file and summarizes the failures at the end. Aborting the whole run at the first bad file once let a single packaging failure mask 114 of 134 files. Failure granularity is therefore per file inside a run, per assertion inside a file.

The runner also lists files with skipped checks, including files that ran some checks or later failed. Skips do not fail a run. When no files fail, a run with skips is reported as completed without failures rather than having passed every check.

## Compositor-dependent tests

Some tests launch Quickshell or query Hyprland, but the suite must stay green on headless machines. `require_compositor "description"` handles this: when no compositor answers it calls `skip` with the reason and exits 0, and otherwise returns so the file proceeds. The skip marker lets the runner distinguish unavailable runtime coverage from checks that passed.

The probe is more than an environment check, because `WAYLAND_DISPLAY` only
proves the variable was inherited. Sandboxes pass the environment through
while blocking `$XDG_RUNTIME_DIR`, so Quickshell clears a bare variable check
and then aborts inside QGuiApplication — a core dump per launch where a skip
belonged. So `compositor_reachable` checks the socket actually exists, then
asks Hyprland itself (`hyprctl -j monitors`, retried, and only when
`HYPRLAND_INSTANCE_SIGNATURE` makes it askable), since a compositor that died
mid-session leaves its socket behind. When the compositor is reachable,
`require_compositor` also sets `ulimit -c 0`: Quickshell leaves through
`qFatal()` if its connection drops mid-run, and the test should fail without
writing a core dump as debris.

Gate only what needs gating — put `require_compositor` in files whose runtime
half needs a live session, and keep static analysis of the same area in code
that runs unconditionally before or beside it.

## Unit-testing shell JavaScript from bash

The Quickshell plugins keep their logic in plain `.js` modules
(`shell/plugins/menu/MenuModel.js`, `bar/BarModel.js`, ...) that end in a
guarded `if (typeof module !== "undefined") module.exports = {...}` block. QML
imports them directly and ignores the guard; Node loads them as CommonJS. That
dual citizenship is what makes the shell's model logic unit-testable without a
compositor.

`run_node_test` is the bridge: it prepends a JS prelude to a heredoc and pipes
the result into `node`. The prelude mirrors the bash assertion protocol
(`pass`, `fail`, `assert`, `assertEqual`, `assertDeepEqual` — same
`ok`/`not ok` lines, same exit-on-first-failure) and provides `root` (from the
exported `ROOT`), `path`, and `requireFromRoot(relativePath)`:

```bash
run_node_test <<'JS'
const menu = requireFromRoot('shell/plugins/menu/MenuModel.js')

const parsed = menu.parseMenuJsonc('{ "items": { "root": { "label": "Go" }, }, }')
assertEqual(parsed.length, 1, 'menu parses JSONC with trailing commas')
JS
```

Roughly a quarter of the shell test files use this to test parsing, merging,
and layout logic as pure functions, reserving compositor-gated tests for what
only a live session can prove.

## Conventions worth copying

- **Redirect background output.** Send background fixtures' stdout to a log or `/dev/null` and clean up the processes on exit. An inherited output pipe can hold the runner open after the test exits.
- **Stub the world, run the real code.** Tests build a scratch `bin/` of stub
  executables (`sudo`, `tmux`, `gsettings`, helper commands) that log their
  arguments to a file, prepend it to `PATH`, and then run the real script
  under test. Assertions grep the call log and the files the script wrote.
- **Fake `$HOME`, real `$OMARCHY_PATH`.** Anything touching user state runs
  with `HOME` pointed at a `mktemp -d` directory (cleaned up via
  `trap ... EXIT`) and `OMARCHY_PATH="$ROOT"`, so tests exercise the checkout
  without touching the developer's machine.
- **Migrations run directly.** A migration test builds the legacy state in a
  fake `$HOME`, runs `bash -euo pipefail "$ROOT/migrations/<ts>.sh"`, and
  asserts the resulting state — including running it twice to prove
  idempotence, and once against non-legacy state to prove it leaves user
  customization alone. Keep that test while the migration is still being written or bugfixed, if it calls an Omarchy helper whose interface can still change, or if it is a security-sensitive privileged repair. Once a one-shot rewrite has shipped in a tagged release and is frozen, drop the test even when that rewrite used sudo, pacman, or limine-mkinitcpio. Keep the migration itself for late-updaters. Tests of `omarchy-migrate`, the login notifier, and `omarchy-upgrade-to-quattro` stay.
- **Assert the invariant, not the snapshot.** Config tests pin the property a
  test is named for (this widget stays adjacent to that one) rather than whole
  structures, so unrelated churn does not fail them.