David Heinemeier HanssonandGitHub
2cc3510d2a
* Offer an AI diagnosis when a process crashes systemd-coredump journals every core dump under a known MESSAGE_ID with the crashing program, pid, and signal as structured fields. omarchy-crash-watch follows that stream and raises a "Process crashed: <program>" toast; clicking it opens omarchy-agent-crash, which briefs the default agent on the crash. The toast goes through omarchy-notification-send --exec rather than a libnotify action, because the shell runs clicks from its own omarchy-exec hint and never emits ActionInvoked. It keeps the default "omarchy-action" app name too, the only one shouldBypassDnd() lets through -- a crash being the last notification worth swallowing. It stays quiet until an agent is configured, since a diagnosis is all it offers. The method lives in a diagnose-crash skill rather than the prompt, so it is edited in one place and works with whichever agent is default. It covers investigating the core, and reporting a confirmed Omarchy bug upstream: scoped to bugs Omarchy controls, searched for duplicates first, only with the user's agreement, and signed with the model and harness that produced it. A migration reaches existing installs, whose skill symlinks and unit enablement would otherwise sit behind one-time setup paths. * Let the diagnosis clean up the core it extracted "Do not modify or delete anything" contradicted the symbolization step right above it, which writes a core to a temp file and deletes it on exit. Read literally, the core survives -- and the same section warns it holds passwords and tokens. The prohibition is about the system, not about your own scratch. * Do not spend a crash toast on a dead notification server The shell owns org.freedesktop.Notifications, so its own crash takes the notification server down with it -- and a shell crash is exactly what you want told about. The toast was sent once into that gap and the dedupe window was recorded regardless, so the rest of the crash loop went quiet for a minute and `journalctl -n 0` never replays what was missed. It now waits for the restarted shell to reclaim the bus name, as omarchy-migrate-notify already does, and only a delivered toast starts the dedupe window.
4.1 KiB
name, description
| name | description |
|---|---|
| diagnose-crash | Diagnose why a program crashed on this machine, from a systemd-coredump core dump. Use when a process has segfaulted, aborted, or otherwise dumped core, when asked why an application crashed or disappeared, or when a "Process crashed:" desktop notification is acted on. Triggers: crash, segfault, SIGSEGV, SIGABRT, core dump, coredumpctl, "why did X crash", "X keeps crashing", backtrace symbolization. Covers reporting a confirmed Omarchy bug upstream — see reporting.md. |
Diagnosing a Crash
Work from evidence. The goal is an honest account of what happened, not a plausible-sounding story.
Establish the facts
coredumpctl info <pid> is the starting point. Beyond the backtrace, note the
command line the process was started with — it usually reveals what the
program was working on when it died, which is often the whole answer.
coredumpctl list shows whether this crash is a one-off or a pattern. Repeated
crashes of the same program, or several programs dying together, point somewhere
different than a single failure does.
Rule out the boring causes first
Check resource exhaustion before blaming the program: free -h, and the journal
for OOM kills. A process killed by the OOM killer is not a bug in that process.
Correlate against the timeline
The crash timestamp is the most underused piece of evidence. Compare it against:
- Filesystem mtimes. A directory or file whose mtime lands on the same second as the crash strongly suggests what triggered it.
- The journal around that moment, for related warnings from the same or neighbouring processes.
- Recent package updates. A crash that starts right after an update points at the update.
Read the whole core, not just frame 0
Thread stacks other than the crashing one show what work was in flight — thumbnailers, image loaders, IPC readers, GPU queues. That context often explains the trigger even when the crashing frame itself cannot be symbolized.
Note any third-party code in the address space: file-manager or browser extensions, plugins, out-of-tree drivers. In-process third-party code is a common crash source and worth flagging — but do not pin blame on it without evidence that it is actually implicated.
Symbolize when you can
This is Arch, which runs a public debuginfod server:
core=$(mktemp -t crash-XXXXXX.core)
trap 'rm -f "$core"' EXIT
coredumpctl dump <pid> --output="$core"
DEBUGINFOD_URLS="https://debuginfod.archlinux.org" \
gdb -q <executable> "$core" \
-batch -ex 'set debuginfod enabled on' -ex 'bt'
A core is a verbatim copy of the process's memory and can hold passwords, tokens,
and private documents. Write it to a fresh mktemp path rather than a predictable
shared one, and delete it when you are done — never leave it lying in /tmp.
Many packages publish no debug symbols. When frames stay unresolved, say so — never invent function names to fill the gap. An unsymbolized stack still has shape: which library each frame belongs to, and whether the crash came from a signal handler, a main loop, or a worker thread.
Report
- What crashed, and what it was doing at the time.
- The most likely mechanism — separating clearly what the evidence proves from what you are inferring.
- Whether any user data was lost, and where it can be recovered from. Check the trash before concluding anything is gone.
- Whether it is likely to recur, and what would avoid or fix it.
Be straight about the limits of the evidence. If the cause is genuinely ambiguous, say so rather than assembling confidence out of guesswork.
Leave the system as you found it. Diagnosis reads; it does not fix, tidy, or reconfigure. The one thing to clean up is your own: delete the core you extracted above, which is a copy of the crashed process's memory.
If it is an Omarchy bug
Most application crashes are upstream bugs in those applications, not Omarchy's
doing. In the minority of cases where the cause really does sit within Omarchy's
sphere of control, read reporting.md before offering to file
anything.