ea6ee9440a
The mute was reachable only as `omarchy-toggle crash-ignore/<program>`, which asks whoever runs it to know the flag layout, to reduce a binary's path to the name the watcher keys on, and to have read the rule that a name climbing out of that directory writes an unrelated toggle. All of that was carried in the skill's prose, which is the wrong place for a rule that has to hold: prose is advice, and the thing being advised about is a name the crashed program chose. So it is a command now. `omarchy crash mute hyprland` silences that program, `off` lifts it, `toggle` flips it, and no argument lists what is muted. It takes the binary's path as readily as the name and reduces it the way the watcher does, so the `Executable:` line from `coredumpctl` can be handed straight to it; it refuses what is not one component of a name, so it cannot be talked into writing outside its own directory whatever it is given; and it re-reads the flag afterwards and reports what is now true rather than what was asked for. The listing counts only regular files, because that is all the watcher honours -- anything else in there would read as muted while the crashes kept arriving. A leading `--` is consumed so a program named `-h`, which the router would otherwise answer with its own help, can still be muted. The watcher gained an unrelated fix that this uncovered. Its fields are read with `IFS=$'\t'`, and tab is IFS whitespace, so an empty field collapsed into the next delimiter and shifted every field after it along one: a crash whose comm was empty had a path read as its pid and was discarded as somebody else's. A process can set its comm to nothing, so that was reachable. Empty fields now arrive as a dash like missing ones, and a dash joins the empty and dot cases that fall back to `unknown`. Co-Authored-By: Codex XHigh <noreply@openai.com>
152 lines
6.8 KiB
Markdown
152 lines
6.8 KiB
Markdown
---
|
|
name: diagnose-crash
|
|
description: >
|
|
Diagnose why a program crashed on this machine, from a systemd-coredump core dump.
|
|
Use when a process has segfaulted, aborted, or otherwise dumped core, when asked
|
|
why an application crashed or disappeared, or when a "Process crashed:" desktop
|
|
notification is acted on. Triggers: crash, segfault, SIGSEGV, SIGABRT, core dump,
|
|
coredumpctl, "why did X crash", "X keeps crashing", backtrace symbolization.
|
|
Covers reporting a confirmed Omarchy bug upstream — see reporting.md.
|
|
---
|
|
|
|
# Diagnosing a Crash
|
|
|
|
Work from evidence. The goal is an honest account of what happened, not a
|
|
plausible-sounding story.
|
|
|
|
## Establish the facts
|
|
|
|
`coredumpctl info <pid>` is the starting point. Beyond the backtrace, note the
|
|
**command line** the process was started with — it usually reveals what the
|
|
program was working on when it died, which is often the whole answer.
|
|
|
|
`coredumpctl list` shows whether this crash is a one-off or a pattern. Repeated
|
|
crashes of the same program, or several programs dying together, point somewhere
|
|
different than a single failure does.
|
|
|
|
## Rule out the boring causes first
|
|
|
|
Check resource exhaustion before blaming the program: `free -h`, and the journal
|
|
for OOM kills. A process killed by the OOM killer is not a bug in that process.
|
|
|
|
## Correlate against the timeline
|
|
|
|
The crash timestamp is the most underused piece of evidence. Compare it against:
|
|
|
|
- **Filesystem mtimes.** A directory or file whose mtime lands on the same second
|
|
as the crash strongly suggests what triggered it.
|
|
- **The journal** around that moment, for related warnings from the same or
|
|
neighbouring processes.
|
|
- **Recent package updates.** A crash that starts right after an update points at
|
|
the update.
|
|
|
|
## Read the whole core, not just frame 0
|
|
|
|
Thread stacks other than the crashing one show what work was **in flight** —
|
|
thumbnailers, image loaders, IPC readers, GPU queues. That context often explains
|
|
the trigger even when the crashing frame itself cannot be symbolized.
|
|
|
|
Note any third-party code in the address space: file-manager or browser
|
|
extensions, plugins, out-of-tree drivers. In-process third-party code is a common
|
|
crash source and worth flagging — but do not pin blame on it without evidence
|
|
that it is actually implicated.
|
|
|
|
## Symbolize when you can
|
|
|
|
This is Arch, which runs a public debuginfod server:
|
|
|
|
```bash
|
|
core=$(mktemp -t crash-XXXXXX.core)
|
|
trap 'rm -f "$core"' EXIT
|
|
coredumpctl dump <pid> --output="$core"
|
|
DEBUGINFOD_URLS="https://debuginfod.archlinux.org" \
|
|
gdb -q <executable> "$core" \
|
|
-batch -ex 'set debuginfod enabled on' -ex 'bt'
|
|
```
|
|
|
|
A core is a verbatim copy of the process's memory and can hold passwords, tokens,
|
|
and private documents. Write it to a fresh `mktemp` path rather than a predictable
|
|
shared one, and delete it when you are done — never leave it lying in `/tmp`.
|
|
|
|
Many packages publish no debug symbols. When frames stay unresolved, say so —
|
|
never invent function names to fill the gap. An unsymbolized stack still has
|
|
shape: which library each frame belongs to, and whether the crash came from a
|
|
signal handler, a main loop, or a worker thread.
|
|
|
|
## Report
|
|
|
|
1. What crashed, and what it was doing at the time.
|
|
2. The most likely mechanism — separating clearly what the evidence **proves**
|
|
from what you are **inferring**.
|
|
3. Whether any user data was lost, and where it can be recovered from. Check the
|
|
trash before concluding anything is gone.
|
|
4. Whether it is likely to recur, and what would avoid or fix it.
|
|
|
|
Be straight about the limits of the evidence. If the cause is genuinely
|
|
ambiguous, say so rather than assembling confidence out of guesswork.
|
|
|
|
**Leave the system as you found it.** Diagnosis reads; it does not fix, tidy, or
|
|
reconfigure. The one thing to clean up is your own: delete the core you extracted
|
|
above, which is a copy of the crashed process's memory. The single change a
|
|
diagnosis may make is the mute below, and only when the user asks for it.
|
|
|
|
## Offer to stop the notifications for this program
|
|
|
|
A crash that is now understood keeps announcing itself, and understanding it
|
|
rarely stops it happening: an upstream bug waiting on a release, a program that
|
|
dumps core every time it exits, a driver that misbehaves on this hardware. Finish
|
|
by offering to silence crash notifications for **that one program**:
|
|
|
|
```bash
|
|
omarchy-crash-mute '<program>'
|
|
```
|
|
|
|
`<program>` is the `process:` name in the crash facts, or the `binary:` path —
|
|
the command reduces a path to the same name the watcher keys on, so passing
|
|
`/usr/lib/chromium/chromium-browser` and passing `chromium-browser` land on the
|
|
same flag. Prefer the binary's path wherever the crash recorded one: the kernel
|
|
truncates the process name to 15 characters and does not truncate a basename, and
|
|
a mute on the truncated form matches nothing, forever, while looking like it
|
|
worked.
|
|
|
|
That is also the answer for a diagnosis started by hand from `omarchy agent crash
|
|
<pid>`, which is handed no name at all: give the command the `Executable:` line
|
|
from `coredumpctl info` and let it do the reducing. Some crashes record no
|
|
executable — pass the process name then, and `unknown` where the crash has
|
|
neither, which is the name such a crash is announced under.
|
|
|
|
The name is still whatever the crashed program's author chose to call a file, so
|
|
handle it as hostile text rather than as a word. The command refuses a name that
|
|
is not one — it cannot be talked into writing a flag outside its own directory —
|
|
but that is no help if the name reaches a shell unescaped first: single quotes
|
|
hold a space or a `$(...)`, and a name containing a single quote closes them and
|
|
runs the rest as your shell. Escape it, or the program that just crashed picks
|
|
the command.
|
|
|
|
Offer it; never run it unprompted. The user may well want to keep being told.
|
|
|
|
Say how to undo it in the same breath, so it is not a one-way door — and the
|
|
command answers both halves itself:
|
|
|
|
```bash
|
|
omarchy-crash-mute '<program>' off # un-mute this one
|
|
omarchy-crash-mute # list what is muted
|
|
```
|
|
|
|
The key is a bare name, so programs sharing one share a mute, and anything run
|
|
through an interpreter is keyed as the interpreter. Muting `python3.13` or `node`
|
|
silences every other Python or Node program on the machine, which is rarely what
|
|
the user means: say so rather than quietly doing it.
|
|
|
|
This silences one program. Every other crash still notifies, and the muted
|
|
program still crashes — nothing here fixes anything, and a mute offered instead
|
|
of a fix that was within reach is the wrong answer. If the user wants crash
|
|
notifications off altogether, that is _Trigger > Toggle > Crash Capture_ instead.
|
|
|
|
## If it is an Omarchy bug
|
|
|
|
Most application crashes are upstream bugs in those applications, not Omarchy's
|
|
doing. In the minority of cases where the cause really does sit within Omarchy's
|
|
sphere of control, read [`reporting.md`](reporting.md) before offering to file
|
|
anything.
|