Offer an AI diagnosis when a process crashes (#6746)
* Offer an AI diagnosis when a process crashes systemd-coredump journals every core dump under a known MESSAGE_ID with the crashing program, pid, and signal as structured fields. omarchy-crash-watch follows that stream and raises a "Process crashed: <program>" toast; clicking it opens omarchy-agent-crash, which briefs the default agent on the crash. The toast goes through omarchy-notification-send --exec rather than a libnotify action, because the shell runs clicks from its own omarchy-exec hint and never emits ActionInvoked. It keeps the default "omarchy-action" app name too, the only one shouldBypassDnd() lets through -- a crash being the last notification worth swallowing. It stays quiet until an agent is configured, since a diagnosis is all it offers. The method lives in a diagnose-crash skill rather than the prompt, so it is edited in one place and works with whichever agent is default. It covers investigating the core, and reporting a confirmed Omarchy bug upstream: scoped to bugs Omarchy controls, searched for duplicates first, only with the user's agreement, and signed with the model and harness that produced it. A migration reaches existing installs, whose skill symlinks and unit enablement would otherwise sit behind one-time setup paths. * Let the diagnosis clean up the core it extracted "Do not modify or delete anything" contradicted the symbolization step right above it, which writes a core to a temp file and deletes it on exit. Read literally, the core survives -- and the same section warns it holds passwords and tokens. The prohibition is about the system, not about your own scratch. * Do not spend a crash toast on a dead notification server The shell owns org.freedesktop.Notifications, so its own crash takes the notification server down with it -- and a shell crash is exactly what you want told about. The toast was sent once into that gap and the dedupe window was recorded regardless, so the rest of the crash loop went quiet for a minute and `journalctl -n 0` never replays what was missed. It now waits for the restarted shell to reclaim the bus name, as omarchy-migrate-notify already does, and only a delivered toast starts the dedupe window.
This commit is contained in:
David Heinemeier Hansson
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/bin/bash
|
||||
|
||||
# omarchy:summary=Diagnose a crashed process with the default coding agent
|
||||
# omarchy:args=<pid> [comm] [exe] [signal]
|
||||
# omarchy:examples=omarchy agent crash 1516893
|
||||
|
||||
# Clicked from a "Process crashed:" notification, or run by hand against any PID
|
||||
# in `coredumpctl list`. The method lives in the diagnose-crash skill so it is
|
||||
# edited in one place and works with whichever agent is default; this only
|
||||
# gathers the facts and points at it.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
pid=${1:?usage: omarchy-agent-crash <pid> [comm] [exe] [signal]}
|
||||
|
||||
if [[ ! $pid =~ ^[0-9]+$ ]]; then
|
||||
echo "Not a PID: $pid" >&2
|
||||
echo "Usage: omarchy agent crash <pid> (see: coredumpctl list)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
comm=${2:-unknown}
|
||||
exe=${3:-unknown}
|
||||
signal=${4:-unknown}
|
||||
|
||||
skill="$OMARCHY_PATH/default/agents/skills/diagnose-crash/SKILL.md"
|
||||
|
||||
# Looked up live so a hand-run PID still gets a timestamp. A rotated-away core
|
||||
# only costs the timestamp, so failure is tolerated.
|
||||
when=$(coredumpctl list "$pid" --no-pager --no-legend 2>/dev/null | tail -1 | cut -d' ' -f1-4) || true
|
||||
when=${when:-unknown}
|
||||
|
||||
prompt=$(
|
||||
cat <<PROMPT
|
||||
A process crashed on this Omarchy machine and I want to know why.
|
||||
|
||||
What systemd-coredump recorded:
|
||||
process: $comm
|
||||
PID: $pid
|
||||
binary: $exe
|
||||
signal: $signal
|
||||
time: $when
|
||||
|
||||
Use the diagnose-crash skill: it covers how to investigate, what to report, and
|
||||
when a crash is worth reporting upstream to Omarchy. If your harness has no skill
|
||||
mechanism, read the skill files directly and follow them instead:
|
||||
|
||||
$skill
|
||||
PROMPT
|
||||
)
|
||||
|
||||
exec omarchy-agent --prompt "$prompt"
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/bash
|
||||
|
||||
# omarchy:summary=Watch for process crashes and offer an AI diagnosis
|
||||
# omarchy:hidden=true
|
||||
|
||||
# systemd-coredump journals every core dump under a known MESSAGE_ID with
|
||||
# structured COREDUMP_* fields, which carry more than the core filenames do.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
# See systemd.journal-fields(7).
|
||||
readonly COREDUMP_MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1
|
||||
|
||||
# nf-md-robot_dead, escaped so this file reads without a Nerd Font.
|
||||
readonly CRASH_GLYPH=$'\U000f16a1'
|
||||
|
||||
# Crash loops dump core repeatedly, so announce each program at most once a
|
||||
# window.
|
||||
readonly dedupe_seconds=${OMARCHY_CRASH_DEDUPE_SECONDS:-60}
|
||||
|
||||
# Extended regex of process names never worth announcing.
|
||||
readonly ignore_pattern=${OMARCHY_CRASH_IGNORE:-}
|
||||
|
||||
declare -A last_notified
|
||||
|
||||
announce() {
|
||||
local comm=$1 pid=$2 exe=$3 signal=$4 exec_command
|
||||
|
||||
exec_command=$(printf 'omarchy-agent-crash %q %q %q %q' "$pid" "$comm" "$exe" "$signal")
|
||||
|
||||
# The shell owns org.freedesktop.Notifications, so a shell crash takes the
|
||||
# notification server down with it and a toast sent into that gap is lost.
|
||||
# Wait for the restarted shell to claim the name again: the crash least
|
||||
# likely to be delivered is the one most worth reporting.
|
||||
omarchy-notification-wait || return 1
|
||||
|
||||
# --exec rather than a libnotify action: the shell runs clicks from its own
|
||||
# omarchy-exec hint and never emits ActionInvoked. Keeps the default
|
||||
# "omarchy-action" app name too, the only one shouldBypassDnd() lets through.
|
||||
omarchy-notification-send \
|
||||
--urgency critical \
|
||||
--glyph "$CRASH_GLYPH" \
|
||||
--exec "$exec_command" \
|
||||
"Process crashed: $comm" \
|
||||
"Click to diagnose with AI"
|
||||
}
|
||||
|
||||
# -n 0 so a restart does not re-announce crashes already dealt with.
|
||||
journalctl -f -n 0 -o json "MESSAGE_ID=$COREDUMP_MESSAGE_ID" 2>/dev/null |
|
||||
while IFS= read -r entry; do
|
||||
IFS=$'\t' read -r uid comm pid exe signal < <(
|
||||
jq -r '[(._UID // "-"),
|
||||
(.COREDUMP_COMM // "-"),
|
||||
(.COREDUMP_PID // "-"),
|
||||
(.COREDUMP_EXE // "-"),
|
||||
(.COREDUMP_SIGNAL_NAME // "-")] | @tsv' <<<"$entry" 2>/dev/null
|
||||
)
|
||||
|
||||
[[ $pid =~ ^[0-9]+$ ]] || continue
|
||||
|
||||
# The toast only offers a diagnosis, so it has nothing to offer until an
|
||||
# agent is chosen. Checked per crash, not at startup, so picking one takes
|
||||
# effect without restarting this service.
|
||||
[[ -n $(omarchy-default-agent) ]] || continue
|
||||
|
||||
# Only this user's crashes; a daemon dumping core is a sysadmin's problem.
|
||||
[[ $uid =~ ^[0-9]+$ ]] || continue
|
||||
((uid == UID)) || continue
|
||||
|
||||
# comm is truncated to 15 characters, so prefer the executable's basename.
|
||||
name=$comm
|
||||
[[ $exe == /* ]] && name=${exe##*/}
|
||||
|
||||
[[ -n $ignore_pattern && $name =~ $ignore_pattern ]] && continue
|
||||
|
||||
# Never announce our own machinery, or it notifies about itself.
|
||||
[[ $name == omarchy-crash-* || $name == omarchy-agent-* ]] && continue
|
||||
|
||||
now=$EPOCHSECONDS
|
||||
(((now - ${last_notified[$name]:-0}) < dedupe_seconds)) && continue
|
||||
|
||||
# Only a delivered toast starts the dedupe window. A failed send that
|
||||
# counted would suppress the rest of a crash loop for a minute, and
|
||||
# `journalctl -n 0` never replays what was missed.
|
||||
announce "$name" "$pid" "$exe" "$signal" && last_notified[$name]=$now
|
||||
done
|
||||
@@ -83,11 +83,16 @@ fi
|
||||
|
||||
# Dev-aware skill symlinks. Cannot live in /etc/skel because OMARCHY_PATH may
|
||||
# point at a dev checkout (omarchy dev link) where the target differs.
|
||||
# Loops every skill directory, so shipping a new one needs no edit here.
|
||||
mkdir -p ~/.agents/skills ~/.claude/skills ~/.codex/skills ~/.pi/agent/skills
|
||||
ln -sfn "$OMARCHY_PATH/default/agents/skills/omarchy" ~/.agents/skills/omarchy
|
||||
ln -sfn "$OMARCHY_PATH/default/agents/skills/omarchy" ~/.claude/skills/omarchy
|
||||
ln -sfn "$OMARCHY_PATH/default/agents/skills/omarchy" ~/.codex/skills/omarchy
|
||||
ln -sfn "$OMARCHY_PATH/default/agents/skills/omarchy" ~/.pi/agent/skills/omarchy
|
||||
for skill in "$OMARCHY_PATH"/default/agents/skills/*/; do
|
||||
skill=${skill%/}
|
||||
name=${skill##*/}
|
||||
ln -sfn "$skill" ~/.agents/skills/"$name"
|
||||
ln -sfn "$skill" ~/.claude/skills/"$name"
|
||||
ln -sfn "$skill" ~/.codex/skills/"$name"
|
||||
ln -sfn "$skill" ~/.pi/agent/skills/"$name"
|
||||
done
|
||||
|
||||
mkdir -p ~/Downloads ~/Pictures ~/Videos ~/.config/gtk-3.0
|
||||
xdg-user-dirs-update --set TEMPLATES "$HOME"
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: diagnose-crash
|
||||
description: >
|
||||
Diagnose why a program crashed on this machine, from a systemd-coredump core dump.
|
||||
Use when a process has segfaulted, aborted, or otherwise dumped core, when asked
|
||||
why an application crashed or disappeared, or when a "Process crashed:" desktop
|
||||
notification is acted on. Triggers: crash, segfault, SIGSEGV, SIGABRT, core dump,
|
||||
coredumpctl, "why did X crash", "X keeps crashing", backtrace symbolization.
|
||||
Covers reporting a confirmed Omarchy bug upstream — see reporting.md.
|
||||
---
|
||||
|
||||
# Diagnosing a Crash
|
||||
|
||||
Work from evidence. The goal is an honest account of what happened, not a
|
||||
plausible-sounding story.
|
||||
|
||||
## Establish the facts
|
||||
|
||||
`coredumpctl info <pid>` is the starting point. Beyond the backtrace, note the
|
||||
**command line** the process was started with — it usually reveals what the
|
||||
program was working on when it died, which is often the whole answer.
|
||||
|
||||
`coredumpctl list` shows whether this crash is a one-off or a pattern. Repeated
|
||||
crashes of the same program, or several programs dying together, point somewhere
|
||||
different than a single failure does.
|
||||
|
||||
## Rule out the boring causes first
|
||||
|
||||
Check resource exhaustion before blaming the program: `free -h`, and the journal
|
||||
for OOM kills. A process killed by the OOM killer is not a bug in that process.
|
||||
|
||||
## Correlate against the timeline
|
||||
|
||||
The crash timestamp is the most underused piece of evidence. Compare it against:
|
||||
|
||||
- **Filesystem mtimes.** A directory or file whose mtime lands on the same second
|
||||
as the crash strongly suggests what triggered it.
|
||||
- **The journal** around that moment, for related warnings from the same or
|
||||
neighbouring processes.
|
||||
- **Recent package updates.** A crash that starts right after an update points at
|
||||
the update.
|
||||
|
||||
## Read the whole core, not just frame 0
|
||||
|
||||
Thread stacks other than the crashing one show what work was **in flight** —
|
||||
thumbnailers, image loaders, IPC readers, GPU queues. That context often explains
|
||||
the trigger even when the crashing frame itself cannot be symbolized.
|
||||
|
||||
Note any third-party code in the address space: file-manager or browser
|
||||
extensions, plugins, out-of-tree drivers. In-process third-party code is a common
|
||||
crash source and worth flagging — but do not pin blame on it without evidence
|
||||
that it is actually implicated.
|
||||
|
||||
## Symbolize when you can
|
||||
|
||||
This is Arch, which runs a public debuginfod server:
|
||||
|
||||
```bash
|
||||
core=$(mktemp -t crash-XXXXXX.core)
|
||||
trap 'rm -f "$core"' EXIT
|
||||
coredumpctl dump <pid> --output="$core"
|
||||
DEBUGINFOD_URLS="https://debuginfod.archlinux.org" \
|
||||
gdb -q <executable> "$core" \
|
||||
-batch -ex 'set debuginfod enabled on' -ex 'bt'
|
||||
```
|
||||
|
||||
A core is a verbatim copy of the process's memory and can hold passwords, tokens,
|
||||
and private documents. Write it to a fresh `mktemp` path rather than a predictable
|
||||
shared one, and delete it when you are done — never leave it lying in `/tmp`.
|
||||
|
||||
Many packages publish no debug symbols. When frames stay unresolved, say so —
|
||||
never invent function names to fill the gap. An unsymbolized stack still has
|
||||
shape: which library each frame belongs to, and whether the crash came from a
|
||||
signal handler, a main loop, or a worker thread.
|
||||
|
||||
## Report
|
||||
|
||||
1. What crashed, and what it was doing at the time.
|
||||
2. The most likely mechanism — separating clearly what the evidence **proves**
|
||||
from what you are **inferring**.
|
||||
3. Whether any user data was lost, and where it can be recovered from. Check the
|
||||
trash before concluding anything is gone.
|
||||
4. Whether it is likely to recur, and what would avoid or fix it.
|
||||
|
||||
Be straight about the limits of the evidence. If the cause is genuinely
|
||||
ambiguous, say so rather than assembling confidence out of guesswork.
|
||||
|
||||
**Leave the system as you found it.** Diagnosis reads; it does not fix, tidy, or
|
||||
reconfigure. The one thing to clean up is your own: delete the core you extracted
|
||||
above, which is a copy of the crashed process's memory.
|
||||
|
||||
## If it is an Omarchy bug
|
||||
|
||||
Most application crashes are upstream bugs in those applications, not Omarchy's
|
||||
doing. In the minority of cases where the cause really does sit within Omarchy's
|
||||
sphere of control, read [`reporting.md`](reporting.md) before offering to file
|
||||
anything.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Reporting a Crash Upstream to Omarchy
|
||||
|
||||
Read this only after concluding that a crash is genuinely Omarchy's to fix.
|
||||
|
||||
## Is it even Omarchy's bug?
|
||||
|
||||
Be strict here. Omarchy is a configuration layer over Arch Linux, so a crash
|
||||
inside a third-party application — a file manager, a browser, a GNOME or Qt
|
||||
library — is almost always an upstream bug in **that** project, not in Omarchy.
|
||||
|
||||
Omarchy's sphere of control is roughly:
|
||||
|
||||
- the `omarchy-*` commands
|
||||
- the Quickshell shell and its plugins
|
||||
- the Hyprland and terminal configuration it ships
|
||||
- its themes
|
||||
- its install and migration scripts
|
||||
- how it packages and configures what it installs
|
||||
|
||||
A crash in a program Omarchy merely installs is **not** an Omarchy bug unless
|
||||
Omarchy's own packaging or configuration is implicated.
|
||||
|
||||
If it is not Omarchy's, say so and stop. Suggesting the right upstream project is
|
||||
useful; filing there yourself is not part of this.
|
||||
|
||||
## Three conditions, all required
|
||||
|
||||
1. **It is a verified bug in Omarchy's sphere**, established on evidence. Issues
|
||||
are for verified bugs only. An "is this even a bug?" belongs on the Discord at
|
||||
<https://omarchy.org/discord>; a feature idea belongs in GitHub Discussions
|
||||
under Suggestions.
|
||||
2. **The user has explicitly agreed.** Show them the exact title and body you
|
||||
propose, and wait for a yes. Never file unprompted.
|
||||
3. **The machine can file it** — `gh auth status` must succeed. If `gh` is missing
|
||||
or unauthenticated, do not install or authenticate it. Say so, and hand the
|
||||
user the finished text to submit themselves.
|
||||
|
||||
## Search before filing
|
||||
|
||||
A duplicate issue costs a maintainer more time than no report at all.
|
||||
|
||||
```bash
|
||||
gh search issues --repo basecamp/omarchy "<program> crash"
|
||||
gh issue list --repo basecamp/omarchy --state all --search "<signal> <program>"
|
||||
```
|
||||
|
||||
Search on the crashing program, the signal, and distinctive symbols from the
|
||||
backtrace — not on the wording of the title you were about to write.
|
||||
|
||||
`gh search issues` accepts only `open` or `closed` for `--state`, and errors on
|
||||
anything else. Leaving it off searches both, which is what you want here.
|
||||
|
||||
Include **closed** issues. A matching issue closed as fixed, when the crash still
|
||||
reproduces on a current system, is a regression — and reporting that is worth far
|
||||
more than another duplicate.
|
||||
|
||||
## Adding to an existing report
|
||||
|
||||
If a plausible match comes back, read it properly first:
|
||||
|
||||
```bash
|
||||
gh issue view <number> --repo basecamp/omarchy --comments
|
||||
```
|
||||
|
||||
Confirm it is genuinely the same failure. The same program crashing is not the
|
||||
same bug if the trigger or the stack differs.
|
||||
|
||||
If it is the same, add to that issue rather than opening a new one — but only
|
||||
when you have something the thread does not already contain: a different
|
||||
reproduction, a symbolized stack where it has none, a narrower trigger, a version
|
||||
where it regressed.
|
||||
|
||||
A comment that only says the bug happens to you too is noise. If that is all you
|
||||
have, tell the user so and file nothing.
|
||||
|
||||
```bash
|
||||
gh issue comment <number> --repo basecamp/omarchy --body "..."
|
||||
```
|
||||
|
||||
## Filing a new issue
|
||||
|
||||
Only when the search turns up nothing that matches:
|
||||
|
||||
```bash
|
||||
gh issue create --repo basecamp/omarchy --title "..." --body "..."
|
||||
```
|
||||
|
||||
Include what happened, what was expected, steps to reproduce, system details from
|
||||
`omarchy version`, and diagnostics from `omarchy debug --no-sudo --print` (which
|
||||
also writes `/tmp/omarchy-debug.log`; the interactive `omarchy debug` can upload
|
||||
it and print a shareable URL worth including).
|
||||
|
||||
`gh` cannot attach media. If a screenshot would help, save one and give the user
|
||||
the path to drag into the web form.
|
||||
|
||||
## Signing
|
||||
|
||||
End the issue or comment with a line naming the model and agent harness that
|
||||
produced it, so a human reader knows it was machine-authored:
|
||||
|
||||
> Filed by \<model name\> via \<agent harness\>.
|
||||
|
||||
Use your actual model and harness names. If you are not certain of them, say so
|
||||
plainly rather than inventing a version string.
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Announce process crashes and offer an AI diagnosis
|
||||
# Needs the session bus to notify, and uwsm-app to open the diagnosis terminal.
|
||||
# Both are up only after graphical-session.target.
|
||||
After=graphical-session.target
|
||||
PartOf=graphical-session.target
|
||||
ConditionEnvironment=WAYLAND_DISPLAY
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/omarchy-crash-watch
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=graphical-session.target
|
||||
@@ -17,4 +17,5 @@ systemctl --user enable --now \
|
||||
omarchy-recover-internal-monitor.service \
|
||||
omarchy-sleep-lock.service \
|
||||
omarchy-migrate-notify.service \
|
||||
omarchy-fcitx5.service
|
||||
omarchy-fcitx5.service \
|
||||
omarchy-crash-watch.service
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
echo "Announce process crashes and offer an AI diagnosis"
|
||||
|
||||
# Both halves are set up in paths that only run once -- omarchy-provision-user
|
||||
# exits after finalize-user is marked, and install/user/first-run is skipped
|
||||
# after the first login -- so existing installs need them done here.
|
||||
|
||||
skills_source="$OMARCHY_PATH/default/agents/skills"
|
||||
|
||||
if [[ -d $skills_source/diagnose-crash ]]; then
|
||||
for skills_dir in ~/.agents/skills ~/.claude/skills ~/.codex/skills ~/.pi/agent/skills; do
|
||||
mkdir -p "$skills_dir"
|
||||
ln -sfn "$skills_source/diagnose-crash" "$skills_dir/diagnose-crash"
|
||||
done
|
||||
fi
|
||||
|
||||
systemctl --user daemon-reload >/dev/null 2>&1 || true
|
||||
|
||||
# `systemctl enable` needs a live user manager, which an update from a TTY does
|
||||
# not have, so fall back to writing the symlink it would have written.
|
||||
if ! systemctl --user enable omarchy-crash-watch.service >/dev/null 2>&1; then
|
||||
wants_dir="$HOME/.config/systemd/user/graphical-session.target.wants"
|
||||
mkdir -p "$wants_dir"
|
||||
ln -sfn /usr/lib/systemd/user/omarchy-crash-watch.service \
|
||||
"$wants_dir/omarchy-crash-watch.service"
|
||||
fi
|
||||
|
||||
# Nothing to start into over SSH; the next graphical login handles it. A failed
|
||||
# start only delays crash toasts, so it stays quiet.
|
||||
if systemctl --user is-active --quiet graphical-session.target; then
|
||||
systemctl --user start omarchy-crash-watch.service >/dev/null 2>&1 || true
|
||||
fi
|
||||
@@ -140,6 +140,7 @@ package_defaults = [
|
||||
("default/systemd/user/omarchy-migrate-notify.service", "/usr/lib/systemd/user/omarchy-migrate-notify.service", "systemd/user/omarchy-migrate-notify.service"),
|
||||
("default/systemd/user/omarchy-tailscale-receive.service", "/usr/lib/systemd/user/omarchy-tailscale-receive.service", "systemd/user/omarchy-tailscale-receive.service"),
|
||||
("default/systemd/user/omarchy-fcitx5.service", "/usr/lib/systemd/user/omarchy-fcitx5.service", "systemd/user/omarchy-fcitx5.service"),
|
||||
("default/systemd/user/omarchy-crash-watch.service", "/usr/lib/systemd/user/omarchy-crash-watch.service", "systemd/user/omarchy-crash-watch.service"),
|
||||
("default/systemd/zram-generator.conf.d/90-omarchy.conf", "/usr/lib/systemd/zram-generator.conf.d/90-omarchy.conf", "systemd/zram-generator.conf.d/90-omarchy.conf"),
|
||||
("default/fonts/omarchy/omarchy.ttf", "/usr/share/fonts/omarchy/omarchy.ttf", "omarchy.ttf"),
|
||||
("default/snapper/root", "/etc/snapper/config-templates/omarchy", "snapper/root"),
|
||||
|
||||
Reference in New Issue
Block a user