Run every test file instead of stopping at the first failure (#6622)

test/shell and test/all inherit `set -euo pipefail`, so the first failing
test file aborts the whole run. One failure then hides every file behind it:
you fix it, rerun, discover the next one, and repeat a file at a time. On a
140-file suite a single unrelated failure can keep most of the suite from ever
reporting.

Keep going after a failing file, then list the files that failed and exit
non-zero. Individual files still stop at their own first failed assertion, so
per-file isolation is unchanged, and a clean run still exits 0.

The files are already independent of each other -- the set of failures is the
same whether the run continues or stops at the first one -- so nothing was
relying on the early abort.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Scott Jones
2026-08-08 08:57:36 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 007d6fcd7d
commit 2b9e2720b3
2 changed files with 23 additions and 2 deletions
+10 -1
View File
@@ -8,7 +8,16 @@ tests=(
"$ROOT/test/shell"
)
# Run every suite even when an earlier one fails, so a single failure can't
# hide whole suites behind it.
failed=()
for test in "${tests[@]}"; do
printf '==> %s\n' "${test#$ROOT/}"
"$test"
"$test" || failed+=("${test#$ROOT/}")
done
if (( ${#failed[@]} > 0 )); then
printf '\n%d of %d suites failed:\n' "${#failed[@]}" "${#tests[@]}" >&2
printf ' %s\n' "${failed[@]}" >&2
exit 1
fi
+13 -1
View File
@@ -18,7 +18,19 @@ if (( ${#tests[@]} == 0 )); then
exit 1
fi
# Keep going after a failing file. A test file exits at its first failed
# assertion, so aborting the run there too would hide every file after it --
# one packaging failure was masking 114 of 134 files.
failed=()
for test in "${tests[@]}"; do
printf '==> %s\n' "${test#$ROOT/}"
bash "$test"
bash "$test" || failed+=("${test#$ROOT/}")
done
if (( ${#failed[@]} > 0 )); then
printf '\n%d of %d test files failed:\n' "${#failed[@]}" "${#tests[@]}" >&2
printf ' %s\n' "${failed[@]}" >&2
exit 1
fi
printf '\nAll %d test files passed.\n' "${#tests[@]}"