Run every test file instead of stopping at the first failure (#6622)
test/shell and test/all inherit `set -euo pipefail`, so the first failing test file aborts the whole run. One failure then hides every file behind it: you fix it, rerun, discover the next one, and repeat a file at a time. On a 140-file suite a single unrelated failure can keep most of the suite from ever reporting. Keep going after a failing file, then list the files that failed and exit non-zero. Individual files still stop at their own first failed assertion, so per-file isolation is unchanged, and a clean run still exits 0. The files are already independent of each other -- the set of failures is the same whether the run continues or stops at the first one -- so nothing was relying on the early abort. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
007d6fcd7d
commit
2b9e2720b3
@@ -8,7 +8,16 @@ tests=(
|
||||
"$ROOT/test/shell"
|
||||
)
|
||||
|
||||
# Run every suite even when an earlier one fails, so a single failure can't
|
||||
# hide whole suites behind it.
|
||||
failed=()
|
||||
for test in "${tests[@]}"; do
|
||||
printf '==> %s\n' "${test#$ROOT/}"
|
||||
"$test"
|
||||
"$test" || failed+=("${test#$ROOT/}")
|
||||
done
|
||||
|
||||
if (( ${#failed[@]} > 0 )); then
|
||||
printf '\n%d of %d suites failed:\n' "${#failed[@]}" "${#tests[@]}" >&2
|
||||
printf ' %s\n' "${failed[@]}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
+13
-1
@@ -18,7 +18,19 @@ if (( ${#tests[@]} == 0 )); then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Keep going after a failing file. A test file exits at its first failed
|
||||
# assertion, so aborting the run there too would hide every file after it --
|
||||
# one packaging failure was masking 114 of 134 files.
|
||||
failed=()
|
||||
for test in "${tests[@]}"; do
|
||||
printf '==> %s\n' "${test#$ROOT/}"
|
||||
bash "$test"
|
||||
bash "$test" || failed+=("${test#$ROOT/}")
|
||||
done
|
||||
|
||||
if (( ${#failed[@]} > 0 )); then
|
||||
printf '\n%d of %d test files failed:\n' "${#failed[@]}" "${#tests[@]}" >&2
|
||||
printf ' %s\n' "${failed[@]}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '\nAll %d test files passed.\n' "${#tests[@]}"
|
||||
|
||||
Reference in New Issue
Block a user