Debugging notes · 30 October 2026

A condition that was true and still failed

The other day, while working on something else, the LLM found the cause of a bug that had puzzled me for years. From time to time, some of my scripts would fail a condition for no reason I could think of. Worst of all, those simple conditions were non-deterministic.

The shape was always this:

if long-output-cmd | grep -q 'magic-string'; then
    [...]
fi

Nothing looks wrong there. I had evidence that long-output-cmd produced deterministic output, and that the magic string was in it. When I ran the same condition by hand, it succeeded.

One piece of context that changes the results between a script and my terminal: most of my scripts run with set -eo pipefail.

set -e is not the problem here, because it is suspended for the command list in an if condition. set -o pipefail is the more interesting one. As I understood it, pipefail only changes which of a pipeline's exit codes the shell reports, and that is not affected by the pipeline sitting inside a conditional.

So I checked that much:

• Pipefail matches non pipefail behavior because failure is in the last command

$ (set -eo pipefail; if echo 's' | false; then echo "Hello"; fi)

• All commands exit with 0

$ (set -eo pipefail; if echo 's' | true; then echo "Hello"; fi)
Hello

• No pipefail but the last command exit code still counts

$ (set -e; if echo 's' | false; then echo "Hello"; fi)

In all of the above pipefail does not change the outcome of the condition.

• Without pipefail the last command matters

$ (set -e; if false | true; then echo reached; fi)
reached

• With pipefail either all succeed or any failure is propagated

$ (set -eo pipefail; if false | true; then echo reached; fi)

Given that mental model check, I honestly thought there was some kind of bug in bash. That is how puzzled I was, and I often work with old software, so it was not an absurd thing to believe. I just could not see how this could fail.

Recently, and in the time of LLMs, the same class of issue appeared again. It came up while validating totally unrelated code. It also only happened on one specific machine, and only inside my script. When I logged in to that machine and ran the command manually, it worked. Whattt???

I fired up the LLM. It spent a while convincing itself the condition really was true, and then found it.

Three things are enough to solve this. If you want the puzzle, stop at the end of this section and give it a thought. Maybe you are cleverer than me and can connect the dots.

The first, from man grep:

-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was detected.

The second, from man bash under PIPELINES:

If pipefail is enabled, the pipeline's return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully.

The third is a question rather than a quote from the manual. What happens to a pipe when the command on the right exits before reading all of its input?

grep -q found its match early, exited, and closed its end of the pipe. long-output-cmd was still writing.

Just like real pipes: if you suddenly close the tap at the downstream end while a lot of fluid is flowing, you can burst the piping system — and the part that bursts is upstream of the valve you closed. The same thing happens here. grep is fine, grep succeeded and closed its file descriptors! The command that dies is the one that was doing nothing wrong.

In bash the mechanism is that writing to a pipe with no reader raises the SIGPIPE signal, and SIGPIPE's default action is to kill the process. So long-output-cmd is not reporting an error it noticed — it is killed by a signal, and bash records 128 + 13 = 141.

I had this part wrong when I first wrote this text down. In my earlier draft I had “the upstream command exits with broken pipe and the exit code gets propagated to the caller”, as if the command had detected a problem and returned an error.

The command usually detects nothing at all! A program only ever sees EPIPE from the write() syscall if it deliberately ignores or handles the SIGPIPE signal; a usual program is simply killed, and 141 is the exit code seen in practice.

Nothing propagates either. Bash collects every command's exit code as it reaps each child, and pipefail only decides which of those already-collected exit codes becomes the pipeline's.

So the pipeline ends with two results: grep -q, which did its job fast enough and exited 0, and long-output-cmd, killed by SIGPIPE at 141. Which one does the pipeline return? By the rule quoted above, the rightmost non-zero status wins, and the only non-zero status is the 141. The conditional takes that as its answer and evaluates false.

$ yes | grep -q y
$ echo ${PIPESTATUS[@]}
141 0
$ (set -o pipefail; yes | grep -q y); echo $?
141
$ (set -e; yes | grep -q y); echo $?
0

The explanation above makes it look like a deterministic phenomenon, but it is not. Most of the time those conditions worked. What decides it is a race: has the producer finished writing by the time the consumer exits?

Four pipelines, each of which you can run yourself:

$ (set -o pipefail; yes | grep -q y)               # 141 every time
$ (set -o pipefail; seq 1 1000  | grep -q '^1$')   # 0 non-zero in 20 runs
$ (set -o pipefail; seq 1 13000 | grep -q '^1$')   # 4 non-zero in 40 runs
$ (set -o pipefail; seq 1 50000 | grep -q '^1$')   # 20 non-zero in 20 runs

Notice that seq 1 1000 does not fail but seq 1 13000 fails 1/10th of the time. seq 1 1000 is 3.9 KB of output, and a Linux pipe holds 65536 bytes by default, so the producer writes everything into the buffer and exits 0 before grep is even relevant. That is why I could never reproduce this with a small test case.

Past the buffer the producer has to block waiting for the reader, and then it matters who gets there first. That is where the machine's load comes in. Same commands, 40 runs each, idle versus 14 busy loops (while :; do :; done &) pinning all 14 cores:

The numbers are noisy, but they only move one way. Load does not create the bug; it widens the window in which you meet it. That is why it showed up on the loaded machine and not on my laptop.

There is one thing in my own account that I should correct while I am here. I said that logging in to that machine and running the command manually worked. If the output was past the buffer, by hand should have failed too. The likely explanation is not load at all: set -eo pipefail lives in the script, not in my interactive shell, so by hand I was in the set -e case the whole time, where the pipeline reports grep's zero and the condition is true.

Nothing in this is specific to grep -q. The condition is about shape: any downstream command that stops reading before EOF can kill a slow upstream producer. grep -q is just the one that bit me.

Same producer, 20 runs each:

PIPESTATUS for seq 1 50000 | head -1 is 141 0, identical to the grep -q case. So cmd | head under pipefail is this same bug, and I suspect that is the version most people have run into without ever working out why.

Once the rule is “either the consumer reads to EOF, or the producer is allowed to die”, the fixes stop being tricks and follow from it. Same producer, 20 runs each:

long-output-cmd | grep 'magic' >/dev/null             #  0/20  no -q, so grep reads to EOF
out=$(long-output-cmd); grep -q magic <<<"$out"       #  0/20  no pipe to break
{ long-output-cmd || [ $? = 141 ]; } | grep -q magic  #  0/20  the death is expected
long-output-cmd | grep -m1 'magic' >/dev/null         # 20/20  still broken

The last line is the one worth remembering. Dropping -q only works because plain grep keeps reading to the end of its input; any flag that makes it exit early puts the bug straight back.