Pipelines, pipefail, Process Substitution and xargs

Small tools, connected

A pipeline cmd1 | cmd2 | cmd3 connects each command's stdout to the next command's stdin; all run concurrently. By default the pipeline's exit status is the last command's status, so false | true succeeds and failures early in a pipeline go unnoticed. set -o pipefail makes the pipeline fail if any command fails, which is what scripts usually want; PIPESTATUS holds each command's status. Process substitution <(cmd) presents a command's output as a file name, so tools that need files can compare outputs directly: diff <(sort a.txt) <(sort b.txt). tee copies a stream to a file while passing it on. xargs turns input lines into command arguments; because names can contain spaces or newlines, pair find -print0 with xargs -0 (NUL-separated), or use find -exec cmd {} +. xargs -P N runs commands in parallel.

Pipelines that handle failures and awkward names

pipefail, NUL-separated names and parallel work.

set -o pipefail

# top 5 client IPs in an access log; fails if the log is missing
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -5

# compare two directory listings without temp files
diff <(ls /srv/a | sort) <(ls /srv/b | sort)

# compress logs older than 7 days, 4 at a time, safe with spaces in names
find /var/log/myapp -name '*.log' -mtime +7 -print0 | xargs -0 -P 4 -n 1 gzip

# keep a copy of output while still processing it
./migrate.sh 2>&1 | tee migrate-$(date +%F).log | grep -i error

A factory assembly line

Each command is a station that does one job and passes the item on. Without pipefail, the line reports success as long as the last station finished, even if a station earlier on broke and passed along nothing.

त्वरित जाँच: Without pipefail, what is the exit status of `false | true`?

• 1, because false failed
• 2
• It depends on the shell version only
• 0, because the status of the last command is used