# Pipelines, pipefail, Process Substitution and xargs — Bash / Shell Scripting

Source: https://www.skillbyai.com/en/bash/io-pipes

> Chain commands safely and handle file names with spaces.

## Small tools, connected

A **pipeline** `cmd1 | cmd2 | cmd3` connects each command's stdout to the next command's stdin; all run concurrently. By default the pipeline's exit status is the **last** command's status, so `false | true` succeeds and failures early in a pipeline go unnoticed. **`set -o pipefail`** makes the pipeline fail if any command fails, which is what scripts usually want; `PIPESTATUS` holds each command's status. **Process substitution** `<(cmd)` presents a command's output as a file name, so tools that need files can compare outputs directly: `diff <(sort a.txt) <(sort b.txt)`. **`tee`** copies a stream to a file while passing it on. **`xargs`** turns input lines into command arguments; because names can contain spaces or newlines, pair **`find -print0`** with **`xargs -0`** (NUL-separated), or use `find -exec cmd {} +`. `xargs -P N` runs commands in parallel.

## Pipelines that handle failures and awkward names

pipefail, NUL-separated names and parallel work.

```bash
set -o pipefail

# top 5 client IPs in an access log; fails if the log is missing
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -5

# compare two directory listings without temp files
diff <(ls /srv/a | sort) <(ls /srv/b | sort)

# compress logs older than 7 days, 4 at a time, safe with spaces in names
find /var/log/myapp -name '*.log' -mtime +7 -print0 | xargs -0 -P 4 -n 1 gzip

# keep a copy of output while still processing it
./migrate.sh 2>&1 | tee migrate-$(date +%F).log | grep -i error
```

## A factory assembly line

Each command is a station that does one job and passes the item on. Without pipefail, the line reports success as long as the last station finished, even if a station earlier on broke and passed along nothing.

**Quiz:** Without pipefail, what is the exit status of `false | true`?

- [ ] 1, because false failed
- [ ] 2
- [ ] It depends on the shell version only
- [x] 0, because the status of the last command is used

*Answer:* 0, because the status of the last command is used. By default a pipeline returns the status of its last command; pipefail changes that.