# Text Processing with grep, sed, awk and jq — Bash / Shell Scripting

Source: https://www.skillbyai.com/en/bash/io-text

> Use the core text tools to search, edit, summarise and parse JSON.

## The Unix toolbox for text

Most scripts glue text tools together. **`grep`** finds lines matching a pattern: `-i` ignore case, `-v` invert, `-c` count, `-E` extended regular expressions, `-r` recursive, `-q` quiet (just the exit status), `-o` print only the match. **`sed`** edits streams: `sed 's/old/new/g'` substitutes, `-n '5,10p'` prints a range, `-i` edits files in place (GNU and BSD/macOS differ: GNU accepts `-i`, macOS needs `-i ''`). **`awk`** processes columns and records: `awk -F, '{sum += $3} END {print sum}'` sums the third CSV column; it handles floating-point maths too. **`cut`**, **`sort`** (`-n` numeric, `-k` key, `-u` unique), **`uniq -c`**, **`wc -l`**, **`tr`** and **`head`/`tail`** complete the set. For **JSON**, use **`jq`** rather than grep: `jq -r '.items[] | select(.status=="active") | .name'`. For CSV with quoted fields containing commas, awk and cut are fragile; use a proper CSV tool or a small Python script.

## Everyday text processing

Each line is a common task in operations scripts.

```bash
grep -E 'ERROR|FATAL' app.log | grep -v 'healthcheck' | tail -20
grep -q 'listen 443' /etc/nginx/nginx.conf && echo "TLS configured"

sed -E 's/password=[^&]+/password=****/g' requests.log > requests.redacted.log
sed -n '/BEGIN CERT/,/END CERT/p' bundle.pem

# total and average response time (column 10) from a log
awk '{ total += $10; n++ } END { if (n) printf "avg %.1f ms over %d requests\n", total/n, n }' access.log

# JSON from an API: names of running instances
curl -fsS "$API_URL/instances" | jq -r '.items[] | select(.state == "running") | .name'

sort -t, -k3,3nr sales.csv | head -10   # top 10 rows by 3rd column, numeric desc
```

## Parse JSON with jq, not grep

Grepping JSON breaks when fields are reordered, reformatted or nested. `jq` understands the structure, handles escaping, and can output raw strings for the next command with `-r`.

**Quiz:** Which tool is best for extracting a field from a JSON API response in a shell script?

- [ ] grep
- [ ] cut
- [x] jq
- [ ] wc

*Answer:* jq. jq parses JSON structurally; text tools break when formatting changes.