Text Processing with grep, sed, awk and jq

The Unix toolbox for text

Most scripts glue text tools together. grep finds lines matching a pattern: -i ignore case, -v invert, -c count, -E extended regular expressions, -r recursive, -q quiet (just the exit status), -o print only the match. sed edits streams: sed 's/old/new/g' substitutes, -n '5,10p' prints a range, -i edits files in place (GNU and BSD/macOS differ: GNU accepts -i, macOS needs -i ''). awk processes columns and records: awk -F, '{sum += $3} END {print sum}' sums the third CSV column; it handles floating-point maths too. cut, sort (-n numeric, -k key, -u unique), uniq -c, wc -l, tr and head/tail complete the set. For JSON, use jq rather than grep: jq -r '.items[] | select(.status=="active") | .name'. For CSV with quoted fields containing commas, awk and cut are fragile; use a proper CSV tool or a small Python script.

Everyday text processing

Each line is a common task in operations scripts.

grep -E 'ERROR|FATAL' app.log | grep -v 'healthcheck' | tail -20
grep -q 'listen 443' /etc/nginx/nginx.conf && echo "TLS configured"

sed -E 's/password=[^&]+/password=****/g' requests.log > requests.redacted.log
sed -n '/BEGIN CERT/,/END CERT/p' bundle.pem

# total and average response time (column 10) from a log
awk '{ total += $10; n++ } END { if (n) printf "avg %.1f ms over %d requests\n", total/n, n }' access.log

# JSON from an API: names of running instances
curl -fsS "$API_URL/instances" | jq -r '.items[] | select(.state == "running") | .name'

sort -t, -k3,3nr sales.csv | head -10   # top 10 rows by 3rd column, numeric desc

Parse JSON with jq, not grep

Grepping JSON breaks when fields are reordered, reformatted or nested. jq understands the structure, handles escaping, and can output raw strings for the next command with -r.

Quick check: Which tool is best for extracting a field from a JSON API response in a shell script?

• grep
• cut
• jq
• wc