Pipes, redirection, and text processing
Linux's power comes from combining small tools. Redirection connects commands to files, pipes connect commands to each other, and text-processing utilities transform data along the way.
Standard streams
Every process starts with three open streams: standard input (stdin, file descriptor 0), standard output (stdout, 1), and standard error (stderr, 2). By default, stdin reads from the terminal and stdout and stderr write to it. Keeping errors on a separate stream lets programs' normal output be processed while error messages still reach the user.
Redirection
> file sends stdout to a file, replacing it; >> file appends; < file reads stdin from a file; 2> file sends stderr to a file; 2>&1 sends stderr wherever stdout currently goes, so command > all.log 2>&1 captures both. /dev/null discards anything written to it, as in command 2>/dev/null to hide errors. A here document (<<EOF ... EOF) supplies multi-line input inline.
Pipes
A pipe, written |, connects one command's stdout to the next command's stdin: ls -l /etc | less pages a long listing. Pipelines can chain many stages, and each command runs concurrently as data flows between them. Doug McIlroy proposed pipes, and Ken Thompson implemented them in Unix in 1973, embodying the idea of small programs that work together.
Filtering with grep
grep prints lines matching a pattern: grep 'error' app.log. Useful options include -i (ignore case), -v (invert, show non-matching lines), -r (search directories recursively), -n (show line numbers), -c (count matches), and -E (extended regular expressions). Regular expressions describe patterns, such as ^ for line start, $ for line end, . for any character, and [0-9]+ for one or more digits.
Transforming text
sort sorts lines (-n numerically, -r reversed, -k to choose a field); uniq removes adjacent duplicate lines (-c counts them), so it usually follows sort; cut extracts fields (cut -d: -f1 /etc/passwd lists user names); tr translates characters; wc counts lines, words, and characters; head and tail take the start or end of input.
sed and awk
sed, the stream editor, edits text as it flows: sed 's/old/new/g' file replaces every occurrence; -i edits files in place. awk is a small language for field-based processing: awk '{print $1, $3}' prints fields 1 and 3, and awk -F, '$3 > 100 {sum += $3} END {print sum}' sums a CSV column above 100. Together with grep they handle most log and data wrangling.
xargs and tee
xargs builds command lines from input: find . -name '*.tmp' | xargs rm removes every file found (use find -print0 | xargs -0 to handle names with spaces). tee writes its input to a file and to stdout at the same time, useful for saving and viewing output simultaneously, as in make 2>&1 | tee build.log.
Worked example
To find the five IP addresses that requested most pages from a web server log where the address is the first field: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5. awk extracts addresses, sort groups identical ones, uniq -c counts each group, sort -rn orders by count descending, and head -5 keeps the top five, each line showing a count and an address.
Key takeaways
Processes have stdin (0), stdout (1), and stderr (2). Redirection uses >, >>, <, 2>, and 2>&1; /dev/null discards output; here documents supply inline input. Pipes (1973, McIlroy and Thompson) chain commands concurrently. grep filters lines with options such as -i, -v, -r, and regular expressions. sort, uniq, cut, tr, wc, head, and tail transform text. sed edits streams and files; awk processes fields and computes sums. xargs turns input into arguments; tee splits output to a file and the screen. Pipelines such as awk | sort | uniq -c | sort -rn | head answer real questions quickly.
Flashcard
Standard input, descriptor 0
Check your understanding
Why does uniq -c usually follow sort in a pipeline?