Pipes and Redirects: The Unix Superpower

Every Unix command does one thing well, then hands its output to the next command. This is the Unix philosophy — and the mechanism that makes it work is pipes and redirects. Once you internalize this, you stop writing scripts for things that should be one-liners.

Three streams every command has

• stdin (file descriptor 0) — input
• stdout (file descriptor 1) — normal output
• stderr (file descriptor 2) — error output

By default, stdin reads from your keyboard, and stdout/stderr both write to your terminal. Redirects change where these go.

Redirect output to a file

ls > files.txt           # write stdout to files.txt (overwrite)
ls >> files.txt          # append stdout to files.txt
ls 2> errors.txt         # write stderr to errors.txt
ls > out.txt 2>&1        # both stdout and stderr to out.txt
ls &> all.txt            # bash shortcut for the above
ls > /dev/null 2>&1      # discard everything

Redirect input from a file

sort < unsorted.txt
mysql -u root < schema.sql
wc -l < /etc/passwd     # count lines without showing filename

Pipes: connect commands

The pipe character | connects one command’s stdout to the next command’s stdin.

# Count how many users are on the system
cat /etc/passwd | wc -l

# Find all listening ports owned by nginx
ss -tlnp | grep nginx

# Largest 5 files in current directory
du -sh * | sort -h | tail -5

# Unique IPs that hit your nginx today
awk '{print $1}' /var/log/nginx/access.log | sort -u | wc -l

Real one-liners that show the power

# Top 10 IPs by request count in nginx logs
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# Disk usage of every subdirectory, sorted
du -sh */ | sort -h

# Find files modified in last 24h, biggest first
find . -mtime -1 -type f -exec ls -lhS {} +

# All running processes by memory, top 10
ps aux --sort=-%mem | head

# Count lines of code in your Python project
find . -name "*.py" | xargs wc -l | tail -1

Tee: split output to file AND screen

# See the output AND save it
ls -la | tee listing.txt

# Useful with sudo when you need to write to a protected file
echo "127.0.0.1 myhost" | sudo tee -a /etc/hosts

xargs: turn output into arguments

Some commands don’t read from stdin (like rm). Use xargs to feed them.

# Delete all *.tmp files
find . -name "*.tmp" | xargs rm

# Safer with -0 for filenames with spaces
find . -name "*.tmp" -print0 | xargs -0 rm

# Run command per input line
cat servers.txt | xargs -I {} ssh {} "uname -r"

Process substitution

Treat command output as a file:

# Compare two commands' outputs
diff <(ls dir1) <(ls dir2)

# Feed multiple commands to a tool that wants files
grep "error" <(journalctl -u nginx) <(journalctl -u php-fpm)

Common mistakes

• cmd > file 2>&1 works. cmd 2>&1 > file doesn’t redirect stderr (order matters). Use &> in bash to avoid this.
• cat file | grep x wastes a process. Use grep x file directly. (This is the famous “useless use of cat.”)
• Pipes go through stdout only. To pipe stderr too, use 2>&1 | or bash’s |&.

What to learn next

Now that you can chain commands, the next big topic is permissions: who can run what, and how to control it.