Bash Redirection and Piping

Bash, a popular shell in Unix-like systems, provides two essential features for managing the flow of data between commands and files. These are redirection and piping. From this writing, you will learn about the basics of Bash redirection and piping, their differences & cases where you can apply them while Bash scripting.

Table of Contents Expand

Redirection in Bash

Redirection is a process that allows you to change the default input source or output destination of a command. With redirection, you can redirect output to a file instead of displaying it on the terminal, or instruct an application to read input from a file rather than the keyboard. You can redirect your used script commands in different ways whenever necessary. So, the following are the ways of redirection available in Linux:

A. Redirecting Input

Redirecting input is a process where the standard input (stdin) of a command is changed from the keyboard to a file. By, using the ‘<’ symbol you can take input data from a file instead of typing it interactively. For example,

cat < file.txt

B. Redirecting Output

Here, the displaying of the standard output (stdout) of a command is changed from the terminal screen to a file or another command’s input. The ‘>’ symbol is used for this purpose. For example:

ls > output.txt

C. Redirecting Standard Error

The standard error (stderr) is a way to control where error messages, generated by commands are sent. You can redirect stderr to a file, discard it, or merge it with the standard output. You can use the redirection operator “2>” to redirect the standard error of a command to a file, “2>>” to append, or “2> /dev/null” to discard it. For example,

whoami -l 2> output.txt

D. Appending Redirected Output

Normally, when you redirect a stdout to a file with the ‘>’ redirection operator, it overwrites the file. You can also append the stdout to the existing file contents while redirecting by using the ‘>>’ redirection operators. For example,

ls >> output.txt

E. Redirecting Standard Output and Standard Error

In Linux, every process has two default output streams, stdout (represented by file descriptor 1) & standard error or stderr (represented by file descriptor 2). You can redirect both of them using the symbol ‘&> or ‘>&’ or ‘2>&1’. For example,

ls /nonexistent_directory &> log.txt

F. Appending Standard Output and Standard Error

In the same way as appending stdout, you can append both stdout & stderr at the same time to the same file too. For that, use the append symbol ‘>>’. For example,

ls >> output.txt 2>&1

Piping in Bash

Bash piping is a powerful process that allows you to connect multiple commands in Linux and other Unix-like shell environments. In a more simple context, a pipe redirects the standard output from one command to another for processing. By using the pipe character ‘|’, you can connect one command’s output to another command’s input.

Pipes are unidirectional, flowing data from left to right through the pipeline.

The Command Syntax >

Command 1 | command 2 | … | Command N

Difference Between Bash Redirection & Piping

Some users may find the operations of Bash redirection & piping quite similar. However, they have some core differences as they serve different purposes. Read out the following key points of their differences:

A pipe (|) directs the output of one program to another program or utility. Whereas, redirection (‘>’, ‘>>’, ‘<’, etc) controls where the output goes or comes from.
Pipe facilitates seamless data flow between commands in a sequence. And, redirection makes sure that output is redirected to a file (‘>’) or read from a file (‘<’).
Piping is unidirectional, sending data from left to right whilst redirection handles input & output separately.
Each serves a distinct purpose. While pipe does is collaboration, redirection emphasizes controlling inputs & outputs.

Let me simplify the two cases a bit more with the following example:

Program 1 > Program 2 vs Program 1 | Program 2

‘Program 1 > Program 2’

As soon as you execute it, your shell initiates the execution of the program labeled as ‘Program 1’. All output generated by ‘Program 1’ is stored within a file designated as ‘Program 2’ (please note, if ‘Program 2’ exists, its contents are overwritten).

‘Program 1 | Program 2’

On the other hand, if the intention is to forward the output from ‘Program 1’ to ‘Program 2’, the following approach can be used:

Program 1 > temp_file && Program 2 < temp_file

Here, first, your shell will launch ‘Program 1’, and save the output to a file name ‘temp_file’. After that, will initiate the execution of ‘Program 2’, treating the contents of ‘temp_files’ as if they were input from the keyboard.

However, this approach could be tedious. Therefore, the concept of pipes was introduced to simplify the process. So, ‘Program 1 | Program 2’ accomplishes the same objective as ‘Program 1 > temp_file && Program 2 < temp_file’.

A Sample Bash Script Example Combining Redirection and Piping

Now, let’s explore a practical example, where I will use both Bash redirection & piping processes in a single command line. To check yourself, copy the following Bash script on your nano editor & save the file with an appropriate name with the .sh extension.

Script (test.sh) >

#! /bin/bash

ls -l | grep ‘.txt’ > text_files
cat text_files

EXPLANATION

The ls command with its –long option, lists all the contents of a directory. After that, the pipe operator ‘|’ takes this output and passes it as the input for the grep command. Where, the grep command searches from those inputs for all the lines containing the specified ‘.txt’ pattern.

Afterward, the redirection operator ‘>’ redirects the standard output of the command sequence ‘ls -l | grep ‘.txt’’ to a file named ‘text_files’. If the file doesn’t exist, it will be created. If it exists, its contents will be overwritten. Finally, the cat command displays the contents of the file.

After that, use the following command to make the script executable:

chmod u+x test.sh

EXPLANATION

chmod: Changes the permission of files and directories.
u+x: Argument with chmod command to add the executable permission for the user.
test.sh: File which you want to make executable.

Finally, run the script by the following command:

./test.sh

The output of the script displays all the files with ‘.txt’ extension.

Conclusion

To sum up, Bash redirection and piping are vital Unix tools. Redirection controls I/O flow, while piping chains commands. Both make the scripting operations & command line interface easier by saving time for users.