Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev |  | Next

# Chapter 20. I/O Redirection

There are always three default files [1] open, `stdin` (the keyboard), `stdout` (the screen), and `stderr` (error messages output to the screen). These, and any other open files, can be redirected. Redirection simply means capturing output from a file, command, program, script, or even code block within a script (see Example 3-1 and Example 3-2) and sending it as input to another file, command, program, or script.

Each open file gets assigned a file descriptor. [2] The file descriptors for `stdin`, `stdout`, and `stderr` are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to `stdin`, `stdout`, or `stderr` as a temporary duplicate link. [3] This simplifies restoration to normal after complex redirection and reshuffling (see Example 20-1).

       1    COMMAND_OUTPUT >
       2       # Redirect stdout to a file.
       3       # Creates the file if not present, otherwise overwrites it.
       4
       5       ls -lR > dir-tree.list
       6       # Creates a file containing a listing of the directory tree.
       7
       8    : > filename
       9       # The > truncates file "filename" to zero length.
      10       # If file not present, creates zero-length file (same effect as 'touch').
      11       # The : serves as a dummy placeholder, producing no output.
      12
      13    > filename
      14       # The > truncates file "filename" to zero length.
      15       # If file not present, creates zero-length file (same effect as 'touch').
      16       # (Same result as ": >", above, but this does not work with some shells.)
      17
      18    COMMAND_OUTPUT >>
      19       # Redirect stdout to a file.
      20       # Creates the file if not present, otherwise appends to it.
      21
      22
      23       # Single-line redirection commands (affect only the line they are on):
      24       # --------------------------------------------------------------------
      25
      26    1>filename
      27       # Redirect stdout to file "filename."
      28    1>>filename
      29       # Redirect and append stdout to file "filename."
      30    2>filename
      31       # Redirect stderr to file "filename."
      32    2>>filename
      33       # Redirect and append stderr to file "filename."
      34    &>filename
      35       # Redirect both stdout and stderr to file "filename."
      36       # This operator is now functional, as of Bash 4, final release.
      37
      38    M>N
      39      # "M" is a file descriptor, which defaults to 1, if not explicitly set.
      40      # "N" is a filename.
      41      # File descriptor "M" is redirect to file "N."
      42    M>&N
      43      # "M" is a file descriptor, which defaults to 1, if not set.
      44      # "N" is another file descriptor.
      45
      46       #==============================================================================
      47
      48       # Redirecting stdout, one line at a time.
      49       LOGFILE=script.log
      50
      51       echo "This statement is sent to the log file, \"$LOGFILE\"." 1>$LOGFILE
      52       echo "This statement is appended to \"$LOGFILE\"." 1>>$LOGFILE
      53       echo "This statement is also appended to \"$LOGFILE\"." 1>>$LOGFILE
      54       echo "This statement is echoed to stdout, and will not appear in \"$LOGFILE\"."
      55       # These redirection commands automatically "reset" after each line.
      56
      57
      58
      59       # Redirecting stderr, one line at a time.
      60       ERRORFILE=script.errors
      61
      62       bad_command1 2>$ERRORFILE       #  Error message sent to $ERRORFILE.
      63       bad_command2 2>>$ERRORFILE      #  Error message appended to $ERRORFILE.
      64       bad_command3                    #  Error message echoed to stderr,
      65                                       #+ and does not appear in $ERRORFILE.
      66       # These redirection commands also automatically "reset" after each line.
      67       #=======================================================================

       1    2>&1
       2       # Redirects stderr to stdout.
       3       # Error messages get sent to same place as standard output.
       4         >>filename 2>&1
       5             bad_command >>filename 2>&1
       6             # Appends both stdout and stderr to the file "filename" ...
       7         2>&1 | [command(s)]
       8             bad_command 2>&1 | awk '{print $5}'   # found
       9             # Sends stderr through a pipe.
      10             # |& was added to Bash 4 as an abbreviation for 2>&1 |.
      11
      12    i>&j
      13       # Redirects file descriptor i to j.
      14       # All output of file pointed to by i gets sent to file pointed to by j.
      15
      16    >&j
      17       # Redirects, by default, file descriptor 1 (stdout) to j.
      18       # All stdout gets sent to file pointed to by j.

       1    0< FILENAME
       2     < FILENAME
       3       # Accept input from a file.
       4       # Companion command to ">", and often used in combination with it.
       5       #
       6       # grep search-word <filename
       7
       8
       9    [j]<>filename
      10       #  Open file "filename" for reading and writing,
      11       #+ and assign file descriptor "j" to it.
      12       #  If "filename" does not exist, create it.
      13       #  If file descriptor "j" is not specified, default to fd 0, stdin.
      14       #
      15       #  An application of this is writing at a specified place in a file.
      16       echo 1234567890 > File    # Write string to "File".
      17       exec 3<> File             # Open "File" and assign fd 3 to it.
      18       read -n 4 <&3             # Read only 4 characters.
      19       echo -n . >&3             # Write a decimal point there.
      20       exec 3>&-                 # Close fd 3.
      21       cat File                  # ==> 1234.67890
      22       #  Random access, by golly.
      23
      24
      25
      26    |
      27       # Pipe.
      28       # General purpose process and command chaining tool.
      29       # Similar to ">", but more general in effect.
      30       # Useful for chaining commands, scripts, files, and programs together.
      31       cat *.txt | sort | uniq > result-file
      32       # Sorts the output of all the .txt files and deletes duplicate lines,
      33       # finally saves results to "result-file".

Multiple instances of input and output redirection and/or pipes can be combined in a single command line.

       1 command < input-file > output-file
       2 # Or the equivalent:
       3 < input-file command > output-file   # Although this is non-standard.
       4
       5 command1 | command2 | command3 > output-file

See Example 16-31 and Example A-14.

Multiple output streams may be redirected to one file.

       1 ls -yz >> command.log 2>&1
       2 #  Capture result of illegal options "yz" in file "command.log."
       3 #  Because stderr is redirected to the file,
       4 #+ any error messages will also be there.
       5
       6 #  Note, however, that the following does *not* give the same result.
       7 ls -yz 2>&1 >> command.log
       8 #  Outputs an error message, but does not write to file.
       9 #  More precisely, the command output (in this case, null)
      10 #+ writes to the file, but the error message goes only to stdout.
      11
      12 #  If redirecting both stdout and stderr,
      13 #+ the order of the commands makes a difference.

Closing File Descriptors

n<&-

  Close input file descriptor `n`.

0<&-, <&-

  Close `stdin`.

n>&-

  Close output file descriptor `n`.

1>&-, >&-

  Close `stdout`.

Child processes inherit open file descriptors. This is why pipes work. To prevent an fd from being inherited, close it.

       1 # Redirecting only stderr to a pipe.
       2
       3 exec 3>&1                              # Save current "value" of stdout.
       4 ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
       5 #              ^^^^   ^^^^
       6 exec 3>&-                              # Now close it for the remainder of the script.
       7
       8 # Thanks, S.C.

For a more detailed introduction to I/O redirection see Appendix F.

# 20.1. Using exec

An exec <filename command redirects `stdin` to a file. From that point on, all `stdin` comes from that file, rather than its normal source (usually keyboard input). This provides a method of reading a file line by line and possibly parsing each line of input using sed and/or awk.

Example 20-1. Redirecting `stdin` using exec

       1 #!/bin/bash
       2 # Redirecting stdin using 'exec'.
       3
       4
       5 exec 6<&0          # Link file descriptor #6 with stdin.
       6                    # Saves stdin.
       7
       8 exec < data-file   # stdin replaced by file "data-file"
       9
      10 read a1            # Reads first line of file "data-file".
      11 read a2            # Reads second line of file "data-file."
      12
      13 echo
      14 echo "Following lines read from file."
      15 echo "-------------------------------"
      16 echo $a1
      17 echo $a2
      18
      19 echo; echo; echo
      20
      21 exec 0<&6 6<&-
      22 #  Now restore stdin from fd #6, where it had been saved,
      23 #+ and close fd #6 ( 6<&- ) to free it for other processes to use.
      24 #
      25 # <&6 6<&-    also works.
      26
      27 echo -n "Enter data  "
      28 read b1  # Now "read" functions as expected, reading from normal stdin.
      29 echo "Input read from stdin."
      30 echo "----------------------"
      31 echo "b1 = $b1"
      32
      33 echo
      34
      35 exit 0

Similarly, an exec >filename command redirects `stdout` to a designated file. This sends all command output that would normally go to `stdout` to that file.

| exec N > filename affects the entire script or current shell. Redirection in the PID of the script or shell from that point on has changed. However . . . N > filename affects only the newly-forked process, not the entire script or shell. Thank you, Ahmed Darwish, for pointing this out.

Example 20-2. Redirecting `stdout` using exec

       1 #!/bin/bash
       2 # reassign-stdout.sh
       3
       4 LOGFILE=logfile.txt
       5
       6 exec 6>&1           # Link file descriptor #6 with stdout.
       7                     # Saves stdout.
       8
       9 exec > $LOGFILE     # stdout replaced with file "logfile.txt".
      10
      11 # ----------------------------------------------------------- #
      12 # All output from commands in this block sent to file $LOGFILE.
      13
      14 echo -n "Logfile: "
      15 date
      16 echo "-------------------------------------"
      17 echo
      18
      19 echo "Output of \"ls -al\" command"
      20 echo
      21 ls -al
      22 echo; echo
      23 echo "Output of \"df\" command"
      24 echo
      25 df
      26
      27 # ----------------------------------------------------------- #
      28
      29 exec 1>&6 6>&-      # Restore stdout and close file descriptor #6.
      30
      31 echo
      32 echo "== stdout now restored to default == "
      33 echo
      34 ls -al
      35 echo
      36
      37 exit 0

Example 20-3. Redirecting both `stdin` and `stdout` in the same script with exec

       1 #!/bin/bash
       2 # upperconv.sh
       3 # Converts a specified input file to uppercase.
       4
       5 E_FILE_ACCESS=70
       6 E_WRONG_ARGS=71
       7
       8 if [ ! -r "$1" ]     # Is specified input file readable?
       9 then
      10   echo "Can't read from input file!"
      11   echo "Usage: $0 input-file output-file"
      12   exit $E_FILE_ACCESS
      13 fi                   #  Will exit with same error
      14                      #+ even if input file ($1) not specified (why?).
      15
      16 if [ -z "$2" ]
      17 then
      18   echo "Need to specify output file."
      19   echo "Usage: $0 input-file output-file"
      20   exit $E_WRONG_ARGS
      21 fi
      22
      23
      24 exec 4<&0
      25 exec < $1            # Will read from input file.
      26
      27 exec 7>&1
      28 exec > $2            # Will write to output file.
      29                      # Assumes output file writable (add check?).
      30
      31 # -----------------------------------------------
      32     cat - | tr a-z A-Z   # Uppercase conversion.
      33 #   ^^^^^                # Reads from stdin.
      34 #           ^^^^^^^^^^   # Writes to stdout.
      35 # However, both stdin and stdout were redirected.
      36 # Note that the 'cat' can be omitted.
      37 # -----------------------------------------------
      38
      39 exec 1>&7 7>&-       # Restore stout.
      40 exec 0<&4 4<&-       # Restore stdin.
      41
      42 # After restoration, the following line prints to stdout as expected.
      43 echo "File \"$1\" written to \"$2\" as uppercase conversion."
      44
      45 exit 0

I/O redirection is a clever way of avoiding the dreaded inaccessible variables within a subshell problem.

Example 20-4. Avoiding a subshell

       1 #!/bin/bash
       2 # avoid-subshell.sh
       3 # Suggested by Matthew Walker.
       4
       5 Lines=0
       6
       7 echo
       8
       9 cat myfile.txt | while read line;
      10                  do {
      11                    echo $line
      12                    (( Lines++ ));  #  Incremented values of this variable
      13                                    #+ inaccessible outside loop.
      14                                    #  Subshell problem.
      15                  }
      16                  done
      17
      18 echo "Number of lines read = $Lines"     # 0
      19                                          # Wrong!
      20
      21 echo "------------------------"
      22
      23
      24 exec 3<> myfile.txt
      25 while read line <&3
      26 do {
      27   echo "$line"
      28   (( Lines++ ));                   #  Incremented values of this variable
      29                                    #+ accessible outside loop.
      30                                    #  No subshell, no problem.
      31 }
      32 done
      33 exec 3>&-
      34
      35 echo "Number of lines read = $Lines"     # 8
      36
      37 echo
      38
      39 exit 0
      40
      41 # Lines below not seen by script.
      42
      43 $ cat myfile.txt
      44
      45 Line 1.
      46 Line 2.
      47 Line 3.
      48 Line 4.
      49 Line 5.
      50 Line 6.
      51 Line 7.
      52 Line 8.

### Notes

[1] | By convention in UNIX and Linux, data streams and peripherals (device files) are treated as files, in a fashion analogous to ordinary files.
[2] | A file descriptor is simply a number that the operating system assigns to an open file to keep track of it. Consider it a simplified type of file pointer. It is analogous to a file handle in C.
[3] | Using `file descriptor 5` might cause problems. When Bash creates a child process, as with exec, the child inherits fd 5 (see Chet Ramey's archived e-mail, SUBJECT: RE: File descriptor 5 is held open). Best leave this particular fd alone.

Prev | Home | Next
Here Documents | Up | Redirecting Code Blocks