Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev | Chapter 16. External Filters, Programs and Commands | Next

# 16.2. Complex Commands

Commands for more advanced users

find

  -exec `COMMAND` \;

  Carries out `COMMAND` on each file that find matches. The command sequence terminates with ; (the ";" is escaped to make certain the shell passes it to find literally, without interpreting it as a special character).

       bash$ find ~/ -name '*.txt'
       /home/bozo/.kde/share/apps/karm/karmdata.txt
       /home/bozo/misc/irmeyc.txt
       /home/bozo/test-scripts/1.txt

  If `COMMAND` contains {}, then find substitutes the full path name of the selected file for "{}".

         1 find ~/ -name 'core*' -exec rm {} \;
         2 # Removes all core dump files from user's home directory.

         1 find /home/bozo/projects -mtime -1
         2 #                               ^   Note minus sign!
         3 #  Lists all files in /home/bozo/projects directory tree
         4 #+ that were modified within the last day (current_day - 1).
         5 #
         6 find /home/bozo/projects -mtime 1
         7 #  Same as above, but modified *exactly* one day ago.
         8 #
         9 #  mtime = last modification time of the target file
        10 #  ctime = last status change time (via 'chmod' or otherwise)
        11 #  atime = last access time
        12
        13 DIR=/home/bozo/junk_files
        14 find "$DIR" -type f -atime +5 -exec rm {} \;
        15 #                          ^           ^^
        16 #  Curly brackets are placeholder for the path name output by "find."
        17 #
        18 #  Deletes all files in "/home/bozo/junk_files"
        19 #+ that have not been accessed in *at least* 5 days (plus sign ... +5).
        20 #
        21 #  "-type filetype", where
        22 #  f = regular file
        23 #  d = directory
        24 #  l = symbolic link, etc.
        25 #
        26 #  (The 'find' manpage and info page have complete option listings.)

         1 find /etc -exec grep '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' {} \;
         2
         3 # Finds all IP addresses (xxx.xxx.xxx.xxx) in /etc directory files.
         4 # There a few extraneous hits. Can they be filtered out?
         5
         6 # Possibly by:
         7
         8 find /etc -type f -exec cat '{}' \; | tr -c '.[:digit:]' '\n' \
         9 | grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$'
        10 #
        11 #  [:digit:] is one of the character classes
        12 #+ introduced with the POSIX 1003.2 standard.
        13
        14 # Thanks, Stéphane Chazelas.

  | The `-exec` option to find should not be confused with the exec shell builtin.

  Example 16-3. Badname, eliminate file names in current directory containing bad characters and whitespace.

         1 #!/bin/bash
         2 # badname.sh
         3 # Delete filenames in current directory containing bad characters.
         4
         5 for filename in *
         6 do
         7   badname=`echo "$filename" | sed -n /[\+\{\;\"\\\=\?~\(\)\<\>\&\*\|\$]/p`
         8 # badname=`echo "$filename" | sed -n '/[+{;"\=?~()<>&*|$]/p'`  also works.
         9 # Deletes files containing these nasties:     + { ; " \ = ? ~ ( ) < > & * | $
        10 #
        11   rm $badname 2>/dev/null
        12 #             ^^^^^^^^^^^ Error messages deep-sixed.
        13 done
        14
        15 # Now, take care of files containing all manner of whitespace.
        16 find . -name "* *" -exec rm -f {} \;
        17 # The path name of the file that _find_ finds replaces the "{}".
        18 # The '\' ensures that the ';' is interpreted literally, as end of command.
        19
        20 exit 0
        21
        22 #---------------------------------------------------------------------
        23 # Commands below this line will not execute because of _exit_ command.
        24
        25 # An alternative to the above script:
        26 find . -name '*[+{;"\\=?~()<>&*|$ ]*' -maxdepth 0 \
        27 -exec rm -f '{}' \;
        28 #  The "-maxdepth 0" option ensures that _find_ will not search
        29 #+ subdirectories below $PWD.
        30
        31 # (Thanks, S.C.)

  Example 16-4. Deleting a file by its inode number

         1 #!/bin/bash
         2 # idelete.sh: Deleting a file by its inode number.
         3
         4 #  This is useful when a filename starts with an illegal character,
         5 #+ such as ? or -.
         6
         7 ARGCOUNT=1                      # Filename arg must be passed to script.
         8 E_WRONGARGS=70
         9 E_FILE_NOT_EXIST=71
        10 E_CHANGED_MIND=72
        11
        12 if [ $# -ne "$ARGCOUNT" ]
        13 then
        14   echo "Usage: `basename $0` filename"
        15   exit $E_WRONGARGS
        16 fi
        17
        18 if [ ! -e "$1" ]
        19 then
        20   echo "File \""$1"\" does not exist."
        21   exit $E_FILE_NOT_EXIST
        22 fi
        23
        24 inum=`ls -i | grep "$1" | awk '{print $1}'`
        25 # inum = inode (index node) number of file
        26 # -----------------------------------------------------------------------
        27 # Every file has an inode, a record that holds its physical address info.
        28 # -----------------------------------------------------------------------
        29
        30 echo; echo -n "Are you absolutely sure you want to delete \"$1\" (y/n)? "
        31 # The '-v' option to 'rm' also asks this.
        32 read answer
        33 case "$answer" in
        34 [nN]) echo "Changed your mind, huh?"
        35       exit $E_CHANGED_MIND
        36       ;;
        37 *)    echo "Deleting file \"$1\".";;
        38 esac
        39
        40 find . -inum $inum -exec rm {} \;
        41 #                           ^^
        42 #        Curly brackets are placeholder
        43 #+       for text output by "find."
        44 echo "File "\"$1"\" deleted!"
        45
        46 exit 0

  The find command also works without the `-exec` option.

         1 #!/bin/bash
         2 #  Find suid root files.
         3 #  A strange suid file might indicate a security hole,
         4 #+ or even a system intrusion.
         5
         6 directory="/usr/sbin"
         7 # Might also try /sbin, /bin, /usr/bin, /usr/local/bin, etc.
         8 permissions="+4000"  # suid root (dangerous!)
         9
        10
        11 for file in $( find "$directory" -perm "$permissions" )
        12 do
        13   ls -ltF --author "$file"
        14 done

  See Example 16-30, Example 3-4, and Example 11-10 for scripts using find. Its manpage provides more detail on this complex and powerful command.

xargs

  A filter for feeding arguments to a command, and also a tool for assembling the commands themselves. It breaks a data stream into small enough chunks for filters and commands to process. Consider it as a powerful replacement for backquotes. In situations where command substitution fails with a too many arguments error, substituting xargs often works. [1] Normally, xargs reads from `stdin` or from a pipe, but it can also be given the output of a file.

  The default command for xargs is echo. This means that input piped to xargs may have linefeeds and other whitespace characters stripped out.

       bash$ ls -l
       total 0
       -rw-rw-r--    1 bozo  bozo         0 Jan 29 23:58 file1
       -rw-rw-r--    1 bozo  bozo         0 Jan 29 23:58 file2

       bash$ ls -l | xargs
       total 0 -rw-rw-r-- 1 bozo bozo 0 Jan 29 23:58 file1 -rw-rw-r-- 1 bozo bozo 0 Jan...

       bash$ find ~/mail -type f | xargs grep "Linux"
       ./misc:User-Agent: slrn/0.9.8.1 (Linux)
       ./sent-mail-jul-2005: hosted by the Linux Documentation Project.
       ./sent-mail-jul-2005: (Linux Documentation Project Site, rtf version)
       ./sent-mail-jul-2005: Subject: Criticism of Bozo's Windows/Linux article
       ./sent-mail-jul-2005: while mentioning that the Linux ext2/ext3 filesystem
       . . .

  `ls | xargs -p -l gzip` gzips every file in current directory, one at a time, prompting before each operation.

  Note that xargs processes the arguments passed to it sequentially, one at a time.

       bash$ find /usr/bin | xargs file
       /usr/bin:          directory
       /usr/bin/foomatic-ppd-options:          perl script text executable
       . . .

  | An interesting xargs option is `-n `NN``, which limits to `NN` the number of arguments passed. `ls | xargs -n 8 echo` lists the files in the current directory in `8` columns.

  Another useful option is `-0`, in combination with `find -print0` or `grep -lZ`. This allows handling arguments containing whitespace or quotes.

  `find / -type f -print0 | xargs -0 grep -liwZ GUI | xargs -0 rm -f`

  `grep -rliwZ GUI / | xargs -0 rm -f`

  Either of the above will remove any file containing "GUI". (Thanks, S.C.)

  Or:

         1 cat /proc/"$pid"/"$OPTION" | xargs -0 echo
         2 #  Formats output:         ^^^^^^^^^^^^^^^
         3 #  From Han Holl's fixup of "get-commandline.sh"
         4 #+ script in "/dev and /proc" chapter.

  The `-P` option to xargs permits running processes in parallel. This speeds up execution in a machine with a multicore CPU.

         1 #!/bin/bash
         2
         3 ls *gif | xargs -t -n1 -P2 gif2png
         4 # Converts all the gif images in current directory to png.
         5
         6 # Options:
         7 # =======
         8 # -t    Print command to stderr.
         9 # -n1   At most 1 argument per command line.
        10 # -P2   Run up to 2 processes simultaneously.
        11
        12 # Thank you, Roberto Polli, for the inspiration.

  Example 16-5. Logfile: Using xargs to monitor system log

         1 #!/bin/bash
         2
         3 # Generates a log file in current directory
         4 # from the tail end of /var/log/messages.
         5
         6 # Note: /var/log/messages must be world readable
         7 # if this script invoked by an ordinary user.
         8 #         #root chmod 644 /var/log/messages
         9
        10 LINES=5
        11
        12 ( date; uname -a ) >>logfile
        13 # Time and machine name
        14 echo ---------------------------------------------------------- >>logfile
        15 tail -n $LINES /var/log/messages | xargs | fmt -s >>logfile
        16 echo >>logfile
        17 echo >>logfile
        18
        19 exit 0
        20
        21 #  Note:
        22 #  ----
        23 #  As Frank Wang points out,
        24 #+ unmatched quotes (either single or double quotes) in the source file
        25 #+ may give xargs indigestion.
        26 #
        27 #  He suggests the following substitution for line 15:
        28 #  tail -n $LINES /var/log/messages | tr -d "\"'" | xargs | fmt -s >>logfile
        29
        30
        31
        32 #  Exercise:
        33 #  --------
        34 #  Modify this script to track changes in /var/log/messages at intervals
        35 #+ of 20 minutes.
        36 #  Hint: Use the "watch" command.

  As in find, a curly bracket pair serves as a placeholder for replacement text.

  Example 16-6. Copying files in current directory to another

         1 #!/bin/bash
         2 # copydir.sh
         3
         4 #  Copy (verbose) all files in current directory ($PWD)
         5 #+ to directory specified on command-line.
         6
         7 E_NOARGS=85
         8
         9 if [ -z "$1" ]   # Exit if no argument given.
        10 then
        11   echo "Usage: `basename $0` directory-to-copy-to"
        12   exit $E_NOARGS
        13 fi
        14
        15 ls . | xargs -i -t cp ./{} $1
        16 #            ^^ ^^      ^^
        17 #  -t is "verbose" (output command-line to stderr) option.
        18 #  -i is "replace strings" option.
        19 #  {} is a placeholder for output text.
        20 #  This is similar to the use of a curly-bracket pair in "find."
        21 #
        22 #  List the files in current directory (ls .),
        23 #+ pass the output of "ls" as arguments to "xargs" (-i -t options),
        24 #+ then copy (cp) these arguments ({}) to new directory ($1).
        25 #
        26 #  The net result is the exact equivalent of
        27 #+   cp * $1
        28 #+ unless any of the filenames has embedded "whitespace" characters.
        29
        30 exit 0

  Example 16-7. Killing processes by name

         1 #!/bin/bash
         2 # kill-byname.sh: Killing processes by name.
         3 # Compare this script with kill-process.sh.
         4
         5 #  For instance,
         6 #+ try "./kill-byname.sh xterm" --
         7 #+ and watch all the xterms on your desktop disappear.
         8
         9 #  Warning:
        10 #  -------
        11 #  This is a fairly dangerous script.
        12 #  Running it carelessly (especially as root)
        13 #+ can cause data loss and other undesirable effects.
        14
        15 E_BADARGS=66
        16
        17 if test -z "$1"  # No command-line arg supplied?
        18 then
        19   echo "Usage: `basename $0` Process(es)_to_kill"
        20   exit $E_BADARGS
        21 fi
        22
        23
        24 PROCESS_NAME="$1"
        25 ps ax | grep "$PROCESS_NAME" | awk '{print $1}' | xargs -i kill {} 2&>/dev/null
        26 #                                                       ^^      ^^
        27
        28 # ---------------------------------------------------------------
        29 # Notes:
        30 # -i is the "replace strings" option to xargs.
        31 # The curly brackets are the placeholder for the replacement.
        32 # 2&>/dev/null suppresses unwanted error messages.
        33 #
        34 # Can  grep "$PROCESS_NAME" be replaced by pidof "$PROCESS_NAME"?
        35 # ---------------------------------------------------------------
        36
        37 exit $?
        38
        39 #  The "killall" command has the same effect as this script,
        40 #+ but using it is not quite as educational.

  Example 16-8. Word frequency analysis using xargs

         1 #!/bin/bash
         2 # wf2.sh: Crude word frequency analysis on a text file.
         3
         4 # Uses 'xargs' to decompose lines of text into single words.
         5 # Compare this example to the "wf.sh" script later on.
         6
         7
         8 # Check for input file on command-line.
         9 ARGS=1
        10 E_BADARGS=85
        11 E_NOFILE=86
        12
        13 if [ $# -ne "$ARGS" ]
        14 # Correct number of arguments passed to script?
        15 then
        16   echo "Usage: `basename $0` filename"
        17   exit $E_BADARGS
        18 fi
        19
        20 if [ ! -f "$1" ]       # Does file exist?
        21 then
        22   echo "File \"$1\" does not exist."
        23   exit $E_NOFILE
        24 fi
        25
        26
        27
        28 #####################################################
        29 cat "$1" | xargs -n1 | \
        30 #  List the file, one word per line.
        31 tr A-Z a-z | \
        32 #  Shift characters to lowercase.
        33 sed -e 's/\.//g'  -e 's/\,//g' -e 's/ /\
        34 /g' | \
        35 #  Filter out periods and commas, and
        36 #+ change space between words to linefeed,
        37 sort | uniq -c | sort -nr
        38 #  Finally remove duplicates, prefix occurrence count
        39 #+ and sort numerically.
        40 #####################################################
        41
        42 #  This does the same job as the "wf.sh" example,
        43 #+ but a bit more ponderously, and it runs more slowly (why?).
        44
        45 exit $?

`expr`

  All-purpose expression evaluator: Concatenates and evaluates the arguments according to the operation given (arguments must be separated by spaces). Operations may be arithmetic, comparison, string, or logical.

  `expr 3 + 5`

    returns `8`

  `expr 5 % 3`

    returns 2

  `expr 1 / 0`

    returns the error message, expr: division by zero

    Illegal arithmetic operations not allowed.

  `expr 5 \* 3`

    returns 15

    The multiplication operator must be escaped when used in an arithmetic expression with expr.

  `y=`expr $y + 1``

    Increment a variable, with the same effect as `let y=y+1` and `y=$(($y+1))`. This is an example of arithmetic expansion.

  `z=`expr substr $string $position $length``

    Extract substring of $length characters, starting at $position.

  Example 16-9. Using expr

         1 #!/bin/bash
         2
         3 # Demonstrating some of the uses of 'expr'
         4 # =======================================
         5
         6 echo
         7
         8 # Arithmetic Operators
         9 # ---------- ---------
        10
        11 echo "Arithmetic Operators"
        12 echo
        13 a=`expr 5 + 3`
        14 echo "5 + 3 = $a"
        15
        16 a=`expr $a + 1`
        17 echo
        18 echo "a + 1 = $a"
        19 echo "(incrementing a variable)"
        20
        21 a=`expr 5 % 3`
        22 # modulo
        23 echo
        24 echo "5 mod 3 = $a"
        25
        26 echo
        27 echo
        28
        29 # Logical Operators
        30 # ------- ---------
        31
        32 #  Returns 1 if true, 0 if false,
        33 #+ opposite of normal Bash convention.
        34
        35 echo "Logical Operators"
        36 echo
        37
        38 x=24
        39 y=25
        40 b=`expr $x = $y`         # Test equality.
        41 echo "b = $b"            # 0  ( $x -ne $y )
        42 echo
        43
        44 a=3
        45 b=`expr $a \> 10`
        46 echo 'b=`expr $a \> 10`, therefore...'
        47 echo "If a > 10, b = 0 (false)"
        48 echo "b = $b"            # 0  ( 3 ! -gt 10 )
        49 echo
        50
        51 b=`expr $a \< 10`
        52 echo "If a < 10, b = 1 (true)"
        53 echo "b = $b"            # 1  ( 3 -lt 10 )
        54 echo
        55 # Note escaping of operators.
        56
        57 b=`expr $a \<= 3`
        58 echo "If a <= 3, b = 1 (true)"
        59 echo "b = $b"            # 1  ( 3 -le 3 )
        60 # There is also a "\>=" operator (greater than or equal to).
        61
        62
        63 echo
        64 echo
        65
        66
        67
        68 # String Operators
        69 # ------ ---------
        70
        71 echo "String Operators"
        72 echo
        73
        74 a=1234zipper43231
        75 echo "The string being operated upon is \"$a\"."
        76
        77 # length: length of string
        78 b=`expr length $a`
        79 echo "Length of \"$a\" is $b."
        80
        81 # index: position of first character in substring
        82 #        that matches a character in string
        83 b=`expr index $a 23`
        84 echo "Numerical position of first \"2\" in \"$a\" is \"$b\"."
        85
        86 # substr: extract substring, starting position & length specified
        87 b=`expr substr $a 2 6`
        88 echo "Substring of \"$a\", starting at position 2,\
        89 and 6 chars long is \"$b\"."
        90
        91
        92 #  The default behavior of the 'match' operations is to
        93 #+ search for the specified match at the BEGINNING of the string.
        94 #
        95 #       Using Regular Expressions ...
        96 b=`expr match "$a" '[0-9]*'`               #  Numerical count.
        97 echo Number of digits at the beginning of \"$a\" is $b.
        98 b=`expr match "$a" '\([0-9]*\)'`           #  Note that escaped parentheses
        99 #                   ==      ==             #+ trigger substring match.
       100 echo "The digits at the beginning of \"$a\" are \"$b\"."
       101
       102 echo
       103
       104 exit 0

  The : (null) operator can substitute for match. For example, `b=`expr $a : [0-9]*`` is the exact equivalent of `b=`expr match $a [0-9]*`` in the above listing.

         1 #!/bin/bash
         2
         3 echo
         4 echo "String operations using \"expr \$string : \" construct"
         5 echo "==================================================="
         6 echo
         7
         8 a=1234zipper5FLIPPER43231
         9
        10 echo "The string being operated upon is \"`expr "$a" : '\(.*\)'`\"."
        11 #     Escaped parentheses grouping operator.            ==  ==
        12
        13 #       ***************************
        14 #+          Escaped parentheses
        15 #+           match a substring
        16 #       ***************************
        17
        18
        19 #  If no escaped parentheses ...
        20 #+ then 'expr' converts the string operand to an integer.
        21
        22 echo "Length of \"$a\" is `expr "$a" : '.*'`."   # Length of string
        23
        24 echo "Number of digits at the beginning of \"$a\" is `expr "$a" : '[0-9]*'`."
        25
        26 # ------------------------------------------------------------------------- #
        27
        28 echo
        29
        30 echo "The digits at the beginning of \"$a\" are `expr "$a" : '\([0-9]*\)'`."
        31 #                                                             ==      ==
        32 echo "The first 7 characters of \"$a\" are `expr "$a" : '\(.......\)'`."
        33 #         =====                                          ==       ==
        34 # Again, escaped parentheses force a substring match.
        35 #
        36 echo "The last 7 characters of \"$a\" are `expr "$a" : '.*\(.......\)'`."
        37 #         ====                  end of string operator  ^^
        38 #  (In fact, means skip over one or more of any characters until specified
        39 #+  substring found.)
        40
        41 echo
        42
        43 exit 0

The above script illustrates how expr uses the escaped parentheses -- \( ... \) -- grouping operator in tandem with regular expression parsing to match a substring. Here is a another example, this time from "real life."

       1 # Strip the whitespace from the beginning and end.
       2 LRFDATE=`expr "$LRFDATE" : '[[:space:]]*\(.*\)[[:space:]]*$'`
       3
       4 #  From Peter Knowles' "booklistgen.sh" script
       5 #+ for converting files to Sony Librie/PRS-50X format.
       6 #  (http://booklistgensh.peterknowles.com)

Perl, sed, and awk have far superior string parsing facilities. A short sed or awk "subroutine" within a script (see Section 36.2) is an attractive alternative to expr.

See Section 10.1 for more on using expr in string operations.

### Notes

[1] | And even when xargs is not strictly necessary, it can speed up execution of a command involving batch-processing of multiple files.

Prev | Home | Next
External Filters, Programs and Commands | Up | Time / Date Commands