Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev | Chapter 37. Bash, versions 2, 3, and 4 | Next

# 37.3. Bash, version 4

Chet Ramey announced Version 4 of Bash on the 20th of February, 2009. This release has a number of significant new features, as well as some important bugfixes.

Among the new goodies:

- Associative arrays. [1]

  Example 37-5. A simple address database

         1 #!/bin/bash4
         2 # fetch_address.sh
         3
         4 declare -A address
         5 #       -A option declares associative array.
         6
         7 address[Charles]="414 W. 10th Ave., Baltimore, MD 21236"
         8 address[John]="202 E. 3rd St., New York, NY 10009"
         9 address[Wilma]="1854 Vermont Ave, Los Angeles, CA 90023"
        10
        11
        12 echo "Charles's address is ${address[Charles]}."
        13 # Charles's address is 414 W. 10th Ave., Baltimore, MD 21236.
        14 echo "Wilma's address is ${address[Wilma]}."
        15 # Wilma's address is 1854 Vermont Ave, Los Angeles, CA 90023.
        16 echo "John's address is ${address[John]}."
        17 # John's address is 202 E. 3rd St., New York, NY 10009.
        18
        19 echo
        20
        21 echo "${!address[*]}"   # The array indices ...
        22 # Charles John Wilma

  Example 37-6. A somewhat more elaborate address database

         1 #!/bin/bash4
         2 # fetch_address-2.sh
         3 # A more elaborate version of fetch_address.sh.
         4
         5 SUCCESS=0
         6 E_DB=99    # Error code for missing entry.
         7
         8 declare -A address
         9 #       -A option declares associative array.
        10
        11
        12 store_address ()
        13 {
        14   address[$1]="$2"
        15   return $?
        16 }
        17
        18
        19 fetch_address ()
        20 {
        21   if [[ -z "${address[$1]}" ]]
        22   then
        23     echo "$1's address is not in database."
        24     return $E_DB
        25   fi
        26
        27   echo "$1's address is ${address[$1]}."
        28   return $?
        29 }
        30
        31
        32 store_address "Lucas Fayne" "414 W. 13th Ave., Baltimore, MD 21236"
        33 store_address "Arvid Boyce" "202 E. 3rd St., New York, NY 10009"
        34 store_address "Velma Winston" "1854 Vermont Ave, Los Angeles, CA 90023"
        35 #  Exercise:
        36 #  Rewrite the above store_address calls to read data from a file,
        37 #+ then assign field 1 to name, field 2 to address in the array.
        38 #  Each line in the file would have a format corresponding to the above.
        39 #  Use a while-read loop to read from file, sed or awk to parse the fields.
        40
        41 fetch_address "Lucas Fayne"
        42 # Lucas Fayne's address is 414 W. 13th Ave., Baltimore, MD 21236.
        43 fetch_address "Velma Winston"
        44 # Velma Winston's address is 1854 Vermont Ave, Los Angeles, CA 90023.
        45 fetch_address "Arvid Boyce"
        46 # Arvid Boyce's address is 202 E. 3rd St., New York, NY 10009.
        47 fetch_address "Bozo Bozeman"
        48 # Bozo Bozeman's address is not in database.
        49
        50 exit $?   # In this case, exit code = 99, since that is function return.

  See Example A-53 for an interesting usage of an associative array.

  Elements of the index array may include embedded space characters, or even leading and/or trailing space characters. However, index array elements containing only whitespace are not permitted.

         1 address[   ]="Blank"   # Error!

- Enhancements to the case construct: the `;;&` and `;&` terminators.

  Example 37-7. Testing characters

         1 #!/bin/bash4
         2
         3 test_char ()
         4 {
         5   case "$1" in
         6     [[:print:]] )  echo "$1 is a printable character.";;&       # |
         7     # The ;;& terminator continues to the next pattern test.      |
         8     [[:alnum:]] )  echo "$1 is an alpha/numeric character.";;&  # v
         9     [[:alpha:]] )  echo "$1 is an alphabetic character.";;&     # v
        10     [[:lower:]] )  echo "$1 is a lowercase alphabetic character.";;&
        11     [[:digit:]] )  echo "$1 is an numeric character.";&         # |
        12     # The ;& terminator executes the next statement ...         # |
        13     %%%@@@@@    )  echo "********************************";;    # v
        14 #   ^^^^^^^^  ... even with a dummy pattern.
        15   esac
        16 }
        17
        18 echo
        19
        20 test_char 3
        21 # 3 is a printable character.
        22 # 3 is an alpha/numeric character.
        23 # 3 is an numeric character.
        24 # ********************************
        25 echo
        26
        27 test_char m
        28 # m is a printable character.
        29 # m is an alpha/numeric character.
        30 # m is an alphabetic character.
        31 # m is a lowercase alphabetic character.
        32 echo
        33
        34 test_char /
        35 # / is a printable character.
        36
        37 echo
        38
        39 # The ;;& terminator can save complex if/then conditions.
        40 # The ;& is somewhat less useful.

- The new coproc builtin enables two parallel processes to communicate and interact. As Chet Ramey states in the Bash FAQ [2] , ver. 4.01:

  > There is a new 'coproc' reserved word that specifies a coprocess:
  > an asynchronous command run with two pipes connected to the creating
  > shell. Coprocs can be named. The input and output file descriptors
  > and the PID of the coprocess are available to the calling shell in
  > variables with coproc-specific names.
  > George Dimitriu explains,
  > "... coproc ... is a feature used in Bash process substitution,
  > which now is made publicly available."
  > This means it can be explicitly invoked in a script, rather than
  > just being a behind-the-scenes mechanism used by Bash.

  Coprocesses use file descriptors. File descriptors enable processes and pipes to communicate.

         1 #!/bin/bash4
         2 # A coprocess communicates with a while-read loop.
         3
         4
         5 coproc { cat mx_data.txt; sleep 2; }
         6 #                         ^^^^^^^
         7 # Try running this without "sleep 2" and see what happens.
         8
         9 while read -u ${COPROC[0]} line    #  ${COPROC[0]} is the
        10 do                                 #+ file descriptor of the coprocess.
        11   echo "$line" | sed -e 's/line/NOT-ORIGINAL-TEXT/'
        12 done
        13
        14 kill $COPROC_PID                   #  No longer need the coprocess,
        15                                    #+ so kill its PID.

  But, be careful!

         1 #!/bin/bash4
         2
         3 echo; echo
         4 a=aaa
         5 b=bbb
         6 c=ccc
         7
         8 coproc echo "one two three"
         9 while read -u ${COPROC[0]} a b c;  #  Note that this loop
        10 do                                 #+ runs in a subshell.
        11   echo "Inside while-read loop: ";
        12   echo "a = $a"; echo "b = $b"; echo "c = $c"
        13   echo "coproc file descriptor: ${COPROC[0]}"
        14 done
        15
        16 # a = one
        17 # b = two
        18 # c = three
        19 # So far, so good, but ...
        20
        21 echo "-----------------"
        22 echo "Outside while-read loop: "
        23 echo "a = $a"  # a =
        24 echo "b = $b"  # b =
        25 echo "c = $c"  # c =
        26 echo "coproc file descriptor: ${COPROC[0]}"
        27 echo
        28 #  The coproc is still running, but ...
        29 #+ it still doesn't enable the parent process
        30 #+ to "inherit" variables from the child process, the while-read loop.
        31
        32 #  Compare this to the "badread.sh" script.

  The coprocess is asynchronous, and this might cause a problem. It may terminate before another process has finished communicating with it.

         1 #!/bin/bash4
         2
         3 coproc cpname { for i in {0..10}; do echo "index = $i"; done; }
         4 #      ^^^^^^ This is a *named* coprocess.
         5 read -u ${cpname[0]}
         6 echo $REPLY         #  index = 0
         7 echo ${COPROC[0]}   #+ No output ... the coprocess timed out
         8 #  after the first loop iteration.
         9
        10
        11
        12 # However, George Dimitriu has a partial fix.
        13
        14 coproc cpname { for i in {0..10}; do echo "index = $i"; done; sleep 1;
        15 echo hi > myo; cat - >> myo; }
        16 #       ^^^^^ This is a *named* coprocess.
        17
        18 echo "I am main"$'\04' >&${cpname[1]}
        19 myfd=${cpname[0]}
        20 echo myfd=$myfd
        21
        22 ### while read -u $myfd
        23 ### do
        24 ###   echo $REPLY;
        25 ### done
        26
        27 echo $cpname_PID
        28
        29 #  Run this with and without the commented-out while-loop, and it is
        30 #+ apparent that each process, the executing shell and the coprocess,
        31 #+ waits for the other to finish writing in its own write-enabled pipe.

- The new mapfile builtin makes it possible to load an array with the contents of a text file without using a loop or command substitution.

         1 #!/bin/bash4
         2
         3 mapfile Arr1 < $0
         4 # Same result as     Arr1=( $(cat $0) )
         5 echo "${Arr1[@]}"  # Copies this entire script out to stdout.
         6
         7 echo "--"; echo
         8
         9 # But, not the same as   read -a   !!!
        10 read -a Arr2 < $0
        11 echo "${Arr2[@]}"  # Reads only first line of script into the array.
        12
        13 exit

- The read builtin got a minor facelift. The `-t` timeout option now accepts (decimal) fractional values [3] and the `-i` option permits preloading the edit buffer. [4] Unfortunately, these enhancements are still a work in progress and not (yet) usable in scripts.

- Parameter substitution gets case-modification operators.

         1 #!/bin/bash4
         2
         3 var=veryMixedUpVariable
         4 echo ${var}            # veryMixedUpVariable
         5 echo ${var^}           # VeryMixedUpVariable
         6 #         *              First char --> uppercase.
         7 echo ${var^^}          # VERYMIXEDUPVARIABLE
         8 #         **             All chars  --> uppercase.
         9 echo ${var,}           # veryMixedUpVariable
        10 #         *              First char --> lowercase.
        11 echo ${var,,}          # verymixedupvariable
        12 #         **             All chars  --> lowercase.

- The declare builtin now accepts the `-l` lowercase and `-c` capitalize options.

         1 #!/bin/bash4
         2
         3 declare -l var1            # Will change to lowercase
         4 var1=MixedCaseVARIABLE
         5 echo "$var1"               # mixedcasevariable
         6 # Same effect as             echo $var1 | tr A-Z a-z
         7
         8 declare -c var2            # Changes only initial char to uppercase.
         9 var2=originally_lowercase
        10 echo "$var2"               # Originally_lowercase
        11 # NOT the same effect as     echo $var2 | tr a-z A-Z

- Brace expansion has more options.

  Increment/decrement, specified in the final term within braces.

         1 #!/bin/bash4
         2
         3 echo {40..60..2}
         4 # 40 42 44 46 48 50 52 54 56 58 60
         5 # All the even numbers, between 40 and 60.
         6
         7 echo {60..40..2}
         8 # 60 58 56 54 52 50 48 46 44 42 40
         9 # All the even numbers, between 40 and 60, counting backwards.
        10 # In effect, a decrement.
        11 echo {60..40..-2}
        12 # The same output. The minus sign is not necessary.
        13
        14 # But, what about letters and symbols?
        15 echo {X..d}
        16 # X Y Z [  ] ^ _ ` a b c d
        17 # Does not echo the \ which escapes a space.

  Zero-padding, specified in the first term within braces, prefixes each term in the output with the same number of zeroes.

       bash4$ echo {010..15}
       010 011 012 013 014 015

       bash4$ echo {000..10}
       000 001 002 003 004 005 006 007 008 009 010

- Substring extraction on positional parameters now starts with $0 as the zero-index. (This corrects an inconsistency in the treatment of positional parameters.)

         1 #!/bin/bash
         2 # show-params.bash
         3 # Requires version 4+ of Bash.
         4
         5 # Invoke this scripts with at least one positional parameter.
         6
         7 E_BADPARAMS=99
         8
         9 if [ -z "$1" ]
        10 then
        11   echo "Usage $0 param1 ..."
        12   exit $E_BADPARAMS
        13 fi
        14
        15 echo ${@:0}
        16
        17 # bash3 show-params.bash4 one two three
        18 # one two three
        19
        20 # bash4 show-params.bash4 one two three
        21 # show-params.bash4 one two three
        22
        23 # $0                $1  $2  $3

- The new ** globbing operator matches filenames and directories recursively.

         1 #!/bin/bash4
         2 # filelist.bash4
         3
         4 shopt -s globstar  # Must enable globstar, otherwise ** doesn't work.
         5                    # The globstar shell option is new to version 4 of Bash.
         6
         7 echo "Using *"; echo
         8 for filename in *
         9 do
        10   echo "$filename"
        11 done   # Lists only files in current directory ($PWD).
        12
        13 echo; echo "--------------"; echo
        14
        15 echo "Using **"
        16 for filename in **
        17 do
        18   echo "$filename"
        19 done   # Lists complete file tree, recursively.
        20
        21 exit
        22
        23 Using *
        24
        25 allmyfiles
        26 filelist.bash4
        27
        28 --------------
        29
        30 Using **
        31
        32 allmyfiles
        33 allmyfiles/file.index.txt
        34 allmyfiles/my_music
        35 allmyfiles/my_music/me-singing-60s-folksongs.ogg
        36 allmyfiles/my_music/me-singing-opera.ogg
        37 allmyfiles/my_music/piano-lesson.1.ogg
        38 allmyfiles/my_pictures
        39 allmyfiles/my_pictures/at-beach-with-Jade.png
        40 allmyfiles/my_pictures/picnic-with-Melissa.png
        41 filelist.bash4

- The new $BASHPID internal variable.

- There is a new builtin error-handling function named command_not_found_handle.

         1 #!/bin/bash4
         2
         3 command_not_found_handle ()
         4 { # Accepts implicit parameters.
         5   echo "The following command is not valid: \""$1\"""
         6   echo "With the following argument(s): \""$2\"" \""$3\"""   # $4, $5 ...
         7 } # $1, $2, etc. are not explicitly passed to the function.
         8
         9 bad_command arg1 arg2
        10
        11 # The following command is not valid: "bad_command"
        12 # With the following argument(s): "arg1" "arg2"

## 37.3.1. Bash, version 4.1

Version 4.1 of Bash, released in May, 2010, was primarily a bugfix update.

- The printf command now accepts a `-v` option for setting array indices.

- Within double brackets, the > and < string comparison operators now conform to the locale. Since the locale setting may affect the sorting order of string expressions, this has side-effects on comparison tests within [[ ... ]] expressions.

- The read builtin now takes a `-N` option (read -N chars), which causes the read to terminate after chars characters.

  Example 37-8. Reading N characters

         1 #!/bin/bash
         2 # Requires Bash version -ge 4.1 ...
         3
         4 num_chars=61
         5
         6 read -N $num_chars var < $0   # Read first 61 characters of script!
         7 echo "$var"
         8 exit
         9
        10 ####### Output of Script #######
        11
        12 #!/bin/bash
        13 # Requires Bash version -ge 4.1 ...
        14
        15 num_chars=61

- Here documents embedded in `$( ... )` command substitution constructs may terminate with a simple ).

  Example 37-9. Using a here document to set a variable

         1 #!/bin/bash
         2 # here-commsub.sh
         3 # Requires Bash version -ge 4.1 ...
         4
         5 multi_line_var=$( cat <<ENDxxx
         6 ------------------------------
         7 This is line 1 of the variable
         8 This is line 2 of the variable
         9 This is line 3 of the variable
        10 ------------------------------
        11 ENDxxx)
        12
        13 #  Rather than what Bash 4.0 requires:
        14 #+ that the terminating limit string and
        15 #+ the terminating close-parenthesis be on separate lines.
        16
        17 # ENDxxx
        18 # )
        19
        20
        21 echo "$multi_line_var"
        22
        23 #  Bash still emits a warning, though.
        24 #  warning: here-document at line 10 delimited
        25 #+ by end-of-file (wanted `ENDxxx')

## 37.3.2. Bash, version 4.2

Version 4.2 of Bash, released in February, 2011, contains a number of new features and enhancements, in addition to bugfixes.

- Bash now supports the the `\u` and `\U` Unicode escape.

         1 echo -e '\u2630'   # Horizontal triple bar character.
         2 # Equivalent to the more roundabout:
         3 echo -e "\xE2\x98\xB0"
         4                    # Recognized by earlier Bash versions.
         5
         6 echo -e '\u220F'   # PI (Greek letter and mathematical symbol)
         7 echo -e '\u0416'   # Capital "ZHE" (Cyrillic letter)
         8 echo -e '\u2708'   # Airplane (Dingbat font) symbol
         9 echo -e '\u2622'   # Radioactivity trefoil
        10
        11 echo -e "The amplifier circuit requires a 100 \u2126 pull-up resistor."
        12
        13
        14 unicode_var='\u2640'
        15 echo -e $unicode_var      # Female symbol
        16 printf "$unicode_var \n"  # Female symbol, with newline
        17
        18
        19 #  And for something a bit more elaborate . . .
        20
        21 #  We can store Unicode symbols in an associative array,
        22 #+ then retrieve them by name.
        23 #  Run this in a gnome-terminal or a terminal with a large, bold font
        24 #+ for better legibility.
        25
        26 declare -A symbol  # Associative array.
        27
        28 symbol[script_E]='\u2130'
        29 symbol[script_F]='\u2131'
        30 symbol[script_J]='\u2110'
        31 symbol[script_M]='\u2133'
        32 symbol[Rx]='\u211E'
        33 symbol[TEL]='\u2121'
        34 symbol[FAX]='\u213B'
        35 symbol[care_of]='\u2105'
        36 symbol[account]='\u2100'
        37 symbol[trademark]='\u2122'
        38
        39
        40 echo -ne "${symbol[script_E]}   "
        41 echo -ne "${symbol[script_F]}   "
        42 echo -ne "${symbol[script_J]}   "
        43 echo -ne "${symbol[script_M]}   "
        44 echo -ne "${symbol[Rx]}   "
        45 echo -ne "${symbol[TEL]}   "
        46 echo -ne "${symbol[FAX]}   "
        47 echo -ne "${symbol[care_of]}   "
        48 echo -ne "${symbol[account]}   "
        49 echo -ne "${symbol[trademark]}   "
        50 echo

  | The above example uses the $' ... ' string-expansion construct.

- When the `lastpipe` shell option is set, the last command in a pipe doesn't run in a subshell.

  Example 37-10. Piping input to a read

         1 #!/bin/bash
         2 # lastpipe-option.sh
         3
         4 line=''                   # Null value.
         5 echo "\$line = "$line""   # $line =
         6
         7 echo
         8
         9 shopt -s lastpipe         # Error on Bash version -lt 4.2.
        10 echo "Exit status of attempting to set \"lastpipe\" option is $?"
        11 #     1 if Bash version -lt 4.2, 0 otherwise.
        12
        13 echo
        14
        15 head -1 $0 | read line    # Pipe the first line of the script to read.
        16 #            ^^^^^^^^^      Not in a subshell!!!
        17
        18 echo "\$line = "$line""
        19 # Older Bash releases       $line =
        20 # Bash version 4.2          $line = #!/bin/bash

  This option offers possible "fixups" for these example scripts: Example 34-3 and Example 15-8.

- Negative array indices permit counting backwards from the end of an array.

  Example 37-11. Negative array indices

         1 #!/bin/bash
         2 # neg-array.sh
         3 # Requires Bash, version -ge 4.2.
         4
         5 array=( zero one two three four five )   # Six-element array.
         6 #         0    1   2    3    4    5
         7 #        -6   -5  -4   -3   -2   -1
         8
         9 # Negative array indices now permitted.
        10 echo ${array[-1]}   # five
        11 echo ${array[-2]}   # four
        12 # ...
        13 echo ${array[-6]}   # zero
        14 # Negative array indices count backward from the last element+1.
        15
        16 # But, you cannot index past the beginning of the array.
        17 echo ${array[-7]}   # array: bad array subscript
        18
        19
        20 # So, what is this new feature good for?
        21
        22 echo "The last element in the array is "${array[-1]}""
        23 # Which is quite a bit more straightforward than:
        24 echo "The last element in the array is "${array[${#array[*]}-1]}""
        25 echo
        26
        27 # And ...
        28
        29 index=0
        30 let "neg_element_count = 0 - ${#array[*]}"
        31 # Number of elements, converted to a negative number.
        32
        33 while [ $index -gt $neg_element_count ]; do
        34   ((index--)); echo -n "${array[index]} "
        35 done  # Lists the elements in the array, backwards.
        36       # We have just simulated the "tac" command on this array.
        37
        38 echo
        39
        40 # See also neg-offset.sh.

- Substring extraction uses a negative length parameter to specify an offset from the end of the target string.

  Example 37-12. Negative parameter in string-extraction construct

         1 #!/bin/bash
         2 # Bash, version -ge 4.2
         3 # Negative length-index in substring extraction.
         4 # Important: It changes the interpretation of this construct!
         5
         6 stringZ=abcABC123ABCabc
         7
         8 echo ${stringZ}                              # abcABC123ABCabc
         9 #                   Position within string:    0123456789.....
        10 echo ${stringZ:2:3}                          #   cAB
        11 #  Count 2 chars forward from string beginning, and extract 3 chars.
        12 #  ${string:position:length}
        13
        14 #  So far, nothing new, but now ...
        15
        16                                              # abcABC123ABCabc
        17 #                   Position within string:    0123....6543210
        18 echo ${stringZ:3:-6}                         #    ABC123
        19 #                ^
        20 #  Index 3 chars forward from beginning and 6 chars backward from end,
        21 #+ and extract everything in between.
        22 #  ${string:offset-from-front:offset-from-end}
        23 #  When the "length" parameter is negative,
        24 #+ it serves as an offset-from-end parameter.
        25
        26 #  See also neg-array.sh.

### Notes

[1] | To be more specific, Bash 4+ has limited support for associative arrays. It's a bare-bones implementation, and it lacks the much of the functionality of such arrays in other programming languages. Note, however, that associative arrays in Bash seem to execute faster and more efficiently than numerically-indexed arrays.
[2] | Copyright 1995-2009 by Chester Ramey.
[3] | This only works with pipes and certain other special files.
[4] | But only in conjunction with readline, i.e., from the command-line.
[5] | And while you're at it, consider fixing the notorious piped read problem.

Prev | Home | Next
Bash, version 3 | Up | Endnotes