Bash bugs

A bash script

#!/usr/bin/env bash

set -euo pipefail

for f in code/*; do
    name=$(basename "$f")
    readarray -t files < <((rg -l '^#!' "$f" && fd -t x . "$f") | sort | uniq)

    if [[ ${#files[@]} -gt 0 ]]; then
        if ! shellcheck "${files[@]}" > /dev/null; then
            echo "❌ $name"
        else
            echo "✅ $name"
        fi
    else
        echo "⚠️ $name"
    fi
done

Results

$ ./check.sh | paste -d',' - - | column -s',' -t

❌ AKelleyCSA       ❌ Beanaddict
⚠️ EitanTreiger1    ✅ Jawtomlb
❌ Lucas1234147     ✅ Malfunnction
✅ REDshrimpp       ❌ TheHerders
❌ YairSas          ❌ antidisestablishmentarianisms
✅ asher46545645    ❌ grappell
❌ jacobmccaughrin  ❌ lashlockbhs
⚠️ okcoder2019      ❌ rift10
❌ sirineelkaroui

Run shellcheck!

A somewhat subtle bug

readarray is a shell builtin that reads its input and puts it in the named variable as an array.

readarray -t bar < lines.txt

cat lines.txt | readarray -t foo

What’s the difference?

Shellcheck may or may not detect this depending on various things.

A quick fix to the subtle bug

It’s good to understand the difference between the two commands on the previous slide.

But it’s easy to make them work essentially the same:

shopt -s lastpipe

Add this line in a script and the last element of each top-level pipeline will be run in the shell executing the script rather than a subshell.

SC2030 and SC2031

ps | readarray -t foo
                  ^-^ SC2030 (info): Modification of foo is
                  local (to subshell caused by pipeline).


In rat.sh line 5:
echo "${#foo[@]}"
      ^--------^ SC2031 (info): foo was modified in
      a subshell. That change might be lost.

SC2094

$ ls -l > foo
$ wc < foo
 11  92 544
$ wc < foo > foo
$ cat foo
0 0 0

Why?

SC2086

cat $1 > input_one

grep -oi $wordToSearchFor

javac $filename

What’s wrong with these?

Quoting is tricky

Mostly just always quote things.

But there are some subtleties.

Particularly around arrays.

A demo (part 1)

#!/usr/bin/env bash

set -euo pipefail

# $# is the length of the array $@ which holds
# the script's command line arguments
echo "num args: $#"

$ ./numargs.sh a b c
num args: 3

But:

$ ./numargs.sh "a b" c
num args: 2

A demo (part 2)

#!/usr/bin/env bash

set -euo pipefail

./numargs.sh $@
./numargs.sh "$@"

$ ./foo.sh "a b" c
num args: 3
num args: 2

Without quotes $@ is expanded into three white space-separated things.

With quotes "$@" is as if the quotes around the individual items are still there.

Note: we don’t need quotes here.

foo=$(somecommand "$f")

You could put quotes around the whole $() expression and bash would be fine with it.

But they’re not necessary because bash does not perform file expansion (globs) or word splitting (splitting on spaces) on the value in a variable assignment.

Nor here

bar=$foo

By the same rule, when used as a value in an assignment $foo is not not subject to word splitting or file expansion.

On the other hand

We should usually quote command substitutions when they occur in other contexts.

touch "$(get-file-name)"

If get-file-name is a command that is going to output one file name we should quote it in case the name contains spaces or *s.

On the third hand

Sometimes a command substitution produces a set of arguments we want to pass.

For instance, in the following command we almost certainly intend to touch all the files returned by find, not one file whose name is made up the names of all the found files.

# shellcheck disable=SC2046
touch $(find . -size +10)

Note the comment to tell shellcheck that we know what we’re doing.

Oh no, there’s a fourth hand!

Actually that’s busted. When we told shellcheck we knew what we were doing we were wrong!

If any of the files returned by find have spaces in their name, this won’t do what we want.

The proper way is to use readarray to read into an array which we can then quote:

readarray -t names < <(find . -size +10); touch "${names[@]}"

SC2164

cd "$1/src/"
^----------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in
case cd fails.

What’s a better way to fix this than the suggestion?

Answer: set -e.

SC2155

#!/usr/bin/env bash

set -euo pipefail

export FOO=$(somecommand)

# do stuff with FOO

Anyone see the problem.

Probably not.

Shellcheck does.

In failure.sh line 5:
export FOO=$(somecommand)
       ^-^ SC2155 (warning): Declare and assign separately
       to avoid masking return values.

Don’t sleep on shell expansions

One of the most powerful parts of bash are the various expansions.

In addition to globs * and tilde ~ expansions, some of you found some of these:

Parameter expansions

$ x='foo.txt.gz'
$ echo "${x##*.}"
gz
$ echo "${x#*.}"
txt.gz
$ echo "${x%%.*}"
foo
$ echo "${x%.*}"
foo.txt
$ echo "${x/foo/bar}"
bar.txt.gz

Can be applied to arrays too!

$ files=("foo.txt" "bar.mp3" "X.java")
$ echo "${files[@]##*.}"
txt mp3 java

Brace expansion

for f in fake-files/{a,b,c}/{x,y,z}.{txt,mp3,jpg}; do \
  mkdir -p $(dirname "$f"); \
  touch "$f"; \
done

Results

$ find fake-files -type f | sort | paste - - - | column -t
fake-files/a/x.jpg  fake-files/a/x.mp3  fake-files/a/x.txt
fake-files/a/y.jpg  fake-files/a/y.mp3  fake-files/a/y.txt
fake-files/a/z.jpg  fake-files/a/z.mp3  fake-files/a/z.txt
fake-files/b/x.jpg  fake-files/b/x.mp3  fake-files/b/x.txt
fake-files/b/y.jpg  fake-files/b/y.mp3  fake-files/b/y.txt
fake-files/b/z.jpg  fake-files/b/z.mp3  fake-files/b/z.txt
fake-files/c/x.jpg  fake-files/c/x.mp3  fake-files/c/x.txt
fake-files/c/y.jpg  fake-files/c/y.mp3  fake-files/c/y.txt
fake-files/c/z.jpg  fake-files/c/z.mp3  fake-files/c/z.txt

Read all about 'em

The Bash manual is quite good.

The documentation on all the kinds of expansions are here.