Bash scripting "common gotchas" for Python/Perl/Ruby programmers

Question

Background:

I grew up on using Perl/Python/Ruby for sysadmin-type tasks and shell scripting. I always avoided Bash scripting whenever I needed anything programmer-ish, like functions, looping or control structures. Back then, I could pick my favorite tool for whatever the job.

Problem:

Now I am working in a situation where the preferred method of doing sysadmin scripts is through Bash only. Since the people I work with have little or no desire to pick up Perl/Python/Ruby, I am forced to start using Bash, and using it for things that normally would be much more straightforward in a regular programming language (it seems).

For example, Perl/Python/Ruby all have pretty good support for XML and YAML, which makes it very very easy to implement config files that go along with a script. They also seem to have much better debugging support and understandable rules for syntax, whitespace etc.

Question:

It seems Bash scripting (when used as a shell-based programming language) has a lot of rules and caveats that one cannot just assume are like any of the "mainstream" programming languages. Stuff that one might expect to "just work" do not.

What are some of the pitfalls and gotchas that make Bash difficult to transition to if you are more accustomed to having the full programming language features of Perl/Python/Ruby? Is there any guide to make the differences more transparent and predictable?

Answer 1

The main one that I've encountered is that whitespace around assignment does not work.

This works:

a=1

This does not:

a = 1

Answer 2

If you're a perl person, this will bite you over and over:

$A=1  # doesn't mean what you think it means
A=1   # what you want

The set command doesn't do assignment in bash: See The Set Builtin. But it does a whole lot of other cool stuff.

Answer 3

Bash makes it really painful to work with multi-dimensional arrays. Bash is clunky enough just working with one-dimensional arrays. In fact, modern support for arrays is why I moved from bash to perl and python.

Answer 4

Yes, follow the bourne shell scripting conventions. Alot of useful information can be found here. I fully agree with this part,

"If you write your scripts for the Bourne Shell and nothing but the Bourne Shell, chances are far better than equal that your script will run straight out of the mail attachment ... on any command shell out there."

Going from Linux to Solaris with Bourne shell only was a painful but extremely good learning experience.

For example, using variables in bash vs bourne shell.

VARIABLE="`cat /dev/null`" # Works everywhere.
VARIABLE=$(cat /dev/null) # Does _not_ work everywhere.

Answer 5

Always use double quotes around variable names when you are using them as an argument to a command or executable when:

• the variable contains a filename
• you want to preserve whitespace in the value as-is
• any time (you can use double quotes even if they're not needed for the first two reasons)

Example (the initial $ represents the shell prompt):

$ a=$'abc     def\nghi  jkl'    # a way to get a newline in the value in a readable way
$ echo $a                       # without quotes, whitespace is collapsed
abc def ghi jkl
$ echo "$a"                     # with quotes it's preserved
abc     def
ghi  jkl

My top general-purpose tip for anyone writing Bash scripts is to read, understand and periodically refer to Greg Wooledge's BashFAQ.

At least equal to that is to understand the Unix (and by extension, Linux) philosophy of one tool, one job and that knowing the tools is like knowing the standard library of a language like C, Python or Perl. In shell programming, you'll often do some simple glue logic to combine the external tools in loops and pipelines. Understanding these tools will save you a lot of work. They aren't specific to Bash, the Bourne Shell or others and can be used with any of them. Examples: grep, find, join, paste, pr, sed, bc and many more.

Answer 6

If you are talking about programming in Bash together with the usual Linux/Unix tools, I would say it can cater to what programming languages like Perl/Python does for you for most of your sysadmin tasks, with the exception of things like programming a GUI (although you can also do that with shell scripting using GUI tools catered for shell programmin), network programming (and depending on what you are doing, networking tools like nc, ftp, ssh/telnet comes with scripting features as well). If you need regular expressions, awk/sed does the job. If you need data structures like arrays(single or multidimensional), awk caters for that. (and a modern bash comes with associative arrays as well). Like Python/Perl, Bash has no lack of looping, control flow structures etc either. The only thing you need is to learn the syntax, that you can pick up by reading about bash through a book or the internet.

Answer 7

Variable assignments in a subshell are not visible (i.e. take no effect) in the parent shell.

One way to solve this is to use Bash process substitution.

See: http://wiki.bash-hackers.org/syntax/expansion/proc_subst

Answer 8

For processing XML on the command line I would recommend xmlstarlet or xml-coreutils.

Some more options here: http://stackoverflow.com/questions/893585/how-to-parse-xml-in-bash

Answer 9

Yes, there are some "common gotchas" you need to get familiar with when it comes to shell scripting, but there are also some "common goodies", especially when you know you can use Bash only.

For example, using Bash you may modify every array item without using a for-loop (which can lead to a noticeable speed-up).

# example taken from: 
# http://floyd-n-milan.blogspot.com/2007/09/messing-with-arrays-in-bash.html
# further information:
# http://www.tldp.org/LDP/abs/html/arrays.html#ARRAYSTROPS

array=( foo bar baz )
echo "${array[@]}"      # foo bar baz

array=( "${array[@]/%/foo}" )
echo "${array[@]}"     #  foofoo barfoo bazfoo

Answer 10

perl was original written as a shell script. So a lot of the useful shell globbing things like:

for i in /home/*/public_html/logs/access*.log; do echo "$i" >>/tmp/loglist.txt;done

are available. Getting your head around head, tail, ifconfig, ping, whois, cut, sort, uniq, find -exec, xargs, sed, awk, and even perl, php, ruby, and python one liners will go a long way to making your life easier when scriping.

Here are some cool networking tricks in bash:

# Check for open TCP ports on localhost
for p in {1..1023};do (echo >/dev/tcp/localhost/$p) >/dev/null 2>&1 && echo $p open;done

# Download an atom XML feed
exec 5<>/dev/tcp/feeds.feedburner.com/80;echo -ne "GET /freegnu HTTP/1.0\r\nHost: feeds.feedburner.com\r\n\r\n" >&5;cat <&5

Answer 11

If you (one day) intend to use the Bourne shell instead of Bash for maximum portability, first make sure your /bin/sh actually is a true Bourne shell implementation (on Mac OS X, for example, it is not).

How to identify a traditional Bourne shell? ... A simple check for an often undocumented but characteristic feature: You can use the circumflex ^ (caret) as replacement for | (pipe).

from: http://www.in-ulm.de/~mascheck/bourne/

Answer 12

The source code of a quite up-to-date Bourne shell version can be downloaded from:

http://freshmeat.net/projects/bournesh/

Answer 13

If you're passing multiple arguments from the shell script's command line to an invoked command (which can happen when dealing with user filenames with spaces in; users produce these things no matter how much you'd rather they didn't) you should not use $* or $@, but rather this:

"$@"

Or, if you're really ultra careful and can't assume the use of bash, this (derived from the above):

${1+"$@"}

Which works around some ancient misfeatures in the handling of empty argument lists that bash has corrected.

How it works

While $@ refers to all arguments, "$@" is magical syntax that substitutes the arguments as individual arguments with appropriate quoting. (If you want them all as one, use "$*" but that's pretty rare.) Thus if the arguments are:

a 'b c' d /home/foo/My\ Tricky\ File\ Name

Then you'll end up with foobar "$@" being expanded like:

foobar "a" "b c" "d" "/home/foo/My Tricky File Name"

(Be careful when testing with echo; it concatenates the arguments anyway.)

The other form builds on top of this. The shell's general syntax ${foo+bar} substitutes bar (whatever it is) if $foo is defined at all; the sequence I used above just uses $1 (the first argument) as the test variable and "$@" as the substitution so it's all only done when necessary, and completely elided otherwise (avoiding the old bug which tended to produce "" instead; not what you want as it created an extra empty argument).

Answer 14

Looks like there have been a lot of good answers to your question, but I would definitly check out the series of articles by Daniel Robbins at IBM's developerWorks site:

http://www.ibm.com/developerworks/library/l-bash.html

He gives a really good overview of BASH and some of the pitfalls associated with it. I have found it to be very useful.

Good Luck!

Answer 15

Any shell variable assignment you make will not be visible to programs you start from your script unless you say "export". They are visible to the shell builtins and subshells created with (), though.

MY_CONTROLING_ENV_VAR=true
echo $MY_CONTROLING_ENV_VAR # prints "true"

bash -c 'echo $MY_CONTROLING_ENV_VAR' # prints nothing

export MY_CONTROLING_ENV_VAR

bash -c 'echo $MY_CONTROLING_ENV_VAR' # prints true now