bash tips needed for understanding how to escape characters in command-line

Question

My knowledge of commandline bash is missing on a particular area: I constantly forget how to properly escape characters. Today I wanted to echo this string into a file:

#!/bin/env bash
python -m SimpleHTTPServer

echo "#!/bin/env bash\npython -m SimpleHTTPServer" > server.sh && chmod +x server.sh -bash: !/bin/env: event not found

That's right: Remember to escape ! or bash will think it's a special bash event command.

But I can't get the escaping right! \! yields \! in the echoed string, and so does \\!.

Furthermore, \n will not translate to a line break.

Do you have some general tips that makes it easier for me to understand escaping rules?

To be very precise, I'll accept an answer which tells me which characters I should escape on the bash command line? Including how to correctly output newline and exclamation mark in my example.

Answer 1

Single quotes inhibit all escaping and substitution:

echo '$hello world!'

You can alternate or disable quoting if you need to mix things up:

echo '$5.00 on '"$horse"'!'
ls -ld ~/'$$'/*

Interpreting escapes is also easy:

echo $'Wake up!\7'
sort -n <<< $'4\n3\n8'

Answer 2

First, shell quoting is not like C strings. Just because you have a \n between quotes does not put a newline in the actual string. It gets a bit confusing sometimes because some versions of echo will interpret the backslash-en combination and output a newline instead. This behavior depends on the exact shell you are using, which options you pass to echo, and which shell options are set, so it is not terribly reliable for cross-platform use. printf is more standardized (it is either a shell builtin or an external program on most systems), but it is not present on (very?) old systems.

As Ignacio Vazquez-Abrams says, single quotes are one answer. The $'' variation he mentions at the end are not portable across all shells though (it is not in the POSIX standard for the shell language, and simpler shells do not support it (though ksh, bash, and zsh all do)).

You can even include literal line breaks in a single quoted string:

echo '#!/bin/env bash
python -m SimpleHTTPServer' >server.sh &&
chmod +x server.sh

Or, with printf:

printf '#!/bin/env bash\npython -m SimpleHTTPServer\n' >server.sh &&
chmod +x server.sh

You should always take care with the first argument to printf though, if there are any uspected % characters, they will be interpreted as format specifiers (i.e. you should generally not use unknown strings as the first argument to printf).

If you need no interpolation, use single quotes. They let you include literals that include anything except a single quote itself. If you need some C-like escape sequences, either put them in the first argument to printf or use $'' (if your shell supports it).

---

If you need to include single quotes, you could use double quotes as the outermost quotes, but then you have to worry about what kinds of interpolations happens inside double quotes (varies by shell, but generally, \, $, and ` are all special and \ has a special meaning when it is immediately before a newline; ! is also special when history expansion is enabled in bash):

echo "The answer is '\$foo'\!"

Or, you could use concatenate multiple single quoted strings with escaped single quotes:

echo 'The answer is '\''foo'\''!'

Either way, it is not too pretty to think about or read if things get too complex.

When faced with wanting to include single quotes inside a literal string, you might change to using a quoted ‘here document’ instead:

cat <<\EOF
The answer is '$foo'!
EOF

The backslash before the EOF makes the here document a literal one instead of the double-quote-ish one that you get without quoting the terminating word.

Here documents are easy to output (cat) or to send to other commands as input, but they are not as convenient to use as arguments:

frob -nitz="$(cat <<\EOF
The answer is '$foo'!
EOF
)"

But if you need lots of literal data that includes both single quotes and characters that are special inside double quotes, such a construct can be a huge boon to readability (and writability). Note: as with all other uses of $(), this will eat trailing newlines from the contents of the here document.

If you need trailing newlines, you can get them, but it is extra work:

s="$(cat <<\EOF


This will have leading and trailing newlines.

It takes as literal everything except a line with only "EOF" on it!$@#&\
(and the "EOF" bit can be changed by using a different word after <<)

The trailing 'xxx' will be trimmed off by the next command.
It is here to protect the otherwise trailing newlines that $() would strip off.


xxx
EOF
)"
s="${s%xxx}"
frob -nitz="$s"

Answer 3

With -e, bashs buildin echo as well as /bin/echo treat \n as a newline:

echo -e '#!/bin/env bash\npython -m SimpleHTTPServer' 

but I only know -e from echo.

Answer 4

The easiest way to avoid "event not found" errors is to turn off history expansion:

set +o histexpand

One thing that hasn't been mentioned yet is that some versions of echo support the -e option which causes interpretation of escapes:

$ echo -e 'two\nlines'
two
lines

Answer 5

You may also store the newline in a variable:

NL='
'
printf "%q\n" "${NL}"
sort -n <<< "4${NL}3${NL}8"


# (yet another) example of shell string concatenation

# As a rule of thumb: 
# Put anything inside single quotes, 
# except "${variables}", "$(commands)" and "'single quotes'".

echo 'The answer is '\''foo'\''!'

echo 'The answer is '"'"'foo'"'"'!'  # concatenate 'str1'"str2"...

echo 'The answer is '\
     "'"\
     'foo'\
     "'"\
     '!'\
     "${NL}`date`"'!'


# ... and sometimes there simply is no need for ${NL} or \n

echo '#!/bin/env bash
python -m SimpleHTTPServer' > server.sh

echo '
line1
line2
line3
' > file