Using Sed to expand environment variables inside files

Question

I'd like to use Sed to expand variables inside a file.

Suppose I exported a variable VARIABLE=something, and have a "test" file with the following:

I'd like to expand this: "${VARIABLE}"

I've been trying commands like the following, but to no avail:

cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile

The result is the "outputfile" with the variable still not expanded:

I'd like to expand this: "${VARIABLE}"

Still, running eval "echo '${VARIABLE}' in bash console results in the value "something" being echoed. Also, I tested and that pattern is trully being matched.

The desired output would be

I'd like to expand this: "something"

Can anyone shed a light on this?

Answer 1

Maybe you can get by without using sed:

$ echo $VARIABLE
something
$ cat test
I'd like to expand this: ${VARIABLE}
$ eval "echo \"`cat test`\"" > outputfile
$ cat outputfile
I'd like to expand this: something

Let shell variable interpolation do the work.

Answer 2

Consider your trial version:

cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile

The reason this doesn't work is because it requires prescience on the part of the shell. The sed script is generated before any pattern is matched by sed, so the shell cannot do that job for you.

I've done this a couple of ways in the past. Normally, I've had a list of known variables and their values, and I've done the substitution from that list:

for var in PATH VARIABLE USERNAME
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

If you want to map variables arbitrarily, then you either need to deal with the whole environment (instead of the fixed list of variable names, use the output from env, appropriately edited), or use Perl or Python instead.

Note that if the value of an environment variable contains a slash in your version, you'd run into problems using the slash as the field separator in the s/// notation. I used the '%' since relatively few environment variables use that - but there are some found on some machines that do contain '%' characters and so a complete solution is trickier. You also need to worry about backslashes in the value. You probably have to use something like '$(eval echo "\$$var" | sed 's/[\%]/\\&/g')' to escape the backslashes and percent symbols in the value of the environment variable. Final wrinkle: some versions of sed have (or had) a limited capacity for the script size - older versions of HP-UX had a limit of about 100. I'm not sure whether that is still an issue, but it was as recently as 5 years ago.

The simple-minded adaptation of the original script reads:

env |
sed 's/=.*//' |
while read var
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

However, a better solution uses the fact that you already have the values in the output from env, so we can write:

env |
sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script
cat test | sed -f sed.script > outputfile

This is altogether safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with shell metacharacters in variable values. This version can only possibly run into any trouble if some output from env is malformed, I think.

Beware - writing sed scripts with sed is an esoteric occupation, but one that illustrates the power of good tools.

All these examples are remiss in not cleaning up the temporary file(s).