perl sort temporary directory

Question

I ran into space issues on my machine and therefore the sort command in unix failed because of lack of space in /tmp. In order to circumvent this, I decided to run sort with the -T option allowing it to use some other directory for creating temporary files. here is the perl script I have

   my $TMPDIR              = "/home/xyz/workspace/";
    my $sortCommand         = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T \$TMPDIR -k1,1\" }' test > test.sort";
    system_call($sortCommand, "Sort");
    sub system_call {
      .......
}

this works perfectly on my desktop but when I run this in a different machine I get the error

"sort: cannot create temporary file: -k1,1/sortFoeXZx: No such file or directory"

Any ideas ?

Answer 1

You have escaped the $ before the TMPDIR variable, which has resulted in $TMPDIR being passed through to the shell. As this is not defined in your environment, the command ends up as follows:

sort -T \ -k1,1

This has inadvertently escaped the following space character thus resulting in your error.

Answer 2

You've got four levels of interpretation here (perl, shell, awk, shell). The sort shell command is

awk 'NR == 1; NR > 1 { print $0 | "sort -T $TMPDIR -k1,1" }' test > test.sort

so $TMPDIR is expanded by the inner shell. Apparently there is an environment variable of that name on your desktop machine but not in the other machine. The best fix is to make TMPDIR an environment variable and quote it properly (expanding it from Perl will cause trouble if its value contains characters that must be protected from awk or shell expansion).

$ENV{TMPDIR} = "/home/xyz/workspace/";
my $sortCommand = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T \\\"\$TMPDIR\\\" -k1,1\" }' test > test.sort";

It's strange to be relying so much on external tools from a Perl program. Ok, if sort overflows /tmp, then Perl's built-in sort is likely to run out of memory. So use Sort::External.

Answer 3

FWIW, you can some of your code by using File::Tempdir, which will let you create a temporary directory or file with the appropriate user permissions, no matter your environment.

use File::Temp 'tempdir';
my $tempdir = tempdir(CLEANUP => 1);
my $sortCommand         = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T $tempdir -k1,1\" }' test > test.sort";
system_call($sortCommand, "Sort");