How to output awk result to file

Question

Hi

I am trying to output 'awk' result to file in my script, with no success. Using '>' does not work, why?

for a in $(find $OUPUT_DIR/ -maxdepth 1 -mindepth 1 -type d -printf "%P\n")
do
    echo $a is a directory
    awk -F, '{ if ($10 == '"$a"') print $0 }' $OUPUT_DIR/CDRNOutput_${CDR_DATE}.csv > $OUPUT_DIR/$a/CDR-${CDR_DATE}.csv
done 

Answer 1

Output redirection is generally a feature of the shell you're working with and, given how much use it gets, I'd be pretty amazed if you'd found a bug in it :-)

Are you sure you're not trying to do redirection with awk itself rather than the shell?

What happens when you do:

echo 'hello' | awk '{print}' >qq.tmp

---

Update:

If this is your code as stated, it's because the $a is not being expanded by your shell script since the awk command is within single quotes.

for a in $(find $OUPUT_DIR/ -maxdepth 1 -mindepth 1 -type d -printf "%P\n")
do
    echo $a is a directory
    awk -F, '{ if ($10 == '"$a"') print $0 }' $OUPUT_DIR/CDRNOutput_${CDR_DATE}.csv > $OUPUT_DIR/$a/CDR-${CDR_DATE}.csv
done

What I tend to do is pass in specific values to awk using the -v option, something like (in your case):

awk -F, -v a=$a '{ if ($10==a) print $0 }' ...

Then the variables become first-class awk citizens without having to worry about who's doing the expansion.

---

Further update:

I'm standing behind my original advice. There's something definitely screwy with the method chosen.

I have a directory in my home directory called XpVm (among others) and I've created the file CDRNOutput_X.csv containing the single line:

1,2,3,4,5,6,7,8,9,XpVm,11

When I execute:

for a in $(find . -maxdepth 1 -mindepth 1 -type d -printf "%P\n" | grep -v '^\.')
do
    echo $a is a directory
    awk -F, '{
        if ($10 == '"$a"') {
            print $0
        } else {
            print "NO";
        }
    }' ./CDRNOutput_X.csv
done

(I've stripped out directories starting with . since they were causing another problem), I get this output:

workspace is a directory
NO
Documents is a directory
NO
XpVm is a directory
NO
Downloads is a directory
NO

which is clearly not what is expected. However, when I use the -v option to awk as I originally suggested, the command:

for a in $(find . -maxdepth 1 -mindepth 1 -type d -printf "%P\n" | grep -v '^\.')
do
    echo $a is a directory
    awk -F, -v a=$a '{
        if ($10 == a) {
            print $0
        } else {
            print "NO"
        }
    }' ./CDRNOutput_X.csv
done

(the only difference being the changes to a), I get:

workspace is a directory
NO
Documents is a directory
NO
XpVm is a directory
1,2,3,4,5,6,7,8,9,XpVm,11
Downloads is a directory
NO

which is correct.

---

Final update (hopefully):

I think I have the problem solved. I'm on a different machine now (so the directory names are simply tmp and tmp2) and, when I run the original script:

for a in $(find . -maxdepth 1 -mindepth 1 -type d -printf "%P\n" | grep -v '^\.')
do
    echo $a is a directory
    awk -F, '{
        if ($10 == '"$a"') {
            print $0
        } else {
            print "NO";
        }
    }' ./CDRNOutput_X.csv
done

with a modified CDRNOutput_X.csv containing tmp instead of XpVm, I get:

tmp is a directory
NO
tmp2 is a directory
NO

That's because the if statement is being seen by awk as:

        if ($10 == tmp) {

(without quotes, since the quotes are actually outside the awk string being used to surround the directory name). This will test $10 for equality against the awk variable called tmp rather than the actual string "tmp". What you need is to make sure that the quotes are inside the awk script, like:

        if ($10 == "tmp") {

and you can do this with the following script (only the if line has changed):

#!/bin/bash
for a in $(find . -maxdepth 1 -mindepth 1 -type d -printf "%P\n" | grep -v '^\.')
do
    echo $a is a directory
    awk -F, '{
        if ($10 == "'"$a"'") {
            print $0
        } else {
            print "NO";
        }
    }' ./CDRNOutput_X.csv
done

Note that the double quotes are duplicated. I've still kept the double quotes immediately around $a in case someone's committed the heinous crime of creating a file with a space in it :-)

Running that script produces:

tmp is a directory
1,2,3,4,5,6,7,8,9,tmp,11
tmp2 is a directory
NO

which is what I think you were aiming for.

So, the upshot is, if you don't want to use awk variables, you can just change your awk string from:

'{ if ($10 == '"$a"') print $0 }'

to:

'{ if ($10 == "'"$a"'") print $0 }'

and it should function okay.

Answer 2

since you have find command with -mindepth and maxdepth set to 1, you can just do it with the shell

#!/bin/bash
CDR_DATE="somedate"
infile=CDRNOutput_${CDR_DATE}.csv
outfile=CDR-${CDR_DATE}.csv
OUPUT_DIR="/some/dir"
cd $OUPUT_DIR
for dir in */
do
    echo "${dir%/*} is a directory"
    dir=${dir%/*}
    while read -r a b c d e f g h i j k
    do
        case "$j" in
            $dir) echo $a $b $c $d $e $f $g $h $i $j $k >> $dir/$outfile;;
        esac
    done < $infile
done