Optimizing grep (or using AWK) in a shell script

Question

Hi -

In my shell script, I am trying to search using terms found in a $sourcefile against the same $targetfile over and over.

My $sourcefile is formatted as such:

pattern1
pattern2
etc...

The inefficient loop I have to search with is:

for line in $(< $sourcefile);do
    fgrep $line $targetfile | fgrep "RID" >> $outputfile
done

I understand it would be possible to improve this by either loading the whole $targetfile into memory, or perhaps by using AWK?

Thanks

Answer 1

A sed solution:

sed 's/\(.*\)/\/\1\/p/' $sourcefile | sed -nf - $targetfile

This transforms every line of $sourcefile to a sed pattern match command:

matchstring

to

/matchstring/p

You'd need to escape special characters to make this robust, however.

Answer 2

Using awk to read in the sourcefile then searching in targetfile (untested):

nawk '
    NR == FNR {patterns[$0]++; next}
    /RID/ {
        for (pattern in patterns) {
            # since fgrep considers patterns as strings not regular expressions, 
            # use string lookup and not pattern matching ("~" operator).
            if (index($0, pattern) > 0) {
                print
                break
            }
        }
    }
' "$sourcefile" "$targetfile" > "$outputfile"

Will also with with gawk.

Answer 3

Am I missing something, or why not just fgrep -f "$sourcefile" "$targetfile"?