what is the best way to find a subset of a list with grep?

Question

My current solution:

#!/bin/sh
while read file2
do
grep $file2 file1
done

the contents of file1 will be something like:

atlanta,blue,20090805
newyork,blue,20090805
washington,blue,20090805
dallas,blue,20090805
jacksonville,blue,20090805

the contents of file2 will be something like:

newyork
dallas
jacksonville

and the desired output to a file would be something like:

newyork,blue,20090805
dallas,blue,20090805
jacksonville,blue,20090805

when seeking a subset of a large list based on names from a second list, what is the best way to do something like this? any recommendations would be appreciated!

Thanks,

Answer 1

What about...

egrep -f file2 file1

for your example case, it should work just like your loop does; and it should apply to the same range of cases to which both your verbal description and your loop do (one "name" per line in the "second list" file, i.e., no punctuation which egrep might misinterpret).

Answer 2

This is cheating since it's perl, but these are two one-liners:

#!/bin/bash
REGEX=`perl -lne 'push(@x,$_);END{print join("|",@x)."\n";}' < file2`
perl -ne 'print $_ if (/\b(?:$ENV{REGEX})\b/o);' < file1

The first line creates a option list of the form: newyork|dallas|jacksonville and stores it in the environment variable REGEX. The second line will then print any line that matches (newyork|dallas|jacksonville). The pieces of the regex here:

• The \b means that it will only work if the city is surrounded by word boundaries. This means that "york" will not match "newyork".
• The ?: means that perl will not try to capture the group, which leads to better performance.
• $ENV{REGEX} takes a parameter from the environment.
• Without the /o, perl will try to reevaluate the $ENV variable on each line, leading to poor performance.

Answer 3

$ awk -F"," 'FNR==NR{a[$1]}NR>FNR && ($1 in a)' file2 file1
newyork,blue,20090805
dallas,blue,20090805
jacksonville,blue,20090805