Egrep acts strange with -f option

Question

Hi all,

I've got a strangely acting egrep -f.

Example:

$ egrep -f ~/tmp/tmpgrep2 orig_20_L_A_20090228.txt | wc -l
3
$ for lines in `cat ~/tmp/tmpgrep2` ; do  egrep $lines orig_20_L_A_20090228.txt ; done | wc -l
12

Could someone give me a hint what could be the problem? No, the files did not changed between executions. The expected answer for the egrep line count is 12.

UPDATE on file contents: the searched file contains cca 13000 lines, each of them are 500 char long, the pattern file contains 12 lines, each of them are 24 char long. The pattern always (and only) occurs on a fixed position in the seached file (26-49).

UPDATE on pattern contents: every pattern from tmpgrep2 are a 24 char long number.

Answer 1

Could it be that the lines read contain something that the shell is expanding/substituting for you, in the second version? Then that doesn't get done by grep when it reads the patterns itself, thus leading to a different sent of patterns being matched.

I'm not totally sure if the shell is doing any expansion on the variable value in an invocation like that, but it's an idea at least.

EDIT: Nope, it doesn't seem to do any substitutions. But it could be quoting issue, if your patterns contain whitespace the for loop will step through each token, not through each line. Take a look at the read bash builtin.

Answer 2

I second @unwind.

Why don't you run without wc -l and see what each search is finding?

And maybe:

for lines in `cat ~/tmp/tmpgrep2` ; do echo $lines ; done

Just to see now the shell is handling $lines?

Answer 3

If the search patterns are found on the same lines, then you can get the result you see:

Suppose you look for:

abc
def
ghi
jkl

and the data file is:

abcdefghijklmnoprstuvwxzy

then the one-time command will print 1 and the loop will print 4.

Answer 4

Do you have any duplicates in ~/tmp/tmpgrep2? Egrep will only use the dupes one time, but your loop will use each occurrence.

Get rid of dupes by doing something like this:

$ for lines in `sort < ~/tmp/tmpgrep2 | uniq` ; do  egrep $lines orig_20_L_A_20090228.txt ; done | wc -l

Answer 5

The others have already come up with most of the things I would look at. The next thing I would check is the environment variable GREP_OPTIONS, or whatever it is called on your machine. I've gotten the strangest error messages or behaviors when using a command line argument that interfered with the environment settings.