comparing files

Question

hi guys i have two pipe delimited files,first file contains 1000 records and second file contains 10 records and records which are present in second file exists first file. Now need a file which which will give me 990 records by excluding 10 records which occur in both files,

i know how to this using SQL .But how can we acheive this in UNIX?

Any help greatly appreciated

Answer 1

see the join command

Answer 2

Suppose the files are all and some. Then do

 fgrep -v -f some all

Answer 3

First, replace the pipes with newlines so we can process the files with line-oriented Unix tools. Then use comm to filter out the lines that appear in both files. Finally, convert the newlines back to pipes.

tr '|' '\n' < file1 | sort > file1.sorted
tr '|' '\n' < file2 | sort > file2.sorted
comm -3 file1.sorted file2.sorted | tr '\n' '|'

Or done all on one line using process substitution <(command) syntax:

comm -3 <(tr '|' '\n' < file1 | sort) <(tr '|' '\n' < file2 | sort) | tr '\n' '|'

Answer 4

The comm utility can be used to filter lines from one file out of another

Answer 5

awk 'FNR==NR{ a[$0++;next}(!($0 in a))' some all