How can I compare two zip format(.tar,.gz,.Z) files in Unix

Question

I have two gz files. I want to compare those files without extracting. for example:

first file is number.txt.gz - inside that file:

1111,589,3698, 
2222,598,4589, 
3333,478,2695, 
4444,258,3694, 

second file - xxx.txt.gz:

1111,589,3698, 
2222,598,4589, 

I want to compare any column between those files. If column1 in first file is equal to the 1st column of second file means I want output like this:

1111,589,3698, 
2222,598,4589,

Answer 1

You cannot compare the files while they remain compressed using different techniques.

You must first decompress the files, and then find the difference between the results.

Decompression can be done with gunzip, tar, and uncompress (or zcat).

Finding the difference can be done with the diff command.

Answer 2

If you need to check and compare your data after it's written to those huge files, and you have time and space constraints preventing you from doing this, then you're using the wrong storage format. If your data storage format doesn't support your process then that's what you need to change.

My suggestion would be to throw your data into a database rather than writing it to compressed files. With sensible keys, comparison of subsets of that data can be accomplished with a simple query, and deleting no longer needed data becomes similarly simple.

Transactionality and strict SQL compliance are probably not priorities here, so I'd go with MySQL (with the MyISAM driver) as a simple, fast DB.

---

EDIT: Alternatively, Blorgbeard's suggestion is perfectly reasonable and feasible. In any programming language that has access to (de)compression libraries, you can read your way sequentially through the compressed file without writing the expanded text to disk; and if you do this side-by-side for two input files, you can implement your comparison with no space problem at all.

As for the time problem, you will find that reading and uncompressing the file (but not writing it to disk) is much faster than writing to disk. I recently wrote a similar program that takes a .ZIPped file as input and creates a .ZIPped file as output without ever writing uncompressed data to file; and it runs much more quickly than an earlier version that unpacked, processed and re-packed the data.

Answer 3

I'm not 100% sure whether it's meant match columns/fields or entire rows, but in the case of rows, something along these lines should work:

comm -12 <(zcat number.txt.gz) <(zcat xxx.txt.gz)

or if the shell doesn't support that, perhaps:

zcat number.txt.gz | { zcat xxx.txt.gz | comm -12 /dev/fd/3 - ; } 3<&0

Answer 4

exact answer i want is this only

nawk -F"," 'NR==FNR {a[$1];next} ($3 in a)' <(gzcat file1.txt.gz) <(gzcat file2.txt.gz)

. instead of awk, nawk works perfectly and it's gzip file so use gzcat