What are the other ways to determine two file contents are identical except for byte-by-byte checking?

Question

To compare byte by byte surely works. But I am wondering if there are any other proven way, say some kind of hashing that outputs unique values for each file. And if there are, what are the advantages and disadvantage of each one in terms of time and memory footprint.

By the way, I found this previous thread http://stackoverflow.com/questions/784585/what-is-the-fastest-way-to-check-if-files-are-identical. However, My question is not about speed, but alternatives.

Please advise. Thanks.

Answer 1

Anyway if your files are n bytes length, you have to compare n bytes, you can't make the problem simpler.

You can only gain speed on n comparisons when files are not identical, by checking length for exemple.

A hash is not a proven method because of collisions, and to make a hash you'll have to read n bytes on each file aswell.

If you want to compare the same file multiple times you can use hashing, then double check with a byte-to-byte

Answer 2

Hashing doesn't output 'unique' values. It can't possibly do so, because there are an infinite number of different files, but only a finite number of hash values. It doesn't take much thought to realise that to be absolutely sure two files are the same, you're going to have to examine all the bytes of both of them.

Hashes and checksums can provide a fast 'these files are different' answer, and within certain probabilistic bounds can provide a fast 'these files are probably the same' answer, but for certainty of equality you have to check every byte. How could there be any way round this?

Answer 3

If you want to compare multiple files then SHA-1 hash algorithm is a very good choice.

Answer 4

The only proven way is to do a byte-by-byte compare. It's also the fastest way and you can cut the memory usage all the way down to 2 bytes if you read a byte at a time. Reading larger chunks at a time is beneficial for performance though.

Hashing will also work. Due to the pigeonhole principle there will be a small chance that you'll get false positives but for all intents and purposes it is negligible if you use a secure hash like SHA. Memory usage is also small, but performance is less than byte-by-byte compare because you'll have the overhead of hashing. Unless you can reuse the hashes to do multiple compares.