MD5 Hashes failing for large Strings

Question

I'm making a MD5 hash of a response, and then signing it with a shared secret.

For most calls this works, but strangely fails (Generates a MD5 hash different from the client) on the only two calls that bring a lot of content in the body.

Can this be because of the size of the body? or maybe because those calls return the content chunked?

Any idea will be appreciated. Thanks a lot.

The hashing code (note that algorithm == MD5 and ENCODING == 'UTF-8'):

private static byte[] hash(String toHash, String algorithm){
      try{
      MessageDigest dg = MessageDigest.getInstance(algorithm);
      dg.update(toHash.getBytes(ENCODING));
      return dg.digest();
    }catch(Exception e){
      throw new ApiInternalException("Error while hashing string: " + toHash,e);
    }
    }

Answer 1

It'd be great if you included your code. Without that, I can only guess what the problem is. Anyway, here's the correct way to create an MD5 hash in Java. If your code differs from this, then you have a problem.

String plainString = "Hash me please";
String md5Hash = "NOTHASHED";
try {
     MessageDigest md5Digest = MessageDigest.getInstance("MD5");
     md5String = new String(md5Digest.digest(plainString.getBytes()));
} catch (NoSuchAlgorithmException nsae) {
     // MD5 is included in all versions of Java, this can never happen
}

Of course, this will return something that looks like: �ǚ���;�f���&fu If you want it to be human readable, it's usually suggested that you Bas64 encode it, in which case just use the line:

new String(Base64Encoder.encode((md5Digest.digest(DESKTOP_STRING.getBytes()))));

Which will give you something that looks like: ssea19zwO6Jm3AiF4SZmdQ==

Keeping in mind that you will need to unencode it later before using it as an md5 hash.

Answer 2

Either explanation is plausible. Other possible explanation include:

• something (maybe a proxy server) is altering content in transit,
• there is a mismatch in the way that character encoding / decoding is being dealt with
• it is actually the MD5 hashes that are being damaged.

You need to gather more information to figure out which it is. I suggest that you modify your client and server sides to capture the data being sent / received into files, move them to the same machine and do a byte-wise comparison. Other things you could try include turning off chunking and dumping / comparing the MD5 checksums at both ends.

EDIT : It would also help us help you if you posted the code that does the MD5 checksum calculation at both ends, and the code that encodes / decodes the checksums for transmission (e.g. using hexadecimal, base64 or whatever).