Why would HMAC SHA-1 return a different digest with the same input?

Question

I am trying to build a working encrypted signature for the Amazon S3 web service, writing a connection library using Objective C.

I have run into HMAC SHA-1 digest problems with the ObjC code, so I'm putting that to the side and looking at existing, working Perl code, to try to troubleshoot digest creation.

I am testing HMAC SHA-1 digest output from the s3ls command of the Net::Amazon::S3 package and comparing that against the _encode subroutine that I pulled out and put into its own perl script:

#!/usr/bin/perl -w                                                                                                                                                                                    

use MIME::Base64 qw(encode_base64);
use Digest::HMAC_SHA1;
use String::Escape qw( printable unprintable );

sub _ascii_to_hex {
    (my $str = shift) =~ s/(.|\n)/sprintf("%02lx", ord $1)/eg;
    return $str;
}

sub _encode {
    my ( $aws_secret_access_key, $str ) = @_;
    print "secret key hex: "._ascii_to_hex($aws_secret_access_key)."\n";
    my $hmac = Digest::HMAC_SHA1->new($aws_secret_access_key);
    $hmac->add($str);
    my $digest = $hmac->digest;
    print "cleartext hex: "._ascii_to_hex($str)."\n";
    print "digest hex: "._ascii_to_hex($digest)."\n";
    my $b64 = encode_base64( $digest, '' );
    print "encoded: ".$b64."\n";
}

my $secret = "abcd1234";
my $cleartext = "GET\n\n\nFri, 12 Dec 2008 10:08:51 GMT+00:00\n/";
_encode($secret, $cleartext);

Here is sample output from this script:

$ ./testhmac.pl 
secret key hex: 6162636431323334
cleartext hex: 4745540a0a0a4672692c2031322044656320323030382031303a30383a353120474d542b30303a30300a2f
digest hex: 63308f9b8a198440d6d8685a3f3f70d0aab02f68
encoded: YzCPm4oZhEDW2GhaPz9w0KqwL2g=

What I am testing is that, if I input the same secret key and cleartext into the same _encode function of the Net::Amazon::S3 package, I should see the very same secret key, cleartext, and digest bytes.

Indeed, I get the same bytes for the secret key and cleartext.

But I get something different for the digest (and of course the base64 encoding), e.g.:

$ s3ls --access-key=foobar --secret-key=abcd1234
...
secret key hex: 6162636431323334
cleartext hex: 4745540a0a0a4672692c2031322044656320323030382031303a30383a353120474d542b30303a30300a2f
digest hex: c0da50050c451847de7ed055c5286de584527a22
encoded: wNpQBQxFGEfeftBVxSht5YRSeiI=

I have verified that the secret key and clear text are the same input to both scripts. The encoding subroutine is virtually identical in both scripts (except for an unused argument passed to the subroutine, which I remove from my custom version).

What would cause the HMAC SHA-1 digest to be computed differently in both cases, if the input bytes and _encode subroutine are the same?

(I have also verified the two scripts against the test cases at RFC 2201.)

Answer 1

I'm afraid I can't help much here, but there's definitely something wrong with what you posted. Your example script produces different output for me and the output you posted really cannot be correct.

How could this

secret key hex: abcd...1234

ever be the result of that

_ascii_to_hex("blahblahblah")

Of course, the whole ascii_to_hex thing is completely irrelevant to your problem, but it shows that you should double-check your results.

Answer 2

For argument's sake, please assume that the secret key is the same in both.

Or, for example, for secret key abcd1234, the secret key bytes are: 6162636431323334.

The secret key bytes are the same in both scripts.

Answer 3

I find the the main problems I have had with hashes in comparisons are:

1. ensure the data and key are the same in both comparisons
2. ensure the data and key are in the same character encoding in both comparisons
3. ensure the key and text are being passed the same in both scripts, i.e. which one is key and which one is text (this has caught me more than once).

Try using the Digest::SHA module to create the hash for you and compare the results with that.

use Digest::SHA qw(hmac_sha1_hex);
my $hash = hmac_sha1_hex($data, $key);

See docs at http://perldoc.perl.org/Digest/SHA.pdf

Answer 4

Divide and conquer?

The test vectors in the RFC are the best place to start. Did they pass in both instances? Which ones did you try? If some work and others don't the most likely problem is that one of the two APIs are improperly marshalling the keys input (Signed vs unsigned arrays, charset conversions..etc)

As an aside its really difficult to help you when your example is nonsense. As others mentioned the hex representation of blah blah is not abc..123. Makes me wonder what else in your example is inaccurate?

Answer 5

To deal with some of the concerns about the question, I've fixed the examples and the results they return to accurately reflect the byte content and the digests returned. Thanks.

Answer 6

The encoding subroutine is virtually identical in both scripts (except for an unused argument passed to the subroutine, which I remove from my custom version).

Since you're not comparing the digests themselves, but Base-64 encoded versions of the digests, I would recommend backing up one step and checking the digests themselves. It may be possible that the Base-64 encoding routines are incorrect.

If you can't compare the digests themselves, then use the same encoding routine in both programs and see what you get.