How can I sum large hexadecimal values in Perl?

Question

I'm iterating over a set of 32-bit hexadecimal strings ("DEADBEEF", "12345678", etc..) and I'm trying to sum them together to form a 32-bit checksum. Assume that the variable $temp is loaded with some hexadecimal string in the example below.

my $temp;
my $checksum;

for (...)
{
    #assume $temp is loaded with a new hex string here
    my $tempNum = hex ($temp);
    $checksum += $tempNum;
    $checksum &= 0xFFFFFFFF;
    print printf("checksum: %08X",$checksum);
}

The first few values are "7800798C", "44444444", and "44444444". The output is:

checksum: 7800798C
checksum: BC44BDD0
checksum: FFFFFFFF
checksum: FFFFFFFF

etc..

as you can see the first two summations are correct and then it seems to saturate. Am I missing something regarding the size limit of Perl variables?

EDIT: This is the actual output from the script (string is the hex string, value is the decimal conversion of that string, and checksum is the resulting output):

string: 7800798C, value: 2013297036, checksum 7800798C
string: 44444444, value: 1145324612, checksum BC44BDD0
string: 44444444, value: 1145324612, checksum FFFFFFFF
string: 44444444, value: 1145324612, checksum FFFFFFFF
string: 78007980, value: 2013297024, checksum FFFFFFFF
string: 44444444, value: 1145324612, checksum FFFFFFFF

Answer 1

If your perl is compiled with 32-bit integers, integer operations which result in numbers greater than 0xffffffff will cause problems. For example:

my $x = hex '0x1234567890';
my $y = hex '0x1234567890';

print $x + $y,  "\n";

You will get:

Integer overflow in hexadecimal number at ...
Hexadecimal number > 0xffffffff non-portable at ...

Use bignum to add transparent support for larger integers:

#!/usr/bin/perl

use strict; use warnings;
use bignum qw/hex/;

my $x = hex '0x7800798C';
my $chk;

for (1 .. 10) {
    $chk += $x;
    $chk &= 0xFFFFFFFF;
    printf("checksum: %08X\n", $chk);
}

Verify that the output matches you expectations:

checksum: 7800798C
checksum: F000F318
checksum: 68016CA4
checksum: E001E630
checksum: 58025FBC
checksum: D002D948
checksum: 480352D4
checksum: C003CC60
checksum: 380445EC
checksum: B004BF78

Without bignum, I get:

checksum: 7800798C
checksum: F000F318
checksum: FFFFFFFF
checksum: FFFFFFFF
checksum: FFFFFFFF
...

Answer 2

What result do you want? The &= will delete any bits of the number larger than 0xffffffff. Continually adding numbers together just gets you a larger and larger number, which is then &ed with 0xffffffff. At some point I would expect you would get something other than 0xffffffff, no? But it can never be bigger!

Perhaps a better way to make a checksum would be to xor your numbers together.

my $temp;
my $checksum;

for (...)
{
    #assume $temp is loaded with a new hex string here
    my $tempNum = hex ($temp);
    $checksum ^= $tempNum;
    print printf("checksum: %08X",$checksum);
}

This will make something unique to those numbers.

Answer 3

To expand on Sinan's answer, both &= operator and and the %X format are affected by the size of integers compiled into Perl. In this case, the maximum size of either is 4294967295, or 0xFFFFFFFF.

While your variables can hold values larger than this, they will be truncated to this maximum when passed through &= or sprintf("%X").

Answer 4

This was asked at Perl Monks before.

The answer seems to be "use integer".

Please see the original answer at Perl Monks and perldoc integer.

$ perl -we 'use integer; printf("%08X\n",  0xFFFF_FFFF + 0xFFFF_FFFF)'