How do I unpack binary files with Perl?

Question

Hi there,

I have two binary files equal in size but not in values. I used unpack as follows but the results are weird.

When I use "big" binary files - Both of the outputs shows part of the results:

One binary file starts good - ends bad,

Second binary file goes wrong at the start.

Where do you think may be the week point ?

open(BIN_FILE1, "<bin_files/BINF1.bin") or die("Cannot open file for writing");
open(BIN_FILE2, "<bin_files/BINF2.bin") or die("Cannot open file for writing");
binmode(BIN_FILE1); 
binmode(BIN_FILE2);
# N An unsigned long (32-bit) in "network" (big-endian) order.
my @values_tmp1 =  unpack("N*", <BIN_FILE1>);
my @values_tmp2 =  unpack("N*", <BIN_FILE2>);
close (BIN_FILE1);
close (BIN_FILE2);
my $tmp_bin1 = @values_tmp1;
my $tmp_bin2 = @values_tmp2;
print "\nBIN FILE1 LENGTH: ",$tmp_bin1,"\n";
print "\nBIN FILE2 LENGTH: ",$tmp_bin2,"\n";

The output is:

BIN FILE1 LENGTH: 1203

BIN FILE2 LENGTH: 124

The input files are:

-rw-rw-r-- 1 yodar yodar   9600 2009-12-23 19:59 BINF1.bin
-rw-rw-r-- 1 yodar yodar   9600 2009-12-27 16:38 BINF2.bin

If there is another simple and safe way to gather binary files data to an array I'll be glad to know.

Thanks a lot, Yodar.

Answer 1

I suspect that you are not reading the entire file (the <> operator tries to read a record, using the record seperator $/ which is newline by default), try changing the read to something like this:

{
    #enable slurp mode
    local $/;
    my @values_tmp1 =  unpack("N*", <BIN_FILE1>);
    my @values_tmp2 =  unpack("N*", <BIN_FILE2>);
}

Answer 2

You said you had "big" files. It looks like they aren't all that big, and will fit easily in memory.

However, if your files are too big to fit in memory all at once you can use read or sysread to read data in chunks.

Example - Read in 4000 bytes at a time:

my $file_path = 'bin_files/BINF1.bin';

open my $fh, '<:encoding(raw)', $file_path 
    or die "Cannot open file '$file_path' for reading";

my $buffer;
while ( my $got = read( $fh, $buffer, 4000 ) ) {

  warn "Read $got bytes from file\n."

  my @integers = unpack "N*", $buffer;

  do_stuff( \@integers );

  $buffer = '';
}

Use a 3 argument open with lexical handles, it avoids certain security and maintainability issues. 3 arg open also allows you to specify file encoding in the mode flag. That way you don't need to call binmode.