Python: strange numbers being pulled from binary file /confusion with hex and decimals

Question

This might be extremely trivial, and if so I apologise, but I'm getting really confused with the outputs I'm getting: hex? decimal? what?

Here's an example, and what it returns:

>>> print 'Rx State: ADC Clk=', ADC_Clock_MHz,'MHz DDC Clk=', DDC_Clock_kHz,'kHz Temperature=', Temperature,'C'
Rx State: ADC Clk= [1079246848L, 0L] MHz DDC Clk= [1078525952L, 0L] kHz Temperature= [1078140928L, 0L] C

Now I admit this is slight guesswork because I don't know exactly what the data is - I have a specification of how to parse it out of the file, but it's giving me very strange answers.

As you can see - the values are very similar, all around the 1078000000 mark, which leads me to believe I might be extracting something strange (like hex, but I don't think it is...)

The structure is read as follows (apologies for length):

#Read block
    more = 1
    while(more == 1):
        a = array.array("L")
        a.fromfile(wholeFile,2)
        if len(a) == 2:
            structure_id = a[0]
            print 'structure_id: ', hex(structure_id)
            structure_length = a[1]
            print 'structure_length: ', structure_length
        else:
            print 'cannot read structure start'

        numDwords = (structure_length/4) - 2 - 1;
        print 'numDwords: ', numDwords

        content = array.array("L")
        content.fromfile(wholeFile,numDwords)
        if len(content) != numDwords:
            print 'cannot read structure'
            more = 0
            ok = 0

and then the above example was retrieved from this by:

pos = 2
v1 = [content[pos+1], content[pos]]
pos = pos+2
v2 = [content[pos+1], content[pos]]
pos = pos+2
v3 = [content[pos+1], content[pos]]
pos = pos+2

ADC_Clock_MHz = v1
DDC_Clock_kHz = v2
Temperature = v3

Right sorry again for how verbose that was, but it's not just those values, it seems some values are ok and some aren't, which leads me to believe that the larger numbers are encoded differently... Also I have no idea why all the values are in pairs either!

Pants question, but if anyone has any insight it'd be much appreciated.

Answer 1

The contents are in pairs because you assign a pair to the variables (e.g. ADC_Clock_MHz = v1 and v1 = [content[pos+1], content[pos]]).

You are basically assigning a list of two elements to v1 where the first element is the element in the index pos+1 in the array content and the second element is the element in the index pos in the array content.

I'm a bit confused why you are constructing this list of those two elements. Are you trying to combine the two elements into a single number? I think you need to tell us a bit more about the file format if you can. Is this homework?

And no, the output is not hex, it's decimal. I think you're reading the data incorrectly from the binary file.

Answer 2

You might be happier with something like the following to read this block.

If your data isn't unpacking properly, the most likely cause is that the header is not simply 2 Long integers. Or, it's possible that the doesn't reflect the "endian-ness" of your platform.

Using struct allows you to add a "<" or ">" to the format string to try different endian-ness. Also, you can easily change the format of the message to use ordinary ints or signed ints or floats without too much real work.

import struct
def read( someFile ):
    header= someFile.read( 8 )
    id, length = struct.unpack( "LL", header )
    print id, length
    body = someFile.read( length-8 ) # Common for length to include the header
    words = (length-8)//4
    content= struct.unpack( "L"*words, body )

You can also print repr(header) and repr(body) to try and get a better sense of what your data actually looks like.

Answer 3

Found out it's incredibly easy to do this with numpy.fromfile

Specify what you want to extract (e.g. uint32, int16 etc.) and it extracts it as an array. You can even specify your own types as a collection of existing types, meaning you can extract known structures in one go (e.g. 2 uint32s then 1 string then 5 int16s as an array of 8 values)