views:

102

answers:

3

I have a 64-bit timestamp unpacked from a file with binary data, where the top 32 bits are the number of seconds and the bottom 32 bits are the fraction of the second. I'm stuck with how to actually convert the bottom 32 bits into a fraction without looping through it bit-by-bit.

Any suggestions?

For reference, the number 4ca1f350 9481ef80 translates to 1285682000.580107659

Edit: For context: the data comes from a packet capture device and the documentation I've seen says that it the fractional part has roughly nano-second precision (specifically it outputs 29 of the 32 bits, giving ~2ns).

+3  A: 

You can just divide the hex number by the maximum possible to get the correct ratio:

>>> float(0x9481ef80) / 0x100000000
0.58010765910148621
Scott Griffiths
I know it's a real nit, but you need 0x100000000 instead of oxffffffff.
Mark Ransom
@Mark: No you're absolutely correct. Spotted it just after I had hit return. Those 2 parts in 10 billion could be crucial!
Scott Griffiths
+2  A: 

To represent the sum of integral and fractional part with enough precision (32 + 29 = 61 bits), you need a Decimal (28 decimal digits by default, which is enough for 93 bits),

>>> from decimal import Decimal
>>> Decimal(0x9481ef80) / Decimal(2**32) + Decimal(0x4ca1f350)
Decimal('1285682000.580107659101486206')

or Fraction (exact),

>>> from fractions import Fraction
>>> Fraction(0x9481ef80, 2**32) + Fraction(0x4ca1f350)
Fraction(43140329262089183, 33554432)
>>> float(_)
1285682000.5801077

Note that a float uses "IEEE double format" so it can only hold 53 bits of precision:

>>> a = 0x9481ef80 / 2**32 + 0x4ca1f350
>>> b = 0x9481ef90 / 2**32 + 0x4ca1f350
>>> a == b

It is fine if you store the fractional part as its own variable, but if that's the case, why not just keep it as-is?

>>> 0x9481ef80 / 2**32
0.5801076591014862
>>> 0x9481ef90 / 2**32
0.5801076628267765
KennyTM
You do not need a Decimal to represent **binary** fractions.
dan04
@dan: Yes, except that IEEE double is insufficient to represent **64** -bits of data.
KennyTM
@dan04, it depends on the precision you need. 64 bits won't fit into most floating point representations. On the other hand not many applications need all 64 bits. Unlike the conversion from decimal to binary, every binary fraction can be perfectly converted to decimal.
Mark Ransom
For context: the data comes from a packet capture device and the documentation I'm looking at says that it the fractional part has nano-second precision (specifically it outputs 29 of the 32 bits). This is for latency calculations so I need it accurate.
davedavedave
That makes a difference. `float` would only give ~microsecond precision.
dan04
So go with decimal then?
davedavedave
+1  A: 

You didn't say seconds since when. It looks like it's since 1970-01-01. You can calculate a fudge factor that is the number of seconds between the epoch (1970-01-01) and your expected lowest value. Then you adjust each value ... vadj = float(hi32 - fudge) + lo32 / 2.0 ** 32

If the difference between max(hi32) and min(lo32) is less than about 6 days worth (should be enough for a packet capture exercise (?)), then you need only 19 bits for hi32 - fudge. 19 bits + 32 bits is 51 bits -- within the precision of a Python float IIRC.

It's late here so I'm not going to do a detailed analysis but the above should give you the picture.

Edit: why @unwind's answer doesn't work:

>>> a = 0x00000001/4294967296.0 + 0x4ca1f350
>>> b = 0x00000002/4294967296.0 + 0x4ca1f350
>>> b - a
0.0
>>>

Edit 2: What operations do you want to do on a timestamp apart from str(), repr(), timestamp_from_str()? Difference is about all that comes to mind. You can use something like this:

>>> class TS64(object):
...   def __init__(self, hi, lo):
...     self.hi = hi
...     self.lo = lo
...   def float_delta(self, other):
...     hi_delta = self.hi - other.hi
...     # check that abs(hi_delta) is not too large, if you must
...     return hi_delta + (self.lo - other.lo) / 4294967296.0
...
>>> a = TS64(0x4ca1f350, 1)
>>> b = TS64(0x4ca1f350, 2)
>>> b.float_delta(a)
2.3283064365386963e-10
>>> repr(_)
'2.3283064365386963e-10'
>>>

About my "if you must" comment: If the observations are more than 6 days apart, do you really need accuracy down to the last (second / 2 ** 32)??? IMHO, if you do float(difference(ts1, ts2)) instead of float(ts1) - float(ts2), you should be OK.

Edit 3: Ambiguity/inconsistency alert

Please edit your question to address the following issues:

You say in a comment that """the documentation I'm looking at says that it the fractional part has nano-second precision (specifically it outputs 29 of the 32 bits)""". Please provide a URL for that documentation.

There are 1000000000 (10**9) nanoseconds in a second. One would expect the fractional part to require math.log(10**9, 2) rounded up (i.e. 29.897352853986263 rounded up i.e. 30) bits, not 29. Please explain.

Please answer: Of the 32 bits available, which 29 or 30 bits contain the fractional part and which 3 or 2 bits are always zero?

Secondly one would expect to convert the nanoseconds to seconds by dividing by 10**9. However your statement in your question """the number 4ca1f350 9481ef80 translates to 1285682000.580107659""" is consistent with dividing by 2**32. In fact 0x9481ef80 is 2,491,543,424 which is greater than twice 10**9. Please explain. What is the source of the "translates to" statement? Do you have any other examples?

John Machin
Re: Point 1: it is just a normal unix timestamp, just with greater accuracy than the usual microsecond accuracy
davedavedave
Re: Edit 2: Yep, I just need the difference between when the packet left the system to the corresponding packet returning
davedavedave
Re: Edit 3: 2^(-29) gives 1.85e-9, so that's ~2ns accuracy [I've updated my question]. I can't find the source where I read about the timestamp, but I remember that it said 29 bits, with the 3 lowest bits being zero. I get that number from Wireshark and from my own calculation in Excel. The reason for dividing by 2^32 rather than 10^9 is because this is a fixed-point binary number, where the lower 32 bits are a fraction of the second.
davedavedave