ansaurus

Question

Answer 1

+1 A:

import base64
coded_string = '''Q5YACgA...'
base64.b64decode(coded_string)

worked for me. At the risk of pasting an offensively-long result, I got:

>>> base64.b64decode(coded_string)
2: 'C\x96\x00\n\x00\x00\x00\x00C\x96\x00\x1b\x00\x00\x00\x00C\x96\x00-\x00\x00\x00\x00C\x96\x00?\x00\x00\x00\x00C\x96\x07M\x00\x00\x00\x00C\x96\x07_\x00\x00\x00\x00C\x96\x07p\x00\x00\x00\x00C\x96\x07\x82\x00\x00\x00\x00C\x96\x07\x94\x00\x00\x00\x00C\x96\x07\xa6Cq\xf0\x7fC\x96\x07\xb8DJ\x81\xc7C\x96\x07\xcaD\xa5\x9dtC\x96\x07\xdcD\xb6\x97\x11C\x96\x07\xeeD\x8b\x8flC\x96\x07\xffD\x03\xd4\xaaC\x96\x08\x11B\x05&\xdcC\x96\x08#\x00\x00\x00\x00C\x96\x085C\x0c\xc9\xb7C\x96\x08GCy\xc0\xebC\x96\x08YC\x81\xa4xC\x96\x08kC\x0f@\x9bC\x96\x08}\x00\x00\x00\x00C\x96\x08\x8e\x00\x00\x00\x00C\x96\x08\xa0\x00\x00\x00\x00C\x96\x08\xb2\x00\x00\x00\x00C\x96\x86\xf9\x00\x00\x00\x00C\x96\x87\x0b\x00\x00\x00\x00C\x96\x87\x1d\x00\x00\x00\x00C\x96\x87/\x00\x00\x00\x00C\x96\x87AA\x0b\xe7PC\x96\x87SCI\xf5gC\x96\x87eC\xd4J\xeaC\x96\x87wD\r\x17EC\x96\x87\x89D\x00F6C\x96\x87\x9bC\x9cg\xdeC\x96\x87\xadB\xd56\x0cC\x96\x87\xbf\x00\x00\x00\x00C\x96\x87\xd1\x00\x00\x00\x00C\x96\x87\xe3\x00\x00\x00\x00C\x96\x87\xf5\x00\x00\x00\x00C\x9cY}\x00\x00\x00\x00C\x9cY\x90\x00\x00\x00\x00C\x9cY\xa4\x00\x00\x00\x00C\x9cY\xb7\x00\x00\x00\x00C\x9cY\xcbC\x1f\xbd\xa3C\x9cY\xdeCCz{C\x9cY\xf1CD\x02\xa7C\x9cZ\x05C+\x9d\x97C\x9cZ\x18C\x03R\xe3C\x9cZ,\x00\x00\x00\x00C\x9cZ?
[stuff omitted as it exceeded SO's body length limits]
\xbb\x00\x00\x00\x00D\xc5!7\x00\x00\x00\x00D\xc5!\xb2\x00\x00\x00\x00D\xc7\x14x\x00\x00\x00\x00D\xc7\x14\xf6\x00\x00\x00\x00D\xc7\x15t\x00\x00\x00\x00D\xc7\x15\xf2\x00\x00\x00\x00D\xc7\x16pC5\x9f\xf9D\xc7\x16\xeeC[\xb5\xf5D\xc7\x17lCG\x1b;D\xc7\x17\xeaB\xe3\x0b\xa6D\xc7\x18h\x00\x00\x00\x00D\xc7\x18\xe6\x00\x00\x00\x00D\xc7\x19d\x00\x00\x00\x00D\xc7\x19\xe2\x00\x00\x00\x00D\xc7\xfe\xb4\x00\x00\x00\x00D\xc7\xff3\x00\x00\x00\x00D\xc7\xff\xb2\x00\x00\x00\x00D\xc8\x001\x00\x00\x00\x00'

What problem are you having, specifically?

Blair Conrad 2010-08-12 17:58:08

I have exactlly the same process as you did, but what are the results ?

ladyfafa 2010-08-12 18:01:33

Answer 2

A:

i used chardet to detect possible encoding of this data ( if its text ), but get {'confidence': 0.0, 'encoding': None}. Then i tried to use pickle.load and get nothing again. I tried to save this as file , test many different formats and failed here too. Maybe you tell us what type have this 16512 bytes of mysterious data?

Odomontois 2010-08-12 19:12:48

I am sorry, I dont know it either, but I got a c++ code to deal with this data, but I dont know how to post the c++ code here in a right formate, seems to be too long (like -200 characters left) to be posted here :(

ladyfafa 2010-08-12 19:19:06

maybe the C++ code is the key to this mysterious data, but I was puzzled by it actually, I mean I need to work with python to decode it correctly

ladyfafa 2010-08-12 19:20:06

@ Odomontois, thanks for your work :)

ladyfafa 2010-08-12 19:20:28

post at pastebin.com and give us the link. Problem bust be in different charmaps for base64 encoding and decoding.

Odomontois 2010-08-12 19:31:39

Ok, I try, thank you

ladyfafa 2010-08-12 19:44:46

http://pastebin.com/XVgCSjAG

ladyfafa 2010-08-12 19:53:49

that's the link of the c++ code :)

ladyfafa 2010-08-12 19:54:20

let me know if you could see it, thanks a lot

ladyfafa 2010-08-12 19:56:38

I'm fairly bad with C++, but it does seem that the code is pulling out a 32-bit float, big-endian, like I was trying to do. I can't tell what the structure of `peak p` is, but I think that's what you just have to feed to `struct.unpack`

Nick T 2010-08-12 20:53:07

me neither, could be a structure for storing those elements

ladyfafa 2010-08-12 21:54:49

Answer 3

+1 A:

Interesting if maddening puzzle...but here's the best I could get:

The data seems to repeat every 8 bytes or so.

import struct
import base64

target = \
r'''Q5YACgAAAABDlgAbAAAAAEOWAC0AAAAAQ5YAPwAAAABDlgdNAAAAAEOWB18AAAAAQ5YH 
[snip.]
ZAAAAABExxniAAAAAETH/rQAAAAARMf/MwAAAABEx/+yAAAAAETIADEAAAAA''' 

dtgt = base64.b64decode(target)

format = ">ff"
for i in range(100):
    print struct.unpack_from(format,dtgt,8*i)

That gives output like (these lines are randomly selected from the first 100):

(300.00030517578125, 0.0)
(300.05975341796875, 241.93943786621094)
(301.05612182617187, 0.0)
(301.05667114257812, 8.7439727783203125)
(326.9617919921875, 0.0)
(326.96826171875, 0.0)
(328.34432983398438, 280.55218505859375)

That first number does seem to monotonically increase through the entire set.

format = 'hhhh' (possibly with various paddings/directions (e.g. '<hhhh', '<xhhhh') also might be worth a look (again, random lines):

(-27069, 2560, 0, 0)
(-27069, 8968, 0, 0)
(-27069, 13576, 3139, -18487)
(-27069, 18184, 31043, -5184)
(-27069, -25721, -25533, -8601)
(-27069, -7289, 0, 0)
(-25533, 31066, 0, 0)
(-25533, -29350, 0, 0)
(-25533, 25179, 0, 0)
(-24509, -1888, 0, 0)
(-24509, -4447, 0, 0)
(-23741, -14725, 32067, 27475)
(-23741, -3973, 0, 0)
(-23485, 4908, -29629, -20922)

Nick T 2010-08-12 19:22:40

@Nick T, thanks, first of all, I think the first value should be positive, because it stands for the mass of a molecule, could never be a negative value; secondly, the toal number of elements in one of these tuples shall bestly be 3, that's all what I could guess about this data, they should be mass, intensity and time respectively

ladyfafa 2010-08-12 19:37:06

it is absolutely right the mass goes in an ascending way "monotonically increase", suppose it is value on the x-axis, from 0 to an adequate maximum, whereas the intensity of this mass might vary sharply

ladyfafa 2010-08-12 19:40:37

So this is data from a GC-MS? Try different formats, see http://docs.python.org/library/struct.html maybe `'>fhh'` or `'>fHH'`

Nick T 2010-08-12 19:54:13

exactlly! you are genious, for sure :)

ladyfafa 2010-08-12 19:57:40

how you know it is a GC-MS? have you worked on it before?

ladyfafa 2010-08-12 20:10:50

I only used a GC-MS a few times in lab, but doesn't it just kick out an intensity vs. time plot? Mass we had to derive from time, calibrated with some known compounds.

Nick T 2010-08-12 20:15:04

But you really knows it, I think the ultimate goal is to find the proteins, still halfway into it though. thanks for all your help :)

ladyfafa 2010-08-12 20:23:29

Answer 4

A:

After decoding, it looks like the data is a repeating structure that's 8 bytes long, or some multiple thereof. It's just binary data though; what it might mean, I have no idea. There are 2064 entries, which means that it could be a list of 2064 8-byte items down to 129 128-byte items.

Walter Mundt 2010-08-12 19:24:16

@Walter Mundt, thank you:))

ladyfafa 2010-08-12 19:49:47

How to know that there are 2064 entries?

ladyfafa 2010-08-13 00:58:52

Oh, i see, length of the decoded divided by 8

ladyfafa 2010-08-13 01:02:41

ansaurus

tags:

views:

answers:

Python base64 data decode

related questions