Read 40 bytes of binary data as ascii text

Question

I have some binary data, in hex editor it looks like: s.o.m.e.d.a.t.a

with all these dots in between each letter

when I read with filehandle.read(40) it shows these dots

I know that the dots aren't supposed to be there, is there a way to unpack some ascii data that is 40 bytes long with struct?

I tried '40s' and 's' but it shows weird data, or only unpacks 1 character instead of 40.

Answer 1

Quick-and-dirty solution is to use s[::2] where s is the 80-characters byte string of which you want to consider only alternate bytes. The "clean: solution, per @fadden's comment, might be to read in the data as UTF-16 (then .encode it to ASCII, etc), but if the Q&D one suffices for your purposes, it might be simpler and faster (if the original data has characters that are not in the lowest 256 range, the Q&D approach will give strange results while the proper one would raise an exception -- which treatment is better depends on you app...).

Answer 2

If your first byte is an ASCII character (as indicated by your example) and your second byte is '\x00', then you probably have data encoded as UTF-16LE.

However it would be a good idea if you showed us unequivocably exactly what's in the first few bytes of your file. Please do this:

python -c "print(repr(open('myfile.txt', 'rb').read(20)))"

and edit your question to show us the result. If any text is confidential, please retain the sense when editing it.

We are especially interested to see if it starts with a UTF-16 BOM ('\xff\xfe' or '\xfe\xff').

For background, what platform (Windows or Linux) are you on? What produced the file?

Update I'm a bit puzzled by your statement """I tried '40s' and 's' but it shows weird data, or only unpacks 1 character instead of 40.""" Examine the following examples:

>>> data = "q\x00w\x00"
>>> unpack("4s", data)
('q\x00w\x00',) # weird? it's effectively tuple([data])
>>> unpack("s", data)
# doesn't produce a string of length 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: unpack requires a string argument of length 1
>>> unpack("ssss", data)
('q', '\x00', 'w', '\x00') # this == tuple(data)
>>>

@pxh commented """You're only getting a single character because those dots are being read as ASCII NULs (and so terminating the string).""" I doubt very much whether @pxh could actually demonstrate that struct.unpack's use of the "s" format depends in any way on the individual byte values in the data, whether NUL ("\x00") or anything else.

Answer 3

For reading binary data in python I am using:

val = f.read(1)
val = struct.unpack( 'c' , val )

And reading byte by byte all that I need. For 40 byte struct I would be

val = f.read(40)
val = struct.unpack( '40c' , val )