Working with bit streams

Question

I have a base64 encoded bit stream, I want to work with. After decoding it with base64.b64decode I get a bytes object (Py3k btw) containing the decoded code. The problem is now, that I would like to work on that bytes object with bit-wise operations, such as shifting, bit wise and etc, but that is not possible as it is a bytes sequence.

Is there any way to use binary operations on that decoded code?

Answer 1

try using

list(bytestring)

Eg.

>>> bstring=b"Hello World"
>>> list( bstring)
[72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100]
>>>

If you want one huge bitfield instead of all those octets

>>> from functools import reduce
>>> reduce(lambda x,y:(x<<8)+y,list(b"Hello World"))
87521618088882533792115812
>>> bin(_)
'0b100100001100101011011000110110001101111001000000101011101101111011100100110110001100100'
>>>

You didn't say how you are counting the bits, perhaps they are supposed to be reversed

>>> reduce(lambda x,y:(x<<8)+y,list(b"Hello World"[::-1]))
121404708493354166158910792
>>> bits=bin(_)[2:]

and pad the string to even bytes

>>> bits=bits.zfill(((len(bits)-1)//8+1)*8)
>>> bits
'0110010001101100011100100110111101010111001000000110111101101100011011000110010101001000'

turn the first 6 bits into an int

>>> int(bits[:6],2)
25

and then the following 4 bits

>>> int(bits[6:10],2)
1

Answer 2

If you're using Python 2.x, you could try using Construct. It can do very elegant parsing of data, including bit data.

It hasn't had so much active development recently, so I'm not sure what would be involved in making it work for Python 3.x. But for 2.x it's great.

Answer 3

If you need to make your bytes object mutable then you can construct a bytearray from it:

mutable = bytearray(b"immutable")

This will let you modify the individual bytes through normal assignments

mutable[0] = mutable[1] = 32

If you need to do bit-wise operations then I suggest trying bitstring (with apologies for recommending my own module). It works for Python 3 and lets you do bit-wise slicing, shifting, logical operations and much more.

>>> s = BitString(bytes=b'your_bytes_object')
>>> s.hex
'0x796f75725f62797465735f6f626a656374'
>>> ten_bits = s[5:15]
>>> print(ten_bits, ten_bits.int)
0b0010110111 183
>>> print(ten_bits << 2)
0b1011011100
>>> print(s[0:6] & '0b110100')
0b010100