views:

799

answers:

11

I am looking for a scripting (or higher level programming) language (or e.g. modules for Python or similar languages) for effortlessly analyzing and manipulating binary data in files (e.g. core dumps), much like Perl allows manipulating text files very smoothly.

Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.

Any suggestions?

+4  A: 

perl's pack and unpack ?

toolkit
And regexps that work happily on binary -- a feature I helped debug back in the day through my (ab)use of it. And you can pack/unpack to binary if you want to match on that. Recent implementations with their unicode support seemed to have muddied the water but I think all that can be turned off.
George Phillips
+2  A: 

Any high-level programming language with pack/unpack functions will do. All 3 Perl, Python and Ruby can do it. It's matter of personal preference. I wrote a bit of binary parsing in each of these and felt that Ruby was easiest/most elegant for this task.

Igor Krivokon
+2  A: 

Why not use a C interpreter? I always used them to experiment with snippets, but you could use one to script something like you describe without too much trouble.

I have always liked EiC. It was dead, but the project has been resurrected lately. EiC is surprisingly capable and reasonably quick. There is also CINT. Both can be compiled for different platforms, though I think CINT needs Cygwin on windows.

R Ubben
This was my thought as well. I don't know EiC, but cint accepts a loose dialect (it does implicitly (but still strong) typing when you don't specify types for newly introduced variable), which makes for a more RAD style when writing macros.
dmckee
+2  A: 

Python's standard library has some of what you require -- the array module in particular lets you easily read parts of binary files, swap endianness, etc; the struct module allows for finer-grained interpretation of binary strings. However, neither is quite as rich as you require: for example, to present the same data as bytes or halfwords, you need to copy it between two arrays (the numpy third-party add-on is much more powerful for interpreting the same area of memory in several different ways), and, for example, to display some bytes in hex there's nothing much "bundled" beyond a simple loop or list comprehension such as [hex(b) for b in thebytes[start:stop]]. I suspect there are reusable third-party modules to facilitate such tasks yet further, but I can't point you to one...

Alex Martelli
+2  A: 

I'm using 010 Editor to view binary files all the time to view binary files. It's especially geared to work with binary files.

It has an easy to use c-like scripting language to parse binary files and present them in a very readable way (as a tree, fields coded by color, stuff like that).. There are some example scripts to parse zipfiles and bmpfiles.

Whenever I create a binary file format, I always make a little script for 010 editor to view the files. If you've got some header files with some structs, making a reader for binary files is a matter of minutes.

Wouter van Nifterick
+23  A: 

Things I want to do include presenting arbitrary chunks of the data in various forms (binary, decimal, hex), convert data from one endianess to another, etc. That is, things you normally would use C or assembly for, but I'm looking for a language which allows for writing tiny pieces of code for highly specific, one-time purposes very quickly.

Well, while it may seem counter-intuitive, I found erlang extremely well-suited for this, namely due to its powerful support for pattern matching, even for bytes and bits (called "Erlang Bit Syntax"). Which makes it very easy to create even very advanced programs that deal with inspecting and manipulating data on a byte- and even on a bit-level:

Since 2001, the functional language Erlang comes with a byte-oriented datatype (called binary) and with constructs to do pattern matching on a binary.

And to quote informIT.com:

(Erlang) Pattern matching really starts to get fun when combined with the binary type. Consider an application that receives packets from a network and then processes them. The four bytes in a packet might be a network byte-order packet type identifier. In Erlang, you would just need a single processPacket function that could convert this into a data structure for internal processing. It would look something like this:

processPacket(<<1:32/big,RestOfPacket>>) ->
    % Process type one packets
    ...
;
processPacket(<<2:32/big,RestOfPacket>>) ->
    % Process type two packets
    ...

So, erlang with its built-in support for pattern matching and it being a functional language is pretty expressive, see for example the implementation of ueencode in erlang:

uuencode(BitStr) ->
<< (X+32):8 || <<X:6>> <= BitStr >>.
uudecode(Text) ->
<< (X-32):6 || <<X:8>> <= Text >>.

For an introduction, see Bitlevel Binaries and Generalized Comprehensions in Erlang.You may also want to check out some of the following pointers:

none
Exactly what I was going to suggest. Erlang can do very cool things with binary data.
JesperE
I'd go with erlang too on this one
Faisal Vali
awesome answer! extremely thorough.
amdfan
I concur. Joe Armstrong always says that he just cannot believe that no other language has Bit literals and pattern matching on Bits. And I completely agree. I mean, they put Float literals (which really only a very few scientific programmers need) in every sh*tty language, but no Bit syntax? Why? If you have String literals and Regexps, adding Bitstrings and Bit Patterns wouldn't be such a big deal, would it? [Rant over.]
Jörg W Mittag
A: 

If you're doing binary level processing, it is very low level and likely needs to be very efficient and have minimal dependencies/install requirements.

So I would go with C - handles bytes well - and you can probably google for some library packages that handle bytes.

Going with something like Erlang introduces inefficiencies, dependencies, and other baggage you probably don't want with a low-level library.

Larry Watanabe
Actually, speed is not that much of an issue for me, as I will mostly use it to "browse" some binary blob (e.g. a core dump, or data recorded from a stream) more or less interactively, with a size that can usually be counted in megabytes.You don't happen to know of any such library packages for C that are worth checking out?
ehdr
+1  A: 

Forth can also be pretty good at this, but it's a bit arcane.

Jason Baker
A *bit* arcane? :)
Paul Nathan
+4  A: 

Take a look at python bitstring, it looks like exactly what you want :)

buster
A: 

Well, if speed is not a consideration, and you want perl, then translate each line of binary into a line of chars - 0's and 1's. Yes, I know there are no linefeeds in binary :) but presumably you have some fixed size -- e.g. by byte or some other unit, with which you can break up the binary blob.

Then just use the perl string processing on that data :)

Larry Watanabe
+2  A: 

The Python bitstring module was written for this purpose. It lets you take arbitary slices of binary data and offers a number of different interpretations through Python properties. It also gives plenty of tools for constructing and modifying binary data.

For example:

>>> from bitstring import BitString
>>> s = BitString('0x00cf')                          # 16 bits long
>>> print(s.hex, s.bin, s.int)                       # Some different views
0x00cf 0000000011001111 207
>>> s[2:5] = '0b001100001'                           # slice assignment
>>> s.replace('0b110', '0x345')                      # find and replace
2                                                    # 2 replacements made
>>> s.prepend(True)                                  # Add 1 bit to the start
>>> s[::-8].hex                                      # Byte reversal
'0xaf6890a281'
>>> ordinary_string = s.bytes                        # Back to Python string

There are also functions for bit-wise reading and navigation in the bitstring, much like in files; in fact this can be done straight from a file without reading it into memory:

>>> s = BitString(filename='somefile.ext')
>>> hex_code, a, b = s.readlist('hex:32, uint:7, uint:13')
>>> s.find('0x0001')         # Seek to next occurence, if found
True

There are also views with different endiannesses as well as the ability to swap endianness.

Scott Griffiths