Splitting a binary file on binary delimiter?

Question

I'm working on a shell script to convert MPO stereographic 3D images into standard JPEG images. A MPO file is just two JPEG images, concatenated together.

As such, you can split out the JPEG files by finding the byte offset of the second JPEG's magic number header (0xFFD8FFE1). I've done this manually using hexdump/xxd, grep, head, and tail.

The problem here is grep: what can I use to search a binary directly for a specific magic number, and get back a byte offset? Or should I not use a shell script for this at all? Thanks.

Answer 1

I think a very simple home brew approach will be your best bet. The code for doing this would be very small, depending on all the special cases of your binary file format.

1. Use mmap to get a convenient view of your file in memory.
2. Start scanning, and save the byte-offset in a variable, say start.
3. Scan until you reach your delimiter, saving the ending offset, in say end.
4. Create a new file
5. Memory-map the new file
6. Copy the byte-range from start to end into the new file.
7. Close the new file and start scanning again.

Answer 2

You can do this using bbe (http://bbe-.sourceforge.net/) which is a sed like program for binary files:

In order to extract the first JPEG use:

bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 2' -o first_jpeg mpo_file

And for the second one:

bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 1' -o second_jpeg mpo_file

Note that this will not work if the JPEG's magic number occurs somewhere else in the MPO file.

Answer 3

I think that Bart is on to your biggest problem.. If that binary sequence repeats during the process, you will get partial JPEGs.

I did a quick test by concatenating some JPEGs and then extracting them with awk (please note that the magic number in my files ended in 0xE0 and not 0xE1):

   # for i in *.jpg ; do cat $i ; done > test.mpo 
   # awk 'BEGIN {RS="\xFF\xD8\xFF\xE0"; FILENUM=-1} {FILENUM++; if (FILENUM == 0) {next}; FILENAME="image0"FILENUM".jpg"; printf "%s",RS$0 > FILENAME;}' test.mpo  
   # file image0*.jpg
    image01.jpg:  JPEG image data, JFIF standard 1.01
    image010.jpg: JPEG image data, JFIF standard 1.01
    image011.jpg: JPEG image data, JFIF standard 1.01

This seemed to work ok for me, but the above mentioned issues are still unhandled and very real.