How to read pixel values of a video?

Question

I recently wrote C programs for image processing of BMP images, I had to read the pixel values and process them, it was very simple I followed the BMP header contents and I could get most of the information of the BMP image.

Now the challenge is to process videos (Frame by Frame), how can I do it? How will I be able to read the headers of continuous streams of image frames in a video clip? Or else, is it like, for example, the mpeg format will also have universal header, upon reading which I can get the information about the entire video and after the header, all the data are only pixels.

I hope I could convey.

Has anyone got experience with processing videos?

Any books or links to tutorials will be very helpful.

Answer 1

A video stream, like MPEG, is composed by a number of frames dependent from (obviously) its duration and its frame-rate. To read a pixel you must start from what is called an Intra frame, which is not dependent from a previous frame into the stream. Any successive frame is a frame which is temporally dependent from its previous frame, so to obtain its pixel, you have to decode the stream from the Intra to the frame you want.
Note that, tipically, an intra frame is inserted periodically to give to the decoder a way to synchronize with the stream. This is very useful in a context where errors can occur.
What you want to do isn't an easy work. You have to use an MPEG decoder and then modify the frame before diplaying it, if you want to do post processing, like a filter or other.
I suggest you to study video coding, and you can find a lot of material on that, starting from the standard MPEG.

Answer 2

I would recommend looking into FFMpeg. It has a command line utility that can grab frames from a movie and dump them to an image like JPG. You can then modify your existing reader to handle JPG files (just use something like libjpeg to decode the JPG to a raw pixel buffer).

Alternatively, you can use the FFMpeg APIs (C, Python, other), and do the frame capture programatically and look at the pixels as you move through the video. Video formats are complex, so unless you want to start understanding all of the different codecs, you might want to grab a library to do the decode to raw pixel buffer for you.

Answer 3

MPEG 1/2/4 videos are much more difficult to handle than bitmaps because they are compressed. With the bitmap data, you have actual color values stored directly to the file. In MPEG, or JPEG for that matter, the color information goes through a number transformations before being written to the file. These include

• RGB -> YUV 420P (sub-sampled chrominance)
• Discrete Cosine Transform
• Weighted quantization
• zig-zag ordering
• differential encoding
• variable-length encoding (Huffman-like)

All of this means that there is no simple way to parse pixel data out of the file. You either have to study every minute detail of the standard and write your own decoder, or use some video decoding library like ffmpeg to do the work for you. ffmpeg can convert your video to still images (see answers this recent question). Also, you could interface directly to the ffmpeg libraries (libavformat and libavcodec). See the answers to this question for good tutorials.