Extracting an image from H.264 sample data (Objective-C / Mac OS X)

Given a sample buffer of H.264, is there a way to extract the frame it represents as an image?

I'm using QTKit to capture video from a camera and using a QTCaptureMovieFileOutput as the output object.

I want something similar to the CVImageBufferRef that is passed as a parameter to the QTCaptureVideoPreviewOutput delegate method. For some reason, the file output doesn't contain the CVImageBufferRef.

What I do get is a QTSampleBuffer which, since I've set it in the compression options, contains an H.264 sample.

I have seen that on the iPhone, CoreMedia and AVFoundation can be used to create a CVImageBufferRef from the given CMSampleBufferRef (Which, I imagine is as close to the QTSampleBuffer as I'll be able to get) - but this is the Mac, not the iPhone.

Neither CoreMedia or AVFoundation are on the Mac, and I can't see any way to accomplish the same task.

What I need is an image (whether it be a CVImageBufferRef, CIImage or NSImage doesn't matter) from the current frame of the H.264 sample that is given to me by the Output object's call back.

Extended info (from the comments below)

I have posted a related question that focusses on the original issue - attempting to simply play a stream of video samples using QTKit: http://stackoverflow.com/questions/3854505/playing-a-stream-of-video-data-using-qtkit-on-mac-os-x

It appears not to be possible which is why I've moved onto trying to obtain frames as images and creating an appearance of video, by scaling, compressing and converting the image data from CVImageBufferRef to NSImage and sending it to a peer over the network.

I can use the QTCapturePreviewVideoOutput (or decompressed) to get uncompressed frame images in the form of CVImageBufferRef.

However, these images references need compressing, scaling and converting into NSImages before they're any use to me, hence the attempt to get an already scaled and compressed frame from the framework using the QTCaptureMovieFileOutput (which allows a compression and image size to be set before starting the capture), saving me from having to do the expensive compression, scale and conversion operations, which kill CPU.

That is what I am currently doing, but using a `QTCaptureVideoPreviewOutput` rather than a decompressed one, since I don't care if frames are dropped. Currently I take the frame, reduce its size, compress it into JPEG format and then carry on. However, I would rather let the framework handle the compression and image reduction and just deal with what it gives me..

Jasarien 2010-10-11 08:53:32

I just woke and made coffee (so don't quote me) but I thought last time I read up on QTKit that you can add multiple outputs. Obviously slower systems might start to chug, but since you're not worried about dropped frames ...

Joshua Nozzi 2010-10-11 15:11:26

That is possible, yes - but I don't see how it helps in this situation. I don't need *more* outputs, I need a more efficient way of obtaining a compressed image of a frame each time the callback is invoked.

Jasarien 2010-10-12 15:22:38

C'mon, now, you didn't mention your awareness of that solution in your post, so it's a valid answer. I believe the answer to *this* version of your question is "you're SOL" but don't quote me. It makes sense, though, since this is not typically needed if you're just dumping straight to a file and may be left out for performance reasons.

Joshua Nozzi 2010-10-12 15:53:11

My question stated that I knew of the method that used `CVImageBufferRef`s which is what the single frame grabbing app uses. You may like to take a look at a previous question of mine that focusses on the original issue - attempting to simply play a stream of video samples using QTKit: http://stackoverflow.com/questions/3854505/playing-a-stream-of-video-data-using-qtkit-on-mac-os-x - It appears not to be possible which is why I've moved onto obtaining frames as images and creating an *appearence of video*.

Jasarien 2010-10-12 22:28:33

I can use the `QTCapturePreviewVideoOutput` (ore decompressed) to get uncompressed frame images in the form of CVImageBufferRef. But these need compressing, scaling and converting into NSImages before they're any use to me, hence the attempt to get an already scaled and compressed frame from the framework and save me having to do the expensive compression, scale and conversion operations, which kill CPU.

Jasarien 2010-10-12 22:31:17

ansaurus

tags:

views:

answers:

Extracting an image from H.264 sample data (Objective-C / Mac OS X)

related questions