How to efficiently render and process video streams using GPU?

Question

I plan to develop a tool for realtime video manipulation using C++, Qt and OpenGL. Video overlay isn't an option since shaders should be used for frame processing. At the moment I imagine a following sequence of steps:

1. Decode video (CPU)
2. Preprocess it (optional, CPU)
3. Transer it to video memory (GPU using DMA)
4. Further process it using vertex and fragment shaders (GPU)
5. Render it (GPU)

I'm looking for some general advice explaning what extentions or technics can be used here. Is there a good reason to use Direct3D instead?

Answer 1

As an alternative you can look at some of the different languages for performing general purpose GPU programming (GPGPU) such as CUDA from NVIDIA or Stream SDK from ATI. Depending on which you choose, you can limit yourself to one brand of GPUs, though. The reason for using these languages would be to work at a level of abstraction which is closer to normal high-level programming instead of working with shaders.

I have no experience with what you want to do so I can't say if shaders are actually better suited for the job but it is something you can consider looking at. It has to be said that algorithm design is still somewhat different than ordinary code and it takes some effort getting the hang of it (I have only worked with CUDA but they all seem to use similar abstractions).

I suppose thaat if you have a decent amount of experience with shader work it might not be worth the hassle for you to learn a new platform.

Answer 2

If you're linux, NVIDIA's recent drivers in the 180.xx series have added support for video decoding via the VDPAU api (Video Decoding and Presentation something). Many major projects have integrated with the api including mplayer, vlc, ffmpeg, and mythtv. I don't know all of the specifics, but they provide api for many codecs including common sub-operations and bitstream manipulation.

I'd look here before going straight to CUDA (which I assume VDPAU may use)

Answer 3

For transferring the frame data from cpu to gpu, you might want to look in to PBOs. Check this also.

Also if you are using shaders anyway, you could ease the cpu burden by doing colorspace transformations in the fragment shaders (YCbCr to RGB).

The "further processing" and "rendering" steps usually are pretty much the same thing, do cool stuff in shaders and blend it to framebuffer. If you want to mix and match videos and different effects, FBOs are useful too.

Answer 4

First thing, on the PC there's no explicit way to use DMA. The driver might use it, or might use something else.

In any case, step 3 will be "change texture data on the graphics card". In OpenGL that's PBO (Pixel Buffer Object) extension or good old glTexSubImage* function. In D3D9 it's LockRect on the texture or other ways (e.g. LockRect on a scratch texture, then blit into a GPU texture). Any of those would potentially use DMA, but you can't be sure.

Then the data is in a texture. You can render it to a screen with some shaders (e.g. doing YCbCr conversion), or render into other texture(s) to do more complex processing effects (e.g. blur/glow/...).

Using Direct3D is easier in a sense that there are clearly defined "wast ways" of doing stuff. In OpenGL there are a lot more options to do anything, and you have to somehow figure out which ones are fast (sometimes the fast paths are different on different platforms or hardware).

Answer 5

The following steps should do it:

1. decode video to YUV

   This is usually what the decoder libs do.

2. Load into OpenGL as a texture

3. convert YUV to RGB

   Since you don't want to use an overlay, you have to convert manually. Here is an example using shaders.

4. put converted texture on a quad and render to screen