Cross-Platform Camera API

Question

Hi,

I'm now building a video transforming filter that have to transform video frames in real-time. One of the key requirements of the filter is to have high performance to minimize the number of dropped frames during the transform.

Another requirement that is of lower priority but also nice to have is to make it cross-platform (both PC's and Mobile devices).

The application is built in C++.

Now my question is:

is there any API that is more portable and has a similar or better performance characteristics than DirectShow? as DirectShow's portability is only limited to Windows-based devices (PCs and Windows Mobile&CE platforms).

Also I've notices that for example using HTC's custom camera API has far better performance than what DirectShow offers. If you want to check this, try to build a filter in DirectShow that will multiply each color by 2 and render that in real-time from camera on the screen. Then do the same with HTC's API. There is almost 4-5x performance boost with vendor's specific API. So it'd be very nice if the library used the device-specific implementation of the driver, as performance is critical when doing this transforms on a mobile device (which is about ~500 MHz).

Answer 1

Consider doing it in OpenGL shaders, then the filters can be hardware accelerated and they can run on desktop and GLES mobile devices (but they won't be fully compatible). The operations that you can (easily) perform this way are somewhat limited, but most things that you would like to do in video filtering can certainly be done.

Answer 2

There is opencv.org, which has an extremely simple cross-platform interface for connecting to video cameras, however this is a complex system designed for image processing applications, and does not directly provide capture. You would need to process your frames and stream them to a file. Also, as to performance, when I looked at openCV it appears that the windows side of it uses a very old 16bit video framework on Windows :(.

You might try Nokia's QT application framework. They are currently in the process of developing a "QT Multimedia" library that does what you want. However, this is currently a beta product, and my experience (3-4 months ago), has been that the Windows port is still needing some work. YMMV.
You will have to install QT, and then the "new QT APIS - Mobility Project" from the "other downloads section of the qt.nokia.com/downloads/. Depending on the timeframe of your project, this may be a good option for you... QT is definitely a nice toolkit IMHO.