First of all, add this to the constructor of your ActiveX control:
// this seemingly innocent line is _extremely_ important.
// This causes the window for the control to be created
// otherwise, you won't get an hWnd to render to!
m_bWindowOnly = true;
Your ActiveX control will have a member variable called m_hWnd that you'll be able to use as a render target. without the m_bWindowOnly variable set true, the ActiveX control won't create its own window.
Finally, pick your renderer (VMR9 for example)
CRect rcClient;
CComPtr<IBaseFilter> spRenderer;
CComPtr<IVMRWindowlessControl9> spWindowless;
// Get the client window size
::GetClientRect(m_hWnd, rcClient);
// Get the renderer filter
spRenderer.Attach( m_pGraph->GetVideoRenderer() );
if( ! spRenderer )
return E_POINTER;
spWindowless = spRenderer;
if( spWindowless )
{
spWindowless->SetVideoClippingWindow( m_hWnd );
spWindowless->SetVideoPosition(NULL, rcClient);
spWindowless.Release();
}
spRenderer.Detach();
Please note that my graph object is a custom object and that GetVideoRenderer() is one of my own functions - it returns an IBaseFilter*.
It took me ages to find this one out. ATL is poorly documented, which is a shame, because it's an excellent technology. Anyways, hope this helps!