How to solve the SelectAtPosition() crash when using VMR-9 in renderless mode?
The SelectAtPosition() crashing problem when renderless mode is used can be solved by implementing IVMRWindowlessControl9 interface in your custom-allocator.
In addition to IVMRSurfaceAllocator9 & IVMRImagePresenter9, implement IMRWindowlessControl9, too! class CAllocator : public IVMRSurfaceAllocator9, IVMRImagePresenter9, IVMRWindowlessControl9
Define all the virtual functions of IVMRWindowlessControl9. In my case, I needed to put extra codes on the following functions to make the mouse handling properly work: GetAspectRatioMode(), SetAspectRatioMode() ,SetAspectRatioMode(),GetVideoPosition() & GetNativeVideoSize(). The other functions just return S_OK.
M$ SDK documentation did not mention that we need to implement this!!! After hours of debugging, I've noticed that quartz.dll tries to query this interface from the custom-allocator. Sometimes it really pays to be patient!