Samplegrabber works fine on AVI/MPEG files but choppy with WMV

Question

I have been using the latest version of the WPFMediaKit. What I am trying to do is write a sample application that will use the Samplegrabber to capture the video frames of video files so I can have them as individual Bitmaps.

So far, I have had good luck with the following code when constructing and rendering my graph. However, when I use this code to play back a .wmv video file, when the samplegrabber is attached, it will play back jumpy or choppy. If I comment out the line where I add the samplegrabber filter, it works fine. Again, it works with the samplegrabber correctly with AVI/MPEG, etc.

 protected virtual void OpenSource()
    {
        FrameCount = 0;
        /* Make sure we clean up any remaining mess */
        FreeResources();

        if (m_sourceUri == null)
            return;

        string fileSource = m_sourceUri.OriginalString;

        if (string.IsNullOrEmpty(fileSource))
            return;

        try
        {
            /* Creates the GraphBuilder COM object */
            m_graph = new FilterGraphNoThread() as IGraphBuilder;

            if (m_graph == null)
                throw new Exception("Could not create a graph");


            /* Add our prefered audio renderer */
            InsertAudioRenderer(AudioRenderer);

            var filterGraph = m_graph as IFilterGraph2;

            if (filterGraph == null)
                throw new Exception("Could not QueryInterface for the IFilterGraph2");

            IBaseFilter renderer = CreateVideoMixingRenderer9(m_graph, 1);                                

            IBaseFilter sourceFilter;

            /* Have DirectShow find the correct source filter for the Uri */
            var hr = filterGraph.AddSourceFilter(fileSource, fileSource, out sourceFilter);
            DsError.ThrowExceptionForHR(hr);

            /* We will want to enum all the pins on the source filter */
            IEnumPins pinEnum;

            hr = sourceFilter.EnumPins(out pinEnum);
            DsError.ThrowExceptionForHR(hr);

            IntPtr fetched = IntPtr.Zero;
            IPin[] pins = { null };

            /* Counter for how many pins successfully rendered */
            int pinsRendered = 0;                

            m_sampleGrabber = (ISampleGrabber)new SampleGrabber();
            SetupSampleGrabber(m_sampleGrabber);
            hr = m_graph.AddFilter(m_sampleGrabber as IBaseFilter, "SampleGrabber");
            DsError.ThrowExceptionForHR(hr);

            /* Loop over each pin of the source filter */
            while (pinEnum.Next(pins.Length, pins, fetched) == 0)
            {
                if (filterGraph.RenderEx(pins[0],
                                         AMRenderExFlags.RenderToExistingRenderers,
                                         IntPtr.Zero) >= 0)
                pinsRendered++;

                Marshal.ReleaseComObject(pins[0]);
            }

            Marshal.ReleaseComObject(pinEnum);
            Marshal.ReleaseComObject(sourceFilter);

            if (pinsRendered == 0)
                throw new Exception("Could not render any streams from the source Uri");

            /* Configure the graph in the base class */
            SetupFilterGraph(m_graph);

            HasVideo = true;
            /* Sets the NaturalVideoWidth/Height */
            //SetNativePixelSizes(renderer);
        }
        catch (Exception ex)
        {
            /* This exection will happen usually if the media does
             * not exist or could not open due to not having the
             * proper filters installed */
            FreeResources();

            /* Fire our failed event */
            InvokeMediaFailed(new MediaFailedEventArgs(ex.Message, ex));
        }

        InvokeMediaOpened();
    }

And:

 private void SetupSampleGrabber(ISampleGrabber sampleGrabber)
    {
        FrameCount = 0;
        var mediaType = new AMMediaType
        {
            majorType = MediaType.Video,
            subType = MediaSubType.RGB24,
            formatType = FormatType.VideoInfo
        };

        int hr = sampleGrabber.SetMediaType(mediaType);

        DsUtils.FreeAMMediaType(mediaType);
        DsError.ThrowExceptionForHR(hr);

        hr = sampleGrabber.SetCallback(this, 0);
        DsError.ThrowExceptionForHR(hr);
    }

I have read a few things saying the the .wmv or .asf formats are asynchronous or something. I have attempted inserting a WMAsfReader to decode which works, but once it goes to the VMR9 it gives the same behavior. Also, I have gotten it to work correctly when I comment out the IBaseFilter renderer = CreateVideoMixingRenderer9(m_graph, 1); line and have filterGraph.Render(pins[0]); -- the only drawback is that now it renders in an Activemovie widow of its own instead of my control, however the samplegrabber functions correctly and without any skipping. So I am thinking the bug is in the VMR9 / samplegrabbing somewhere.

Any help? I am new to this.

Answer 1

Some decoders will use hardware acceleration using DXVA. This is implemented by negotiating a partly-decoded format, and passing this partly decoded data to the renderer to complete decoding and render. If you insert a sample grabber configured to RGB24 between the decoder and the renderer, you will disable hardware acceleration.

That, I'm sure, is the crux of the problem. The details are still a little vague, I'm afraid, such as why it works when you use the default VMR-7, but fails when you use VMR-9. I would guess that the decoder is trying to use dxva and failing, in the vmr-9 case, but has a reasonable software-only backup that works well in vmr-7.

I'm not familiar with the WPFMediaKit, but I would think the simplest solution is to replace the explicit vmr-9 creation with an explicit vmr-7 creation. That is, if the decoder works software-only with vmr-7, then use that and concentrate on fixing the window-reparenting issue.

Answer 2

It ended up the the code I have posted (which in itself was fairly shamelessly slightly modified by me from Jeremiah Morrill's WPFMediakit source code) was in fact adequate to render the .WMV files and be sample grabbed.

It seems the choppiness has something to do with running through the VS debugger, or VS2008 itself. After messing around with the XAML for a while in the visual editor, then running the app, I will have this choppy behavior introduced. Shutting down VS2008 seems to remedy it. :P

So not much of an answer, but at least a (annoying - restarting VS2008) fix when it crops up.