C#: Object with custom marshaller not containing data after PInvoke call

Question

I am having a problem with PInvoking some WinAPI functions that accept WAVEFORMATEX structures as parameters. Since the length of the WAVEFORMATEX structure can vary, I implemented a WaveFormatEX class that is marshalled by a custom marshaller class (which implements ICustmoMarshaller). This is following an example provided by Aaron Lerch in his Blog (Part 1, Part 2), but with a few modifications from my side.

When I call the API function from my code, the methods MarshalManagedToNative and MarshalNativeToManaged of the custom marshaller are called, and at the end of MarshalNativeToManaged, the managed object contains the correct values. But when the execution returns to my calling code, the WaveFormatEx object does not contain the values read during the API call.

So the question is: Why does the data that is correctly marshalled back from native to managed not show up in my WaveFormatEx object after the native API call? What am I doing wrong here?

Edit:
To clarify, the function call succeeds, so does the marshalling of the WaveFormatEx object back to managed code. Just when the execution returns from the marshalling method to the scope from where the method was called, the WaveFormatEx object that was declared in that calling scope does not contain the result data.

Here are the function prototype and the WaveFormatEx class:

[DllImport("avifil32.dll")]
public static extern int AVIStreamReadFormat(
    int Stream,
    int Position,
    [In, Out, MarshalAs(UnmanagedType.CustomMarshaler, 
        MarshalTypeRef = typeof(WaveFormatExMarshaler))]
    WaveFormatEx Format,
    ref int Size
    );

[StructLayout(LayoutKind.Sequential)]
public class WaveFormatEx
{
    public int FormatTag;
    public short Channels;
    public int SamplesPerSec;
    public int AvgBytesPerSec;
    public short BlockAlign;
    public short BitsPerSample;
    public short Size;
    public byte[] AdditionalData;

    public WaveFormatEx(short AdditionalDataSize)
    {
        WaveFormat.Size = AdditionalDataSize;
        AdditionalData = new byte[AdditionalDataSize];
    }
}

The marshalling methods look like this:

public object MarshalNativeToManaged(System.IntPtr NativeData)
{
    WaveFormatEx ManagedObject = new WaveFormatEx(0);
    ManagedObject = (WaveFormatEx)Marshal.PtrToStructure(
       NativeData, typeof(WaveFormatEx));

    ManagedObject.AdditionalData = new byte[ManagedObject.Size];

    // If there is extra data, marshal it
    if (ManagedObject.WaveFormat.Size > 0)
    {
        NativeData = new IntPtr(
            NativeData.ToInt32() + 
            Marshal.SizeOf(typeof(WaveFormatEx)));
        ManagedObject.AdditionalData = new byte[ManagedObject.WaveFormat.Size];
        Marshal.Copy(NativeData, ManagedObject.AdditionalData, 0, 
            ManagedObject.WaveFormat.Size);
    }
    return ManagedObject;
}

public System.IntPtr MarshalManagedToNative(object Object)
{
    WaveFormatEx ManagedObject = (WaveFormatEx)Object;

    IntPtr NativeStructure = Marshal.AllocHGlobal(
        GetNativeDataSize(ManagedObject) + ManagedObject.WaveFormat.Size);

    Marshal.StructureToPtr(ManagedObject, NativeStructure, false);

    // Marshal extra data 
    if (ManagedObject.WaveFormat.Size > 0)
    {
        IntPtr dataPtr = new IntPtr(NativeStructure.ToInt32() 
            + Marshal.SizeOf(typeof(WaveFormatEx)));
        Marshal.Copy(ManagedObject.AdditionalData, 0, dataPtr,  Math.Min(
            ManagedObject.WaveFormat.Size,
            ManagedObject.AdditionalData.Length));
    }
    return NativeStructure;
}

And this is my calling code:

WaveFormatEx test = new WaveFormatEx(100);
int Size = System.Runtime.InteropServices.Marshal.SizeOf(test);

// After this call, test.FormatTag should be set to 1 (PCM audio), 
// but it is still 0, as well as all the other members
int Result = Avi.AVIStreamReadFormat(AudioStream, 0, test, ref Size);

Answer 1

There are several mistakes in the code and the declarations that prevents this code from working on a 64-bit operating system. Be sure to set the Platform Target to x86.

Are you sure the native function actually returns data? What is the Result return value? A non-zero value indicates failure.

The proper way to call this function is to call it twice. First with the lpFormat argument set to null (IntPtr.Zero) so it tells you how large a buffer it needs (returned by lpbcFormat). Then you create the buffer and call it again.

Instead of a custom marshaller, I would just create the buffer with Marshal.AllocHGobal after the first call and pass the IntPtr it returns as the lpFormat argument in the second call. Then, iff you get a success return code, use Marshal.PtrToStructure to write the WaveFormatEx. And Marshal.Copy to get the additional data.

Fwiw, using ref causes the P/Invoke marshaller to pass a WaveFormatEx** to the function but it expects a WaveFormatEx*. Which will cause it to overwrite data in the garbage collected heap, destroying its internal format. A kaboom is next when the CLR notices this.

Check out the NAudio project as a good alternative for doing this yourself.