P/Invoke AccessViolationException that makes no sense

Question

First of all, sorry to post a question like this when so many other have been asked on this topic, but I've been reading all the questions I could find (+google), and none has really given me any hints as to what is happening in my case.

I have a 3rd party .dll (libFLAC) with two exported functions with similar name:

FLAC__StreamEncoderInitStatus FLAC__stream_encoder_init_file (FLAC__StreamEncoder *encoder, const char *filename, FLAC__StreamEncoderProgressCallback progress_callback, void *client_data)

FLAC__StreamEncoderInitStatus FLAC__stream_encoder_init_FILE (FLAC__StreamEncoder *encoder, FILE *file, FLAC__StreamEncoderProgressCallback progress_callback, void *client_data)

The purpose of the first one is to initialize an encoder with a filename. It calls fopen() which on windows (with unicode filenames) is not a good thing in some cases. That's why the developer provides the second function, which can be passed an open FILE* in "w+b" mode (which i'm opening with a call to MSVCRT.DLL _wfopen_s)

I have defined my P/Invoke declarations as follows:

[DllImport(Globals.LIBFLAC_DLL, EntryPoint = "FLAC__stream_encoder_init_file", CharSet = CharSet.Ansi)]
public static extern InitStatus InitFilenameAnsi(IntPtr Encoder, string Filename, ProgressDelegate ProgressCallback, ref Object ClientData);

[DllImport(Globals.LIBFLAC_DLL, EntryPoint = "FLAC__stream_encoder_init_FILE")]
public static extern InitStatus InitFileHandle(IntPtr Encoder, IntPtr FileHandle, ProgressDelegate ProgressCallback, ref Object ClientData);

The (strange?) problem is that the call to the "file" function (lowercase) works great, while the one that takes a FILE* (IntPtr) doesn't work (It throws the AccessViolationException).

I am passing exactly the same 1st, 3rd and 4th parameters to both, so I'm pretty sure the problem is the IntPtr FileHandle parameter (3rd and 4th are null and can be null as per the docs. Providing real values doesn't help either). I am also sure the actual file handle is ok: I am using the exact same code to _wfopen_s() in another project and it's working perfectly. The file is also created before the crash, so it's not a problem with that either.

EDIT: The return value is just a public enum InitStatus: int.

For a moment, i thought that there could be a problem with the functions having almost the same name, so I tried calling them by ordinal, but it didn't work either:

[DllImport(Globals.LIBFLAC_DLL, EntryPoint = "#259", CharSet = CharSet.Ansi)] //"FLAC__stream_encoder_init_file" OK

[DllImport(Globals.LIBFLAC_DLL, EntryPoint = "#258")] //"FLAC__stream_encoder_init_FILE" FAILS

I also thought that maybe the DLL I was using could have some sort of problem, so I switched it for a version compiled by another 3rd party (150K bigger in size), but I get exactly the same problem.

The code I'm using to call:

[TestMethod]
public void ThisFailsMiserably()
{
    Object ClientData = null;
    IntPtr FileHandle = IntPtr.Zero;

    try
    {
        FILE.Open(out FileHandle, "test.flac", "w+b");
        Assert.AreNotEqual(IntPtr.Zero, FileHandle); //Works great! the file is created, and the debugger shows the file handle value.
        StreamEncoder.InitFileHandle(FlacStreamEncoder, FileHandle, null, ref ClientData); //AccessViolationException
        Assert.IsTrue(StreamEncoder.Finish(FlacStreamEncoder));
    }
    finally
    {
        FILE.Close(FileHandle);
    }
}

[TestMethod]
public void ThisWorksGreat()
{
    Object ClientData = null;
    Assert.AreEqual(StreamEncoder.InitStatus.OK, StreamEncoder.InitFilenameAnsi(FlacStreamEncoder, "test.flac", null, ref ClientData));
    Assert.IsTrue(StreamEncoder.Finish(FlacStreamEncoder));
}

PS: I don't know if it matters, but I'm using VS2010 under Win7 x64.

Thank you for your time in advance

Answer 1

    FILE.Open(out FileHandle, "test.flac", "w+b");

What does this actually do? The flac code seems to require a FILE*. That's a pointer, not a handle. The CRT is completely agnostic of Windows handles. If FileHandle is actually a Windows handle then you're guaranteed to get the big Kaboom, handles are obfuscated pointer values.

You cannot get a FILE* from the CRT without pinvoking a function that you write that returns the required pointer. That function should call fopen (or _wfopen_s) and ought to use the exact same version of the CRT as the flac code. In the end you're not ahead of course, might as well call the function that works.

Answer 2

"I don't know if it matters, but I'm using VS2010 under Win7 x64." - it matters a lot! Are you using a 32-bit libFLAC DLL? If so, you need to set your "Platform target" under the build tab of the project properties to "x86" in order to force the managed assembly to run in 32-bit mode, so it matches the 32-bit DLL you're p-invoking into.

Or you could obtain a 64-bit libFLAC DLL, and then you'll have to include both the 32-bit and 64-bit versions, and call the appropriate one depending on the value of IntPtr.Size (to determine if you're running in 32- or 64-bit mode).