P/Invoke errors only on Vista/Windows Server 2008

Question

I have code that uses methods from the SSPI dll (security.dll) via P/Invoke, which has worked perfectly on every platform tested (Windows XP, Windows Server 2003 both x86 and x64), however we are in the process of migrating to Windows Server 2008, and have found that the P/Invoke calls are crashing the process.

I have put together the following reproduce code:

using System;
using System.Runtime.InteropServices;

namespace TestPInvoke
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                // The following code works on all platforms tested (Windows Server 2003 x86/x64, Windows Server 2008 x64, Windows XP, Windows Vista)
                var table1 = (SecurityFunctionTable)Marshal.PtrToStructure(InitReturningPtr(), typeof(SecurityFunctionTable));
                Console.WriteLine(table1.dwVersion);
                Console.WriteLine(table1.EnumerateSecurityPackages.ToInt64().ToString("x16"));
                Console.ReadLine();

                // This call crashes only on Windows Server 2008 and Windows Vista (but works fine on XP and 2K3)
                var table2 = InitReturningClass();
                Console.WriteLine(table2.dwVersion);
                Console.WriteLine(table2.EnumerateSecurityPackages.ToInt64().ToString("x16"));
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Console.WriteLine(e.StackTrace);
            }
            Console.ReadLine();
        }

        [DllImport("security.dll", EntryPoint = "InitSecurityInterfaceW")]
        public static extern IntPtr InitReturningPtr();

        [DllImport("security.dll", EntryPoint = "InitSecurityInterfaceW")]
        public static extern SecurityFunctionTable InitReturningClass();

        [StructLayout(LayoutKind.Sequential)]
        public class SecurityFunctionTable
        {
            public uint dwVersion;
            public IntPtr EnumerateSecurityPackages;
            public IntPtr QueryCredentialsAttributes;
            // ...omitted for brevity
        }
    }
}

The problem is not isolated to this particular DLL or function either, but any P/Invoke call I've tried where the return value of the native function is a pointer to a structure which is implicitly marshalled into a class exhibits the issue.

Since I have a functional workaround (using Marshal.PtrToStructure) this isn't a major problem, but I am curious to know why it works with no (apparent) issues on XP and 2k3, but not Vista and 2k8, and whether there is any way to fix the class-returning method to avoid the rather uglier explicit marshalling.

Answer 1

Using Marshal.PtrToStructure is the correct thing to do. The fact that this ever worked on XP/Server 2003 is pure coincidence, as the code has always been buggy.

Answer 2

Edit: I didn't realize it is an API function. You cannot declare the function returning a structure. The P/Invoke marshaller will deallocate the memory for the structure with CoTaskMemFree(). Doesn't work, it wasn't allocated with CoTaskMemAlloc().

The heap manager in XP and 2003 is forgiving, it just ignores the faulty release request. But not the one in Vista and 2008, it bombs the program because it obviously is using memory incorrectly. Important because these kind of memory shenanigans are security problems.

Yes, by declaring the return type as IntPtr, you avoid the P/Invoke marshaller releasing the memory. Which is appropriate, the API function really does return a pointer, not a structure. Marshal.PtrToStructure is required to marshal the pointed-to structure to a managed struct.

The security APIs are in general troublesome like this. The notion of a security API returning a pointer into internal kernel security structure does boggle the mind a bit...