Marshaling from C# to C++

Question

Hi everyone. I gotta pass an InParameter from my C# application to an exported function from a VC++ DLL. The function accepts 2 parameters :

int func_name (FILE* fp, BYTE& by);

fp is In and by is Out parameter.

I was thinking of marshaling using IntPtr for the FILE* and using byte for BYTE. Is it correct? If I write the following in C#

[DllImport("name_of_project.dll", CharSet = CharSet.Ansi)] public static extern int func_name(IntPtr FilePointer, [MarshalAs(UnmanagedType.BYTE&)] byte by);

will it work? I think it will give an error for the '&' sign in the marshaling statement. How do I pass the out parameter by reference?

Your help would be much appreciated.

Thanks,Viren

Answer 1

No, this won't work as written.

1. Where are you going to get a FILE* in a C# program? Unless some other function in your unmanaged dll gives you this pointer, you're in trouble. BTW I strongly recommend wrapping this FILE* in a SafeHandle; then your p/invoke signature will use a SafeFilePointer instead of a bare IntPtr.
2. How many bytes are you marshaling? If just one, put out byte by in the managed signature (no attributes required).
3. You don't need to specify a charset, since you are not marshaling any chars or strings.

Answer 2

If native function expects reference, you can marshal it using it ref/out. So in your situation, you could use: out byte by. I've checked it, and it works for me.

Edit again: It just came to my mind, that advices I gave you won't work, as FILE is a struct, that you won't be able to marshall from c# that easily. So scratch those, you can use this approach if you will threat SafeFileHandle as HANDLE object inside your unmanaged library. If you can't modify this dll, one solution is to create own wrapper for creating files using stdio, it could look like this:

1. Create library that exposes functionality of fopen, fclose etc. Fe. (in c++):

   FILE* CreateFile(char* name);

2. Marhall this function, in this example you would use (c#):

   public static extern IntPtr CreateMyFile([MarshalAs( UnmanagedType.LPStr)] string name);

3. Marshall all FILE* parameters to IntPtr, and then just pass result of function CreateMyFile as FILE*.