Answering my own question.
When you return an IStream from your own COM server the .NET interop DLL puts an IStream into it's interface. For example lets say the type library is MyComServer, then the interop will contain a class MyComServer.Interop.IStream.
this IStream class has functions like RemoteRead, RemoteWrite etc. These take a ref byte as the first parameter. Using these worked fine on 32bit to 32bit, but that ref probably becomes a pointer and so on 64 to 32 something went wrong.
The solution is to convert the MyComServer.Interop.IStream to System.Runtime.InteropServices.ComTypes.IStream using "as" (or just cast probably). This then takes a more familiar byte[], int count and IntPtr for the return size.
The IntPtr is annoying as it could be an out int, but I used (typing this in by hand so not compiler checked..)
byte[] buffer = new byte[100];
IntPtr ptr = Marshall.AllocHGlobal( sizeof(int) );
stream.Read( buffer, 100, ptr );