According to this question it's possible to seamlessly combine managed and unmanaged code using C++/CLI. I don't quite get it - shouldn't there be marshalling between managed and unmanaged anyway?
For example, I have InnerLibrary that is compiled as a native C++ .dll with a header published and C++/CLI OuterLibrary that calls code from InnerLibrary. Will there be marshalling? Who will implement it and how costly will it be?
There doesn't have to be any marshalling because C++/CLI is able to emit unsafe code that makes the call directly. Take a look at some C++/CLI code in Reflector -- it'll look very different from C#.
This is something that C# can't do (at least, not without the unsafe keyword and some pointer hacks), and it's also something that pure-mode C++/CLI can't do (for the same reason as C#).
.NET unsafe code is able to make direct calls to unmanaged functions; it's just that this ability isn't conveniently available except through C++/CLI.
It depends on the data types involved.
Intrinsic types such as int, double, and so on (string does not qualify) have the same representation in both native and managed code, no marshaling is required. Arrays of intrinsic types are laid out the same way as well (if we ignore the metadata .NET stores, but that's separate from the array content).
Value types using explicit layout attributes where all members are intrinsic types, are also memory layout-compatible.
Pinning may be required if the data is stored within an object on the managed heap (this is true for all arrays).
Class types, on the other hand, have to be converted/translated back and forth.
There is marshalling invovled, but you (i.e. the programmer) must do it explicitly.
If your C++CLI OuterLibrary calls has a function that takes a System.String/System::String^, the C++ type system requires that you perform a type conversion before passing it to an InnerLibrary function that takes a const char*. You have to do the conversion yourself - which is the marshalling.
Microsoft ship something called the C++ Support Library, which provides functions that help with C++ <-> C++CLI interaction.
There are two points here:
1) Managed/unmanaged code transition: every transition has its fixed cost. When C++/CLI code is compiled in a single assembly, compiler tries to make all the code managed, when it is possible, to minimize such transitions. When external unmanaged Dll is called from C++/CLI code, such optimization is impossible. Therefore, it is a good idea to minimize such transitions, at least in time-critical sections. See more about this here: http://msdn.microsoft.com/en-us/magazine/dd315414.aspx, Performance and the Location of Your Interop Boundary
2) Parameters marshalling. This depends on parameters type. Some parameters don't need to be marshalled, for example, simple types like int. Strings should be marshalled. Some tricks, like pinning pointers, allows to prevent simple type array marshalling.
Well, it's a feature that's built into the C++/CLI compiler, called C++ Interop. There's a lot less black magic involved that you might think. The JIT compiler generates the exact same kind of machine code as your C++ compiler generates. All of the .NET value types have a direct equivalent in C++ so no conversion is needed. It does not automatically handle reference types, you have to do that yourself. pin_ptr<>, typically.
All that it really does is inject a bit of code that handles the transition from a managed stack frame to an unmanaged stack frame. That code puts a special "cookie" on the stack, recognized by the garbage collector. Which prevents it from blundering into the unmanaged frames and mis-identify unmanaged pointers as object references. There's not much to that code, takes about 5 nanoseconds in the Release build, give or take.