In a manner of speaking, yes.
Continue to compile the C++/CLI code as 32-bit so it can use the native library using C++ interop.
Then you will have to configure it to load as an out-of-process COM server when acting as an Office 64 plugin. With native COM code, midl automatically generates the 64-bit proxy. There should be some similar capability to create a proxy when registering .NET classes marked COMVisible
.
The 64-bit COM interface will be contained in the auto-generated 64-bit proxy DLL, so this doesn't violate the rule that the bitness of all modules in a process must be the same.