An InvalidCastException is raised when the IUnknown::QueryInterface() method of the COM interface returns the E_NOINTERFACE error return code. Two basic reasons for that. The usual one is that the COM object simply doesn't implement the interface you are casting to. That's however very repeatable, it cannot be affected by a debugging session.
The second reason is trickier and is related to threading. When you use the RCW in another thread then the CLR will automatically marshal the interface pointer. This requires the COM server to either explicitly implement the IMarshal interface, have a proxy/stub DLL registered (built from the IDL) or to support the standard marshaller (uses the type library).
This is not universally done, there are a lot of COM servers that simply assume they will only be used from a single thread. The debugger comes in to play because of the way it evaluates expressions in the Watch window. It actually runs code to obtain the value of the expression. The exact rules are not well documented that I know of, but sometimes that code will run on a helper thread. Typical for example if you used Debug + Break All to break into the program. That helper thread is going to bomb if the COM server doesn't support marshaling. One thing to check is that the thread that owns the COM object is the current thread. Debug + Windows + Threads and double-click the owning thread (usually the main thread of your app).
Not a great explanation, it is not a slam-dunk for your observation, but I suspect this is close to the problem. Not much you can do about it, avoid using the Watch window to display COM server properties. Or write a bit of debug code to copy the server property into a local variable, put a Watch on that variable instead.