I cannot provide an explanation for the error, only my initial suspicion that there is a permission difference between the context in which your code is run as a service and the context in which it is run when you place it in a console application. The E_HANDLE HRESULT is my clue here. Assuming that you run the console application as your logged-in user, you could try configuring the service to start as that user as well. If it works in that configuration, then you can attempt to narrow down what required resource is unavailable when it fails.
I can suggest a workaround. If there is an oddity of the DLL in question that prevents it from working in a hosted service, you can take the sacrificial process approach, so named because it is typically used to isolate a DLL that crashes often. Briefly, you create a proxy program whose only purpose is to load and call the DLL on behalf of your main process, using named pipes or some other IPC method to pass requests and results. If the DLL crashes, you start a new instance of the proxy program. In your case it would have the added benefit that only the wrapper program would need to be 32-bit.