You seem to be asking for two things -
- a way to host a long-running, persistent application on Windows, which will fiddle with some device or other over the serial port.
- a way to communicate to that application from other applications.
A Windows Service is a nice solution to the first. The Windows Service acts as an application host for your custom logic. It gives you start/stop and lightweight monitoring and config capability through a common UI and API. It's easy to build Windows Services in C#. It is easy to start and stop Windows Services from the command line, from an app, from a GUI tool.
Think of WCF as primarily a communication interface. WCF can be used to construct the external interface exposed by the Windows Service. If you want to connect to the app (service) from .NET or HTTP clients (REST or SOAP), then WCF will work for you.
COM .... The confusion you are experiencing around where COM fits in, is due to the fact that COM couples an interface mechanism (IDispatch, IUnknown, etc) with a hosting + lifecycle model (Local servers, Inproc servers, Remote servers, auto start upon request, etc). The hosting and lifecycle stuff in COM worked, but it was always pretty slim. Hard to know which services were up, and for how long. Hard to start and stop on demand.
COM, in this situation, like WCF, is primarily of interest as a communication interface. If you want to use a component-based interface to connect to the service, COM may be a good complement to the Windows Service for hosting.
COM and WCF (REST or SOAP) are not mutually exclusive, and in fact many apps expose multiple interfaces. You might elect to do one, or the other, or both. There's no wrong answer, it depends on your requirements.
If I were doing this, I would use a Windows Service, and also I would expose a COM interface from that service, as described here. The lifecycle and hosting stuff is taken care of by the Windows Service. The service is then accessible via COM, which means via virtually any programming or script language / environment. Javascript, VB6, Excel or OFfice, Python, Perl, etc.
I'd also think about exposing a WCF interface from the service. This would allow any REST client to connect, as well. Think of WCF as the wireless phone, and COM as the landline. Sometimes you want only one. Sometimes you want both.