You might find that PortMon by Mark Russinovich is helpful in your work, although it isn't provided with source code. It will log all system calls related to a port, and has extensive filtering capabilities to keep that log to a manageable size.
Edit: PortMon works by injecting a kernel mode device driver into the system when it runs. That driver hooks the ports that it wants to monitor by inserting a filter driver into the stack. That filter driver reports all IRPs that pass through it to the application. This is not an easy beast to implement by any stretch, and it really can't be done in user mode.
The Windows DDK does have sample code for a port filter driver, but it is a lot of work to go from the sample to something useful.
A pure user mode solution that might work out is to use two additional serial ports to eavesdrop on the wire. A pair of USB to serial adapters and a bit of wiring will do what you need. Then it is only a "small matter of programming" to monitor and correlate the send and receive lines. A single chip solution could be based on the FT2232H device from FTDI, which has an available evaluation module. Add a couple of RS232 level translators and some D connectors and you've got a serial eavesdropper on USB that looks like two COM ports to Windows.
Another outside the box approach is that many logic analyzers and mixed signal oscilloscopes can do serial protocol decoding, often as an optional component. One source of inexpensive USB+software solutions is Saelig. I've bought chips and modules from them myself, but don't have any direct experience with any of the USB-based logic analyzers they sell despite thinking that I should have one in my bag along with the Netbook...