Repeatedly opening and closing the port is not recommended. Check the Remarks section in the MSDN Library article for SerialPort.Close(). There's a background thread that needs to shut down before a port can be opened again, that takes time. The amount of time is not predictable.
The Close() method can easily deadlock if the DataReceived event handler is currently running. The most typical way to get the deadlock is calling Control.Invoke() in the event handler. Make sure you don't use any code in the event handler that blocks or requires a thread context switch. Using BeginInvoke() is fine.
Not being able to kill the program is caused by a problem in the serial port device driver. Start Taskmgr.exe, Process tab, View + Select Columns and tick "Handles". If after killing the program, you see the Handles column showing 1 then the serial port driver is hanging on to an I/O request that it doesn't complete. The process cannot terminate until all its kernel mode threads are exited.
There's little you can do about that particular problem, other than hoping for a driver update or switching to another vendor. Especially USB serial port emulators are notorious for having lousy device drivers. You get rid of a troublemaker like that by taking it out the parking lot and running it over with your car several times.
Another typical problem with USB emulators is that they are so easy to disconnect while they are in use. That works about as well as jerking a flash drive out of the socket while Windows is writing to it. It would also be a good way to get the device driver hung .NET versions prior to version 4.0 suffer from a heart attack in a background thread when the device suddenly disappears. Short from upgrading, a small sign next to the connector that says "Do not disconnect while in use!" is a practical workaround. They will anyway but get bored with it after a couple of times.