Which port? Parallel port is my favorite choice since it outputs +5V (TTL logic level) and is very straightforward to program. Most parallel ports have enough power to drive an LED. It's important to remember that computer ports in general are designed to only output signaling voltages, and not to produce enough current to actually power most devices.
Which compiler? Doesn't matter. This kind of hardware hacking is more fun and easy under Linux, though, so GCC is a good choice.
How do I send data? Depends on the port and the operating system. USB is frightfully complicated for a simple project, so forget it. Serial and parallel ports can be controlled via a variety of different interfaces. My preference is to use the ioctl()
system call under Linux to directly control the parallel-port pins. Here's info on how to do that: http://www.linuxfocus.org/common/src/article205/ppdev.html
Do I need a microprocessor? No, you don't need a microprocessor in the external device (obviously your computer has a microprocessor :-P). If you use the parallel or serial ports, you can just use the LED and a resistor or two and the necessary parts to connect the LED directly.
(Also: The Linux Device Drivers book, available for free online, has information on interfacing simple electronic devices to parallel ports and writing kernel drivers for them.)
EDIT: There seems to be massive confusion in this thread about what the OP means by, "Do I need a microprocessor?" Emphatically, the parallel port alone can drive an LED based on the software in the computer. No microprocessor is needed in the device. However, if you want the device to be able to control itself without being connected to the computer, a microprocessor or some other digital logic is required.