IM Programs don't require any open ports because they are client programs. As a general rule (watch out for outbound firewalls) all ports are open for outgoing connections on a client. In order to actually accept connections, the port has to allow incoming traffic. Most servers, for example, can accept incoming traffic on all ports. Network communications and protocols are designed with this in mind: the client (your browser) always initiates the connection to the server (the website), not the other way around.
As for a protocol, you should use either TCP/IP or UDP, depending on what you need to do. UDP is great if you plan to have your client for the game account for missing information by using approximation (most FPS games do this, as well as just about any game that requires very quick reflexes) where TCP/IP is simpler, will greatly reduce errors in transmission, and be more prone to lag. The main difference between the two is that UDP is "fire and forget," meaning that it doesn't bother to check to see if the message actually arrived at its destination.
As for the protocol on top of that, you mentioned http and ftp. I would greatly advice using neither. When you're using direct socket communication on a custom port, you have all the freedom of crafting your own protocol for the game to use. So long as it gets its job done by sending packets back and forth, everything else about it is completely up to you, and should be. After all, a game protocol needs to be fast, you don't want all this extra header garbage from http clogging up your network transmissions unless you really need it.
Just remember: network connections require two parts, a client and a server. This means that you either have a central server set up for the game, or you have one player "host" the game (acting as a server, he will need his ports open). The other players (all players, in a central server setup) connect to the central guy, and pass communication through him. This means that your game protocol needs two parts, a server end and a client end. The client in most games sort of runs blind, in that it sends off player information about the user, and then relies on server updates to tell it where everything else in the game is. The server should be in charge of collecting player information from all clients, and then distributing that information to everyone else. This can include validation (most do) to prevent "illegal" things from happening within the game, but that's up to you.