I'm creating a client-server application which communicates via a custom socket protocol. I'd like the client to be usable from within networks that have a restrictive firewall (corporate, school, etc.). Usually this is done by connecting via HTTP, since that's always available.
If I want to do that, do I really have to use HTTP or is it enough to use my custom protocol via server port 80?
If it is a custom socket protocol then it is not HTTP.
But you can still use TCP on port 80 to escape the firewall but then you would have to handle the proxy situation as well. Proxies are HTTP aware and custom TCP might not work and they probably would not forward your requests.
I do not know about the reasons you want to do this (if it is legal or not) but there are software that are used to bypass the filtering in countries such as Iran. One of the softwares (Haystack) uses a sophisticated encryption to masquerade the request as an innocent looking packet.
The firewall may have more restricted checks than just restricting ports, and you might also have proxies along the way, and they will deal in HTTP.
Still, using a well-known port for something other than its normal use is still far better than so many schemes which do inherently non-HTTP stuff over HTTP, and essentially implement RFC 3093 (when people implement April Fools RFCs it normally shows a combination of humour and technical acumen, RFC 3093 is the exception).
To get around the proxy issue, you could use 443 rather than 80, as HTTPS traffic can't be proxied in quite the same way. Indeed, you often don't even need to use SSL, as the proxy will just assume that they can't see it.
None of this needs to be done with your application though. What your application needs to do is to have its port be configurable (that should be done with any server application anyway). The default should be something away from well-known ports, but the sysadmin will be able to use 80 or 443 or whatever if they need to.
You would be better off investigating tunneling with SSH. It is designed for precisely this. An HTTP proxy isn't likely to work for a number of reasons including those given in the other answers.