The first question is: why can you not get an SSL certificate? I can think of two reasons:
- SSL certificates are too expensive
- You don't want to have a certificate issued by a third party
If your problem is #1, StartSSL provides free certificates with a 1-year validity or charges $50 for unlimited certificates valid for 2 years (including wildcards). They are recognized by both Mozilla and the Microsoft trust store.
If the issue is #2, why not issue a self-signed certificate and hard-code it into your application? That does not compromise the security of the system at all (only your particular cert will be accepted by the app), but eliminates the need to "get" an SSL certificate from somewhere else.
If you really really can't use SSL, look at challenge-response systems such as Kerberos or anonymous key-material generators like Diffie-Helman (with an asymmetric key for server identity validation). Many methods exist for secure two-party authentication over an insecure line. The key is that the ID verification step must be challenge-response instead of a "send me your secret" scheme.