Secure password transmission over unencrypted tcp/ip

Question

I'm in the designing stages of a custom tcp/ip protocol for mobile client-server communication. When not required (data is not sensitive), I'd like to avoid using SSL for overhead reasons (both in handshake latency and conserving cycles).

My question is, what is the best practices way of transmitting authentication information over an unencrypted connection?

Currently, I'm liking SRP or J-PAKE (they generate secure session tokens, are hash/salt friendly, and allow kicking into TLS when necessary), which I believe are both implemented in OpenSSL. However, I am a bit wary since I don't see many people using these algorithms for this purpose. Would also appreciate pointers to any materials discussing this topic in general, since I had trouble finding any.

Edit

Perhaps the question should have been: is there a best practices approach for secure passwords over unencrypted tcp/ip? If not, what are the reasons for selecting a particular method over others? (The Rooks answer is closest in spirit to this question so far, even if it does violate the letter).

Edit, part deux

I'm primarily interested in the case of client-server authentication, where there is an expectation that both parties have a shared secret (password) a priori.

Answer 1

You should have a look at "Diffie-Hellman key exchange":

Diffie–Hellman key exchange (D–H) is a cryptographic protocol that allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure communications channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher.

Once you have exchanged a key, you can encrypt your password with this key and transmit it over the insecure protocol.

Answer 2

You could use a challenge-response algorithm. The algorithm goes like this:

• The server sends a random string to the client.
• The client combines this string with the password (by combining, you can xor them or just append them).
• The client calculates a hash (for example, SHA1) of the result, and sends it to the server.
• The server calculates the same hash using this random number and the real password.
• The server compares the two hashes.

Since you shouldn't store a password in plain text, but as a hash instead, the client should calculate this hash at the very beginning.

There are possibly several libraries implementing this, so you probably don't need to code it yourself.

Answer 3

I still think that SSL is by far your best choice, after all why reinvent the wheal when so much can go wrong? You don't have to buy an expensive certificate if your have a list of "good" and "bad" (compromised) certificates. openSSL is completely free, and i don't see a good reason not to use it.

Some things you might not know: ssl handshakes can be resumed.

Also you can use SSL/TLS over UDP to reduce overhead its called DTLS.