I've inherited a mobile app which sends auth credentials (userid/password) in the clear.
I'd imagine that I have 2 choices: a) use TLS. b) write my own auth protocol.
If I choose (b) what are the key guidelines that I must follow to make it it secure. e.g. how to avoid replay attacks, encryption strategies. Thanks
If you use b), the key guidelines are: Don't. If you want it secure, that is.
Try to stick with a).
For (b) I guess you do a challenge-response thingy.
Server generates a random string, sends it to the client. Client appends it to the password and hashes the whole thing, sends the hash back to the server. Server does the same calculation, compares the result with what it got from the client. If they match then the client sent the correct password.
The most obvious vulnerability is that if someone snoops both sides of the exchange, they can then run an offline dictionary attack against the password.
For both "you can't get sued for it" and "reasonably protected" definitions of 'safe', for a mobile application, you can assume that the line is secure vs man-in-the-middle attacks and wide open to eavesdropping. SSL/TLS sounds the easiest way to go, but this might depend on your carrier and target phones.
If you can't make TLS work and you need to roll your own, use Diffie-Hellman key exchange and established crypto library (Legion of the Bouncy Castle have a jightweight implementation that is J2ME compliant.)
Writing your own security protocol is not necessary, and a bad idea. It will almost definitely have exploitable flaws. If all you need is to protect the confidentiality of the login credentials, then SSL/TLS is what you should use. It also allows you to more easily upgrade to client certificate-based authentication in the future.