When writing an average new app in 2009, what's the most reasonable digest function to use, in terms of security and performance? (And how can I determine this in the future, as conditions change?)
When similar questions were asked previously, answers have included SHA1, SHA2, SHA-256, SHA-512, MD5, bCrypt, and Blowfish.
I realize that to a great extent, any one of these could work, if used intelligently, but I'd rather not roll a dice and pick one randomly. Thanks.
I'd follow NIST/FIPS guidelines:
March 15, 2006: The SHA-2 family of hash functions (i.e., SHA-224, SHA-256, SHA-384 and SHA-512) may be used by Federal agencies for all applications using secure hash algorithms. Federal agencies should stop using SHA-1 for digital signatures, digital time stamping and other applications that require collision resistance as soon as practical, and must use the SHA-2 family of hash functions for these applications after 2010. After 2010, Federal agencies may use SHA-1 only for the following applications: hash-based message authentication codes (HMACs); key derivation functions (KDFs); and random number generators (RNGs). Regardless of use, NIST encourages application and protocol designers to use the SHA-2 family of hash functions for all new applications and protocols.
You say "digest function"; presumably that means you want to use it to compute digests of "long" messages (not just hashing "short" "messages" like passwords). That means bCrypt and similar choices are out; they're designed to be slow to inhibit brute-force attacks on password databases. MD5 is completely broken, and SHA-0 and SHA-1 are too weakened to be good choices. Blowfish is a stream cipher (though you can run it in a mode that produces digests), so it's not such a good choice either.
That leaves several families of hash functions, including SHA-2, HAVAL, RIPEMD, WHIRLPOOL, and others. Of these, the SHA-2 family is the most thoroughly cryptanalyzed, and so it would be my recommendation for general use. I would recommend either SHA2-256 or SHA2-512 for typical applications, since those two sizes are the most common and likely to be supported in the future by SHA-3.
It really depends on what you need it for.
If you are in need of actual security, where the ability to find a collision easily would compromise your system, I would use something like SHA-256 or SHA-512 as they come heavily recommended by various agencies.
If you are in need of something that is fast, and can be used to uniquely identify something, but there are no actual security requirements (ie, an attacker wouldn't be able to do anything nasty if they found a collision) then I would use something like MD5.
MD4, MD5, and SHA-1 have been shown to be more easily breakable, in the sense of finding a collision via a birthday attack method, than expected. RIPEMD-160 is well regarded, but at only 160 bits a birthday attack needs only 2^80 operations, so it won't last forever. Whirlpool has excellent characteristics and appears the strongest of the lot, though it doesn't have the same backing as SHA-256 or SHA-512 does - in the sense that if there was a problem with SHA-256 or SHA-512 you'd be more likely to find out about it via proper channels.