Does RADIUS’s use of the MD5 algorithm make it not FIPS compliant for an application that allows RADIUS authentication?

Question

The issue is if you enforce FIPS compliance in the Windows security policy settings, an exception will be thrown because RADIUS protocol uses the MD5 algorithm to hash the request authenticator. There is not an alternative MD5 algorithm that is FIPS compliant so it does not appear any code implementation of RADIUS would be possible on a machine enforcing FIPS compliance.

Does this mean RADIUS is mutually exclusive with FIPS compliance?

The code implements the RADIUS protocol as specified by the official RFC (http://tools.ietf.org/html/rfc2865).

Answer 1

When you enable FIPS compliance in Windows, you're asserting that you are now going to use only the FIPS-certified encryption and hash algorithms. More specifically, it's the cryptographic module in Windows that has been certified only to allow users to use approved FIPS algorithms. The list of acceptable algorithms is defined in Annex A: Approved Security Functions for FIPS PUB 140-2, Security Requirements for Cryptographic Modules.

MD5 is not an approved hash algorithm, so no, applications cannot use it. For hashing, you're limited to the SHA family of algorithms. So MD5-based Radius is out because it cannot use MD5 from a FIPS-approved security module.

If you peruse the FIPS-certified modules, you may notice that some declare MD5 as a non-approved algorithm. What this means is that the certified module internally uses MD5, but does not expose the functionality to applications, or use it for communication. For example, a hardware encryption module running embedded linux may use MD5 to hash passwords in /etc/passwd. That's OK because users of the module cannot use MD5.