Windows 2000 and Windows 2003 RC2 encryption/decryption using windows Crypto API?

Question

Our app encrypts a value using RC2 in C++ code and I wrote a decryption routine in .NET

The problem is that it works fine on our dev server, which is Windows 2003 but fails on the Windows 2000 one. It's running the same code and I checked everything else and it seems there are differences in the way the 2 encrypt.

Has anyone had any experience with that?

Answer 1

1. Make sure Windows 2000 has SP3, or encryption is far less likely to work, particularly if using Capicom (or the api it wraps).
2. Not all of the encryption algorithms and keylengths are supported on Windows 2000 if using Capicom (or the api it wraps).

Answer 2

There's an obscure issue with win2000/winxp encryption/decryption. The Cryptography Service Provider on win2000 has 40 for the effective key length and that's used by default even if you use a 128 bit key. So to fix that you have to manually reset the effective key length on 2000 or you can set it to 40 before you decrypt on the winxp (or win2003 in my case).

If you're using C# to decrypt you can't set the effective key size to something different than the key size if you use the RC2CryptoServiceProvider class.

Answer 3

I have face the same problem between XP SP1 and XP SP2 using crypto service provider, the encrypted value is different for 1 particular crypto which I can't recall at the moment.