Wincrypt: Unable to decrypt file which was encrypted in C#. NTE_BAD_DATA at CryptDecrypt

Question

I am trying to decrypt a piece of a file with wincrypt and I cannot seem to make this function decrypt correctly. The bytes are encrypted with the RC2 implementation in C# and I am supplying the same password and IV to both the encryption and decryption process (encrypted in C#, decrypted in c++).

All of my functions along the way are returning true until the final "CryptDecrypt" function. Instead of me typing out any more, here is the function:

static char* DecryptMyFile(char *input, char *password, int size)
{
    HCRYPTPROV provider = NULL;

    if(CryptAcquireContext(&provider, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, 0))
    {printf("Context acquired.");}
    else
    {
     if (GetLastError() == NTE_BAD_KEYSET)
     {
     if(CryptAcquireContext(&provider, 0, NULL, PROV_RSA_FULL, CRYPT_NEWKEYSET))
      {printf("new key made.");}
      else
      {
       printf("Could not acquire context.");
      }
     }
     else
     {printf("Could not acquire context.");}
    }

    HCRYPTKEY key = NULL;
    HCRYPTHASH hash = NULL;

    if(CryptCreateHash(provider, CALG_MD5, 0, 0, &hash))
    {printf("empty hash created.");}
    else
    {printf("could not create hash.");}

    if(CryptHashData(hash, (BYTE *)password, strlen(password), 0))
    {printf("data buffer is added to hash.");}
    else
    {printf("error. could not add data buffer to hash.");}

    if(CryptDeriveKey(provider, CALG_RC2, hash, 0, &key)) 
    {printf("key derived.");}
    else
    {printf("Could not derive key.");}

    DWORD dwKeyLength = 128;

if(CryptSetKeyParam(key, KP_EFFECTIVE_KEYLEN, reinterpret_cast<BYTE*>(&dwKeyLength), 0))
    {printf("success");}
    else
    {printf("failed.");}

    BYTE IV[8] = {0,0,0,0,0,0,0,0};

    if(CryptSetKeyParam(key, KP_IV, IV, 0))
    {printf("worked");}
    else
    {printf("faileD");}

    DWORD dwCount = size;
    BYTE *decrypted = new BYTE[dwCount + 1];

    memcpy(decrypted, input, dwCount);
    decrypted[dwCount] = 0;


    if(CryptDecrypt(key,0, true, 0, decrypted, &dwCount))
    {printf("succeeded");}
    else
    {printf("failed");}

return (char *)decrypted;
}

input is the data passed to the function, encrypted. password is the same password used to encrypt the data in C#. size is the size of the data while encrypted.
All of the above functions return true until CryptDecrypt, which I cannot seem to figure out why. At the same time, I'm not sure how the CryptDecrypt function would possibly edit my "decrypted" variable, since I am not passing a reference of it.

Any help or advice onto why this is not working would be greatly appreciated. This is my first endeavour with wincrypt and first time using C++ in years.

If it is of any more help, as well, this is my encryption (in C#):

 public static byte[] EncryptString(byte[] input, string password)
    {
        PasswordDeriveBytes pderiver = new PasswordDeriveBytes(password, null);
        byte[] ivZeros = new byte[8];
        byte[] pbeKey = pderiver.CryptDeriveKey("RC2", "MD5", 128, ivZeros);

        RC2CryptoServiceProvider RC2 = new RC2CryptoServiceProvider();

        //using an empty initialization vector for convenience.
        byte[] IV = new byte[8];
        ICryptoTransform encryptor = RC2.CreateEncryptor(pbeKey, IV);

        MemoryStream msEncrypt = new MemoryStream();
        CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write);
        csEncrypt.Write(input, 0, input.Length);
        csEncrypt.FlushFinalBlock();

        return msEncrypt.ToArray();
    }

I have confirmed that my hash value in C++ is identical to my key in C#, created by PasswordDeriveBytes.CryptDeriveKey

Answer 1

First, as in my comment, use GetLastError() so you know what it failed. I'll assume that you get NTE_BAD_DATA, all the other errors are much more easier to deal with since they basically mean you missed some step int he API call sequence.

The typical reason why CryptDecrypt would fail with NTE_BAD_DATA would be that you're decrypting the last block of a block cypher (as you are) and the decrypted padding bytes are incorrect. This can happen if the input is truncated (not all encrypted bytes were saved to the file) or if the key is incorrect.

I would suggest you take this methodically since there are so many places where this can fail that will all manifest only at CryptDecrypt time:

1. Ensure that the file you encrypt in C# can be decrypted in C#. This would eliminate any file save truncation issues.
2. Try to encrypt and decrypt with fixed hard codded key first (no password derived), this will ensure that your key set code IV initialization are correct (as well as padding mode and cypher chaining mode).
3. Ensure that the password derivation process arives at the same hash. Things like ANSI vs. Unicode or terminal 0 can wreak havok on the MD5 hash and result in wildly different keys from apparently the same password hash.