C# encryption, C++ decryption. Final few bytes failing on decryption.

Question

My apologies for the length of the code I'm about to list.

I need to encrypt the contents of an xml file on the C# end of my code, and decrypt it in C++. I'm using RC2, with RC2CryptoServiceProvider and CryptoStream on the C# side, with Wincrypt on the C++ side. Encryption seems to be working fine, it looks like such:

public static byte[] EncryptString(byte[] input, string password)
{
    PasswordDeriveBytes pderiver = new PasswordDeriveBytes(password, null);
    byte[] ivZeros = new byte[8];
    byte[] pbeKey = pderiver.CryptDeriveKey("RC2", "MD5", 128, ivZeros);

    RC2CryptoServiceProvider RC2 = new RC2CryptoServiceProvider();

    byte[] IV = new byte[8];
    ICryptoTransform encryptor = RC2.CreateEncryptor(pbeKey, IV);

    MemoryStream msEncrypt = new MemoryStream();
    CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write);
    csEncrypt.Write(input, 0, input.Length);
    csEncrypt.FlushFinalBlock();

    return msEncrypt.ToArray();
}

MY decryption code almost works perfectly. It is missing the final two characters of the file, and instead spitting out garbage characters. I have tried null-terminating the decrypted string, but no dice. It is as follows:

char* FileReader::DecryptMyFile(char *input, char *password, int size, int originalSize) 
{
UNREFERENCED_PARAMETER(password);
HCRYPTPROV provider = NULL;
if(CryptAcquireContext(&provider, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, 0))
{
 printf("Context acquired.");
}
else
{
 if (GetLastError() == NTE_BAD_KEYSET)
 {
  if(CryptAcquireContext(&provider, 0, NULL, PROV_RSA_FULL, CRYPT_NEWKEYSET))
  {
   printf("new key made.");
  }
  else
  {
   printf("Could not acquire context.");
  }
 }
 else
 {
  DWORD check = GetLastError();
  UNREFERENCED_PARAMETER(check);
  printf("Could not acquire context.");
 }
}

HCRYPTKEY key = NULL;
HCRYPTHASH hash = NULL;
if(CryptCreateHash(provider, CALG_MD5, 0, 0, &hash))
{
 printf("empty hash created.");
}

if(CryptHashData(hash, (BYTE *)password, strlen(password), 0))
{
 printf("data buffer is added to hash.");
}

HCRYPTHASH duphash = NULL;
CryptDuplicateHash(hash, 0, 0, &duphash);
BYTE *mydata = new BYTE[512];
DWORD mydatasize = 512;
CryptGetHashParam(hash, HP_HASHVAL, mydata, &mydatasize, 0);

BYTE *mydata2 = new BYTE[512]; //these duplicates were made to test my hash.
DWORD mydatasize2 = 512;  
CryptGetHashParam(duphash, HP_HASHVAL, mydata2, &mydatasize2, 0); 

if(CryptDeriveKey(provider, CALG_RC2, hash, 0, &key)) 
{
 printf("key derived."); 
}

DWORD dwKeyLength = 128;
if(CryptSetKeyParam(key, KP_EFFECTIVE_KEYLEN, reinterpret_cast<BYTE*>(&dwKeyLength), 0))
{
 printf("CryptSetKeyParam success");
}

BYTE IV[8] = {0,0,0,0,0,0,0,0};
if(CryptSetKeyParam(key, KP_IV, IV, 0))
{
 printf("CryptSetKeyParam worked");
}

DWORD dwCount = size;
BYTE *somebytes = new BYTE[dwCount + 1];

memcpy(somebytes, input, dwCount);

if(CryptDecrypt(key,0, true, 0, somebytes, &dwCount))
{
 printf("CryptDecrypt succeeded.");
}
else
{
 if(GetLastError() == NTE_BAD_DATA)
 {
  printf("NTE_BAD_DATA");
 }
 printf("CryptDecrypt failed.");
 DWORD testest = NULL;
 testest = GetLastError();
 testest = 3;
}
somebytes[originalSize] = '\0';

    return (char *)somebytes;
}

The resulting xml file should end with </LudoData>. Currently, it ends with </LudoDat[funny looking s]b

Why might this be? How can I stop this? I'm terribly confused as to why this is occurring. Since I am null-terminating the decryption and still getting a problem only on the final characters, I don't believe that the decryption is the problem (although I would love to be wrong). Is it possible that my encryption is having troubles when finishing the encryption of the file?

Upon returning from CryptDecrypt, dwCount is equal to size, which is 11296. Meanwhile, originalSize is equal to 11290.

Answer 1

Shouldn't you be doing just the following

DWORD dwCount = size;
BYTE *somebytes = new BYTE[dwCount];

memcpy(somebytes, input, dwCount);

before calling CryptDecrypt? Why do you need to add a null character when CryptDecrypt already takes the number of bytes as it's final argument? I suspect that's what's breaking things as you've changed the input by clobbering the character at position originalSize - 1.

Edit:

Which is bigger, originalSize or size? Since CryptDecrypt re-uses the array it must be dimensioned with the bigger of the two. BTW, why do you think you need the extra one in the dimension?

Answer 2

This line is your problem (before the call to CryptDecrypt):

somebytes[originalSize - 1] = '\0';

It is overwriting part of the encrypted padding in the last block with a zero, which is causing the last block to decrypt to rubbish. Just remove the line - the ciphertext isn't nul-terminated anyway (it almost certainly contains plenty of embedded nuls), the length parameter is used by the decryption routine to know how much data there is.

Oh, and... RC2? Seriously?