Padding error - when using AES Encryption in Java and Decryption in C

Question

I have a problem while decrypting the xl file in rijndael 'c' code (The file got encrypted in Java through JCE) and this problem is happening only for the excel files types which having formula's. Remaining all file type encryption/decryption is happening properly.

(If i decrypt the same file in java the output is coming fine.)

While i dumped a file i can see the difference between java decryption and 'C' file decryption.

od -c -b filename(file decrypted in C)

0034620  005 006  \0  \0  \0  \0 022  \0 022  \0 320 004  \0  \0 276   4

005 006 000 000 000 000 022 000 022 000 320 004 000 000 276 064

0034640   \0  \0  \0  \0  \f  \f  \f  \f  \f  \f  \f  \f  \f  \f  \f  \f

000 000 000 000 014 014 014 014 014 014 014 014 014 014 014 014 0034660 

od -c -b filename(file decrypted in Java)

0034620  005 006  \0  \0  \0  \0 022  \0 022  \0 320 004  \0  \0 276   4

005 006 000 000 000 000 022 000 022 000 320 004 000 000 276 064

0034640   \0  \0  \0  \0 000 000 000 000 0034644

(the above is the difference between the dumped files)

The following java code i used to encrypt the file.

public class AES {

    /**
     * Turns array of bytes into string
     * 
     * @param buf
     *            Array of bytes to convert to hex string
     * @return Generated hex string
     */

    public static void main(String[] args) throws Exception {

        File file = new File("testxls.xls");

        byte[] lContents = new byte[(int) file.length()];
        try {
            FileInputStream fileInputStream = new FileInputStream(file);
            fileInputStream.read(lContents);
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e1) {
            e1.printStackTrace();
        }
        try {
            KeyGenerator kgen = KeyGenerator.getInstance("AES");
            kgen.init(256); // 192 and 256 bits may not be available
            // Generate the secret key specs.
            SecretKey skey = kgen.generateKey();
            // byte[] raw = skey.getEncoded();
            byte[] raw = "aabbccddeeffgghhaabbccddeeffgghh".getBytes();
            SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES");
            Cipher cipher = Cipher.getInstance("AES");
            cipher.init(Cipher.ENCRYPT_MODE, skeySpec);
            byte[] encrypted = cipher.doFinal(lContents);
            cipher.init(Cipher.DECRYPT_MODE, skeySpec);
            byte[] original = cipher.doFinal(lContents);
            FileOutputStream f1 = new FileOutputStream("testxls_java.xls");
            f1.write(original);

        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }
}

I used the following file for decryption in 'C'.

#include <stdio.h>
#include "rijndael.h"


#define KEYBITS 256

#include <stdio.h>
#include "rijndael.h"

#define KEYBITS 256 

    int main(int argc, char **argv)  
    {  
     unsigned long rk[RKLENGTH(KEYBITS)];  
     unsigned char key[KEYLENGTH(KEYBITS)];  
     int i;  
     int nrounds;  
     char dummy[100] = "aabbccddeeffgghhaabbccddeeffgghh";  
     char *password;  
     FILE *input,*output;  
     password = dummy;  
     for (i = 0; i < sizeof(key); i++)  
          key[i] = *password != 0 ? *password++ : 0;  
     input = fopen("doc_for_logu.xlsb", "rb");  
     if (input == NULL)  
     {  
         fputs("File read error", stderr);  
          return 1;  
     }  
    output = fopen("ori_c_res.xlsb","w");  
    nrounds = rijndaelSetupDecrypt(rk, key, 256);  
     while (1)  
     {
      unsigned char plaintext[16];  
      unsigned char ciphertext[16];  
      int j;  
      if (fread(ciphertext, sizeof(ciphertext), 1, input) != 1)  
            break;  
      rijndaelDecrypt(rk, nrounds, ciphertext, plaintext);  
      fwrite(plaintext, sizeof(plaintext), 1, output);    
     }  
     fclose(input);  
     fclose(output);  
    }

Answer 1

In order to remove the PKCS padding that Java is adding, the C side needs to read the value of the last byte in the final decypted block, then trim that many bytes from the end of the decrypted stream.

This means that you can't do your fwrite until after you try to read the next block (because you need to know whether the current block is the last one or not):

unsigned char plaintext[16];  
unsigned char ciphertext[16];
int last_block;

last_block = (fread(ciphertext, sizeof(ciphertext), 1, input) != 1);  
while (!last_block)
{
    size_t plaintext_size = sizeof plaintext;

    rijndaelDecrypt(rk, nrounds, ciphertext, plaintext);
    last_block = (fread(ciphertext, sizeof(ciphertext), 1, input) != 1);

    if (last_block)
    {
        /* Remove padding */
        plaintext_size -= plaintext[(sizeof plaintext) - 1];
    }

    if (plaintext_size)
    {
        fwrite(plaintext, plaintext_size, 1, output);
    }
}

Answer 2

Hi,

I modified the file read operation in 'c' file. Now it is coming fine.

Thanks for your support. :-)