RSA C# Encrypt Java Decrypt

Question

Hi guys,

In my program (server side - Java) I've created keystore file, with command:

keytool -genkey -alias myalias -keyalg RSA -validity 10000 -keystore my.keystore

and exported related X509 certificate with:

keytool -export -alias myalias -file cert.cer -keystore my.keystore

After I saved cert.cer on client side (C#) and I write this code:

X509Certificate2 x509 = new X509Certificate2();
byte[] rawData = ReadFile("mycert.cer");
x509.Import(rawData);

RSACryptoServiceProvider rsa = (RSACryptoServiceProvider)x509.PublicKey.Key;
byte[] plainbytes = System.Text.Encoding.ASCII.GetBytes("My Secret");
byte[] cipherbytes = rsa.Encrypt(plainbytes, true);
String cipherHex = convertToHex(cipherContent);
byte[] byteArray = encoding.GetBytes(cipherHex);

....

I write this Java code on server side:

keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(new FileInputStream("C:\\my.keystore"), "mypass".toCharArray());
Key key = keyStore.getKey("myalias", "mypass".toCharArray());
if (key instanceof PrivateKey) {
    Certificate cert = keyStore.getCertificate("myalias");
    PublicKey pubKey = cert.getPublicKey();
    privKey = (PrivateKey)key;
}
byte[] toDecodeBytes = new BigInteger(encodeMessageHex, 16).toByteArray();
Cipher decCipher = Cipher.getInstance("RSA");
decCipher.init(Cipher.DECRYPT_MODE, privKey);
byte[] decodeMessageBytes = decCipher.doFinal(toDecodeBytes);
String decodeMessageString = new String(decodeMessageBytes);

I receive this error:

javax.crypto.BadPaddingException: Data must start with zero

Can you help me, please? Thanks thanks,

Answer 1

Hm, I haven't done exactly what you are here, I had an DES one I had to do, and I was getting the same error. The trick was I had to get compatible CipherModes and PaddingModes on both sides. For mine it was PaddingMode.PKCS7 and CipherMode.CBC on the csharp side, and on the java side I used the DESede/CBC/PKCS5Padding xform.

hth Mike

Answer 2

Your method of hex-decoding in Java using BigInteger will produce the wrong result much of the time because the Java BigInteger class encodes a value whose high-order byte is >= 128 with an extra zero byte at the front. Use the Apache commons codec for hex en/de-coding.

EDIT: Your C# code is not correct. There is a .NET class System.Runtime.Remoting.Metadata.W3cXsd2001.SoapHexBinary that will do the hex en/de-coding for your C# code. The following C# code fragment should work better

public static String execute(String content)
{
    ASCIIEncoding encoding = new ASCIIEncoding();
    byte[] plainbytes = System.Text.Encoding.ASCII.GetBytes(content);
    byte[] cipherbytes = rsa.Encrypt(plainbytes, false);
    SoapHexBinary hexBinary = new SoapHexBinary(cipherbytes);
    String cipherHex = hexBinary.ToString();

    // This String is used on java side to decrypt
    Console.WriteLine("CIPHER HEX: " + cipherHex);
    return cipherHex;
}

EDIT 2:

Seems that SoapHexBinary class might have had a short life in .NET. There are a number of good solutions to the problem at this msdn link.

Answer 3

Thanks to GregS answers I found solution of my problem. Now I post this.

C# SIDE (Client-Side)

X509Certificate2 x509 = new X509Certificate2();
byte[] rawData = ReadFile("mycert.cer");
x509.Import(rawData);

After I loads my X509Certificate I call my execute method:

public static String execute(String content)
{
    ASCIIEncoding encoding = new ASCIIEncoding();
    byte[] plainbytes = System.Text.Encoding.ASCII.GetBytes(content);
    byte[] cipherbytes = rsa.Encrypt(plainbytes, false);
    SoapHexBinary hexBinary = new SoapHexBinary(cipherbytes);
    String cipherHex = hexBinary.ToString();

    // This String is used on java side to decrypt
    Console.WriteLine("CIPHER HEX: " + cipherHex);
    return cipherHex;
}

JAVA SIDE (Server-Side)

keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(new FileInputStream("C:\\my.keystore"), "mypass".toCharArray());
Key key = keyStore.getKey("myalias", "mypass".toCharArray());
if (key instanceof PrivateKey) {
    Certificate cert = keyStore.getCertificate("myalias");
    PublicKey pubKey = cert.getPublicKey();
    privKey = (PrivateKey)key;
}
byte[] toDecodeBytes = new BigInteger(encodeMessageHex, 16).toByteArray();
Cipher decCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
decCipher.init(Cipher.DECRYPT_MODE, privKey);
byte[] decodeMessageBytes = decCipher.doFinal(toDecodeBytes);
String decodeMessageString = new String(decodeMessageBytes);

The problem was in Hex-Encryption on C# Side that are completely different on Java side. Thanks GregS, you are the best ;)