RSA Encryption in .NET - decryption in JAVA -> Java throws "modulus not positive" error.

Question

I am having problems with our third party vendor implimenting an sso. They are receiving the following error when verifying my signature:

java.lang.ArithmeticException: BigInteger: modulus not positive--at java.math.BigInteger.modPow(BigInteger.java:1556)

I have no control over their Java code. Here is what I am doing now:

I created a key pair in C# using this code:

        CspParameters csp = new CspParameters();
        csp.KeyNumber = (int)KeyNumber.Signature;
        using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(1024, csp))
        {

            File.AppendAllText(path + "PrivateKey.xml", rsa.ToXmlString(true));
            File.AppendAllText(path + "PublicKey.xml", rsa.ToXmlString(false));
        }

Here is the code for the signature:

public string MD5withRSASignature(string encryptedStringToSign)
    {

        byte[] signature;
        using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(1024))
        {
            XmlDocument xDoc = new XmlDocument();
            xDoc.Load(PRIVATE_KEY_PATH);
            rsa.FromXmlString(xDoc.OuterXml);
            byte[] bytes = Encoding.UTF8.GetBytes(encryptedStringToSign);
            signature = rsa.SignData(bytes, new MD5CryptoServiceProvider());
        }

        return Convert.ToBase64String(signature);
    }

(Yes I know the private key should be in a key store).

Here is the code they use to convert xml keys (this is Java)

private static RSAPublicKey ReadXMLKey(String fileName)
{

        DocumentBuilderFactory factory =     DocumentBuilderFactory.newInstance();
        DocumentBuilder builder = factory.newDocumentBuilder();
        Document document = builder.parse( new File(fileName) );
        byte[] modBytes = GetBytesFromElement(document, "Modulus");
        byte[] expBytes = GetBytesFromElement(document, "Exponent");
        RSAPublicKeySpec rsaKeyspec = new RSAPublicKeySpec(new BigInteger(modBytes), new BigInteger(expBytes));
        RSAPublicKey key = (RSAPublicKey)KeyFactory.getInstance("RSA").generatePublic(rsaKeyspec);

        return key;
         }

private static byte[] GetBytesFromElement(Document doc, String tag) throws IOException
{
    BASE64Decoder decoder = new BASE64Decoder();
    NodeList list = doc.getElementsByTagName(tag);
    byte[] results = null;
    if (list.getLength() == 1)
    {
        Element item = (Element)list.item(0);
        Text text = (Text)item.getFirstChild();
        results = decoder.decodeBuffer(text.getNodeValue().trim());
    }
    return results;
}

Answer 1

The exception has to do with the RSA public key that Java is using. None of your code addresses that point. How did the Java side obtain that key, what format is being used?

One common mistake that can account for the error is if the modulus is converted to an array of bytes but a leading zero byte is not present when it needs to be. Basically, this BigInteger constructor is a little trickier to use than it may first appear. It is designed for compatibility with DER-encoded ASN.1 integers. The upshot of it all is that if the first byte b of your modulus has the high bit set, i.e. 128 <= b < 256, you must prepend a leading zero byte or your modulus will be interpreted as a negative number. For simplicity you can always prepend a leading zero byte; no harm will come if it wasn't necessary.