Encrypting socket communication with RSA in C#

Question

I'm currently trying to implement a socket server that enables the clients to send some commands to start and stop various services that the server provides. I got the communication between client and server running, and got the server to respond to the commands the clients send. The next step would be to encrypt the communication between client and server. I'm trying to acomplish this with the RSACryptoServiceProvider.

Heres the code i use to enrypt the data:

CspParameters parameter = new CspParameters();
parameter.KeyContainerName = "keycontainer";
rsaProvider = new RSACryptoServiceProvider(parameter);
StreamReader r = new StreamReader(@"C:\tmp\rsakey.xml");
rsaProvider.FromXmlString(r.ReadToEnd());
string command = "server reply goes here";
ASCIIEncoding bc = new ASCIIEncoding();
byte[] cmd = rsaProvider.Encrypt(bc.GetBytes(command), false);
handler.Send(cmd);

And heres the code i use to decrypt my commands

rsaProvider = new RSACryptoServiceProvider(parameter);
StreamReader r = new StreamReader(@"C:\tmp\rsakey.xml");
string xml = r.ReadToEnd();
rsaProvider.FromXmlString(xml);
ASCIIEncoding bc = new ASCIIEncoding();
string test = bc.GetString(state.buffer);
byte[] tmp = rsaProvider.Decrypt(bc.GetBytes(test), false);

I tried directly decrypting my string after I encrypted them, and it works. But the transfer from client to server or the reverse doesn't work and the Decrypt function throws an exception that tells me that the data to decrypt exceeds the maximum for the module.

Whats the correct way to encrypt the traffic between client and server?

Answer 1

Asymmetric encryption (e.g. with RSA) is good only for short messages. With a 1024-bit RSA key (a typical size), the maximum input message size is 117 bytes.

The "normal" setup is to select a random secret key (i.e. a bunch of random bits), encrypt that key with RSA, and encrypt the rest of the message symmetrically (i.e. with a symmetric cipher such as AES) using that key. Symmetric ciphers do not have such size limits and are much faster than asymmetric encryption anyway.

Now it so happens that even if the theory may look simple, there are an awful lot of details which can make an implementation go wrong in many ways, including making the code appear to run properly, while being open to attacks (and the point of using cryptography is to resist attacks). So the correct way is to use an existing protocol which already covers those details, along with an implementation which has been designed and tuned for that as well. The most common protocol for that is TLS (also known as SSL). C#/.NET appears to come with a System.Net.Security.SslStream class which is probably what you want.