Protecting an Application's Memory From Tampering

Question

We are adding AES 256 bit encryption to our server and client applications for encrypting the TCP/IP traffic containing sensitive information. We will be rotating the keys daily. Because of that, the keys will be stored in memory with the applications.

Key distribution process:

1. Each server and client will have a list of initial Key Encryption Key's (KEK) by day

2. If the client has just started up or the server has just started up, the client will request the daily key from the server using the initial key. The server will respond with the daily key, encrypted with the initial key. The daily key is a randomly generated set of alphanumeric characters. We are using AES 256 bit encryption.

3. All subsequent communications will be encrypted using that daily key.

4. Nightly, the client will request the new daily key from the server using the current daily key as the current KEK. After the client gets the new key, the new daily key will replace the old daily key.

Is it possible for another bad application to gain access to this memory illegally or is this protected in Windows? The key will not be written to a file, only stored in a variable in memory.

If an application can access the memory illegally, how can you protect the memory from tampering?

We are using C++ and XP (Vista/7 may be an option in the future so I don't know if that changes the answer).

Answer 1

I think that you may have a more fundamental problem on your hands.

If there is even the faintest chance that this machine might catch a rootkit, then your all of your keys are ours, as it were.

On Windows, Process A can read the memory of Process B if any of the below are true:

1. it has privileges to open the memory device.
2. it has privileges to open the virtual memory of process B.
3. it has a friend in the kernel.

If you have complete control over what's running on the machine, and complete confidence that no one can introduce any surprises, you're golden.

This is, of course, not unique to Windows. What is unique to Windows is the volume of rootkit malware.

Answer 2

Your key distribution process isn't secure: if someone gets that initial key set, it's all over.

Yes, other applications can get access to the process memory on Windows using the debugger hooks, and there's pretty much no way of preventing that. Essentially, if the box is compromised in any way, there's very little you can do about it.

Answer 3

One more bit of information as well is that even if you don't write the key to disk yourself it might very well end up there anyway. All that is needed is for the virtual memory system to page out the memory holding the key.

As others have pointed out if you cannot trust the box you are on you are screwed anyway. There are ways of making this sort of thing more secure using some extra hardware, basically moving the encryption task onto a daughter card that you can better control. That is the technique used by some financial institutions and some government entities. It might be out of your budget spect though.

Answer 4

There's really no way to prevent other processes from reading the memory of your process.

To make things a little more resistant to an unmotivated attacked, you could consider using a randomly generated key to encrypt the real key while it is not in use. You could generate a new random KEK every time the program is run, perhaps even changing while the program is running every few seconds. Only decrypt the "real" key when you actually need to use it, and then immediately zero out the memory where the key is when you are done.

Of course, this won't really work if you're in constant communication. But if your communication is idle most of the time and only sees occasional spikes in traffic, this approach might work.