Creating an encrypted log file

Question

I am creating a client side application which needs to create a log of the user activity but for various reasons this log must not be human readable.

Currently for my development I am creating a plain text log which looks something like this:

12/03/2009 08:34:21 -> User 'Bob' logged in 12/03/2009 08:34:28 -> Navigated to config page 12/03/2009 08:34:32 -> Option x changed to y

When I deploy my application the log must not be in plain text, so all text must be encrypted. This doesn't appear to be straightforward to achieve as I need the log file to dynamically update as each entry is added. The approach I was thinking about was to create a binary file, encrypt each log entry in isolation and then append it to the binary file with some suitable demarcation between each entry.

Does anyone know of any common approaches to this problem, I'm sure there must be a better solution!

Answer 1

This is not really my thing, I'll admit that readily, but can't you encrypt each entry individually and then append it to the logfile? If you that refrain from encrypting the timestamp, you can easily find entries your are looking for and decrypt those when needed.

My point being mainly that appending individual encrypted entries to a file does not necessarily need to be binary entries appended to a binary file. Encryption with (for example) gpg will yield ascii garble that can be appended to an ascii file. Would that solve you problem?

Answer 2

Assuming you're using some sort of logging framework, e.g., log4j et al, then you should be able to create a custom implementation of Appender (or similar) that encrypts each entry, as @wzzrd suggested.

Answer 3

FWIW, the one time I needed an encrypted logger I used a symmetric key (for performance reasons) to encrypt the actual log entries.

The symmetric 'log file key' was then encrypted under a public key and stored at the beginning of the log file and a separate log reader used the private key to decrypt the 'log file key' and read the entries.

The whole thing was implemented using log4j and an XML log file format (to make it easier for the reader to parse) and each time the log files were rolled over a new 'log file key' was generated.

Answer 4

I'm wondering what kind of application you write. A virus or a Trojan horse? Anyway ...

Encrypt each entry alone, convert it to some string (Base64, for example) and then log that string as the "message".

This allows you to keep parts of the file readable and only encrypt important parts.

Notice that there is another side to this coin: If you create a fully encrypted file and ask the user for it, she can't know what you will learn from the file. Therefore, you should encrypt as little as possible (passwords, IP addresses, costumer data) to make it possible for the legal department to verify what data is leaving.

A much better approach would be to an obfuscator for the log file. That simply replaces certain patterns with "XXX". You can still see what happened and when you need a specific piece of data, you can ask for that.

[EDIT] This story has more implications that you'd think at first glance. This effectively means that a user can't see what's in the file. "User" doesn't necessarily include "cracker". A cracker will concentrate on encrypted files (since they are probably more important). That's the reason for the old saying: As soon as someone gets access to the machine, there is no way to prevent him to do anything on it. Or to say it another way: Just because you don't know how doesn't mean someone else also doesn't. If you think you have nothing to hide, you haven't thought about yourself.

Also, there is the issue of liability. Say, some data leaks on the Internet after you get a copy of the logs. Since the user has no idea what is in the log files, how can you prove in court that you weren't the leak? Bosses could ask for the log files to monitor their pawns, asking to have it encoded so the peasants can't notice and whine about it (or sue, the scum!).

Or look at it from a completely different angle: If there was no log file, no one could abuse it. How about enabling debugging only in case of an emergency? I've configured log4j to keep the last 200 log messages in a buffer. If an ERROR is logged, I dump the 200 messages to the log. Rationale: I really don't care what happens during the day. I only care for bugs. Using JMX, it's simple to set the debug level to ERROR and lower it remotely at runtime when you need more details.

Answer 5

For .Net see Microsoft Application blocks for log and encrypt functionality: http://msdn.microsoft.com/en-us/library/dd203099.aspx

I would append encrypted log entries to a flat text file using suitable demarcation between each entry for the decryption to work.

Answer 6

Encrypting each log entry individually would decrease the security of your ciphertext a lot, especially because you're working with very predictable plaintext.

Here's what you can do:

1. Use symmetric encryption (preferably AES)
2. Pick a random master key
3. Pick a security window (5 minutes, 10 minutes, etc.)

Then, pick a random temporary key at the beginning of each window (every 5 minutes, every 10 minutes, etc.)

Encrypt each log item separately using the temporary key and append to a temporary log file.

When the window's closed (the predetermined time is up), decrypt each element using the temporary key, decrypt the master log file using the master key, merge the files, and encrypt using the master key.

Then, pick a new temporary key and continue.

Also, change the master key each time you rotate your master log file (every day, every week, etc.)

This should provide enough security.

Answer 7

It is not clear to me wheter your concern is on the security, or the implement.

A simple implement is to hook up with a stream encryptor. A stream encryptor maintains its own state and can encrypt on the fly.

StreamEncryptor<AES_128> encryptor;
encryptor.connectSink(new std::ofstream("app.log"));
encryptor.write(line);
encryptor.write(line2);
...