How can you encrypt users' data server-side without ruining the experience?

Question

Many users – myself included – would like the security of having everything they do on a web service encrypted. That is, they don't won't any one at the web service to be able to look at their: posts, info, tasks, etc...

This is also major complaint in this discussion of an otherwise cool service: http://news.ycombinator.com/item?id=1549115

Since this data needs to be recoverable, some sort of two-way encryption is required. But unless you're prompting the user for the encryption key on every request, this key will need to be stored on the server, and the point of encrypting the data is basically lost.

What is a way to securely encrypt user data without degrading the user experience (asking for some key on every request)?

-- UPDATE --

From @Borealid's answer, I've focused on two possibilities: challenge-response protocols, where no data (password included) is sent in the "clear", and non-challenge-response protocols, where data (password included) is sent in the "clear" (although over HTTPS).

Challenge-response protocols (specifically SRP: http://srp.stanford.edu/)

It seems that its implementation would need to rely on either a fully AJAX site or using web storage. This is so the browser can persist the challenge-response data during encryption and also the encryption key between different "pages". (I'm assuming after authentication is completed I would send them back the encrypted encryption key, which they would decrypt client-side to obtain the real encryption key.)

The problem is that I'm either:

• fully AJAX, which I don't like because I love urls and don't won't a user to live exclusively on a single url, or
• I have to store data encryption keys in web storage, which based on http://dev.w3.org/html5/webstorage/ will persist even after the browser is closed and could be a security vulnerability

In addition, as SRP takes more than one request ( http://srp.stanford.edu/design.html ), there needs to be some persistence on the server-side. This is just another difficulty.

Traditionally

If I'm ok transmitting passwords and data in the clear (although over HTTPS), then the client-side issues above are not present.

On registration, I'll generate a random unique encryption key for the user, and encrypt it using their password and a random salt.

In the database, I'll store the user's password hash and salt (through bcrypt), encrypted encryption key, encryption key salt, and encryption iv.

After an authentication, I'll also need to use their password to decrypt the encryption key so that they may view and enter new data. I store this encryption key only temporarily and delete it when they explicitly "log out".

The problems with this approach is that (like @Borealid points out) evil sysadmins can still look at your data when you are logged in.

I'm also not sure how to store the encryption keys when users are logged in. If they are in the same data store, a stolen database would reveal all data of those who were logged in at the time of theft.

Is there a better in-memory data store for storing these encryption keys (and challenge data during an SRP authentication)? Is this something Redis would be good for?

Answer 1

If the data need to be recoverable in the event of user error, you can't use something like a cookie (which could get deleted). And as you point out, server-side keys don't actually secure the user against malicious sysadmins; they only help with things like databases stolen offline.

However, if you're running a normal web service, you've already gotten pretty lucky - the user, in order to be unique and non-ephemeral, must be logged in. This means they go through some authentication step which proves their identity. In order to prove their identity, most web sites use a passed credential (a password).

So long as you don't use a challenge-response authentication protocol, which most web sites don't, you can use an encryption key derived from a combination of a server-side secret and the user's password. Store the encryption key only while the user is authenticated.

If you do this, the users are still vulnerable to sysadmins peeking while they're using the service (or stealing their passwords). You might want to go a step further. To go one up, don't send the password to the server at all. Instead, use a challenge-response protocol for authentication to your website, and encrypt the data with a derivative of the user's password via JavaScript before uploading anything.

This is foolproof security: if you try to steal the user's password, the user can see what you're doing because the code for the theft is right there in the page you sent them. Your web service never touches their data unencrypted. This is also no hindrance to the normal user experience. The user just enters their password to log in, as per normal.

This method is what is used by Lacie's storage cloud service. It's very well done.

Note: when I say "use foo to encrypt", I really mean "use foo to encrypt a secure symmetric key which is then used with a random salt to encrypt". Know your cryptography. I'm only talking about the secret, not the methodology.

Answer 2

If you want to perform computations on a server without even the server being able to see the data, you may be interested in knowing about fully homomorphic encryption. A fully homomorphic encryption scheme lets you perform arbitrary computations on encrypted data, even if you can't decrypt it. However, this is still a topic of research.

For now, I guess your best bet would be to encrypt all posts and assign meaningless (e.g. sequential) IDs to each one. For a more in-depth discussion of how to encrypt server-side data with today's technology, look up.

Answer 3

None of those other solutions are going to maintain the feature set requested -- which specifically wants to preserve the user experience. If you look at the site referenced in the link, they email you a nightly past journal entry. You're not going to get that with JavaScript trickery per above because you don't have the browser to depend on. So basically this is all leading you down a path to a degraded user experience.

What you would want, or more precisely the best solution you're going to find in this space, is not so much what wuala does per above, but rather something like hush.com. The handling of user data needs to be done on the client side at all times -- this is generally accomplished via full client-side Java (like the Facebook photo uploader, etc), but HTML/JavaScript might get you there these days. JavaScript encryption is pretty poor, so you may be better off ignoring it.

OK, so now you've got client-side Java running a Journal entry encryption service. The next feature was to email past journal entries to users every night. Well, you're not going to get that in an unencrypted email obviously. This is where you're going to need to change the user experience one way or the other. The simplest solution is not to email the entry and instead to provide for instance a journal entry browser in the Java app that reminds them of some old entry once they get to the website based on a link in the daily email. A much more complex solution would be to use JavaScript encryption to decrypt the entry as an attachment inline in the email. This isn't rocket science but there is a fairly huge amount of trickery involved. This is the general path used by several web email encryption services such as IronPort. You can get a demo email by going to http://www.ironport.com/securedemo/.

As much as I'd love to see a properly encrypted version of all this, my final comment would be that journal entries are not state secrets. Given a solid privacy policy and good site security semantics, I'm sure 99% of your users will feel just fine about things. Doing all this right with true security will take an enormous amount of effort per above and at least some design/UE changes.