Locking with S3

Question

What's the recommended way to implement a simple locking mechanism to be used in conjunction with S3?

Example of what I want to do:

• acquire lock by object id
• read object from S3
• modify data
• write object to S3
• release lock

Ideally looking for a cloud based locking mechanism. I could use memcached locally, but then I have to deal with scaling that. I don't see an obvious way to implement lightweight locking with any AWS APIs, but it seems like a common problem.

I wonder if you could use SimpleDB to do an atomic acquire lock operation. Has anyone tried that?

Answer 1

Ok, I spent some time this morning playing with boto and I think I have a solution that works using SimpleDB. You need the latest boto release so that conditional puts and consistent reads are supported.

Example code here: http://pastebin.com/3XzhPqfY

Please post comments/suggestions. I believe this code should be fairly safe -- my test in main() tries it with 10 threads.

One thing I haven't addressed is that S3 reads are not consistent (right?), so in theory a thread may be operating on an old copy of the S3 value. It looks like there may be a workaround for that as described here:

http://www.shlomoswidler.com/2009/12/read-after-write-consistency-in-amazon.html

Answer 2

i dont think that you can do this using S3 only, using simpleDB's consistency enhancements as james said is a good way that works

you can look for some examples here : Amazon SimpleDB Consistency Enhancements

another approach that might be fine is using the versioning feature of S3
so basically, store an object id/version id pair in simpleDB as the most "valid" version
and assure that all GET requests will retrieve that version
after a successful PUT of a modified object, update the version id in the DB

this way you could also use the ability to retrieve previous versions of an object for restoring if needed.