Python Service File Caching Apache Race Condition

Question

I am writing a python service (pyamf) through which a user can access images. All images are stored on a central server. The python services will be running on satellite machines which have network access to server. The service should work as follows:

1. check locally to see if the file exists, if so, use it.
2. check locally to see if file is currently being transferred from server ( file.part exists and size is changing ). If so, wait for download to finish, then use file.
3. if file does not exist and file is not being downloaded, download the file via urlretrieve.

The problem is with Apache's multiple threads. Threads are reaching the file presence check at the same time and therefore they all think the file needs to be downloaded. Needless to say, this is not good.

What is the right way to handle this race condition?

Thanks!

Answer 1

I'm guessing its either a threaded or a forked apache, but the effect would be the same since they are accessing a remote resource.

This problem is sometimes called the "dog pile" problem and its one of the issues addressed by the Beaker caching library (http://beaker.groovie.org). It provides a system bywhich you can create a callable that "creates" a new cached value, in this case a URL corresponding to some image that is fetched, if a value doesn't already exist. Locking is used such that concurrent threads or processes wait for the single process elected as the "creator" to finish what its doing. Beaker will use lockfiles if configured on a unix-like multi-process oriented system or mutexes if on a windows system.

I'm the original author of Beaker's guts along with Ben Bangert who packaged it up for usage with Pylons.