Locking a file in Python

Question

Hello, I need to lock a file for writing in Python. It will be accessed from multiple Python processes at once. I have found some solutions online, but most fail for my purposes as they are often only Unix based or Windows based.

Any help would be appreciated. Thanks.

Answer 1

Locking a file is usually a platform-specific operation, so you may need to allow for the possibility of running on different operating systems. For example:

import os

def my_lock(f):
    if os.name == "posix":
        # Unix or OS X specific locking here
    elif os.name == "nt":
        # Windows specific locking here
    else:
        print "Unknown operating system, lock unavailable"

Answer 2

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don't want to solve.

Your best bet is have a separate process that coordinates read/write access to that file.

Answer 3

There is a cross-platform file locking module here: Portalocker

Although as Kevin says, writing to a file from multiple processes at once is something you want to avoid if at all possible.

If you can shoehorn your problem into a database, you could use SQLite. It supports concurrent access and handles its own locking.

Answer 4

Locking is platform and device specific, but generally, you have a few options:

1. use flock(), or equivilent (if your os supports it). This is advisory locking, unless you check for the lock, its ignored.
2. Use a lock-copy-move-unlock methodology, where you copy the file, write the new data, then move it (move, not copy - move is an atomic operation), and you check for the existence of the lock file.
3. Use a directory as a "lock". This is necessary if you're writing to NFS, since NFS doesn't support flock().
4. There's also the possibility of using shared memory between the processes, but I've never tried that; its very os-specific.

For all these methods, you'll have to use a spin-lock (retry-after-failure) technique for acquiring and testing the lock. This does leave a small window for mis-synchronization, but its generally small enough to not be an major issue.

If you're looking for a solution that is cross platform, then you're better off logging to another system via some other mechanism (the next best thing is the NFS technique above).

Note that sqlite is subject to the same constraints over NFS that normal files are, so you can't write to an sqlite database on a network share and get synchronization for free.

Answer 5

Alright, so I ended up going with the code I wrote here, on my website. I can use it in the following fashion:

from filelock import FileLock

with FileLock("myfile.txt"):
    # work with the file as it is now locked
    print("Lock acquired.")

If anybody sees any problem with this, please let me know.

Answer 6

Hello

Kernel-level file locking is an extremly complex subject, since different types of locks have very different semantic, and the main lock of unix systems (fcntl) has horrible flaws.

But I've just released a python library which deals with the issue in a very portable way. It can achieve shared/exclusive file record locking in a system-wide manner, and provides workaround for fcntl flaws.

Enjoy RSFile - http://bitbucket.org/pchambon/python-rock-solid-tools/

Answer 7

i prefer lockfile — Platform-independent file locking doc here