What is the best way to open a file for exclusive access in Python?

Question

What is the most elegant way to solve this:

• open a file for reading, but only if it is not already opened for writing
• open a file for writing, but only if it is not already opened for reading or writing

The built-in functions work like this

>>> path = r"c:\scr.txt"
>>> file1 = open(path, "w")
>>> print file1
<open file 'c:\scr.txt', mode 'w' at 0x019F88D8>
>>> file2 = open(path, "w")
>>> print file2
<open file 'c:\scr.txt', mode 'w' at 0x02332188>
>>> file1.write("111")
>>> file2.write("222")
>>> file1.close()

scr.txt now contains '111'.

>>> file2.close()

scr.txt was overwritten and now contains '222' (on Windows, Python 2.4).

The solution should work inside the same process (like in the example above) as well as when another process has opened the file.
It is preferred, if a crashing program will not keep the lock open.

Answer 1

To make you safe when opening files within one application, you could try something like this:

import time
class ExclusiveFile(file):
    openFiles = {}
    fileLocks = []

    class FileNotExclusiveException(Exception):
        pass

    def __init__(self, *args):

        sMode = 'r'
        sFileName = args[0]
        try:
            sMode = args[1]
        except:
            pass
        while sFileName in ExclusiveFile.fileLocks:
            time.sleep(1)

        ExclusiveFile.fileLocks.append(sFileName)

        if not sFileName in ExclusiveFile.openFiles.keys() or (ExclusiveFile.openFiles[sFileName] == 'r' and sMode == 'r'):
            ExclusiveFile.openFiles[sFileName] = sMode
            try:
                file.__init__(self, sFileName, sMode)
            finally:
                ExclusiveFile.fileLocks.remove(sFileName)
         else:
            ExclusiveFile.fileLocks.remove(sFileName)
            raise self.FileNotExclusiveException(sFileName)

    def close(self):
        del ExclusiveFile.openFiles[self.name]
        file.close(self)

That way you subclass the file class. Now just do:

>>> f = ExclusiveFile('/tmp/a.txt', 'r')
>>> f
<open file '/tmp/a.txt', mode 'r' at 0xb7d7cc8c>
>>> f1 = ExclusiveFile('/tmp/a.txt', 'r')
>>> f1
<open file '/tmp/a.txt', mode 'r' at 0xb7d7c814>
>>> f2 = ExclusiveFile('/tmp/a.txt', 'w') # can't open it for writing now
exclfile.FileNotExclusiveException: /tmp/a.txt

If you open it first with 'w' mode, it won't allow anymore opens, even in read mode, just as you wanted...

Answer 2

If you have multiple Python scripts accessing the same file there are a couple of simple options:

Read-only method

1. set the file as read-only when it is not in use
2. before reading/writing to the file check if it is read-only
3. if it is not read-only then that means it is in use and you will have to wait until it is read-only again
4. if it is read-only then remove the read-only flag, read/write the file, then re-apply the read-only flag

Lockfile method

Same as above except instead of using the read-only flag you create a file named 'scr.txt.lock' when you want to access the file and then delete the file when done.

Answer 3

I don't think there is a fully crossplatform way. On unix, the fcntl module will do this for you. However on windows (which I assume you are by the paths), you'll need to use the win32file module.

Fortunately, there is a portable implementation (portalocker) using the platform appropriate method at the python cookbook.

To use it, open the file, and then call:

portalocker.lock(file, flags)

where flags are portalocker.LOCK_EX for exclusive write access, or LOCK_SH for shared, read access.

Answer 4

Here's a start on the win32 half of a portable implementation, that does not need a seperate locking mechanism.

Requires the Python for Windows Extensions to get down to the win32 api, but that's pretty much mandatory for python on windows already, and can alternatively be done with ctypes. The code could be adapted to expose more functionality if it's needed (such as allowing FILE_SHARE_READ rather than no sharing at all). See also the MSDN documentation for the CreateFile and WriteFile system calls, and the article on Creating and Opening Files.

As has been mentioned, you can use the standard fcntl module to implement the unix half of this, if required.

import winerror, pywintypes, win32file

class LockError(StandardError):
 pass

class WriteLockedFile(object):
 """
 Using win32 api to achieve something similar to file(path, 'wb')
 Could be adapted to handle other modes as well.
 """
 def __init__(self, path):
  try:
   self._handle = win32file.CreateFile(
    path,
    win32file.GENERIC_WRITE,
    0,
    None,
    win32file.OPEN_ALWAYS,
    win32file.FILE_ATTRIBUTE_NORMAL,
    None)
  except pywintypes.error, e:
   if e[0] == winerror.ERROR_SHARING_VIOLATION:
    raise LockError(e[2])
   raise
 def close(self):
  self._handle.close()
 def write(self, str):
  win32file.WriteFile(self._handle, str)

Here's how your example from above behaves:

>>> path = "C:\\scr.txt"
>>> file1 = WriteLockedFile(path)
>>> file2 = WriteLockedFile(path) #doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
 ...
LockError: ...
>>> file1.write("111")
>>> file1.close()
>>> print file(path).read()
111

Answer 5

The solution should work inside the same process (like in the example above) as well as when another process has opened the file.

If by 'another process' you mean 'whatever process' (i.e. not your program), in Linux there's no way to accomplish this relying only on system calls (fcntl & friends). What you want is mandatory locking, and the Linux way to obtain it is a bit more involved:

Remount the partition that contains your file with the mand option:

# mount -o remount,mand /dev/hdXY

Set the sgid flag for your file:

# chmod g-x,g+s yourfile

In your Python code, obtain an exclusive lock on that file:

fcntl.flock(fd, fcntl.LOCK_EX)

Now even cat will not be able to read the file until you release the lock.