os.path.exists() lies

Question

I'm running a number of python scripts on a linux cluster, and the output from one job is generally the input to another script, potentially run on another node. I find that there is some not insignificant lag before python notices files that have been created on other nodes -- os.path.exists() returns false and open() fails as well. I can do a while not os.path.exists(mypath) loop until the file appears, and it can take upwards of a full minute, which is not optimal in a pipeline with many steps and potentially running many datasets in parallel.

The only workaround I've found so far is to call subprocess.Popen("ls %s"%(pathdir), shell=True), which magically fixes the problem. I figure this is probably a system problem, but any way python might be causing this? Some sort of cache or something? My sysadmin hasn't been much help so far.

Answer 1

os.path.exists() just calls the C library's stat() function.

I believe you're running into a cache in the kernel's NFS implementation. Below is a link to a page that describes the problem as well as some methods to flush the cache.

http://www.unixcoding.org/NFSCoding#File_Handle_Caching

Answer 2

The problem is related to the fact that the Python process runs in its own shell. When you run subprocess.Popen(shell=True) you are spawning a new shell, which is working around the issue you're experiencing.

Python is not causing this issue. It's a combination of how NFS (file storage) and directory listings function in Linux.