how to make sure not to read a file before finishing the write to it

views:

126

answers:

+3 Q:

how to make sure not to read a file before finishing the write to it

When trying to monitor a directory using inotify on Linux, as we know, we get notified as soon as the file gets created (before the other process finish writing to it)

Is there an effective way to make sure that the file is not read before writing to it is complete by the other process?

We could potentially add a delayed read; but as we all know, it is flawed.

For a little bit more clarity on the scenario; the two processes are running as different users; the load expected is about a few hundred files created per second.

You can check /proc/<pid>/fd to see if the file is still opened. If it is not listed there, you can be sure that the process is no longer using it.

Gianni 2010-07-16 18:32:00

@Gianni, is there a way to find the fd of the file, which is opened by a foreign process; none that I have heard of. AFAIK, the fd is process specific and only the OS and the process will know of the relation between an fd and file on disk; if at all there is a way, I am pretty sure that will be available only to root.

CodeMedic 2010-07-16 18:53:37

@CodeMedic Look at the dir. The fd is a symbolik link to the file itself, so you don't need to convert fd -> name, all you have to do is use `lstat` to see to which file the link points to.

Gianni 2010-07-16 18:56:14

eg: user@box:~# ls -l /proc/1433/fd/0 lr-x------ 1 root root 64 2010-07-16 15:31 /proc/1433/fd/0 -> /dev/null

Gianni 2010-07-16 18:57:20

@Gianni that would work only if the two processes are running as the same user. Normally /proc/<PID>/fs/* are user only readable. On top of that we will have scalability issues by having to iterate through the list every second or so. Thanks anyways

CodeMedic 2010-07-16 19:21:55

@CodeMedic Well, you didn't specify that, so I suggest you edit your question and add that.

Gianni 2010-07-16 19:32:12

+1 A:

Create it somewhere else, write to it, close it, then rename it - or am I missing something obvious?

Dipstick 2010-07-16 19:54:41

Yeah, this is the right way. Or create it under a temporary name and rename when finished.

MarkR 2010-07-17 12:43:50

Thats only if I have control on the process! Here I dont! Its a third party piece.

CodeMedic 2010-07-19 10:01:56

+1 A:

Based on your question, it sounds like you're currently monitoring the directory with the IN_CREATE (and maybe IN_OPEN) flag. Why not also use the IN_CLOSE flag so that you get notified when the file is closed? From there, it should be easy to keep track of whether something has the file open and you'll know that you don't want to try reading it yet.

jamessan 2010-07-16 20:58:06

Thanks jamessan; your reply helped me arrive at a solution that seems to work for me.I now monitor for IN_CREATE and IN_CLOSE; combine them to pick the new arrivals into the directory that I am monitoring! Thanks for the help.

CodeMedic 2010-07-18 22:24:06

maybe the lsof command can help. It lists all the opened files. $ man lsof

Adil Butt 2010-07-25 01:16:18

the question here is not about doing the copy from shell.

CodeMedic 2010-07-27 00:03:22

ansaurus

tags:

views:

answers:

how to make sure not to read a file before finishing the write to it

related questions