CAMEL is processing the file before the "mv" command completes

Question

We are facing a problem in our monitor script.

The flow of the program is

1. Customer ftp/sftp the file (in .csv format) to "source" directory
2. Bash script renames the completed .csv file to .aaa file
3. Another Bash script moves the ".aaa" file to "destination" directory and renames the file back to ".csv"
4. CAMEL is monitoring the "destination" directory. If any file comes to "destination" folder and having extension not equal to ".aaa" it will start processing the file.

Here the step 3 is failing. Bash script moves the file successfully. But while renaming the file from ".aaa" to ".csv" CAMEL fetches the file and start processing. CAMEL fails with the error "the footer is not present". This is because the "mv" command was not complete and CAMEL processed before the "mv" is completed. So it processed an incomplete file.

Since we use scripting we have little flexibility in using the commands. We are using Linux "mv" command to rename the file.

Is there a way to lock the file in scripting while renaming which will stop the CAMEL from accessing it?

Or is there a better way of handling this scenario?

Appreciate the help in advance.

Thanks, Mathew Liju

Answer 1

How do you do step 3?

mv foo.aaa dest/foo.csv

or

mv foo.aaa dest/foo.aaa
mv dest/foo.aaa dest/foo.csv

The latter should be atomic I think, while the first is a copy if origin and dest are on different partitions / filesystems.

Answer 2

I don't know what CAMEL is, but i think you are looking for a lock-free queue mechanism on the file system. Maildir (part of qmail) implements such a lock-free mail delivery queue.

You can also have a look all around inotify.

Depending on your OS and partition layout it would be necesary to cp, mv or ln the files.

Answer 3

A good tip is to always write what version of the various products you are using.

Anyway Camel 2.x got 5 or so different readlock options you can choose from to remedy such an issue. And you can implement your own if you have special supports such as people having to only start reading files if there is a special .DONE file etc.

http://camel.apache.org/file2.html