Java I/O over an NFS mount

Question

I have a bit of Java code that outputs an XML file to a NFS mounted filesystem. On another server that has the filesytem mounted as a Samba share, there is a process running that polls for new XML files every 30 seconds. If a new file is found, it is processed and then renamed as a backup file. 99% of the time, the files are written without an issue. However, every now and then the backup file contains a partially written file.

After some discussion with some other people, we guessed that the process running on the external server was interfering with the Java output stream when it read the file. They suggested first creating a file of type .temp which will then be renamed to .xml after the file write is complete. A common industry practice. After the change, the rename fails every time.

Some research turned up that Java file I/O is buggy when working with NFS mounted filesystems.

Help me Java gurus! How do I solve this problem?

Here is some relevant information:

• My process is Java 1.6.0_16 running on Solaris 10
• Mounted filesystem is a NAS
• Server with polling process is Windows Server 2003 R2 Standard, Service Pack 2

Here is a sample of my code:

//Write the file
XMLOutputter serializer = new XMLOutputter(Format.getPrettyFormat());
FileOutputStream os = new FileOutputStream(outputDirectory + fileName + ".temp");
serializer.output(doc, os);//doc is a constructed xml document using JDOM
os.flush();
os.close();

//Rename the file
File oldFile = new File(outputDirectory + fileName + ".temp");
File newFile = new File(fileName + ".xml");
boolean success = oldFile.renameTo(newFile);
if (!success) {
    // File was not successfully renamed.
    throw new IOException("The file " + fileName + ".temp could not be renamed.");
}//if

Answer 1

You probably have to specify the complete path in the new file name:

File newFile = new File(outputDirectory + fileName + ".xml");

Answer 2

This looks like a bug to me:

File oldFile = new File(outputDirectory + fileName + ".temp");
File newFile = new File(fileName + ".xml");

I would have expected this:

File oldFile = new File(outputDirectory + fileName + ".temp");
File newFile = new File(outputDirectory + fileName + ".xml");

In general, it sounds like there is a race condition between the writing of the XML file and the read/process/rename task. Can you have the read/process/rename task only operate on files > 1 minute old or something similar?

Or, have the Java program write out an additional, empty file once it has completed writing out the XML file that signals that the writing to the XML file has completed. Only read/process/rename the XML file when the signal file is present. Then delete the signal file.

Answer 3

The original bug definitely sounds like an issue with concurrent access to the file -- your solution should have worked, but there are alternate solutions too.

For example, put a timer on your auto-read process so it when a new file is detected it records filesize, sleeps X seconds, and then if the sizes don't match restarts the timer. That should avoid problems with partial file transfer.

EDIT: or check the timestamps as pre above to check this, but make sure it's old enough that any imprecision in the timestamp doesn't matter (say, 10 seconds to 1 minute since last modified).

Alternately, try this:

File f = new File("foo.xml");
FileOutputStream fos = new FileOutputStream(f);
FileChannel fc = fos.getChannel();
FileLock lock = fc.lock();
(DO FILE WRITE)
fis.flush();
lock.release();
fos.close();

This SHOULD use native OS file locking to prevent concurrent access by other programs (such as your XML reader daemon).

As far as NFS glitches: there is a documented "feature" (bug) where files can't be moved between filesystems via "rename" in Java. Could there be confusion, since it is on a NFS filesystem?

Answer 4

Possibly due to "The rename operation might not be able to move a file from one filesystem to another" from http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#renameTo%28java.io.File%2) Try using apache commons io FiltUtils.copyFileToDirectory http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#copyFileToDirectory%28java.io.File,%20java.io.File) instead

Answer 5

Some information to NFS in general. Depending on your NFS settings, locks might not work at all and a lot of big NFS installations are tuned for read performance, therefore new data might turn up later than expected, due to caching effects.

I have seen effects where you created a file, added data (this was seen on another machine), but all data after that appeared with a 30 sec delay.

Best solution by the way is a rotating file schema. So that the last one is assumed to be written and the one before was safely written and can be read. I would not work on a single file and use it as a "pipe".

You can alternatively use an empty file that is written after the large file was written and closed properly. So if the small guys is there, the big guy was definitively done and can be read.