How to handle incomplete files? Getting exception

Question

Hi All,

I need to create a java program which will create thread to search for a file in particular folder(source folder) and pick the file immediately for process work(convert it into csv file format) once it found the file in the source folder. Problem i am facing now is file which comes to source folder is big size(FTP tool is used to copy file from server to source folder), thread is picking that file immediately before it copies fully to source folder and throwing exception. How do i stop thread until the file copy into source folder completely?. It has to pick the file for processing only after the file is copied completely into source folder.

Answer 1

Assuming you have no control over FTP process...

Let it be like this. When you get the exception, then try to process it again next time. Repeat it until the file gets processed. Its good to keep few attributes in case of exception to check it later, like; name, last-modified, size.

Check the exact exception before deciding to process it later, the exception might occur for some other reason.

Answer 2

If you have some control on the process that does the FTP you could potentially have it create a "flag file" in the source directory immediately AFTER the ftp for the big file is finished.

Then your Java thread has to check the presence of this flag file, if it's present then there is a file ready to be processed in the source directory. Before processing the big file, the thread should remove the flag file.

Flag file can be anything (even an empty file).

Answer 3

You could try different things:

• Repeatedly check the last modification date and the size of the file until it doesn’t change anymore for a given amount of time, then process it. (As pointed out by qbeuek this is neither safe nor deterministic.)
• Only process files with names that match certain criteria (e.g. *.dat). Change the FTP upload/download process to upload/download files with a different name (e.g. *.dat.temp) and rename the files once they are complete.
• Download the files to a different location and move them to your processing directory once they’re complete.
• As Vinegar said, if it doesn’t work the first time, try again later. :)

Answer 4

Tha safest way is to download the file to a different location and then move it to the target folder.

Another variation mentioned by Bombe is to change the file name to some other extension after downloading and look only for files with that extension.

Answer 5

I only read the file which is not in write mode. This is safest as this means no other process is writing in this file. You can check if file is not in write mode by using canWrite method of File class.

This solution works fine for me as I also have the exact same scenario you facing.

Answer 6

If your OS is Linux, and your kernel > 2.6.13, you could use the filesystem event notification API named inotify. There's a Java implementation here : http://code.google.com/p/inotify-java/.

Here's a sample code (heavily inspired from the website).

        try {
        Inotify i = new Inotify();
        InotifyEventListener e = new InotifyEventListener() {

            @Override
            public void filesystemEventOccurred(InotifyEvent e) {
                System.out.println("inotify event occurred!");
            }

            @Override
            public void queueFull(EventQueueFull e) {
                System.out.println("inotify event queue: " + e.getSource() +
                        " is full!");
            }

        };
        i.addInotifyEventListener(e);
        i.addWatch(System.getProperty("user.home"), Constants.IN_CLOSE_WRITE);
    } catch (UnsatisfiedLinkError e) {
        System.err.println("unsatisfied link error");
    } catch (UserLimitException e) {
        System.err.println("user limit exception");
    } catch (SystemLimitException e) {
        System.err.println("system limit exception");
    } catch (InsufficientKernelMemoryException e) {
        System.err.println("insufficient kernel memory exception");
    }