In Java, what is the best/safest pattern for monitoring a file being appended to?

Question

Someone else's process is creating a CSV file by appending a line at a time to it, as events occur. I have no control over the file format or the other process, but I know it will only append.

In a Java program, I would like to monitor this file, and when a line is appended read the new line and react according to the contents. Ignore the CSV parsing issue for now. What is the best way to monitor the file for changes and read a line at a time?

Ideally this will use the standard library classes. The file may well be on a network drive, so I'd like something robust to failure. I'd rather not use polling if possible - I'd prefer some sort of blocking solution instead.

Edit -- given that a blocking solution is not possible with standard classes (thanks for that answer), what is the most robust polling solution? I'd rather not re-read the whole file each time as it could grow quite large.

Answer 1

Probably won't help, just thinking loud, but on UNIX one would use tail -f to see any lines added to a file - you need something similar but in terms of java classes. tail -f is itself implemented with polling I believe. It encounters EOF, then waits some small amount of time (100ms) then tries to read again up to EOF. So it always gets the latest data, because while other process writes - EOF moves forward.

Answer 2

This is not possible with standard library classes. See this question for details.

For efficient polling it will be better to use Random Access. It will help if you remember the position of the last end of file and start reading from there.

Answer 3

Use Java 7's WatchService, part of NIO.2

The WatchService API is designed for applications that need to be notified about file change events.

Answer 4

I tried the following (polling on a 1 sec interval) and it works (just prints in processing):

  private static void monitorFile(File file) throws IOException {
    final int POLL_INTERVAL = 1000;
    FileReader reader = new FileReader(file);
    BufferedReader buffered = new BufferedReader(reader);
    try {
      while(true) {
        String line = buffered.readLine();
        if(line == null) {
          // end of file, start polling
          Thread.sleep(POLL_INTERVAL);
        } else {
          System.out.println(line);
        }
      }
    } catch(InterruptedException ex) {
     ex.printStackTrace();
    }
  }

As no-one else has suggested a solution which uses a current production Java I thought I'd add it. If there are flaws please add in comments.

Answer 5

Just to expand on Nick Fortescue's last entry, below are two classes that you can run concurrently (e.g. in two different shell windows) which shows that a given File can simultaneously be written to by one process and read by another.

Here, the two processes will be executing these Java classes, but I presume that the writing process could be from any other application. (Assuming that it does not hold an exclusive lock on the file-are there such file system locks on certain operating systems?)

I have successfully tested these two classes on both Windoze and Linux. I would very much like to know if there is some condition (e.g. operating system) on which they fail.

Class #1:

import java.io.File;
import java.io.FileWriter;
import java.io.PrintWriter;

public class FileAppender {

    public static void main(String[] args) throws Exception {
        if ((args != null) && (args.length != 0)) throw
            new IllegalArgumentException("args is not null and is not empty");

        File file = new File("./file.txt");
        int numLines = 1000;
        writeLines(file, numLines);
    }

    private static void writeLines(File file, int numLines) throws Exception {
        PrintWriter pw = null;
        try {
            pw = new PrintWriter( new FileWriter(file), true );
            for (int i = 0; i < numLines; i++) {
                System.out.println("writing line number " + i);
                pw.println("line number " + i);
                Thread.sleep(100);
            }
        }
        finally {
            if (pw != null) pw.close();
        }
    }

}

Class #2:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;

public class FileMonitor {

    public static void main(String[] args) throws Exception {
        if ((args != null) && (args.length != 0)) throw
            new IllegalArgumentException("args is not null and is not empty");

        File file = new File("./file.txt");
        readLines(file);
    }

    private static void readLines(File file) throws Exception {
        BufferedReader br = null;
        try {
            br = new BufferedReader( new FileReader(file) );
            while (true) {
                String line = br.readLine();
                if (line == null) { // end of file, start polling
                    System.out.println("no file data available; sleeping..");
                    Thread.sleep(2 * 1000);
                }
                else {
                    System.out.println(line);
                }
            }
        }
        finally {
            if (br != null) br.close();
        }
    }

}