How to read a block of n lines of a very big text file in both directions, using Java

Question

I need to read a very big log file (about 400MB) and display its content in a textarea.

Obviously I can't read the whole file first, because of its dimension, so I am looking for a method that can read the first n lines (or portion of bytes) and then, when asked, start reading the next n lines, and so on. The method needs to work in both directions: i.e. if the lines from 1000 to 1500 are displayed and the user wants to read back, I have to display the lines from 500 to 1000.

I am trying with RandomAccessFile, LineIterator and classic BufferedReader approach, but those don't seem to be good solutions.

Thanks in advance.

Answer 1

If your file doesn't have strict structure (i.e. fixed number of characters per line or per some logical block as Martijn noticed) then you cannot use RandomAccessFile advantages.

In that case you can try to split you huge file into smaller files with fixed number of lines in each (1000 for example). In this case when user wants to see lines from 1800 to 2100 you need to read only 2 files (from several thousands probably if you say that the total size is about 400 Mb).

If you cannot do this the only thing you can do is to use NIO to increase performance at least a bit.

Answer 2

In similar cases, others have suggested to store the file in a database. For example each row as a separate record. Then reading data in either direction and in big or small chunks is much easier.