Turning a remote file into an input stream over HTTP in Java

Question

Hi.

I have the URL of a text file and I want my Java program to read that text file. But the plot thickens! The file is constantly being appended with new lines and I want to read these lines as they come in.

I think the right approach is to open a URLConnection to the URL of the file and somehow put that URLConnection under the 'supervision' of some sort of StreamReader or StreamBuffer type of object.

This is where my Java skills become questionable and I was wondering if anyone cares to donate an answer or two.

Thanks.

Answer 1

Get the InputStream from the URLConnection and wrap it in an InputStreamReader and then the ISR in a BufferedReader:

InputStream is = urlConnection.getInputStream();
InputStreamReader isr = new InputStreamReader(is, <encoding>);
BufferedReader br = new BufferedReader(isr);

You can now use br.readLine() to read each line of text from the resource until it returns null (EOF). You have to get the character encoding from somewhere though, either from the HTTP response content type header or if it's known, you can specify it directly.

Answer 2

To make this implement an InputStream you would need to make multiple http requests in a loop and keep track of how much you have read so far so the consumer of the stream gets consistent output.

pseudocode:

int read_bytes = 0;
while (should_be_reading) {
  # make http request

  # read or scan to read_bytes
  # emit any new bytes
  # update read_bytes

}

Answer 3

If you keep track of the total number of bytes read, you can use an http range header to tell the server to start serving the file from a given position. This functionality is mainly used to resume downloads, but should be applicable here.

I realize this doesn't give you an input stream, but I think it is a more robust solution.

Answer 4

According to the comments above, I think I've solved this problem. I am just not so sure that this doesn't mean that I am downloading the entire file every time:

                long charsRead = 0;

                while(keepRunning)
                {
                    URL url = new URL(finalUrlString);
                    URLConnection connection = url.openConnection();
                    InputStreamReader stream = new InputStreamReader(connection.getInputStream());
                    BufferedReader reader = new BufferedReader(stream);
                    long skipped = reader.skip(charsRead);
                    String line = reader.readLine();
                    if(line != null)
                    {
                        charsRead += line.length() + 1;
                        process(line);
                    }
                    reader.close();
                }

This piece of code runs inside its own thread. I am using the process method to fill up a vector of objects generated by parsing each line.

A different piece of code, on a different thread, looks at this vector - reads the objects - and empties it.

Of course the this thread and the other one are synchronized around that vector instance.