Parsing a text file on BlackBerry takes forever

Question

I was originally using RIM's native xml parser methods to parse a 150k text file, approximately 5000 lines of xml, however it was taking about 2 minutes to complete, so I tried a line based format:

Title: Book Title
Line 1
Line 2
Line 3

I should be able to read the file in less time than it takes to blink, but it is still slow.

Identifier books is a Vector of Book objects and lines are stored in a vector of strings in the Book object.

class classs = Class.forName("com.Gui.FileLoader");
InputStream is = classs.getResourceAsStream( fileName );

int totalFileSize = IOUtilities.streamToBytes( is ).length;
int totalRead = 0;

//Thought that maybe a shared input stream would be faster, in this case it't not.
SharedInputStream sis = SharedInputStream.getSharedInputStream( classs.getResourceAsStream( fileName ) );

LineReader lr = new LineReader( sis );
String strLine = new String( lr.readLine() );
totalRead += strLine.length();

Book book = null;

//Loop over the file until EOF is reached, catch EOF error move on with life after that.
while(1 == 1){

    //If Line = Title: then we've got a new book add the old book to our books vector.
    if (strLine.startsWith("Title:")){

        if (book != null){
            books.addElement( book );
        }

         book = new Book();

         book.setTitle( strLine.substring( strLine.indexOf(':') + 1).trim() );

         strLine = new String( lr.readLine() );
         totalRead += strLine.length();
         continue;
    }

    int totalComplete = (int) (  ( (double) totalRead / (double) totalFileSize ) * 100.00);
    _observer.processStatusUpdate( totalComplete , book.getTitle() );

    book.addLine( strLine );

    strLine = new String( lr.readLine(), "ascii" );
    totalRead += strLine.length();
}

Answer 1

Try using new BufferedInputStream(classs.getResourceAsStream(fileName));

EDIT:

Apparently the documentation that says they have BufferedInputStream is wrong.

I am going to leave this wrong answer here just so people have that info (doc being wrong).

Answer 2

Where does the profiler say you spend your time?

If you do not have a preferred profiler there is jvisualvm in the Java 6 JDK.

(My guess is that you will find all the time being spent on the way down to "read a character from the file". If so, you need to buffer)

Answer 3

For one thing, you're reading in the file twice - once for determining the size and then again for parsing it. Since you're already reading it into a byte array for determining the size, why not pass that byte array into a ByteArrayInputStream constructor? For example:

//Used to determine file size and then show in progress bar, app is threaded.
byte[] fileBytes = IOUtilities.streamToBytes( is );
int totalFileSize = fileBytes.length;
int totalRead = 0;

ByteArrayInputStream bais = new ByteArrayInputStream( fileBytes );
LineReader lr = new LineReader( bais);

This way it won't matter if the rest of the classes reading from the stream are reading a byte at a time - it's all in-memory.

Answer 4

It is easy to assume that all the operations you've elided from the code sample finish in constant time. I am guessing that one of them is doing something inefficiently, such as book.addLine( strLine ); or perhaps _observer.processStatusUpdate( totalComplete , book.getTitle() ); If those operations are not able to complete in constant time, then you could easily have a quadratic parsing algorithm.

Just thinking about the operations is the best way to figure it out, but if you're stumped, try using the BlackBerry profiler. Run your program in the Eclipse debugger and get it to stop at a breakpoint just before parsing. Then, in Eclipse, select 'window .. show view .. other .. BlackBerry .. BlackBerry Profiler View' Select the 'setup options' button from the profiler view toolbar. It has a blue triangle in the icon. Set 'method attribution' to cumulative, and 'what to profile' to 'time including native methods'

then continue your program. once parsing is finished, you'll need to pause program execution, then click on the 'method' tab of the profiler view. You should be able to determine your pain point from there.