ibatis: improve insert performace

Question

Hi

I am using ibatis for my sql insert stmt. In my code i am parsing files line by line from a folder. Each line that matches criteria, need to be inserted into database. Total number of insert in a single run of program can be any where along 200k.

    SqlSession sess = null;    
    this.sess = sf.openSession(ExecutorType.BATCH, false);
    for (each file) {
     for( each line matching criteria ){
         this.sess.insert("com.logs.util.insertFileInfo", fileData);
         insertcount++;
         if(insert count == 10)
              this.sess.commit();
         }    
      }
      if(insert count > 0){
           this.sess.commit();
      }   
    }

This style slowly takes up lot of memory and after some times throws OutOfMemory exception. How can i improve performance here.

Answer 1

Is it your intention to commit after every 10 inserts? It looks like you only do so after the first 10 inserts. I guess that you would need something like

if ((insertCount % 10) == 0) {
   this.sess.commit();
}

Those uncommitted changes have to be stored somewhere. I'm not familiar with Ibatis but if the uncommitted changes are being stored in a buffer allocated by Ibatis then you will eventually run out of memory if you don't commit the changes.