ansaurus

Question

OutOfMemory when reading big amounts of data using hibernate.

Answer 1

A:

Can you post the Exception stacktrace? It may be solved by passing suitable JVM options for GC.

I think this is related - http://stackoverflow.com/questions/1707350/java-stringbuilder-huge-overhead.

Looks from the StackTrace that a very large String is being created and causing the exception.

Padmarag 2010-02-11 07:59:45

Have you tried storing in LOB and using Streams for output?

Padmarag 2010-02-11 08:40:38

Answer 2

+1 A:

At the risk of appearing stupid - have you considered doing this another way?

Personally I would avoid doing batch processing that "far away" from the database. I don't know what database you're using but there's usually a mechanism for efficiently pulling a dataset out of the database & into a file even if it involves moderately simple manipulation on the way out. Stored procedures, specific export utilities. Investigate what else is available from your database vendor.

mcottle 2010-02-11 08:34:26

Answer 3

+1 A:

It looks like you are calling getProductIterator() with the starting and ending row numbers, while getProductIterator() is expecting the starting row and a row count. As your "upper limit" gets higher you are reading data in bigger chunks. I think you mean to pass batchSize as the second argument to getProductIterator().

KeithL 2010-02-11 18:24:19

Oh, yes! Stupid me. You're 100% right:)

Vladimir 2010-02-12 13:57:32

Answer 4

A:

Not a direct answer but for this kind of data manipulation, I would use the StatelessSession interface.

Pascal Thivent 2010-02-11 18:35:12

Answer 5

+1 A:

KeithL is right - you're passing an ever-increasing limit. But breaking it up that way doesn't make sense anyway. The whole point of a scroll cursor is that you process a row at a time so there's no need to break it up into chunks. The fetch size reduces the trips to the database at the cost of using up more memory. The general pattern should be:

Query q = session.createCriteria(... no offset or limit ...);
q.setCacheMode(CacheMode.IGNORE);
q.setFetchSize(1000);  // experiment with this to optimize performance vs. memory
ScrollableResults iterator = query.scroll(ScrollMode.FORWARD_ONLY);
while (iterator.next()) {
  Product p = (Product)iterator.get();
  ...
  // session.evict(p);  // an alternative to setting the cache mode above
}

That said, the error is getHtmlSources so the problem may be completely unrelated to the session/cursor/scroll issue. If those html strings are huge and they're being referenced the entire time, you may just be running out of contiguous memory.

Btw, I don't see a getScrollableResults method on ScrollableResults.

Brian Deterling 2010-02-11 19:38:12

ansaurus

tags:

views:

answers:

OutOfMemory when reading big amounts of data using hibernate.

related questions