Is Hibernate good for batch processing? What about memory usage?

Question

I have a daily batch process that involves selecting out a large number of records and formatting up a file to send to an external system. I also need to mark these records as sent so they are not transmitted again tomorrow.

In my naive JDBC way, I would prepare and execute a statement and then begin to loop through the recordset. As I only go forwards through the recordset there is no need for my application server to hold the whole result set in memory at one time. Groups of records can be feed across from the database server.

Now, lets say I'm using hibernate. Won't I endup with a bunch of objects representing the whole result set in memory at once?

Answer 1

Hibernate does also iterate over the result set so only one row is kept in memory. This is the default. If it to load greedily, you must tell it so.

Reasons to use Hibernate:

• "Someone" was "creative" with the column names (PRXFC0315.XXFZZCC12)
• The DB design is still in flux and/or you want one place where column names are mapped to Java.
• You're using Hibernate anyway
• You have complex queries and you're not fluent in SQL

Reasons not to use Hibernate:

• The rest of your app is pure JDBC
• You don't need any of the power of Hibernate
• You have complex queries and you're fluent in SQL
• You need a specific feature of your DB to make the SQL perform

Answer 2

In my opinion I would NOT use Hibernate, since it makes your application a whole lot bigger and less maintainable and you do not really have a chance of optimizing the generated sql-scripts in a quick way. Furthermore you could use all the SQL functionality the JDBC-bridge supports and are not limited to the hibernate functionality. Another thing is that you have the limitations too that come along with each layer of legacy code.

But in the end it is a philosophical question and you should do it the way it fits you're way of thinking best.

Answer 3

If there are possible performance issues then stick with the JDBC code.

There are a number of well known pure SQL optimisations which which would be very difficult to do in Hibernate.

Only select the columns you use! (No "select *" stuff ).

Keep the SQl as simple as possible. e.g. Dont include small reference tables like currency codes in the join. Instead load the currency table into memory and resolve currency descriptions with a program lookup.

Depending on the DBMS minor re-ordering of the SQL where predicates can have a major effect on performance.

If you are updateing/inserting only commit every 100 to 1000 updates. i.e. Do not commit every unit of work but keep some counter so you commit less often.

Take advantage of the aggregate functions of your database. If you want totals by DEPT code then do it in the SQL with " SUM(amount) ... GROUP BY DEPT ".

Answer 4

Hibernate as any ORM framework is intended for developing and maintaining systems based on object oriented programming principal. But most of the databases are relational and not object oriented, so in any case ORM is always a trade off between convenient OOP programming and optimized/most effective DB access.

I wouldn't use ORM for specific isolated tasks, but rather as an overall architectural choice for application persistence layer.

Answer 5

Hibernate offers some possibilities to keep the session small.

You can use Query.scroll(), Criteria.scroll() for JDBC-like scrolling. You can use Session.evict(Object entity) to remove entities from the session. You can use a StatelessSession to suppress dirty-checking. And there are some more performance optimizations, see the Hibernate documentation.