Spring + Hibernate transaction takes 25 seconds doing nothing

Question

I have a Java application using Spring + Hibernate. I have a pretty simple transaction that just recently started to take REALLY long to execute (~25 secs) though it's not making any obscure / complex queries, and according to the log, those 25 seconds are spent within Hiberante's code. The browser just hangs there waiting until it's done...

The transaction is done though annotations, using org.springframework.orm.hibernate3.LocalSessionFactoryBean as session factory and org.springframework.orm.hibernate3.HibernateTransactionManager as transaction manager.

For Hibernate's cache I'm using Memcached, but it shouldn't be an issue, according to the log at least it's not touching the cache in the meantime....

I here submit the relevant piece of the log:

 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Found thread-bound Session [org.hibernate.impl.SessionImpl@a04a093] for Hibernate transaction
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Using transaction object [org.springframework.orm.hibernate3.HibernateTransactionManager$HibernateTransactionObject@35ca1808]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Creating new transaction with name [com.quebicoca.api.service.PaymentService.buy]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Preparing JDBC Connection of Hibernate Session [org.hibernate.impl.SessionImpl@a04a093]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Exposing Hibernate transaction as JDBC transaction [com.mysql.jdbc.JDBC4Connection@66efd0ce]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Bound value [org.springframework.jdbc.datasource.ConnectionHolder@a75b92e] for key [org.springframework.jdbc.datasource.DriverManagerDataSource@1420ca8b] to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Initializing transaction synchronization
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionInterceptor] Getting transaction for [com.quebicoca.api.service.PaymentService.buy]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@3538ce1b] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] bound to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Found thread-bound Session [org.hibernate.impl.SessionImpl@a04a093] for Hibernate transaction
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder@a75b92e] for key [org.springframework.jdbc.datasource.DriverManagerDataSource@1420ca8b] bound to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Using transaction object [org.springframework.orm.hibernate3.HibernateTransactionManager$HibernateTransactionObject@36d0805f]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|HibernateTransactionManager] Participating in existing transaction
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionInterceptor] Getting transaction for [com.quebicoca.api.service.PaymentService.buy]
 [DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@3538ce1b] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] bound to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|HashCodeKeyStrategy] Transformed key [UserOrder_holders] to hashCode [-1959252437]
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|HashCodeKeyStrategy] Final cache key: [org.hibernate.cache.UpdateTimestampsCache:0:-1959252437]
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|MemcachedCache] Memcache.set(org.hibernate.cache.UpdateTimestampsCache:0:-1959252437)
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|SpyMemcache] MemcachedClient.set(org.hibernate.cache.UpdateTimestampsCache:0:-1959252437)
 ...
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|DispatcherServlet] Successfully completed request
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|TransactionSynchronizationManager] Removed value [org.springframework.orm.hibernate3.SessionHolder@5862af29] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] from thread [http-8080-25]
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|OpenSessionInViewFilter] Closing single Hibernate Session in OpenSessionInViewFilter
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|SessionFactoryUtils] Closing Hibernate Session
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|ExceptionTranslationFilter] Chain processed normally
 [DEBUG] [2010-10-09 @ 07:25:55] [http-8080-25|SecurityContextPersistenceFilter] SecurityContextHolder now cleared, as request processing completed
 [DEBUG] [2010-10-09 @ 07:25:56] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@3538ce1b] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] bound to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:56] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@3538ce1b] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] bound to thread [http-8080-12]
 [DEBUG] [2010-10-09 @ 07:25:56] [http-8080-12|HibernateTemplate] Found thread-bound Session for HibernateTemplate
 [DEBUG] [2010-10-09 @ 07:25:56] [http-8080-12|HashCodeKeyStrategy] Transformed key [UserOrder] to hashCode [-1966312925]
 [DEBUG] [2010-10-09 @ 07:25:56] [http-8080-12|HashCodeKeyStrategy] Final cache key: [org.hibernate.cache.UpdateTimestampsCache:0:-1966312925]

The last thing logged before the 25 seconds gap is:

[DEBUG] [2010-10-09 @ 07:25:30] [http-8080-12|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@3538ce1b] for key [org.hibernate.impl.SessionFactoryImpl@c4be179] bound to thread [http-8080-12]

I was the only one logged in during this log, I shut down access for everyone except myself from the Apache running in front of the Tomcat.

Any ideas are welcomed.

EDIT:

Added my Order class according to requested:

@Entity
@Table(name = "UserOrder")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
public class Order {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;

    @ManyToOne(optional = false)
    private Deal deal;
    @CollectionOfElements
    private Map<String, Integer> holders;
    @ManyToOne(optional = false)
    private User user;
    @org.hibernate.annotations.Type(type = "org.joda.time.contrib.hibernate.PersistentDateTime")
    @Column(nullable = false)
    private DateTime buyingDate;
    @Column(nullable = false)
    @Enumerated(EnumType.STRING)
    private OrderState state;
    private BigDecimal payedWithUserCredit;
    @Column(nullable = false)
    private int quantity;
    @Column(nullable = false)
    private String secret;

    private boolean mailSent;

    @Enumerated(EnumType.STRING)
    private PaymentMethod paymentMethod;
}

EDIT 2:

Afeter suggestions Spy Memcached may have something to do I decided to go back to EHCache.

It turned to be even worse (1:16 hanging). The log:

 [DEBUG] [2010-10-13 @ 04:28:47] [http-8080-1|HibernateTransactionManager] Triggering beforeCommit synchronization
 [DEBUG] [2010-10-13 @ 04:28:47] [http-8080-1|HibernateTransactionManager] Triggering beforeCompletion synchronization
 [DEBUG] [2010-10-13 @ 04:28:47] [http-8080-1|HibernateTransactionManager] Initiating transaction commit
 [DEBUG] [2010-10-13 @ 04:28:47] [http-8080-1|HibernateTransactionManager] Committing Hibernate transaction on Session [org.hibernate.impl.SessionImpl@20360e46]
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|HibernateTransactionManager] Triggering afterCommit synchronization
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|HibernateTransactionManager] Triggering afterCompletion synchronization
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|TransactionSynchronizationManager] Clearing transaction synchronization
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|TransactionSynchronizationManager] Removed value [org.springframework.jdbc.datasource.ConnectionHolder@30c2ea0b] for key [org.springframework.jdbc.datasource.DriverManagerDataSource@614a616] from thread [http-8080-1]
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|HibernateTransactionManager] Not closing pre-bound Hibernate Session [org.hibernate.impl.SessionImpl@20360e46] after transaction
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@839c6c7] for key [org.hibernate.impl.SessionFactoryImpl@6fbae5f5] bound to thread [http-8080-1]
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|DriverManagerDataSource] Creating new JDBC DriverManager Connection to [jdbc:mysql://localhost/quebicoca]
 [DEBUG] [2010-10-13 @ 04:30:03] [http-8080-1|TransactionSynchronizationManager] Retrieved value [org.springframework.orm.hibernate3.SessionHolder@839c6c7] for key [org.hibernate.impl.SessionFactoryImpl@6fbae5f5] bound to thread [http-8080-1]

I'm compeltely lost...

Answer 1

Figured it out!

In the @Transactional method doing this I would do something like this:

orderDAO.save(order);
user.addOrder(order);

So the order would be saved, and added to the user's Set of orders.

When the save was called, the insert for the order was placed, but not for the holder.

When exiting the method, Hibernate would go over all models in session to see what else needs to be persisted (which should indicate only the UserOrder_holder is remaining). It seems it would take FOREVER trying to figure this out with the User's Orders, and associated Deals (most of which are repeated among different Orders). And I'm not talking about a huge database, the query used to retrieve the orders would just return 17 rows from the database... It's just lame.

Anyway, making the Order's relation to the Deal lazy solved the issue, at least for now, t may resurface later on with more Orders.