views:

477

answers:

1

We are using Quartz with Spring and our configuration is throwing deadlocks when quartz has more than 1 thread configured. I'm starting to believe that it's because we don't have our quartz configured correctly with Spring, but I can't find enough documentation on how to configure the two to play nicely.

We are running on both Windows and Linux environments - pointing at MSSQL and Oracle DBs. With both OS, using either DB, we can throw the following deadlock errors...

We're consistently throwing these deadlock errors. We run under heavy load, inserting hundreds of quartz triggers in a matter of minutes.

2010-03-17 18:52:31,737 [] [] ERROR [DFScheduler_Worker-42] core.ErrorLogger core.ErrorLogger (QuartzScheduler.java:2185) - An error occured while marking executed job complete. job= 'BPM.6e41a6567f0000020100362a51dc7a50'

org.quartz.JobPersistenceException: Couldn't remove trigger: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. [See nested exception: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1469)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2978)at org.quartz.impl.jdbcjobstore.JobStoreSupport$39.execute(JobStoreSupport.java:2962) at org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3713)at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3747)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3709)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2958)at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1727)at org.quartz.core.JobRunShell.run(JobRunShell.java:273)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534)

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(Unknown Source at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown Source) at 
...
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteSimpleTrigger(StdJDBCDelegate.java:1820) at org.quartz.impl.jdbcjobstore.JobStoreSupport.deleteTriggerAndChildren(JobStoreSupport.java:1345 at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1453 ... 9 more

This is what my quartz.properties file looks like:

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 50
org.quartz.threadPool.threadPriority = 5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = false
org.quartz.jobStore.selectWithLockSQL = SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?
A: 

Do you use a SchedulerFactoryBean? The documentation is: http://static.springsource.org/spring/docs/2.5.x/reference/scheduling.html It would completely remove the need for the actual quartz.properties and you can use the datasource you use in the rest of the application. This has obvious advantages (like reusable connections in a connection pool) and at least for me offered a somewhat easier time debugging.

Here's a snippet of how mine looked like in an older project:

<bean id="schedulerFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="quartzProperties"> <!--  quartz attributes, configurable -->
        <value>
            org.quartz.scheduler.rmi.export = false
            org.quartz.scheduler.rmi.proxy = false
            org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
            org.quartz.threadPool.threadCount = ${threadCount}
            org.quartz.threadPool.threadPriority = 5
            org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true
            org.quartz.jobStore.useProperties = false
            org.quartz.jobStore.driverDelegateClass = ${driverDelegate}
            org.quartz.jobStore.tablePrefix = QRTZ_
            org.quartz.jobStore.isClustered = false
        </value>
    </property>
    <property name="autoStartup" value="false" /> 
    <property name="waitForJobsToCompleteOnShutdown" value="false" />
    <property name="dataSource" ref="dataSource" />
</bean>

and the datasource the usual:

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    ....
    <property name="validationQuery" value="SELECT 1" />
    <property name="testWhileIdle" value="true" />
    <property name="testOnBorrow" value="true" />
    <property name="testOnReturn" value="true" />
</bean>

Obviously the quartzProperties are similar to the ones in the file, but I've found the general behavior of the application better once I switched from the quarts.properties to the Spring classes. It also helped that I had only one datasource actually accessing the database.

laura
We've been using the spring classes too. However, I'll try your configuration as well. I notice that you don't have a non-transactional datasource configured. Why is that? I'm just trying to wrap my head around all this.
Khandelwal
Interesting question. Having not actually done the datasource setup I can't really answer it though. The particular project which contained these snippets needed a transactional datasource due to the type of db queries it had, but the other projects used the same setup (one transactional datasource per project).
laura