I've got a number of tasks/servlets that are hitting the HardDeadlineExceededError which is leaving everything hanging in an 'still executing' state.
The work being done can easily exceed the 29 second threshold.
I try to catch the DeadlineExceededException and base Exception in order to save the exit state but neither of these exception handlers are being caught...
Is there a way to determine which tasks are in the queue or currently executing?
Are there any other strategies for dealing with this situation?
The situation I'm dealing with is documented under "The Request Timer" heading.
// task handler for retrieving information from external web services
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String taskRetryCountParam = req.getParameter( "X-AppEngine-TaskRetryCount" );
int taskRetryCount = (taskRetryCountParam==null) ? 0 : Integer.parseInt(taskRetryCountParam);
// look up the persistent 'task' and mark it as 'running'
logger.info( this.getClass().getName() + ".doPost( "+ taskId + " ) retryCount=" + taskRestryCount );
// Do lots of heavy lifting here
// like calling external web services using URL fetch service
// and saving the contents into our database.
// look up the persistent 'task' and mark it as 'completed'
} catch ( DeadlineExceededException deadline ) {
// got this deadline exception
// look up the persistent 'task' and mark it as 'errored - try again'
logger.warning( "DeadlineExceeded Exception while loading content " + deadline.getMessage() );
resp.setStatus( HttpServletResponse.SC_REQUEST_TIMEOUT );
}
} catch ( Exception unknown ) {
// got some unknown exception
// look up the persistent 'task' and mark it as 'errored - cancelled'
logger.severe( "General Exception while loading content exception:" + unknown.getMessage() );
resp.setStatus( HttpServletResponse.SC_OK );
}
}
Here is the log file entries when I get into this situation... It appears that my database transactions are taking too long when it comes time to .
W 05-30 12:42PM 09.535
Error for /loadstatus
com.google.apphosting.runtime.HardDeadlineExceededError: This request (083793d1091c2ca3) started at 2010/05/30 19:41:39.814 UTC and was still executing at 2010/05/30 19:42:09.529 UTC.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:443)
at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:326)
at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:217)
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:131)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:43)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:104)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:102)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:102)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:43)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:58)
at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:42)
at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:56)
at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:66)
at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
at org.datanucleus.jdo.JDOTransaction.commit(JDOTransaction.java:83)
at org.datanucleus.store.appengine.jdo.DatastoreJDOTransaction.commit(DatastoreJDOTransaction.java:56)
<snip>
C 05-30 12:42PM 09.629
Uncaught exception from servlet
com.google.apphosting.runtime.HardDeadlineExceededError: This request (083793d1091c2ca3) started at 2010/05/30 19:41:39.814 UTC and was still executing at 2010/05/30 19:42:09.529 UTC.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:443)
at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:326)
at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:217)
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:131)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:43)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:104)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:102)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:102)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:43)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:58)
at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:42)
at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:56)
at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:66)
at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
at org.datanucleus.jdo.JDOTransaction.commit(JDOTransaction.java:83)
at org.datanucleus.store.appengine.jdo.DatastoreJDOTransaction.commit(DatastoreJDOTransaction.java:56)
<snip>
W 05-30 12:42PM 09.644
A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)