Oracle data provider pegs IIS worker process when web site is stopped

Question

We're experiencing a nasty issue in Oracle 11g Release 2 where the w3wp process takes over and entire processor core, and debugging shows that the Oracle data provider is throwing ThreadAbortExceptions infinitely. A developer found this issue by doing the following:

1) Browse a web site that uses Oracle data connections locally (http://localhost/OracleWebSite - we use IIS, not the ASP.NET dev server, for all of our sites). This ensures that the w3wp process is running and that an active Oracle pool of connections exists in the app pool.

2) Stop the web site (or perform a Rebuild All operation in Visual Studio on the web site in question).

Our Oracle connection handling in the affected applications (all Oracle web apps) is well-established and robust. This issue does not occur if we disable connection pooling. This issue does not occur in Oracle 11g Release 1.

Answer 1

Anything that triggers a recompile (web.config change, app_offline.htm, .aspx file change, etc.) causes the CPU usage on the core to max out. If you repeat the process, it maxes out the CPU usage on the next core, until the overall CPU usage is at 100%.

I hooked up windbg with sos extensions and it looks like for each maxed out core there is 1 thread stuck in System.AppDomain.Unload(System.AppDomain) and another stuck on Oracle.DataAccess.Client.OracleTuningAgent.DoScan().

First thread

• Oracle.DataAccess.Client.OracleTuningAgent.DoScan()
• Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction()
• System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
• System.Threading.ThreadHelper.ThreadStart()

Second thread

• System.AppDomain.Unload(System.AppDomain)
• System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain(System.Object)
• System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
• System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading. _ThreadPoolWaitCallback)
• System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)

It looks like AppDomain.Unload is waiting on OracleTuningAgent.DoScan to finish, but that thread is blocked or sleeping.

Oracle has confirmed the issue (bug # 9648040) and it is a top priority. In the meantime, the possible workarounds are:

1. Roll back to 11gR1/earlier client
2. Add 'Self Tuning=false' to the connection string. You will of course lose the benefits of the automatic tuning.

-Scott