C# service with Oracle .NET provider gets slower and slower

Question

I have a C# service written for .NET 2.0 that uses the Oracle data access provider for .NET 2.102.2.20. The service runs multiple threads and runs lots of queries to an Oracle 9.2 database. I am using NHibernate.

What I am seeing is that when it starts up it runs fast, then gets slower and slower. The CPU usage starts low then goes up and up. After a few minutes it is crawling and the CPU is at 100%. I have looked in my code and find nothing that could be doing this. Percent time in GC is < 5%. I have tried varying the ODP.NET parameters to no avail.

Anyone have an idea what could be doing this?

More detail: The threads are worker threads and need to keep running all the time. I have no runaway threads, they are doing real work. I have profiled the program and it looks like it is spending a lot of its time inside the Oracle provider, which you would expect, but why would it use so much CPU? It's as if it's spinning waiting for resultsets or something, but it doesn't happen right away, only after a while.

Update: It may be something to do with the .NET CLR on the server. A multi-threaded test program that does a lot of string manipulation also shows the same behavior on this machine, starting out fast, then slowing down over the course of 15 minutes to about 1/3 of the speed. The test program does not show this slowdown behavior on a another identical server with the same OS version and the same (we think) .NET CLR version.

Update: It is now looking like this is a problem with the server overheating and thermal protection kicking in and slowing down the CPUs' frequency.

Update: A firmware update for the server fixed this. I still think it was an overheating problem because if I stopped running the process and let the server "rest" for a little while it would start running fast when I restarted the process, but if I killed the process and restarted it right away the process would start already running slow. My guess is that the firmware control for the fans was malfunctioning so the CPUs would heat up and slow down. If I stopped running for a little while they would cool down and run fast again, until they heated up again.

Answer 1

Sounds like your threads aren't terminated and keep running all the time or you've some serious memory leaks. Without more information, this could be anything.

Answer 2

If CPU is at 100% you're probably facing a runaway thread. You can diagnose this using WinDbg + SoS. Attach to the process and use the !runaway command to get an overview of how much CPU each thread is using. Then use !clrstack to find out what the runaway thread is doing. Let me know if you need additional details.

Answer 3

Hi Carlos,

I'm getting the exact same issue. The OracleConnection gets progressively slower and slower. What's interesting is that if I call:

cn.Close(); OracleConnection.ClearPool(cn);

every time, it never slows down.

It must have something to do with the oracle connection (caching??)