Diagnosing Cause of 100% CPU Usage by "System" Process

Question

I have a Windows server application, implemented in C++ using the Win32 API, that does a lot of serial and TCP/IP communication. As it runs, CPU usage gradually increases, until it reaches 100%. Task Manager indicates that most (>75%) of the CPU usage is by the "System" process. If I kill my server process, then CPU usage returns to normal.

Are there any "easy" ways to diagnose exactly what the problem is?

I suspect that I/O connections are being opened and never closed, so the OS is spending more and more time servicing those requests, but I'd like to verify that is the case before I try to fix the problem.

---

Update: After playing around with xperf, I've found that the System process is spending more than half of its time in ntoskrnl.exe!KxWaitForSpinLockAndAcquire. I don't know anything about this, but the name of the function suggests to me that there might be a deadlock/contention issue.

Other functions that System is using a lot include NETIO.SYS!FilterMatchEnum, NETIO.SYS!MatchConditionOverlap, NETIO.SYS!IsFilterVisible, and MpNWMon.sys!NetFlowUpendByCompletionHandle.

Answer 1

I recommend checking out the sysinternals tools if you haven't already.

One tool there I like very much is the Handle tool, which shows you all the files that are open in the system.

Another one which seems directly applicable to your scenario is ProcDump, which allows you to dump process information when a given process exceeds x% of CPU usage.

Answer 2

What's your handle count - if that's rising, it would point to something not being closed.

Answer 3

Use process explorer, but open your own process rather than the System process. The system process is likely just responding to your requests. Since you don't have the code for it, you are going to have a hard time knowing what it's doing

for your own process:

• Open the performance tab that shows the handle counts etc,
• Open the tcp-ip tab that shows the connections.
• Check also the pane on the lower half of the process explorer for your process showing events, files, etc..

Hopefully you can see what resource is being over used from one of those.

Portmon may also be worth a try to see if the activity is as expected.

Answer 4

I suggest using sysinternal's Process Monitor and Portmon, which should show you the status of TCP connections and serial port activity.

If you see a correlation between that and the CPU usage then you have a good lead.

Answer 5

Userdump. Get a userdump of your process when it is in this situation. Then load the dump in winDbg and see what it is doing. One really useful command is !runaway. This will tell you which threads are using the most CPU. Then look at those threads to see what they are doing.