Am I deadlocking? Why?

Question

I have a Silverlight 3 application which needs to call WCF service. The WCF service in turn calls an ASMX web service. When the WCF service call completes, the silverlight UI needs to be updated.

WCF is being called in async.

The thing is that the Silverlight app needs to call the WCF method(which later calls asmx internally) hundreds of time. I understand that because it is in aSync, hundreds of threads will be spawned, so I have coded in a check to ensure not more than X time does the WCF function gets called. Only when 1 call complets, do I add one more call. I hope this will keep a check on the total number of threads. What is the ideal value of that X value? I am ona simple XP machine with dual core CPU and 4 GB ram.

As the WCF calls get completed, I need to show progress bar on the silverlight UI.

This works fine for smaller number of calls, but when i need to make say around 10000 calls, after some time I get a timeout in the Silverlight's WCFCompleted method. I feel I may be going in a deadlock?

My WCF is configured to be multiple concurrent.

Every time one WCF call completes, it updates the UI... can that be causing this deadlock?

Any ideas anyone? I am stuck bad and lost here.

Answer 1

Deadlock is only caused when one thread allocates a resource and waits on another, while a different thread allocates the second resource and waits on the first. Neither one can continue.

The problem is that when multiple threads are calling the Windows UI, none of them are aware of the internal resources that Windows is trying to use. It's impossible to predict what the outcome might be; deadlock is one possibility.

Answer 2

So hundreds of threads are not being spun up unless you are doing it. The point of the async call is that the call is made and nothing is in existence (since nothing could happen) until the operation is finished and posted to the completion port. This assumes that you are using callbacks. If you call EndXXX on an IAsyncResult the thread will block until the operation completes.

Now a couple of things could be happening. First if you have a complicated service backend you could have a deadlock somewhere in your service. However second and probably more likely your WCF service configuration needs tuning. WCF tuning is something of a black art that once you understand you generally swear that you will never use WCF again.

How many outbound connections are simultaneously allowed on your client and conversely inbound connections allowed on your service? Using WCF tracing on both the client and the server is usually what you need to solve these types of problems. I recommend starting there as i will enable you to get the detail you need to see what is actually going on.

Answer 3

I would be doubtful that you are encountering a threading deadlock in this scenario. WPF, and I also believe Silverlight, use a threaded dispatcher to dispatch messages from worker threads to the UI. Its one of the reasons why such UI's are so responsive. What is more likely is you are overloading the WCF endpoint stack, its buffers, or the ASP.NET pipeline stack and/or buffers. When it comes to handling network connections, there are limitations on how rapidly and frequently you can create a considerable amount of connections. If you are creating 10,000 connections within a couple minutes of each other, it is also entirely possible that you are overrunning the TCP stack. When a socket is closed, it is not immediately reusable, and enters a final wait state for around 4 minutes. If you fire off several batches of 10,000 connections within a period of 4 minutes, you'll consume all of the available sockets, leaving WCF (and any other windows application outside of those with reserved ports) unable to communicate. I can't be certain you are encountering this (you really need to use a lot of sockets), but I answered another question in detail about how to increase the default thresholds for the windows TCP stack here:

WCF: System.Net.SocketException - Only one usage of each socket address (protocol/network address/port) is normally permitted

If you really wish to get down to the bottom of the issue, I recommend enabling WCF trace logging. The WCF trace logs capture a considerable amount of information, and provide extremely detailed information about each and every connection and message processed. Once enabled, you should be able to identify the problem unless it is occurring in the ASP.NET pipeline or HTTP.sys before even reaching WCF. I recommend reading the following articles:

• Configuring Tracing
• Configuring Message Logging
• Recommended Settings for Tracing and Message Logging
• Auditing and Logging
• Service Trace Viewer Tool (SvcTraceViewer.exe)

Answer 4

I later seemingly solved the issue by : 1) converted asmx service to WCF 2) Increased WCF timeouts from 1 minute to 5 minute 3) Instead of updating UI progress bar on every Completed callback, I am now using a timer which updated UI say every 30 seconds only.