Hi, I am developing a multithread server which works nice so far - 1 separate thread for client accepting, threadpool for data reading and processing. Today I have added new thread for doing some stuff and sending messages to client every 500 ms (just 2-5 messages). I have noticed quite massive slowdown but Im not sure why - its separate thread and its not due to iteration and locking collections, because when I add //before SendMessage call, it was still as fast as before. The SendMessage basically iterates all connected clients and for each of them calls SendData method which writes data to their networkstream. What am I missing? I still think those are different threads and I hope its not due to stream.write.. Thank you in advance!
A:
If you can try to post a code sample or a summary, your message sending implementation would make a good candidate.
First, purely general advice.
This is a good time to whip out a profiler. This kind of guessing is tempting, and often a good mental excercise, but most of the time programmers are wrong about what they think is making their software slow. A profiler will tell you, for example, if your program is spending 90% of its execution time inside of one method.
Second, a speculative guess.
It sounds like your message command runs off a timer. Make sure that you aren't having issues with reentrancy - for example if your sendmessage loop takes longer than 500ms to complete (and together with creating a new thread and multiple unpredictable latency network calls it could well do that), and you have the whole operation in a lock, then the timer will keep spawning off threadpool threads that are sitting in that lock waiting for the previous operation to complete - and there is a finite number of available threadpool threads. To check if this is a problem you don't even need a profiler, when latency gets bad pause the debugger and check up on your list of currently executing threads.
If this is the case consider doing something else - like have a single thread that runs in an infinite loop using a waithandle as a blocking mechanism and timer that sets the waithandle every 500ms.
But it will be much easier to help you if you post some code snippets, and run a profiler (Ants or DotTrace both work great).
Egor
2009-11-01 09:43:42
A:
Threads & threadpools for things like socket servers is the old way to do things. It's very unscalable (optimally you would like to not have more threads than cores), and full of locks.
Try converting your code to asynchronous code. You only need 1 thread, and you get callbacks whenever input arrives or when new data can be sent. The resulting code is much faster and doesn't have these bottleneck problems.
I know the advice of: no no, rewrite everything you should do it like this, is not really helpful, since it doesn't answer the exact question you asked. But if you do have the time, I still think it's a good advice. Or else, it's good advice for the next server you'll make ;^)
Toad
2009-11-01 10:01:07
I second this, with sockets using the asynchronous model is the way to go -http://msdn.microsoft.com/en-us/library/bbx2eya8.aspx. Using the sendCallBack you can make sure that you're not sending data to a socket before it had a chance to complete the previous operation on this socket. So, for example every 500ms you can fire off a socket send on every socket, unless the socket in question is still writing the previous message.
Egor
2009-11-01 11:14:45
Unless you have to do some IO which will block all other sockets from doing something useful. In this case it actually is better to have some threads, but probably not one per connection.
Glenner003
2009-11-02 09:16:39
It's strange you say you shouldn't have more threads than cores when the ATL:Threadpool (when left to its own devices) creates 2x number of cores you have in threads.
L2Type
2009-11-03 08:40:21
@DotNetWill: The advice to not have more threads than CPU's is only valid when you have cpu intensive threads, when there are other limiting elements such as IO it practical to have more threads as some threads will be waiting for input. But to have a thread per session can become dangerous when lots of clients connect and the cost of switching threads becomes just too much to cope with.
Glenner003
2009-11-03 12:47:24
glenner: this is not true. When using Asynchronous calls to do IO (including sockets, files, etc) the thread does NOT block. You'll receive a callback when data arrives or is ready to be sent. Asynchronous calls can be recognized with the begin.... and end... methods.
Toad
2009-11-03 12:58:51
reinier, by using Asynchronous calls you just are creating or using threads allowing yours to do something else. In a lot of cases there is nothing useful to do in the context of that thread anyway. Using asynchronous calls is perfect when you will have one worker thread (per CPU) to do it all and thats not always the most performant or manageable way to do things. You have to look at the whole picture before deciding what is the best strategy to follow. In this case it is pretty clear that the OP has chosen to create task specific threads instead of generic worker threads.
Glenner003
2009-11-04 10:50:49
Glenner: Asynchronous calls do NOT create threads. that's the whole point. The OS only calls your callback on an available threadpool thread. However during the asynchronous call NO extra thread is running.
Toad
2009-11-04 11:51:42
Hi Rennier,I'm sorry your right. It used to say that there were threads created in the .Net documentation, but it actually is relying on the OS async IO calls, which are interrupt steered.
Glenner003
2009-11-04 12:56:33
glenner no worries...usually the harder I shout, the less likely I'm right. Glad for once it's the other way around ;^)
Toad
2009-11-04 14:38:40
A:
Run your application with profiler (for example YourKit http://www.yourkit.com/dotnet/. It will immediately shown the reason of problem.
Serge
2009-11-03 08:27:47