How many threads is too many?

views:

3107

answers:

+14 Q:

How many threads is too many?

I am writing a server, and I branch each action off into a thread when the request is incoming. I do this because almost every request is a database query. I am using a threadpool library to cut down on construction/destruction of threads.

My question is though - what is a good cutoff point for I/O threads like these? I know it would just be a rough estimate, but are we talking hundreds? thousands?

EDIT:

Thank you all for your responses, it seems like I am just going to have to test it to find out my thread count ceiling. The question is though: how do I know I've hit that ceiling? What exactly should I measure?

As many threads as the CPU cores is what I've heard very often.

masfenix 2009-01-27 00:48:45

This is extremely wrong.

Geoffrey Chetwood 2009-01-27 00:50:15

@Rich, at least explain why:-). This rule-of-thumb only applies when all threads are CPU-bound; they get one 'CPU' each. When many of the threads are I/O bound, it's usually better to have many more threads than 'CPU's (CPU is quoted since it applies to physical threads of execution, eg cores).

paxdiablo 2009-01-27 00:54:53

@Pax: This is not at all how threading works. The number of CPUs has no bearing. Either you need a new thread or not.

Geoffrey Chetwood 2009-01-27 00:57:37

You can't run multiple threads on different CPUs/cores in parallel, not in Python! Global Interpreter Lock is there to block it!

Abgan 2009-01-27 01:04:15

@Abgan, I wasn't sure about that, thinking maybe Python would create "real" OS threads (run on multiple CPUs). If what you say is true (I have no reason to doubt), then CPU quantity has no bearing - threading is useful then only when most threads are waiting for something (eg DB I/O).

paxdiablo 2009-01-27 01:17:13

@Rich: when (real) threading, CPU count DOES have bearing since you can run multiple non-waiting threads truly concurrently. With one CPU, only one runs and the benefit accrues from having many other threads waiting for a non-CPU resource.

paxdiablo 2009-01-27 01:18:46

@Pax: You don't understand the concept of thread pools then I guess.

Geoffrey Chetwood 2009-01-27 01:21:00

@Rich, I understand thread pools fine; it appears I (and others here) also understand hardware better than you. With one CPU, only one execution thread can run, even if there's others waiting for a CPU. Two CPUs, two can run. Iff all threads are waiting for a CPU, ideal thread count is equal to...

paxdiablo 2009-01-27 01:27:35

...the number of CPUs. When threads are waiting for a non-CPU resource, ideal number is higher.

paxdiablo 2009-01-27 01:28:29

@Pax: That doesn't handle the problem of a queue for incoming data requests.

Geoffrey Chetwood 2009-01-27 01:29:09

@Rich, see my comment to your answer, 'nuff said.

paxdiablo 2009-01-27 01:31:08

I've heard the same thing. On one of many of Jeffrey Richter's interviews he states the same.

Chuck Conway 2009-01-27 02:18:12

lol i got voted down to obvilion. Obviously I didnt know so correct me>

masfenix 2009-01-27 05:40:13

@masfenix: Of course you got downvoted. First of all, your answer makes absolutely /no/ sense in relation to thread pools. Second, your answer makes no sense even in the context of regular threading.

Geoffrey Chetwood 2009-01-27 17:35:17

+26 A:

Measure, don't guess. One suggestion is to make it configurable and initially set it to 100, then release your software to the wild and monitor what happens.

If your thread usage peaks at 3, then 100 is too much. If it remains at 100 for most of the day, bump it up to 200 and see what happens.

You could actually have your code itself monitor usage and adjust the configuration for the next time it starts but that's probably overkill.

EDIT 1 for clarification and elaboration:

Despite the somewhat animated discussions taking part on this question, I still believe this answer to be relevant.

I'm not advocating rolling your own thread pooling subsystem, by all means use the one you have. But, since you were asking about a good cut-off point for threads, I assume your thread pool implementation has the ability to limit the maximum number of threads created (which is a good thing).

I may be assuming too much, but I've written thread and database connection pooling code and they have the following features (which I believe are essential for performance):

a minimum number of active threads.
a maximum number of threads.
shutting down threads that haven't been used for a while.

The first sets a baseline for minimum performance in terms of the thread pool client (this number of threads is always available for use). The second sets a restriction on resource usage by active threads. The third returns you to the baseline in quiet times so as to minimise resource use.

You need to balance the resource usage of having unused threads (A) against the resource usage of not having enough threads to do the work (B).

(A) is generally memory usage since a thread doing no work will not be using much of the CPU. (B) will generally be a delay in the processing of requests as they arrive as you need to wait for a thread to become available.

That's why you measure. As you state, the vast majority of your threads will be waiting for a response from the database so they won't be running. There are two factors that affect how many threads you should allow for.

The first is the number of DB connections available. This may be a hard limit unless you can increase it at the DBMS - I'm going to assume your DBMS can take an unlimited number of connections in this case (although you should ideally be measuring that as well).

Then, the number of threads you should have depend on your historical use. The minimum you should have running is the minimum number that you've ever had running + A%, with an absolute minimum of (for example, and make it configurable just like A) 5.

The maximum number of threads should be your historical maximum + B%.

You should also be monitoring for behaviour changes. If, for some reason, your usage goes to 100% of available for a significant time (so that it would affect the performance of clients), you should bump up the maximum allowed until it's once again B% higher.

EDIT 2 in response to the "what exactly should I measure?" question:

What you should measure specifically is the maximum amount of threads in concurrent use (e.g., waiting on a return from the DB call) under load. Then add a safety factor of 10% for example (emphasised, since other posters seem to take my examples as fixed recommendations).

In addition, this should be done in the production environment for tuning. It's okay to get an estimate beforehand but you never know what production will throw your way (which is why all these things should be configurable at runtime). This is to catch a situation such as unexpected doubling of the client calls coming in.

paxdiablo 2009-01-27 00:51:10

If threads are spawned on incoming requests then thread-usage will mirror the number of unserviced requests. There's no way to determine the "optimal" number from this.Indeed you will find more threads cause more resource contention and thus the number of active threads will increase.

Andrew Grant 2009-01-27 01:34:13

@Andrew, thread creation takes time, and you *can* determine the optimal number based on historical data [+ N%] (hence measure, don't guess). In addition, more threads only cause resource contention when they're doing work, not waiting on a signal/semaphore.

paxdiablo 2009-01-27 01:38:24

Where is this data on 'thread creation' causing a performance problem when using a thread pool? A good thread pool would not be creating and destroying threads in between tasks.

Geoffrey Chetwood 2009-01-27 01:40:07

@Pax If all your threads are waiting on the same semaphores to run DB queries then that's the very definition of contention. It's also not true to say threads don't cost anything if they're waiting on a semaphore.

Andrew Grant 2009-01-27 01:46:32

@Andrew, I can't see why'd you'd semaphore-block the DB queries, any decent DB will allow concurrent access, with many threads waiting on the responses. And threads shouldn't cost any *execution time* while semaphore-blocked, they should sit in the blocked queue until the semaphore is released.

paxdiablo 2009-01-27 01:50:52

I am just extremely glad that servers with similar functionality and behaviors like IIS or Apache would not follow this advice. Just thinking about the mechanics of it is laughable.

Geoffrey Chetwood 2009-01-27 01:51:58

Your opinion, Rich, and you seem to be in the minority. Neither of us seems to be going to move on this issue, so let's just agree to leave it there. Arguments become counter-productive at some point, and I think this one has reached that point. Have the last word if you wish, I will not respond.

paxdiablo 2009-01-27 01:55:05

One thing to consider is how many cores exist on the machine that will be executing the code. That represents a hard limit on how many threads can be proceeding at any given time. However, if, as in your case, threads are expected to be frequently waiting for a database to execute a query, you will probably want to tune your threads based on how many concurrent queries the database can process.

ElectricDialect 2009-01-27 00:51:42

um, no. The whole point of threads was (before multicore and multiple processors became prevalent) is to be able to mimic having multiple processors on a machine that has just one. That's how you get responsive user interfaces-- a main thread and ancillary threads.

mmr 2009-01-27 00:53:56

@mmr: Um no. The idea of threads is to allow for blocking I/O and other tasks.

Geoffrey Chetwood 2009-01-27 00:55:30

The statement I made was that the number of cores on a machine represents a hard limit on the number of threads that can be doing work at a given time, which is a fact. Of course other threads can be waiting on I/O operations to complete, and for this question that is an important consideration.

ElectricDialect 2009-01-27 01:01:26

@ElectricDialect: What you are saying has no relevance to a thread pool.

Geoffrey Chetwood 2009-01-27 01:02:35

Anyway - you have GIL in Python, which makes threads only theoretically parallel. No more than 1 thread can run simultaneously, so it's only the responsiveness and blocking operations that matters.

Abgan 2009-01-27 01:02:43

@Rich B -- umm, no. Os/2 was sold as a multithreaded OS because you could have a responsive UI with threads. Blocking I/O is just one aspect of not having a responsive UI; basically, you just agreed with what I said, without having the history to understand it.

mmr 2009-01-27 01:05:49

@mmr: I would never say what you said because so far you haven't said anything accurate.

Geoffrey Chetwood 2009-01-27 01:08:14

@Rich B-- um, what? Have you done any UI coding with serious, hard work done in the background? Blocking I/O is just one task, and there can be many tasks (as you said)-- having a responsive UI means threading when you do those tasks. As for OS/2: http://www.os2hq.com/os2faq.htm

mmr 2009-01-27 01:14:13

@mmr: Your comments like your answer have nothing at all to do with the topic of thread pools. I am not sure what you don't understand here, but I assure I and most other people here are fully aware of what OS2 is. I must say I am a bit intrigued by the fact you keep referencing it though.

Geoffrey Chetwood 2009-01-27 01:18:15

@RichB-- Because I was pointing out the flaw of assuming that the number of threads should be linked to the number of cores (ie, that simultaneous execution is the only value to threads). And I maintain that a thread pool is not the right answer to this problem, but that's just my preference.

mmr 2009-01-27 01:23:52

@mmr: If you don't think a thread pool is the answer to handling incoming server requests to external I/O, then I guess I have nothing left to say to you. That says it all.

Geoffrey Chetwood 2009-01-27 01:25:01

+1 For actually understanding how computers work. @mmr: You need to understand the difference between appears to have multiple processors, and does have multiple processors. @Rich B: A thread pool is just one of many ways to handle a collection of threads. It is a good one, but certainly not the only one.

grieve 2009-04-17 05:19:15

In most cases you should allow the thread pool to handle this. If you post some code or give more details it might be easier to see if there is some reason the default behavior of the thread pool would not be best.

You can find more information on how this should work here: http://en.wikipedia.org/wiki/Thread_pool_pattern

Geoffrey Chetwood 2009-01-27 00:53:37

@Rich, you appear to be rubbing quite a few people up the wrong way :-). We'll have to let the votes decide, I guess (agree to disagree).

paxdiablo 2009-01-27 01:30:06

@Pax: This would not be the first time the majority of people didn't want to answer the question at hand (or understand it). I am not worried.

Geoffrey Chetwood 2009-01-27 01:30:59

I think this is a bit of a dodge to your question, but why not fork them into processes? My understanding of networking (from the hazy days of yore, I don't really code networks at all) was that each incoming connection can be handled as a separate process, because then if someone does something nasty in your process, it doesn't nuke the entire program.

mmr 2009-01-27 00:55:16

For Python that's especially true, as multiple processes can run in parallel, while multiple threads - don't. The cost is however quite high. You have to start new Python interpreter each time, and connect to DB with each process (or use some pipes redirection, but it also comes at a price).

Abgan 2009-01-27 01:05:29

Switching between processes is - most of the time - more expensive than switching between threads (whole context switch instead of some registers).At the end it depends heavily on your threading-lib. As the questions revolved around threading, I assume that processes are out of question already.

Leonidas 2009-01-27 01:15:52

Fair enough. I'm not sure why that's why I'm getting a -2 ding to the score, though, unless people really want to see thread-only answers, rather than including other answers that work.

mmr 2009-01-27 01:15:56

@mmr: Considering the question was about /thread/ pools, yes, I think people should be expecting an answer about threads.

Geoffrey Chetwood 2009-01-27 01:19:30

Process creation can be done once at startup (ie, a process pool instead of a thread pool). Amortized over the application duration, this may be small. They can't share info easily but it DOES buy them the possibility of running on multi-CPUs so this answer is useful. +1.

paxdiablo 2009-01-27 01:22:07

@Rich, I've answered many questions about raw JS with the comment "use jQuery" even when they specifically excluded jQuery as an option, the intent being to change their mind. The answer just has to be useful, not 100% relevant.

paxdiablo 2009-01-27 01:23:51

+3 A:

If your threads are performing any kind of resource-intensive work (CPU/Disk) then you'll rarely see benefits beyond one or two, and too many will kill performance very quickly.

The 'best-case' is that your later threads will stall while the first ones complete, or some will have low-overhead blocks on resources with low contention. Worst-case is that you start thrashing the cache/disk/network and your overall throughput drops through the floor.

A good solution is to place requests in a pool that are then dispatched to worker threads from a thread-pool (and yes, avoiding continuous thread creation/destruction is a great first step).

The number of active threads in this pool can then be tweaked and scaled based on the findings of your profiling, the hardware you are running on, and other things that may be occurring on the machine.

Andrew Grant 2009-01-27 01:03:53

He /is/ running in a pool.

Geoffrey Chetwood 2009-01-27 01:32:51

Yes, and it should be used in conjunction with a queue or pool of requests.

Andrew Grant 2009-01-27 01:35:24

@Andrew: Why? It should add a task to the thread pool each time it receives a request. It is up to the thread pool to allocate a thread for the task when there is one available.

Geoffrey Chetwood 2009-01-27 01:41:44

So what do you do when you have hundreds of requests coming in and are out of threads? Create more? Block? Return an error?Place your requests in a pool that can be as large as need be, and then feed these queued requests to your thread pool as threads become free.

Andrew Grant 2009-01-27 01:51:08

@Andrew: Again, your thread pool should be handling this.

Geoffrey Chetwood 2009-01-27 01:54:17

"a number of threads are created to perform a number of tasks, which are usually organized in a queue. Typically, there are many more tasks than threads. As soon as a thread completes its task, it will request the next task from the queue until all tasks have been completed."

Geoffrey Chetwood 2009-01-27 01:54:59

@Andrew: I am not sure what python thread pool the OP is using, but if you want a real world example of this functionality I am describing: http://msdn.microsoft.com/en-us/library/system.threading.threadpool.aspx

Geoffrey Chetwood 2009-01-27 02:02:17

+2 A:

As Pax rightly said, measure, don't guess. That what I did for DNSwitness and the results were suprising: the ideal number of threads was much higher than I thought, something like 15,000 threads to get the fastest results.

Of course, it depends on many things, that's why you must measure yourself.

Complete measures (in French only) in Combien de fils d'exécution ?.

bortzmeyer 2009-01-27 08:42:24

15,000? That's a tad higher than I would have expected as well. Still, if that's what you got, then that's what you got, I can't argue with that.

paxdiablo 2009-01-27 12:46:15

For this specific application, most threads are just waiting a response from the DNS server. So, the more parallelism, the better, in wall-clock time.

bortzmeyer 2009-01-27 16:46:14

ryeguy, I am currently developing a similar application and my threads number is set to 15. Unfortunately if I increase it at 20, it crashes. So, yes, I think the best way to handle this is to measure whether or not your current configuration allows more or less than a number X of threads.

hyperboreean 2009-01-27 12:36:16

+2 A:

One thing you should keep in mind is that python (at least the C based version) uses what's called a global interpreter lock that can have a huge impact on performance on mult-core machines.

If you really need the most out of multithreaded python, you might want to consider using Jython or something.

Chad Okere 2009-09-18 06:01:59

After reading this, I tried running sieve of Eratosthenes tasks on three threads. Sure enough, it was actually 50% *slower* than running the same tasks in a single thread. Thanks for the heads up. I was running Eclipse Pydev on a virtual machine that was allocated two CPU's. Next, I'll try a scenario that involves some database calls.

Don Kirkby 2010-05-28 00:31:46

ansaurus

tags:

views:

answers:

How many threads is too many?

EDIT:

related questions