Clojure lots of threads

Question

I just got done watching Rick Hickey's "Clojure Concurrency" talk, and I have a few questions about threads.

Let's say I have a situation with lots of Agents, let's say 10,000 of them running one machine. I'd rather not have 10,000 CPU threads running at once, but I don't want threads to be blocked by the actions of other threads.

In this example I won't really be waiting for replies, instead each Agent will be sending a message or two, and then waiting until it gets a message.

How would I structure a program like this without getting 10k OS threads which would probably end up slowing the system down.

Answer 1

Keep in mind that Clojure is running on top of a JVM. So you may have 10,000 Java threads, but that doesn't equate to 10,000 OS processes. I suspect that the garbage collector could end up being your bottleneck, so I would focus on tuning the footprints of each agent. (It goes without saying that you should test and verify this before tuning.)

Answer 2

Check out this article on agents and thread pools.