The more threads that changes the Clojure's ref are, the more does the rate of retries per threads rise?

Question

I worry about this a little.
Imagine the simplest version controll way that programmers just copy all directory from the master repository and after changing a file do reversely if the master repository is still the same. If it has been changed by another, they must try again.

When the number of programmers increases, it is natural that retries also increase, but it might not be proportional to the number of programmers.
If ten programmers work and a work takes an hour per person, to complete all work ten hours are needed at least.
If they are earnest, about 9 + 8 + 7 + ... 1 = 45 man-hours come to nothing.
In a hundread of programmers, about 99 + 98 + ... 1 = 4950 man-hours come to nothing.

I tried to count the number of retries and got the results.

Source

(defn fib [n] 
   (if  (or (zero? n) (= n 1)) 
       1 
      (+  (fib (dec n) )  (fib (- n 2))))) 

(defn calc! [r counter-A counter-B counter-C n]
  (dosync
   (swap! counter-A inc)
   ;;(Thread/sleep n)
   (fib n)
   (swap! counter-B inc)
   (alter r inc)
   (swap! counter-C inc)))

(defn main [thread-num n]
  (let [r (ref 0)
        counter-A (atom 0)
        counter-B (atom 0)
        counter-C (atom 0)]
    (doall (pmap deref
              (for [_ (take thread-num (repeat nil))]
                (future (calc! r counter-A counter-B counter-C n)))))
    (println thread-num " Thread. @ref:" @r)
    (println "A:" @counter-A ", B:" @counter-B ", C:" @counter-C)))

CPU: 2.93GHz Quad-Core Intel Core i7
result

user> (time (main 10 25))
10  Thread. @ref: 10
A: 53 , B: 53 , C: 10
"Elapsed time: 94.412 msecs"
nil
user> (time (main 100 25))
100  Thread. @ref: 100
A: 545 , B: 545 , C: 100
"Elapsed time: 966.141 msecs"
nil
user> (time (main 1000 25))
1000  Thread. @ref: 1000
A: 5507 , B: 5507 , C: 1000
"Elapsed time: 9555.165 msecs"
nil

I changed the job to (Thread/sleep n) instead of (fib n) and got similar results.

user> (time (main 10 20))
10  Thread. @ref: 10
A: 55 , B: 55 , C: 10
"Elapsed time: 220.616 msecs"
nil
user> (time (main 100 20))
100  Thread. @ref: 100
A: 689 , B: 689 , C: 117
"Elapsed time: 2013.729 msecs"
nil
user> (time (main 1000 20))
1000  Thread. @ref: 1000
A: 6911 , B: 6911 , C: 1127
"Elapsed time: 20243.214 msecs"
nil

In Thread/sleep case, I think retries could increase more than this result because CPU is available.
Why don't retries increase?

Thanks.

Answer 1

Because you are not actually spawning 10, 100 or 1000 threads! Creating a future does not always create a new thread. It uses a thread pool behind the scenes where it keeps queuing the jobs (or Runnables to be technical). The thread pool is a cached thread pool which reuses the threads for running the jobs.

So in your case, you are not actually spawning a 1000 threads. If you want to see the retries in action, get a level below future - create your own thread pool and push Runnables into it.

Answer 2

self answer

I have modified main function not to use pmap and got results, which work out as calculated.

(defn main [thread-num n]
  (let [r (ref 0)
        counter-A (atom 0)
        counter-B (atom 0)
        counter-C (atom 0)]
    (doall (map deref (doall (for [_ (take thread-num (repeat nil))]
                   (future (calc! r counter-A counter-B counter-C n))))))
    (println thread-num " Thread. @ref:" @r)
    (println "A:" @counter-A ", B:" @counter-B ", C:" @counter-C)))

fib

user=> (main 10 25)
10  Thread. @ref: 10
A: 55 , B: 55 , C: 10
nil
user=> (main 100 25)
100  Thread. @ref: 100
A: 1213 , B: 1213 , C: 100
nil
user=> (main 1000 25)
1000  Thread. @ref: 1000
A: 19992 , B: 19992 , C: 1001
nil

Thread/sleep

user=> (main 10 20)
10  Thread. @ref: 10
A: 55 , B: 55 , C: 10
nil
user=> (main 100 20)
100  Thread. @ref: 100
A: 4979 , B: 4979 , C: 102
nil
user=> (main 1000 20)
1000  Thread. @ref: 1000
A: 491223 , B: 491223 , C: 1008
nil