Python multiprocessing easy way to implement a simple counter?

Question

Hey everyone, I am using multiprocessing in python now. and I am just wondering whether there exists some sort of simple counter variable that each process when they are done processing some task could just increment ( kind of like how much work done in total).

I looked up the API for Value, don't think it's mutable.

Answer 1

If you are using this variable to maintain which processes can access which resources, you might be interested in a semaphore.

Otherwise, you will need to have a global variable 'count', have each process lock it, increment it, then release it, such as:

import thread

def updateCount():
    mutex.acquire()
    globalVar = globalVar + 1
    mutex.release()

globalVar = 0
mutex = thread.allocate_lock()
for i in range(YOUR_PROCESSES):
    thread.start_new(yourThread, args)

Then in yourThread, just call updateCount() as appropriate.

Answer 2

Value is indeed mutable; you specify the datatype you want from the ctypes module and then it can be mutated. Here's a complete, working script that demonstrates this:

from time import sleep
from ctypes import c_int
from multiprocessing import Value, Lock, Process

counter = Value(c_int)  # defaults to 0
counter_lock = Lock()
def increment():
    with counter_lock:
        counter.value += 1

def do_something():
    print("I'm a separate process!")
    increment()

Process(target=do_something).start()
sleep(1)
print counter.value   # prints 1, because Value is shared and mutable

EDIT: Luper correctly points out in a comment below that Value values are locked by default. This is correct in the sense that even if an assignment consists of multiple operations (such as assigning a string which might be many characters) then this assignment is atomic. However, when incrementing a counter you'll still need an external lock as provided in my example, because incrementing loads the current value and then increments it and then assigns the result back to the Value.

So without an external lock, you might run into the following circumstance:

• Process 1 reads (atomically) the current value of the counter, then increments it
• before Process 1 can assign the incremented counter back to the Value, a context switch occurrs
• Process 2 reads (atomically) the current (unincremented) value of the counter, increments it, and assigns the incremented result (atomically) back to Value
• Process 1 assigns its incremented value (atomically), blowing away the increment performed by Process 2