ansaurus

Question

Are Python built-in container thread-safe ?

Answer 1

+1 A:

They are thread-safe as long as you don't disable the GIL in C code for the thread.

Ignacio Vazquez-Abrams 2010-02-09 06:28:08

This is an implementation detail of CPython you shouldn't relay on. It's possibly going to chance in the future and other implementations don't have it.

Georg 2010-02-09 06:32:36

Georg - this aspect of python kind of terrefies me. Never mind all the bugs that will drop out of java programs when 8 cores become common on the desktop - what happens when GIL is removed and multithreaded python apps are suddenly running on 8 core boxes?

Ben 2010-02-09 12:11:04

It shouldn't terrify anybody if they don't pretend their code is thread-safe when it clearly isn't. :)

Kylotan 2010-02-09 15:12:16

Kylotan - I take this as tongue in cheek (hope that's correct!), but still the following is worth saying.Can you (or anybody) hand on heart say they have never written multithreaded code that has is free of race conditions? I may +think+ I have, but I am realistic enough to accept that I have not achieved this in all cases.GIL is a safety net that could give false confidence in this regard, inculcating bad habits that are then swept away when/if the default python container purges GIL at a time when 8+ PCs are common.This may be a moot point as some say GIL will never go. Time will tell!

Ben 2010-02-09 19:07:34

Answer 2

+7 A:

You need to implement your own locking for all shared variables that will be modified in Python. You don't have to worry about reading from the variables that won't be modified (ie, concurrent reads are ok), so immutable types (frozenset, tuple, str) are probably safe, but it wouldn't hurt. For things you're going to be changing - list, set, dict, and most other objects, you should have your own locking mechanism (while in-place operations are ok on most of these, threads can lead to super-nasty bugs - you might as well implement locking, it's pretty easy).

By the way, I don't know if you know this, but locking is very easy in Python - create a threading.lock object, and then you can acquire/release it like this:

with list1Lock.acquire():
    # change or read from the list here
# continue doing other stuff (the lock is released when you leave the with block)

In Python 2.5, do from __future__ import with_statement; Python 2.4 and before don't have this, so you'll want to put the acquire()/release() calls in try:...finally: blocks.

Here is some very good information about thread synchronization in Python: http://effbot.org/zone/thread-synchronization.htm

Daniel G 2010-02-09 06:29:50

I believe that, for someone that hasn't used threading locks before, it should be noted that the lock (in your example, `list1Lock`) should be *shared* between the threads, in order for it to work correctly. Two independent locks, one for each thread, would lock nothing, just add silly overhead.

ΤΖΩΤΖΙΟΥ 2010-02-18 08:44:53

Answer 3

+4 A:

Yes, but you still need to be careful of course

For example:

If two threads are racing to pop() from a list with only one item, One thread will get the item successfully and the other will get an IndexError

Code like this is not thread-safe

if L:
    item=L.pop() # L might be empty by the time this line gets executed

You should write it like this

try:
    item=L.pop()
except IndexError:
    # No items left

gnibbler 2010-02-09 06:31:19

ansaurus

tags:

views:

answers:

Are Python built-in container thread-safe ?

related questions