views:

429

answers:

3

I'm writing a Python program with a lot of file access. It's running surprisingly slowly, so I used cProfile to find out what was taking the time.

It seems there's a lot of time spent in what Python is reporting as "{built-in method acquire}". I have no idea what this method is. What is it, and how can I speed up my program?

+5  A: 

Without seeing your code, it is hard to guess. But to guess I would say that it is the threading.Lock.acquire method. Part of your code is trying to get a threading lock, and it is waiting until it has got it.

There may be simple ways of fixing it by

  • restructuring your file access,
  • not locking,
  • using blocking=False,
  • or even not using threads at all.

But again, without seeing your code, it is hard to guess.

Ali A
It's _not_ the threading.Lock.acquire method. It turns out to be the threading.Condition.acquire method. Unfortunately, there doesn't seem to be any way to tell them apart in the profile output.
Chris B.
The two things represent similar situations.
Ali A
A: 

Using threads for IO is a bad idea. Threading won't make your program wait faster. You can achieve better results by using asynchronous I/O and an event loop; Post more information about your program, and why you are using threads.

nosklo
I'm using threads because I've got a slow network read operation going on, and I need to do some processing on the data after it's loaded. With the current design, I can load two datasets, generate a merge key on each, and merge and save them. Maybe an event loop's better for that, but I doubt it.
Chris B.
-1, threading multiple I/O operations can be beneficial in python because each thread will release the GIL while it's waiting for the OS to process the request.
David Locke
@David: And how exactly that is better than using asynchronous I/O?
nosklo
A: 

you want to look for cpu used, not for "total time used" from within that method--that might help. Sorry I don't use python but that's how it is for me in ruby :) -r

rogerdpack