Speeding up the python "import" loader

Question

Hi Folks,

I'm getting seriously frustrated at how slow python startup is. Just importing more or less basic modules takes a second, since python runs down the sys.path looking for matching files (and generating 4 stat() calls - ["foo", "foo.py", "foo.pyc", "foo.so"] - for each check). For a complicated project environment, with tons of different directories, this can take around 5 seconds -- all to run a script that might fail instantly.

Do folks have suggestions for how to speed up this process? For instance, one hack I've seen is to set the LD_PRELOAD_32 environment variable to a library that caches the result of ENOENT calls (e.g. failed stat() calls) between runs. Of course, this has all sorts of problems (potentially confusing non-python programs, negative caching, etc.).

Cheers!

/YGA

Answer 1

The first things that come to mind are:

• Try a smaller path
• Make sure your modules are pyc's so they'll load faster
• Make sure you don't double import, or import too much

Other than that, are you sure that the disk operations are what's bogging you down? Is your disk/operating system really busy or old and slow?

Maybe a defrag is in order?

Answer 2

zipping up as many pyc files as feasible (with proper directory structure for packages), and putting that zipfile as the very first entry in sys.path (on the best available local disk, ideally) can speed up startup times a lot.

Answer 3

I would suggest running python in interactive mode.

python -i my_program.py

The file my_program.py should contain all the imports and code known to work. Once that stuff is done, you're dumped into interactive mode, where you can try out the behavior you are testing. If it fails, you can just move on to your next test without having to reimport your files.

Answer 4

If you run out of options, you can create a ramdisk to store your python packages. A ramdisk appears as a directory in your file system, but will actually be mapped directly to your computer's RAM. Here are some instructions for Linux/Redhat.

Beware: A ramdisk is volatile, so you'll also need to keep a backup of your files on your regular hard drive, otherwise you'll lose your data when your computer shuts down.

Answer 5

Something's missing from your premise--I've never seen some "more-or-less" basic modules take over a second to import, and I'm not running Python on what I would call cutting-edge hardware. Either you're running on some seriously old hardware, or you're running on an overloaded machine, or either your OS or Python installation is broken in some way. Or you're not really importing "basic" modules.

If it's any of the first three issues, you need to look at the root problem for a solution. If it's the last, we really need to know what the specific packages are to be of any help.