ansaurus

Question

Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"

Answer 1

+1 A:

swap may not be the red herring previously suggested. How big is the python process in question just before the ENOMEM?

Under kernel 2.6, /proc/sys/vm/swappiness controls how aggressively the kernel will turn to swap, and overcommit* files how much and how precisely the kernel may apportion memory with a wink and a nod. Like your facebook relationship status, it's complicated.

...but swap is actually available on demand (according to the web host)...

but not according to the output of your free(1) command, which shows no swap space recognized by your server instance. Now, your web host may certainly know much more than I about this topic, but virtual RHEL/CentOS systems I've used have reported swap available to the guest OS.

Adapting Red Hat KB Article 15252:

A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. .... Systems with 4GB of ram or less [are recommended to have] a minimum of 2GB of swap space.

Compare your /proc/sys/vm settings to a plain CentOS 5.3 installation. Add a swap file. Ratchet down swappiness and see if you live any longer.

pilcrow 2009-09-03 03:55:58

What's the best way to check the size of the python process? ps?

DavidM 2009-09-03 23:54:15

something like `ps -o user,pid,vsz="Mem(Kb)" -o cmd $PYTHON_PID`, or top(1), should do it.

pilcrow 2009-09-04 01:30:24

Answer 2

A:

munmap(0xb7d28000, 4096) = 0
write(2, "OSError", 7) = 7

I've seen sloppy code that looks like this:

serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

You should check to see if this is what is happening in the python code. Errno is only valid if the proceeding system call failed.

Edited to add:

You don't say how long this process lives. Possible consumers of memory

forked processes
unused data structures
shared libraries
memory mapped files

codeDr 2009-09-03 21:43:09

Yes, but we see from the OP's strace that the first syscall failure -- from clone() -- is ENOMEM as is reported. This error is preserved along python's low-memory stumble through traceback construction, even though the C-library `errno` is reset many times along the way.

pilcrow 2009-09-03 21:48:47

hmm.... good catch.

codeDr 2009-09-04 03:58:35

Answer 3

A:

The relevant line in your strace is:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)

... I know others have talked about swap and memory availability (and I would recommend that you set up at least a small swap partition, ironically even if it's on a RAM disk ... the code paths through the Linux kernel when it has even a tiny bit of swap available have been exercised far more extensively than those (exception handling paths) in which there is zero swap available.

However I suspect that this is still a red herring.

The fact that free is reporting 0 (ZERO) memory in use by the cache and buffers is very disturbing. I suspect that the free output ... and possibly your application issue here, are caused by some proprietary kernel module which is interfering with the memory allocation in some way.

According to the man pages for fork()/clone() the fork() system call should return EAGAIN if your call would cause a resource limit violation (RLIMIT_NPROC) ... however, it doesn't say if EAGAIN is to be returned by other RLIMIT* violations. In any event if your target/host has some sort of weird Vormetric or other security settings (or even if your process is running under some weird SELinux policy) then it might be causing this -ENOMEM failure.

It's pretty unlikely to be a normal run-of-the-mill Linux/UNIX issue. You've got something non-standard going on there.

Jim Dennis 2009-09-04 04:02:46

The server is running on a Media Template (dv) base which uses Virtuozzo for virtualisation.

DavidM 2009-09-04 18:42:11

ansaurus

tags:

views:

answers:

Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"

related questions