Resident Set Size (RSS) limit has no effect

Question

The following problem occurs on a machine running Ubuntu 10.04 with the 2.6.32-22-generic kernel: Setting a limit for the Resident Set Size (RSS) of a process does not seem to have any effect. I currently set the limit in Python with the following code:

import resource
# (100, 100) is the (soft, hard) limit. ~100kb.
resource.setrlimit(resource.RLIMIT_RSS, (100, 100))
memory_sink = ['a']*10000000   # this should fail

The list, memory_sink, succeeds every time. When I check RSS usage with top, I can easily get the process to use 1gb of RAM, which means that the limit is not working. Do RSS limits not work with this kernel or distro? If it helps, resource.RLIMIT_NPROC (user process limit) does work.

Answer 1

Form the getrlimit manpage:

RLIMIT_RSS
Specifies the limit (in pages) of  the  process's  resident  set
(the  number of virtual pages resident in RAM).  This limit only
has effect in Linux 2.4.x, x < 30, and there only affects  calls
to madvise(2) specifying MADV_WILLNEED.

It seems this is just not supported on Linux kernel 2.6.