How to increase connection pool size for Twisted?

Question

Hello

I'm using Twisted 8.1.0 as socket server engine. Reactor - epoll. Database server is MySQL 5.0.67. OS - Ubuntu Linux 8.10 32-bit

in /etc/mysql/my.cnf:

max_connections = 1000

in source code:

adbapi.ConnectionPool("MySQLdb", ..., use_unicode = True, charset='utf8', cp_min=3, cp_max=700, cp_noisy=False)

But in reality I can see only 200 (or less) open connections (SHOW PROCESSLIST) when application is running under heavy load. It is not enough for my app :(

As I see this is limit for the thread pool. Any ideas?

Alex

Answer 1

As you suspect, this is probably a threading issue. cp_max sets an upper limit for the number of threads in the thread pool, however, your process is very likely running out of memory well below this limit, in your case around 200 threads. Because each thread has its own stack, the total memory being used by your process hits the system limit and no more threads can be created.

You can check this by adjusting the stack size ulimit setting (I'm using bash) prior to running your program, i.e.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 32750
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32750
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

You can see that the default stack size is 10240K on my machine and I have found that I can create about 300 threads with this setting. Adjusting the stack size down to 1024K (using ulimit -s 1024) I can create about 3000 threads.

You can get some idea about the thread creation limits on you system using this script:

from thread import start_new_thread
from time import sleep

def sleeper():
    try:
        while 1:
            sleep(10000)
    except:
        if running: raise

def test():
    global running
    n = 0
    running = True
    try:
        while 1:
            start_new_thread(sleeper, ())
            n += 1
    except Exception, e:
        running = False
        print 'Exception raised:', e
    print 'Biggest number of threads:', n

if __name__ == '__main__':
    test()

Whether this solves your problem will depend on the memory requirements of the ConnectionPool threads.